Page MenuHomeWildfire Games

build-osx-libs: Compare lib version instead of dir mtime
ClosedPublic

Authored by Krinkle on Mar 1 2020, 11:39 PM.

Details

Reviewers
Itms
Group Reviewers
Restricted Owners Package(Owns No Changed Paths)
Commits
rP23686: Check the library version instead of the mtime of the already-built guard file…
Summary

There are numerous problems with using directory mtimes for comparisons:

  • They don't always change when the source code changes.

    For libraries that are fetched over HTTP, the comparison was mostly fine because for those the source directory that the script compared against was the versioned subdirectory that either doesn't exist or is named after the lib version (e.g. "libsodium/libsodium-1.0.18/" compared to "libsodium/.already-built").

    However, directories for which the source is checked into version control (such as SpiderMonkey), this doesn't work so well as individual files within source/spidermonkey/ may change without the directory mtime changing per-se (mtimes of files don't propagate to parent directories on most file systems, as far as I know).

    This was raised in Trac issue #2551 and worked around in rP15696 by adding a second mtime comparison for source/spidermonkey/README.txt which just so happens to contain the mozjs version number.

    Instead of these indirect measures, I propose to write the LIB_VERSION value to the .already-built file and compare against that directly.
  • They don't work properly when going backwards in time.

    When checking out an older revision, branch, or when using something like git-bisect, the time comparisons might not work as expected, especially if the older version still exists on disk with its older timestamp.

    By performing strict comparisons we remove the assumption that a newer source directory is always want we are currently interested in. Thus reducing our assumption to only that we are interested in what the currently executing version of the build script is meant to produce.

Also, there were two libraries that were lacking any kind of mtime or other comparison. These were NVTT and FColladda. This meant that once built, no matter how long ago, they would never be rebuild unless the user passed the --force-rebuild option. These change quite rarely, so perhaps that isn't too bad, but I've added a LIB_VERSION for these as well so that we at least have the option of making this work automatically if we remember to bump them, which could even be done after the fact if forgotten and still helps other people. Rebuilding only one library takes significantly less time than rebuilding everything from scratch.

Test Plan
  • When run for the first time, all libraries are built.
  • When run again no libraries are rebuilt.
  • Edit one of the .already-built files and modify its contents, and then run it again, only that library should be rebuilt.
  • When run again with --force-rebuild, all libraries should be rebuilt.

Diff Detail

Repository
rP 0 A.D. Public Repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Krinkle created this revision.Mar 1 2020, 11:39 PM
Owners added a subscriber: Restricted Owners Package.Mar 1 2020, 11:39 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1835/display/redirect

Stan awarded a token.Mar 1 2020, 11:41 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/412/display/redirect

Krinkle edited the summary of this revision. (Show Details)
Krinkle added a subscriber: Itms.
Stan added reviewers: Restricted Owners Package, Itms.Mar 18 2020, 12:11 PM
Krinkle updated this revision to Diff 11887.May 17 2020, 1:22 AM

Rebased.

Krinkle updated this revision to Diff 11888.May 17 2020, 1:25 AM
Krinkle added a subscriber: wraitii.

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2136/display/redirect

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2137/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/721/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/722/display/redirect

Itms added a comment.May 17 2020, 11:44 AM

Hi Krinkle, very nice patch! It will help the CI considerably.

I am only concerned about the lib versions for the bundled libraries. Currently if devs forget to bump the value of LIB_VERSION for SpiderMonkey, it won't build, because it won't find the archive.

However, people upgrading nvtt or patch fcollada might forget to bump the version. There is a good chance that they encounter bugs, due to the lib not being rebuilt, and so they will fix the version string in this script. But in order to help them and to make things consistent, I think a good improvement would be to move the lib version definitions (the two new ones and also the spidermonkey one) to the top of the file, line 50.

Itms added a comment.May 17 2020, 12:30 PM

Additionally I had this idea: P203

Since update-workspaces.sh does not build the 3 bundled libs on macOS, I think clean-workspaces.sh should not clean them either.

That change would create issues, but I think that in conjunction with this diff, it would work well. What do you think?

Itms added a comment.May 17 2020, 4:27 PM

And one final idea: maybe the nvtt version should have a wildfiregames suffix so that it gets rebuilt when we add some patches, like I will do in D2563.

I would prefer to have this committed before committing D2563 ?

Krinkle updated this revision to Diff 11915.EditedMay 17 2020, 9:35 PM

Hoist bundled lib versions to top of file. Add version suffix for nvtt as well.

In D2649#115710, @Itms wrote:

[…] Currently if devs forget to bump the value of LIB_VERSION for SpiderMonkey, it won't build, because it won't find the archive.

Right, it's not ideal, but at least CI should find such issue quick enough. It was like this before as well, but I'll think about it some more, perhaps something for a future patch? By the way, do you know the reason why this lib is stored in version control, rather than fetched on-demand? I mean, I like self-hosting, I'm curious given it's different from all the others.

In D2649#115710, @Itms wrote:

people upgrading nvtt or patch fcollada might forget to bump the version. There is a good chance that they encounter bugs, due to the lib not being rebuilt, and so they will fix the version string in this script. But in order to help them and to make things consistent, I think a good improvement would be to move the lib version definitions (the two new ones and also the spidermonkey one) to the top of the file, line 50.

Thanks, I didn't realise nvtt was patched as well. Done!

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2154/display/redirect

Krinkle added inline comments.May 17 2020, 10:05 PM
libraries/osx/build-osx-libs.sh
914 ↗(On Diff #11915)

This isn't/wasn't versioned, but we do have patches. If you want I could add a suffix here as well, but it would be extra work to maintain.

Based on past changes, it seems like we usually don't patch this ourselves, except during updates which means it would not need to be tracked (the same way that we don't track all the make flags etc).

Let me know if that is incorrect.

Another way could be to try to do it automatically. Maybe this is too hacky, but something like the follwing could work:

LIB_VERSION="${SPIDERMONKEY_VERSION} $(md5sum ../source/spidermonkey/patch.sh)"
LIB_ARCHIVE="${SPIDERMONKEY_VERSION}.tar.bz2"
LIB_DIRECTORY="${SPIDERMONKEY_VERSION}"
Itms added a comment.May 17 2020, 10:33 PM

Thanks! I'll try to commit this during the week, we have some bank holidays...

Right, it's not ideal, but at least CI should find such issue quick enough. It was like this before as well, but I'll think about it some more, perhaps something for a future patch? By the way, do you know the reason why this lib is stored in version control, rather than fetched on-demand? I mean, I like self-hosting, I'm curious given it's different from all the others.

I think it's because on Linux and Windows we download nothing, nvtt and windows libs are under version control. macOS is the odd one about it. But given the size of the SM bundle it might be interesting to consider fetching it.

Maybe a suffix to spidermonkey would be nice as well. I don't have a strong opinion, it's your fcollada suffix, plus the current nvtt bugs, which gave me the idea. For consistency I guess we'd need a suffix for SM. You're right that in general we don't make patches in between SM upgrades, but it could happen.

Also, suffixing with the commit date might not be the best. Usually when one writes a patch, they don't know yet the date on which it will be committed. This will be something people will forget to bump when they commit. Maybe we can be a bit more generic about this suffix and use something like +wfg-1, then +wfg-2 and so on.

Itms accepted this revision.May 21 2020, 5:06 PM

I am going to commit this, but with a suffix for spidermonkey, as well as using numbers (instead of tentative dates) for the suffixes.

This revision is now accepted and ready to land.May 21 2020, 5:06 PM
This revision was landed with ongoing or failed builds.May 21 2020, 5:29 PM
This revision was automatically updated to reflect the committed changes.