Page MenuHomeWildfire Games

Use Homebrew to fetch OSX libraries instead of compiling them all
AbandonedPublic

Authored by wraitii on May 5 2018, 7:05 PM.

Details

Summary

This is a bit of a design decision issue.

For about 3 years or so now I've been using this hack (in an even more hacky way) to avoid recompiling every OSX library. This installs them using brew and symlinks.

The only exception is gloox as brew's requires openSSL and I tried brew-compiling without, it didn't work. So keep using the old way.

We may want to have both scripts in parallel, possibly?

Test Plan

Review this and answer the above question.

Diff Detail

Repository
rP 0 A.D. Public Repository
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 6632
Build 10932: Vulcan BuildJenkins
Build 10931: arc lint + arc unit

Event Timeline

wraitii created this revision.May 5 2018, 7:05 PM
Owners added a subscriber: Restricted Owners Package.May 5 2018, 7:05 PM
Vulcan added a subscriber: Vulcan.May 5 2018, 7:08 PM

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/462/display/redirect

vladislavbelov requested changes to this revision.May 6 2018, 12:31 AM

Isn't it still valid?

# patch zlib's configure script to use our CFLAGS and LDFLAGS
(patch -p0 -i ../../patches/zlib_flags.diff && CFLAGS="$CFLAGS" LDFLAGS="$LDFLAGS" ./configure --prefix="$ZLIB_DIR" --static && make ${JOBS} && make install) || die "zlib build failed"

If it's not needed (I don't know, it should be tested), then the patches/zlib_flags.diff file should be removed.

Also we shouldn't forget to update our wiki too.

This revision now requires changes to proceed.May 6 2018, 12:31 AM
wraitii updated this revision to Diff 7172.Jan 1 2019, 10:19 AM

Now using a pythonized version, which I think is the way to go. This isn't complete since it doesn't incorporate the brew part, and we still need to build gloox I believe.

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/867/

vladislavbelov requested changes to this revision.Apr 2 2019, 4:43 PM

I suppose you plan changes here.

libraries/osx/build-osx-libs.sh
152

Why is it commented here and below? Do I understand correctly that you plan changes?

This revision now requires changes to proceed.Apr 2 2019, 4:43 PM

I'm mostly looking for comments on whether we actually want to do this or not.

I think this means versions are no longer controlled and libs no longer statically linked, thus the build no longer standalone and deterministic.

I'm still new, but I very much liked the idea of it being controlled in the repo, statically linked, and deterministic. I don't know how often this is an issue in CPP code bases, but for other code bases this typically increases bugs, things working for one dev but not another, harder to contribute due to rare issues specific to ones own environment, waste more triaging/patching/maintenance cost on lib stuff.

Having said that, Homebrew is much faster by comparison due to having most of these pre-compiled and pre-bottled, so they're very quick to install. It might be possible to access some of those versions and formulas directly, allowing us to get the best of both?

I think this means versions are no longer controlled and libs no longer statically linked, thus the build no longer standalone and deterministic.
I'm still new, but I very much liked the idea of it being controlled in the repo, statically linked, and deterministic. I don't know how often this is an issue in CPP code bases, but for other code bases this typically increases bugs, things working for one dev but not another, harder to contribute due to rare issues specific to ones own environment, waste more triaging/patching/maintenance cost on lib stuff.

You're correct on these two accounts, guess it's not said in this diff but it's noted elsewhere. The issue mostly comes up when building the Release bundle - where we want some degree of backwards compatibility. Brew libraries being new, we can't guarantee so easily that the game would work on 10.10 for example.
Another drawback is that OSX always chooses dynamic libraries over static libraries - and for 0 A.D.'s bundle we need static libraries or the application won't be standalone. So that's annoying. An alternative would be bundling the libraries in the Frameworks/ part of the bundle, but that will likely greatly increase executable size.

Having said that, Homebrew is much faster by comparison due to having most of these pre-compiled and pre-bottled, so they're very quick to install. It might be possible to access some of those versions and formulas directly, allowing us to get the best of both?

Kinda fragile, sadly.

There are two realistic options imo:

  • Use a package manager like Conan - might be annoying to setup up, dunno
  • Have both scripts. Use brew for dev versions - it helps make sure that we're compatible with upstream versions, and such. Have the "build" script for releases.

The drawback of (2) is that the less used script is more likely to break.

Stan added a subscriber: Stan.Jun 20 2019, 8:55 AM

Well maybe you guys could try to use my conan patch ? You will need to change the copy to osx instead of win32 and likely edit premake libs.

historic_bruno added a subscriber: historic_bruno.EditedJul 10 2019, 8:05 PM

So just to add onto this discussion, since myself and leper originally wrote this script.

I realize a full build-osx-libs.sh takes a long time. And that you can't use the fact that packages have already been built for other dependencies in, say, Homebrew or another package manager. It's also more maintenance on our end to keep the script updated, check all the build flags, etc.

The reason this script was created is that it forces all macOS users to have a single precisely controlled build path, where we specify the lib versions, macOS SDK, and macOS API versions we support. AFAIK, Homebrew does not offer this, if it (or some other package manager) does we could revisit the topic.

As for why that matters? As stated above, for official releases (or even for unofficial community bundles), you must have an SDK and API-based build. This means system libraries are picked up from the SDK you specify, not your local system dirs, and that you are aware of the minimum API (macOS version) you choose to support. The risk of not doing this is that your bundle only runs on your machine, and nobody else's.

My concern is that if we support Homebrew-based builds as an extension, then we have multiple macOS build paths. This means less reviewers for changes to build-osx-libs.sh (mostly people would only catch them after release or during RC, which is a much shorted period of time, since they use the more convenient, faster Homebrew build path). That's why we never officially supported Homebrew, at least since we starting releasing bundles. It required a re-think of the macOS build process.

Other ideas, concerns, suggestions are always welcome :)

historic_bruno added a comment.EditedJul 10 2019, 8:11 PM

Another reason is that it allows building for other Apple platforms, including iOS. Even though we don't support that at all, I have tinkered with it in the past, and it helped substantially to have this infrastructure in place. Only a few differences needed between building for macOS and iOS.

See: https://wildfiregames.com/forum/index.php?/topic/15436-android-port/page/11/&tab=comments#comment-299683

historic_bruno added a comment.EditedJul 10 2019, 8:16 PM

Also, requiring Homebrew to run a bundle (also an option to consider) breaks the Apple vision of apps as self-contained bundles. The 0A.D. bundle should rely only on what is inside it, and the standard macOS system libraries. That was always the design goal, and at least in the past, that was how it worked.

https://developer.apple.com/app-store/app-bundles/

wraitii abandoned this revision.Aug 15 2019, 12:36 PM

Abandoning since won't get committed.