Page MenuHomeWildfire Games

Update fmt to the most recent release
Needs ReviewPublic

Authored by s0600204 on Apr 3 2020, 12:56 AM.

Details

Reviewers
Itms
Krinkle
Group Reviewers
Windows Developers
macOS Developers
Trac Tickets
#3190
Summary

Support the most recent released version of fmt (previously known as cppformat) available.

This revision also de-bundles the source, as we no longer need to modify it to get the library to work with pyrogenesis.

Linux/BSD: Users now need to have fmt installed from their distribution's package repository. Minimum required version of fmt is 4.0.0.

OSX: The source is acquired and compiled (in build-osx-libs.sh) and included/linked automatically,

Windows: The relevant header files are retained, and along with a pre-built library (not actually included in the revision yet) are to be the only thing bundled.

Test Plan

On Linux/BSD:

  • Make sure fmt (at least version 4.0.0) is installed (e.g. via your distro's package manager).
  • Rebuild pyrogenesis
    • You may need to make clean and clean-workspaces.sh.
  • Report back on successes, fails, and version of fmt used.

On OSX:

  • Run build-osx-libs.sh
  • Rebuild pyrogenesis.
  • Report back successes and fails.

On Windows:

  • Acquire and build the library:
  • Place the built library (fmt.lib) into ./libraries/win32/fmt/lib/
    • Could you also upload a copy of it somewhere online so I can include it in this revision?
  • Follow usual method of building pyrogenesis on Windows

Event Timeline

s0600204 created this revision.Apr 3 2020, 12:56 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1962/display/redirect

Vulcan added a comment.Apr 3 2020, 1:00 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1435/display/redirect

Vulcan added a comment.Apr 3 2020, 1:03 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/542/display/redirect

Krinkle added a subscriber: Krinkle.Apr 3 2020, 4:58 AM

I can help with testing on macOS. Unfortunately, it seems trunk is not compiling well currently on macOS due to a problem with NVTT. I've filed https://trac.wildfiregames.com/ticket/5710.

Angen added a subscriber: Angen.Apr 3 2020, 8:35 AM

\libraries\source\fmt\include\fmt\format.h(1721): error C2039: '_finite': is not a member of 'std' (compiling source file ..\..\..\source\network\NetClient.cpp)

Krinkle requested changes to this revision.Apr 4 2020, 8:46 PM

I'm happy to help, with we do have a macOS worker in the Jenkins pipeline already. It'd be good to see that passing first:
https://jenkins.wildfiregames.com/blue/organizations/jenkins/macos-differential/detail/macos-differential/542/pipeline/41

It seems to fail right now on:

build/workspace/gcc$ make
Linking pyrogenesis

ld: library not found for -lfmt

It's possible that this is due to a limitation in the CI setup (maybe it is too eagerly re-using the library compilation cache and not taking this change into account).

But, looking at the earlier "Libraries build" stage on the above Jenkins link, I actually don't see it even mention anything about fmt.

Does this need to be added to libraries/osx/build-osx-libs.sh?

This revision now requires changes to proceed.Apr 4 2020, 8:46 PM
s0600204 updated this revision to Diff 11632.Apr 4 2020, 11:10 PM

Build fmt on OSX (I hope)

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1969/display/redirect

It occurs to me that:

  • We no longer modify the fmt source;
  • The source is available from publicly accessible location;
  • Many, if not most, linux distro package repositories have a sufficiently recent version built and available;

So... we probably don't actually need to bundle the source. Headers: yes (for Windows), but the source: no.

We/I would need someone on Windows to build and provide a prebuilt library file, like that which is provided for the other dependencies on Windows.

Thoughts?

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1442/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/549/display/redirect

s0600204 updated this revision to Diff 11633.Apr 5 2020, 1:04 AM

Build on OSX, second attempt

The Linux build fails due to needing a make clean.

The Windows build fails due to not having a prebuilt library file to include.

Vulcan added a comment.Apr 5 2020, 1:06 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1970/display/redirect

Vulcan added a comment.Apr 5 2020, 1:11 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1443/display/redirect

s0600204 planned changes to this revision.Apr 9 2020, 6:03 AM
s0600204 updated this revision to Diff 11652.Apr 9 2020, 7:02 AM

Hopefully get this compiling on the Windows Jenkins slave

Probably won't link, but meh.

Vulcan added a comment.Apr 9 2020, 7:04 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1983/display/redirect

Vulcan added a comment.Apr 9 2020, 7:11 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1456/display/redirect

s0600204 updated this revision to Diff 11653.Apr 9 2020, 7:26 AM

Try de-bundling fmt

As mused above, this is an attempt to de-bundle fmt almost entirely so we're not having to maintain a 3rd party library.

I am prepared to roll this back by one commit - I'm just curious to see how far it gets through the various autobuilders.

Anyhow... with this latest state:

Linux:

  • Users are now required to have fmt pre-installed (from package repositories ideally).

OSX:

  • The source is downloaded and compiled automatically (in build-osx-libs.sh).

Windows:

  • The required headers are retained to include during compilation.
  • Someone will need to provide a built library file to link against.
Vulcan added a comment.Apr 9 2020, 7:27 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1984/display/redirect

Vulcan added a comment.Apr 9 2020, 7:28 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/564/display/redirect

Vulcan added a comment.Apr 9 2020, 7:35 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1457/display/redirect

s0600204 updated this revision to Diff 11654.Apr 9 2020, 7:38 AM

Fix OSX fetching of the fmt archive

Vulcan added a comment.Apr 9 2020, 7:39 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1985/display/redirect

Vulcan added a comment.Apr 9 2020, 7:48 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1458/display/redirect

Vulcan added a comment.Apr 9 2020, 8:09 AM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/565/display/redirect

I tried updating the workspaces with this patch and got:

Package fmt was not found in the pkg-config search path.
Perhaps you should add the directory containing `fmt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fmt' found

on my Lubuntu 18.04 with fmt-dev 4.0.0 installed.
And I could not find fmt.pc. Sorry in advance if this is a noob question though,,,

s0600204 updated this revision to Diff 11657.Apr 9 2020, 5:37 PM

Get working with fmt 4.0 - 5.2.x on Linux

Vulcan added a comment.Apr 9 2020, 5:38 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1988/display/redirect

on my Lubuntu 18.04 with fmt-dev 4.0.0 installed.
And I could not find fmt.pc. Sorry in advance if this is a noob question though,,,

Not a noob (or newb) question.

Looking into it, fmt.pc is generated with fmt 5.3.0 or better. Unfortunately, the take-up of that is roughly ~65%[^1].

Annoyingly, among the distros that don't have it is the entire (released[^2]) Ubuntu line. This will have a knock-on effect to every distro that uses Ubuntu as a baseline (e.g. Linux Mint) and thus affect a lot of our user base on Linux.

The latest version of this revision doesn't use pkg-config, so as to support all the systems who are lacking a version of fmt recent enough.


^1 - 58 out of 88 distro repositories that provide a fmt package, as tracked by https://repology.org/project/fmt/versions (colour-coded list)
^2 - Ubuntu 20.04 "Focal" supposedly will have it, but that's not out yet...

Stan added a subscriber: Stan.Apr 9 2020, 5:43 PM
Stan added inline comments.
build/premake/extern_libs5.lua
277

It might be a stupid question. but can we check the version?

Vulcan added a comment.Apr 9 2020, 5:45 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1461/display/redirect

Vulcan added a comment.Apr 9 2020, 5:52 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/568/display/redirect

s0600204 added inline comments.Apr 9 2020, 8:26 PM
build/premake/extern_libs5.lua
277

To check the version, we'd need some (ideally non-convoluted) way of finding out what the version is.

Freagarach added a comment.EditedApr 9 2020, 8:31 PM

Thanks for the update and explanation :)
It works now :)

s0600204 retitled this revision from Update fmt to version 6.1.2 to Update fmt to the most recent release.Apr 9 2020, 8:48 PM
s0600204 edited the summary of this revision. (Show Details)
s0600204 edited the test plan for this revision. (Show Details)
Krinkle requested changes to this revision.Apr 13 2020, 2:06 AM

I have done the following on macOS 10.14 Mojave

  • Checkout latest trunk with D2563 and D2649 applied.
  • Perform a complete and clean build (libraries/osx/build-osx-libs.sh -j3, clean-workspace.sh, update-workspaces.sh -3, gcc$ make pyrogenesis -j3).
  • Apply this patch (D2689).
  • Re-run libraries/osx/build-osx-libs.sh -j3, which will only (re)compile fmt.
    • This step passed without errors.
  • Re-run clean-workspace.sh, update-workspaces.sh -3.
    • This step passed without errors.
  • Run gcc$ make pyrogenesis -j3
    • This step fails. Error attached below.
libraries
Building libsodium...
Skipping - already built (use --force-rebuild to override)
Building fmt...
Downloading 6.1.2.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   117  100   117    0     0    371      0 --:--:-- --:--:-- --:--:--   371
100  706k  100  706k    0     0   266k      0  0:00:02  0:00:02 --:--:--  882k
~/Development/0ad-git/libraries/osx/fmt/fmt-6.1.2 ~/Development/0ad-git/libraries/osx/fmt ~/Development/0ad-git/libraries/osx
~/Development/0ad-git/libraries/osx/fmt/fmt-6.1.2/build ~/Development/0ad-git/libraries/osx/fmt/fmt-6.1.2 ~/Development/0ad-git/libraries/osx/fmt ~/Development/0ad-git/libraries/osx
-- CMake version: 3.16.2
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/clang++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Version: 6.1.2
-- Build type: Release
-- CXX_STANDARD: 11
-- Performing Test has_std_11_flag
-- Performing Test has_std_11_flag - Success
-- Performing Test has_std_0x_flag
-- Performing Test has_std_0x_flag - Success
-- Performing Test SUPPORTS_VARIADIC_TEMPLATES
-- Performing Test SUPPORTS_VARIADIC_TEMPLATES - Success
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS
-- Performing Test SUPPORTS_USER_DEFINED_LITERALS - Success
-- Performing Test FMT_HAS_VARIANT
-- Performing Test FMT_HAS_VARIANT - Success
-- Performing Test HAS_NULLPTR_WARNING
-- Performing Test HAS_NULLPTR_WARNING - Success
-- Looking for strtod_l
-- Looking for strtod_l - found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/krinkle/Development/0ad-git/libraries/osx/fmt/fmt-6.1.2/build
Scanning dependencies of target fmt
[ 66%] Building CXX object CMakeFiles/fmt.dir/src/posix.cc.o
[ 66%] Building CXX object CMakeFiles/fmt.dir/src/format.cc.o
[100%] Linking CXX static library libfmt.a
[100%] Built target fmt
~/Development/0ad-git/libraries/osx/fmt/fmt-6.1.2 ~/Development/0ad-git/libraries/osx/fmt ~/Development/0ad-git/libraries/osx
~/Development/0ad-git/libraries/osx/fmt ~/Development/0ad-git/libraries/osx
Building Spidermonkey...
Skipping - already built (use --force-rebuild to override)
Building NVTT...
Skipping - already built (use --force-rebuild to override)
Building FCollada...
make
$ make pyrogenesis -j3
==== Building mocks_real (release) ====
Creating obj/mocks_real_Release
mocks_real.cpp
Linking mocks_real
==== Building network (release) ====
Creating obj/network_Release
precompiled.h
In file included from ../../../source/pch/network/precompiled.h:19:
In file included from ../../../source/lib/precompiled.h:111:
../../../source/ps/CLogger.h:27:10: fatal error: 'fmt/printf.h' file not found
#include <fmt/printf.h>
         ^~~~~~~~~~~~~~
1 error generated.
make[1]: *** [obj/network_Release/precompiled.h.gch] Error 1
make: *** [network] Error 2
This revision now requires changes to proceed.Apr 13 2020, 2:06 AM
s0600204 updated this revision to Diff 11675.Apr 13 2020, 8:06 PM
s0600204 edited the test plan for this revision. (Show Details)

Put the headers in a place where they're findable on OSX

Also:

  • Update to fmt 6.2.0 [win/osx]
  • Improve does-library-need-to-be-rebuilt check (like D2649) [osx]

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/1998/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/1471/display/redirect

Stan added a comment.Apr 13 2020, 9:59 PM

Looks like it failed to find it on linux ?

In D2689#113619, @Stan wrote:

Looks like it failed to find it on linux ?

In the summary, @s0600204 wrote:

Linux/BSD: Users now need to have fmt installed from their distribution's package repository.

This applies to the Linux image used on Jenkins too.

I've build this with D2671 aplied beforehand on alpine linux, afterwards i was able to run the game (i had some js errors in game setup view, but likely not related, wasn't on the highest revision but could do that if needed)