Page MenuHomeWildfire Games

SM91 upgrade
Needs RevisionPublic

Authored by wraitii on Jan 9 2022, 3:24 PM.

Details

Reviewers
Stan
Trac Tickets
#5986
Summary

This upgrades 0 A.D. to use Spidermonkey ESR 91.
API changes are essentially trivial, basically renaming:

The most important change is the introduction of Warp: https://hacks.mozilla.org/2020/11/warp-improved-js-performance-in-firefox-83/
Based on some rather trivial AI replays (both a 1v1 and a 4-player FFA), I think we actually see greater speed improvements than claimed by Mozilla. My very basic profiling suggests JS code might be up to 2x faster, resulting in substantial speed boosts for the simulation as a whole (up to 30% for 'sim update'). Note that this won't translate to 'the game is 30% faster' since this will mostly affect simturns and not raw frames, but overall the upgrade seems _excellent_.

Edit -> Scratch that, I'm seeing the effect of x86 to ARM compilation. Obviously the game is much faster when it's not running in rosetta...

It appears that SM91 remains replay-compatible with SM78 (it worked in the above AI game anyways) so getting some perf comparisons will be easy enough.

NB: a subsequent diff handles file removals & ought handle binary changes such & windows headers.

NB2: Download the tarball from here: https://github.com/wraitii/spidermonkey-tarballs/releases/download/mozjs-91.5.0/mozjs-91.5.0.tar.xz

Test Plan

Ideally, this would be compiled & tested on Linux & Windows. I haven't had time to try and compile this on Windows yet, so no idea whether it works there.
I've tested it on Mac OS 12 however, compiling for both x86 & ARM.

Event Timeline

wraitii created this revision.Jan 9 2022, 3:24 PM
Owners added a subscriber: Restricted Owners Package.Jan 9 2022, 3:25 PM
Vulcan added a comment.Jan 9 2022, 3:28 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/5475/display/redirect

I tried to apply the patch, but neither with Arcanist nor with pull0ad I am able to apply the patch successfully.

Terminal Output - Arcanist
https://dpaste.com/FENKKZMPS.txt

Terminal Output - pull0ad
https://dpaste.com/2238BZ8MK.txt

Is SM91 supported by our low/old platforms (like GCC7 + 4.15.0 or Windows 7)?

libraries/source/spidermonkey/build.sh
111

Is it ok store the archive on github? Maybe gitlab or our server?

Vulcan added a comment.Jan 9 2022, 5:49 PM

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/7179/display/redirect

wraitii added inline comments.Jan 9 2022, 6:07 PM
libraries/source/spidermonkey/build.sh
111

This was mostly done as a hack to make the CI work until the tarballs got committed to SVN.

Stan published this revision for review.Jan 10 2022, 12:56 AM
Stan added a subscriber: Stan.

Windows CI choked.

nwtour added a subscriber: nwtour.Jan 10 2022, 2:42 PM

I have not build this version out of the box (linux/clang-11)

In file included from /home/nwtour/ff/0ad/libraries/source/spidermonkey/mozjs-91.5.0/modules/fdlibm/src/e_acos.cpp:44:
/home/nwtour/ff/0ad/libraries/source/spidermonkey/mozjs-91.5.0/modules/fdlibm/src/math_private.h:34:21: error: typedef redefinition with different types ('__double_t' (aka 'double') vs 'long double')
typedef __double_t  double_t;
                    ^
/usr/include/math.h:156:21: note: previous definition is here
typedef long double double_t;
                    ^
1 error generated.
ERROR: SpiderMonkey build failed
libraries/source/spidermonkey/build.sh
111
113

bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2

-j, --bzip2
       Filter the archive through bzip2(1).

-J, --xz
       Filter the archive through xz(1)

I have not build this version out of the box (linux/clang-11)
/home/nwtour/ff/0ad/libraries/source/spidermonkey/mozjs-91.5.0/modules/fdlibm/src/math_private.h:34:21: error: typedef redefinition with different types ('__double_t' (aka 'double') vs 'long double')

Did you manage to fix that error? I didn't run into it.
The others I'm aware of, need to fix em.

Did you manage to fix that error? I didn't run into it.
The others I'm aware of, need to fix em.

I looked at the trunk https://hg.mozilla.org/mozilla-central/file/7c1dcab226105b4d4dab172021483381689dd6d7/modules/fdlibm/src/math_private.h
These lines are not there:

/* Emulate FreeBSD internal double types. Adapted from https://github.com/freebsd/freebsd-src/search?q=__double_t  */

typedef double      __double_t;
typedef __double_t  double_t;

In trunk revision history - i don't see them either: https://hg.mozilla.org/mozilla-central/log/7c1dcab226105b4d4dab172021483381689dd6d7/modules/fdlibm/src/math_private.h

Looks like hack added to the ESR (?)

Stan added a comment.EditedJan 10 2022, 4:41 PM

Some files are missing from the tarball, see https://bugzilla.mozilla.org/show_bug.cgi?id=1749306

I built with SM's 78 headers but there are only a few commits missing https://github.com/mozilla/gecko-dev/commits/esr91/mozglue/dllservices (3abb99c8e2abd5f2b327ef488231683efdc22673 is 78.*)

Here are the files that are working(tm) for me on Windows after fixing it.

Can't compile though https://pastebin.com/eLXQKS70
Same error for this guy (no answer) https://discourse.mozilla.org/t/sm-91-compatibility-with-msvc/83338

Stan added a comment.Jan 11 2022, 2:22 PM

wraitii edited the summary of this revision. (Show Details)Jan 11 2022, 2:25 PM

Did you manage to fix that error? I didn't run into it.
The others I'm aware of, need to fix em.

https://bugzilla.mozilla.org/show_bug.cgi?id=1729459

wraitii updated this revision to Diff 19435.Jan 13 2022, 3:53 PM

Should fix issues noted above.

Build failure - The Moirai have given mortals hearts that can endure.

Debug:
     6>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\rlinterface.vcxproj]
     8>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\i18n\L10n.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\lobby.vcxproj]
     8>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\lobby\Globals.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\lobby.vcxproj]
     9>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\network.vcxproj]
     8>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\lobby\scripting\GlooxScriptConversions.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\lobby.vcxproj]
     8>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\lobby\XmppClient.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\lobby.vcxproj]
     8>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\lobby\scripting\JSInterface_Lobby.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\lobby.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\ScriptContext.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\JSON.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    12>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\simulation2.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\ScriptExceptions.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\ScriptConversions.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\ScriptStats.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    14>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\graphics.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\ScriptInterface.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    11>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. (compiling source file ..\..\..\source\scriptinterface\StructuredClone.cpp) [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\scriptinterface.vcxproj]
    13>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\engine.vcxproj]
    15>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\atlas.vcxproj]
    16>e:\jenkins\workspace\vs2015-differential\source\scriptinterface\scripttypes.h(81): fatal error C1189: #error:  Your compiler is trying to use an incorrect major version of the SpiderMonkey library. The only version that works is the one in the libraries/spidermonkey/ directory, and it will not work with a typical system-installed version. Make sure you have got all the right files and include paths. [E:\Jenkins\workspace\vs2015-differential\build\workspaces\vs2017\gui.vcxproj]

Link to build: https://jenkins.wildfiregames.com/job/vs2015-differential/6590/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/5494/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/7197/display/redirect

Linux x32/Clang 11
build, start game, save-load, atlas, js-console: all successfull, found no errors

libraries/source/spidermonkey/build.sh
8

In SVN 78.6.0+2. Not automatically applied:

patching file libraries/source/spidermonkey/build.sh
Hunk #1 FAILED at 3.
Hunk #4 succeeded at 101 (offset -3 lines).
Hunk #5 FAILED at 119.
Hunk #6 succeeded at 135 (offset 4 lines).
Hunk #7 succeeded at 148 (offset 4 lines).
2 out of 7 hunks FAILED -- saving rejects to file libraries/source/spidermonkey/build.sh.rej
patching file libraries/source/spidermonkey/patch.sh
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file libraries/source/spidermonkey/patch.sh.rej
source/gui/Scripting/JSInterface_GUIProxy.h
173
In file included from ../../../source/gui/ObjectBases/IGUIObject.cpp:25:
../../../source/gui/Scripting/JSInterface_GUIProxy.h:173:98: warning: unused parameter 'desc' [-Wunused-parameter]
                                              JS::MutableHandle<mozilla::Maybe<JS::PropertyDescriptor>> desc) const override
Langbart added a comment.EditedJan 15 2022, 12:40 AM
  • Patch applied successfully
  • Spidermoneky was successfully built
  • Game successfully compiled and started
Performance

I used the following replay and counted the time for reaching the inGame time of 10mins, 20mins and 30mins.

10min20min30min
rP2621110:42min21:54min33:13min
rP26211+D442810:38min21:43min32:46min
4 sec11sec27sec

PS: Default settings applied in both cases.

Stan added a comment.Jan 18 2022, 3:12 PM

Shellcheck output

$ shellcheck myscript
 
Line 56:
  CONF_OPTS="${CONF_OPTS}"
  ^-- SC2269 (info): This variable is assigned to itself, so the assignment does nothing.
 
Line 59:
if [ "`uname -s`" = "Darwin" ]
      ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: (apply this, apply all SC2006)
if [ "$(uname -s)" = "Darwin" ]
 
Line 82:
  [ ! -z "$(command -v rustc)" ] || (echo "Error: rustc is not available. Install the rust toolchain (rust + cargo) before proceeding." && exit 1)
    ^-- SC2236 (style): Use -n instead of ! -z.
 
Line 83:
  [ ! -z "${LLVM_OBJDUMP}" ] || (echo "Error: LLVM objdump is not available. Install it (likely via LLVM-clang) before proceeding." && exit 1)
    ^-- SC2236 (style): Use -n instead of ! -z.
 
Line 103:
if $REBUILD = true;
            ^-- SC2283 (error): Remove spaces around = to assign (or use [ ] to compare, or quote '=' if literal).
 
Line 121:
  . ../patch.sh
    ^-- SC1091 (info): Not following: ../patch.sh was not specified as input (see shellcheck -x).
 
Line 135:
    ${CONF_OPTS} \
    ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
    "${CONF_OPTS}" \
 
Line 139:
  ${MAKE} ${MAKE_OPTS}
          ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
  ${MAKE} "${MAKE_OPTS}"
 
Line 147:
  ${CONF_OPTS} \
  ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
  "${CONF_OPTS}" \
 
Line 149:
${MAKE} ${MAKE_OPTS}
        ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
${MAKE} "${MAKE_OPTS}"
 
Line 167:
  if [ "`uname -s`" = "OpenBSD" ];
        ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: (apply this, apply all SC2006)
  if [ "$(uname -s)" = "OpenBSD" ];
 
Line 170:
  elif [ "`uname -s`" = "Darwin" ];
          ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: (apply this, apply all SC2006)
  elif [ "$(uname -s)" = "Darwin" ];
 
Line 181:
  pushd "${FOLDER}/build-release/dist/include"
  ^-- SC3044 (warning): In POSIX sh, 'pushd' is undefined.
 
Line 183:
  popd
  ^-- SC3044 (warning): In POSIX sh, 'popd' is undefined.
 
Line 184:
  pushd "${FOLDER}/build-debug/dist/include"
  ^-- SC3044 (warning): In POSIX sh, 'pushd' is undefined.
 
Line 186:
  popd
  ^-- SC3044 (warning): In POSIX sh, 'popd' is undefined.
 
Line 210:
if [ "`uname -s`" = "Darwin" ]
      ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: (apply this, apply all SC2006)
if [ "$(uname -s)" = "Darwin" ]

Can't fix all of them, but at least some.

libraries/source/spidermonkey/build.sh
154

Would be nice to use set that OS variable on non Windows instead of calling uname -s everywhere.

228–231

It seems we don't need those anymore? I can't even find the DLLs for SM78.
https://trac.wildfiregames.com/browser/ps/trunk/binaries/system

Deleted in rP24332

Stan added inline comments.Feb 1 2022, 1:58 PM
libraries/source/spidermonkey/FixFpNormIssue.diff
61

This breaks on Windows.

error: use of undeclared identifier '__CONCAT'
63

This breaks as well.

c:/Dev/Perso/0ad-git/libraries/source/spidermonkey/mozjs-91.5.0/modules/fdlibm/src/math_private.h(681,24): error: exponent has no digits
                WRAPPED__CONCAT(0x1.8p, LDBL_MANT_DIG) / 2);
libraries/source/spidermonkey/build.sh
53–57

IIRC it doesn't do anything anymore.

80–84
103
181–186

popd and pushd are not posix commands, one can use subshells instead.

Stan requested changes to this revision.Feb 1 2022, 1:58 PM
This revision now requires changes to proceed.Feb 1 2022, 1:58 PM
Stan added a subscriber: jprahman.Aug 9 2022, 8:12 AM

@jprahman the issue is that the new patches applied on top of sm91 prevented it to build on Windows. We first need to address those before switching to 102.