Page MenuHomeWildfire Games

Version and update the Windows autobuilder script
ClosedPublic

Authored by Itms on May 19 2019, 8:57 PM.

Details

Summary

Puts into version control the script used by Jenkins to autobuild on Windows. Update it to use VS 2015 and Jenkins Pipeline.

This could be adapted into a script that builds patches on Windows.

Test Plan

Test that all stages of script works as intended

Diff Detail

Repository
rP 0 A.D. Public Repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/1463/display/redirect

vladislavbelov added inline comments.
build/jenkins/0ad_win.bat
6 ↗(On Diff #8088)

Does it work? Spaces in the name?

Also, wouldn't it be good to use a later version? It has less problems.

Stan added a subscriber: Stan.May 19 2019, 9:44 PM
Stan added inline comments.
build/jenkins/0ad_win.bat
6 ↗(On Diff #8088)

We should always use the lowest supported version to detect breakages. One could also use multiple version. I have a powershell script that does that for me :)

There is no space in the name btw, it's src, dest :)

8 ↗(On Diff #8088)

I thought autobuild wasnt' large adress aware, interesting.

10 ↗(On Diff #8088)

Might want to delete more ?

build/jenkins/0ad_win.bat
6 ↗(On Diff #8088)

We should always use the lowest supported version to detect breakages. One could also use multiple version. I have a powershell script that does that for me.

I suppose it's not useful for Windows, because we pack wxWidgets for releases by ourselves. Also such detecting doesn't work very good for wxWidgets, because it has different behaviour for different versions.

There is no space in the name btw, it's src, dest :)

Ah, thx, I didn't notice :)

Itms added a comment.May 20 2019, 8:34 PM

@Angen Marking for you the places where the scripts are actually cut on Jenkins.

build/jenkins/0ad_win.bat
8 ↗(On Diff #8088)

It is cut after this;

38 ↗(On Diff #8088)

It is also cut after here.

Stan added inline comments.May 20 2019, 8:37 PM
build/jenkins/0ad_win.bat
38 ↗(On Diff #8088)

I usually pass

/t:test;pyrogenesis /p:Configuration="Release" /m /p:PlatformToolset=v120_xp /maxCpuCount /clp:Warningsonly

Stan added inline comments.May 20 2019, 8:39 PM
build/jenkins/0ad_win.bat
38 ↗(On Diff #8088)

One could also check if Msbuild is in the path, sometimes it isn't.

Angen commandeered this revision.Wed, May 29, 7:40 PM
Angen updated this revision to Diff 8205.
Angen edited reviewers, added: Itms; removed: Angen.

my first pipeline version of the bat file

Stan added inline comments.Wed, May 29, 7:53 PM
win_autobuild
14 ↗(On Diff #8205)

One could delete *.pdb, *.ilk, *.exp, *.lib as well
for Collada and AtlasUI when building a new one only
Same for pyrogenesis

57 ↗(On Diff #8205)

Add newline.

Itms requested changes to this revision.Wed, May 29, 9:14 PM

Very nice! I'm proposing the following changes. They are either changes from the scripted pipeline syntax to declarative, or improvements that we could make to the current bat script (so, not a criticism towards your work). The only bug/puzzling thing is the absence of the repo address, which makes me think like this pipeline shouldn't work? Unless you checked the repo out manually once?

win_autobuild
1 ↗(On Diff #8205)

Let's use the

pipeline {
    agent { label 'WindowsSlave' }
    stages {

syntax for consistency with the Linux pipeline.

You can add that label to your local agent to make it work on your machine.

5 ↗(On Diff #8205)

We could get rid of all these variable definitions by just reverting everything after commits, and using the build parameters only when composing the changelist in the last step.

7 ↗(On Diff #8205)

The address of the source repository seems to be missing, how do you checkout the repo the first time?

8 ↗(On Diff #8205)

You can remove all the parenthesis when the directives have only one parameter (like here bat "svn up"). We don't have style conventions yet but I think that makes things more readable 🙂

16 ↗(On Diff #8205)

Use when (https://jenkins.io/doc/book/pipeline/syntax/#when) instead of scripted pipeline syntax as much as possible.

36 ↗(On Diff #8205)

Let's move to VS2015 at the same time.

This revision now requires changes to proceed.Wed, May 29, 9:14 PM
Angen marked 5 inline comments as done.Thu, May 30, 9:52 PM
Angen added inline comments.
win_autobuild
16 ↗(On Diff #8205)

as I understand syntax is
when {}
steps {}

problem is that I can have only one when steps in stage
so I dont think it is possible if we do not want about 8 more stages ?

Itms added inline comments.Thu, May 30, 10:14 PM
win_autobuild
16 ↗(On Diff #8205)

Oh sorry you're right, here it would be a waste. However this seems quite verbose, doesn't if ($atlas == true) or even if ($atlas == "true") work? 🤔

I need help, or this is going to be without tests:
I cannot call test.exe from jenkins pipeline somehow.
It runs forever:
bat "binaries\\system\\test.exe > cxxtest_release.xml"
the same forever run for:
bat "binaries\\system\\test.exe"

Itms added a comment.Fri, May 31, 7:27 AM

You can upload with Planned Changes and I'll test.

Angen updated this revision to Diff 8230.Fri, May 31, 9:57 AM
Angen edited the test plan for this revision. (Show Details)

There is problem with testing stage, test.exe does not end nor provides output

Angen planned changes to this revision.Fri, May 31, 9:57 AM
Angen planned changes to this revision.Fri, May 31, 9:11 PM
Angen updated this revision to Diff 8254.Fri, May 31, 10:16 PM

move output to script thnx @Stan

Angen planned changes to this revision.Fri, May 31, 10:16 PM
Stan added a comment.EditedSat, Jun 1, 2:45 PM

Here is my version of the script. Builds debug and release and the tests

	def atlasPrj = ""
	def output = ""
	def jobs = "8"
	def atlasOption = ""
	def glooxOption = "--use-shared-glooxwrapper"
	pipeline {
		agent { label 'WindowsSlave' }
		stages {
			stage("Checkout") {
				options {
					retry(3)
				}
				steps {
					script {
						try {
							svn "https://svn.wildfiregames.com/public/ps/trunk"
						} catch(e) {
							bat "svn cleanup"
							sleep 300
							throw e
						}
					}
					bat "svn cleanup"
				}
			}
			stage('Setup workspace') {
				steps {
					bat 'robocopy /MIR C:\\wxwidgets3.0.2\\lib libraries\\win32\\wxwidgets\\lib & IF %ERRORLEVEL% LEQ 1 exit 0'
					bat 'robocopy /MIR C:\\wxwidgets3.0.2\\include libraries\\win32\\wxwidgets\\include & IF %ERRORLEVEL% LEQ 1 exit 0'
					script {
						if (env.atlas.toBoolean() == true) {
							atlasPrj = ";AtlasUI"
							atlasOption = "--atlas"
							bat "del binaries\\system\\AtlasUI.dll"
							echo "atlas is enabled"
						}
						if (env.gloox.toBoolean() == true) {
							bat "del binaries\\system\\glooxwrapper.pdb binaries\\system\\glooxwrapper.lib binaries\\system\\glooxwrapper.dll"
							glooxOption = "--build-shared-glooxwrapper"
							echo "gloox is enabled"
						}
						if(env.collada.toBoolean() == true) {
							bat "del binaries\\system\\Collada.dll"
							echo "collada is enabled"
						}

						bat "cd build\\premake && if not exist ..\\workspaces\\vc2015\\SKIP_PREMAKE_HERE premake5\\bin\\release\\premake5 --outpath=\"..\\workspaces\\vc2015\" ${atlasOption} ${glooxOption} --large-address-aware --jenkins-tests%* vs2015"
						bat 'del binaries\\system\\pyrogenesis.pdb'
                        bat 'del binaries\\system\\pyrogenesis_dbg.exe'
						bat 'del binaries\\system\\pyrogenesis.exe'
						bat 'del binaries\\system\\test_dbg.exe'
						bat 'del binaries\\system\\test.exe'
						output = bat(returnStdout: true, script: 'svnversion source -n').trim()
						output = output.readLines().drop(1).join("")
					}
					bat "echo L\"${output}\" > build\\svn_revision\\svn_revision.txt"
				}
			}
			stage ('Build') {
				steps {
					bat "cd build\\workspaces\\vc2015 && MSBuild.exe pyrogenesis.sln -m:${jobs} -t:pyrogenesis;test%atlasPrj% -clp:Warningsonly -clp:ErrorsOnly -p:Configuration=\"Release\""
					bat "cd build\\workspaces\\vc2015 && MSBuild.exe pyrogenesis.sln -m:${jobs} -t:pyrogenesis;test%atlasPrj% -clp:Warningsonly -clp:ErrorsOnly -p:Configuration=\"Debug\""
				}
			}
			stage ('Tests') {
				steps {
					script {
						bat 'del .\\cxxtest_debug.xml'
						bat 'del .\\cxxtest_release.xml'
						bat 'binaries\\system\\test_dbg.exe > cxxtest_debug.xml'
						bat 'binaries\\system\\test.exe > cxxtest_release.xml'
					}
				}
				post {
					always {
						junit "cxxtest_*.xml"
					}
				}
			}
			stage ('Commit') {
				steps {
					bat "svn changelist --remove --recursive --cl commit"
					script {
						if (env.pyrogenesis.toBoolean() == true) {
							bat "svn changelist commit binaries\\system\\pyrogenesis.pdb binaries\\system\\pyrogenesis.exe"
						}
						if (env.atlas.toBoolean() == true) {
							bat "svn changelist commit binaries\\system\\AtlasUI.dll"
						}
						if (env.collada.toBoolean() == true) {
							bat "svn changelist commit binaries\\system\\Collada.dll"
						}
						if (env.gloox.toBoolean() == true) {
							bat "svn changelist commit binaries\\system\\glooxwrapper.dll binaries\\system\\glooxwrapper.lib binaries\\system\\glooxwrapper.pdb"
						}
					}
					bat 'svn ci --changelist commit --non-interactive -m "[Windows] Automated build."'
				}
			}
		}
		post {
			always {
				bat "svn revert -R ."
			}
		}
	}

I had the same issue as @Angen with the test not running, it seems a system account doesn't have sufficient privileges to run them...
So one needs to use a specific account in services.msc

Also the build won't run if the repo is in a specific folder with spaces such as program files.

I also added

  • number of cores,
  • fixed options, though we could add large address aware and nvtt.
  • I don't call the bat and instead call premake directly.
  • One could use to download and update specific wxwidgets versions, (maybe a pipeline script ?)
Angen updated this revision to Diff 8275.Sat, Jun 1, 6:50 PM
Angen edited the summary of this revision. (Show Details)

thnx to @Stan testing stage works ( like worked before but now I know it works )

Owners added a subscriber: Restricted Owners Package.Sat, Jun 1, 6:50 PM
Vulcan added a comment.Sat, Jun 1, 6:52 PM

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/1580/display/redirect

Itms added a comment.Sun, Jun 2, 8:40 AM

Good work! I am currently testing the pipeline on the autobuilder with the following changes.

Side note: I am wondering about the deletion of binaries that we already performed in the previous script. It doesn't seem needed since we create a changelist properly... What do you think Angen?

build/jenkins/win_autobuild.Jenkinsfile
15 ↗(On Diff #8275)

I'm using the https://svn.wildfiregames.com/public/ps/trunk in order to be able to commit.

31 ↗(On Diff #8275)

I have built the newer version 3.0.4 on the autobuilder.

41 ↗(On Diff #8275)

extra space

48 ↗(On Diff #8275)

I am not going to send the test results to Jenkins for the autobuilder, so I'm removing --jenkins-tests. This will be useful when writing a pipeline for testing patches though.

55 ↗(On Diff #8275)

removed the debug build, also useful for patches

65 ↗(On Diff #8275)

removed, also useful for patches

Angen updated this revision to Diff 8280.Sun, Jun 2, 10:06 AM

commit debuged gloox, remove some tests

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/1585/display/redirect

Itms added inline comments.Sun, Jun 2, 9:53 PM
build/jenkins/win_autobuild.Jenkinsfile
6 ↗(On Diff #8280)

Adding a parameters section is a good habit, it allows them to be documented in the pipeline script.

32 ↗(On Diff #8280)

I had forgotten to copy the files at the correct location and this didn't fail properly. I got

2019/06/02 15:56:28 ERROR 3 (0x00000003) Accessing Source Directory C:\wxwidgets3.0.4\lib\

The system cannot find the path specified.

but the build wasn't marked as failed.

55 ↗(On Diff #8280)

It's a bit irritating not to know when the build is likely to finish, so I removed -clp:Warningsonly -clp:ErrorsOnly in order to have the build output along the way.

66 ↗(On Diff #8280)

Hum I have an issue here, the step never finishes (and I don't have the output from the tests, so maybe it actually never starts). Can you reproduce that?

Angen marked an inline comment as done.Sun, Jun 2, 11:32 PM
Angen added inline comments.
build/jenkins/win_autobuild.Jenkinsfile
32 ↗(On Diff #8280)

fixed: & -> ^&

Stan added inline comments.Mon, Jun 3, 8:46 AM
build/jenkins/win_autobuild.Jenkinsfile
66 ↗(On Diff #8280)

That's because the Jenkins service doesn't have enough permissions to run the tests as a system account. Msbuild is somehow whitelisted. But that means you need to run Jenkins as another user the easiest being the user that usually logs in to the machine.

Angen updated this revision to Diff 8284.Mon, Jun 3, 9:08 AM
Angen marked an inline comment as done.

fix robocopy return code

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/1588/display/redirect

Itms commandeered this revision.Tue, Jun 4, 4:34 PM
Itms edited reviewers, added: Angen; removed: Itms.
Itms updated this revision to Diff 8303.Tue, Jun 4, 4:37 PM
Itms retitled this revision from Version the Windows autobuilder script to Version and update the Windows autobuilder script.
Itms edited the summary of this revision. (Show Details)

Final version currently in place on Jenkins.

Stan added inline comments.Tue, Jun 4, 4:41 PM
build/jenkins/pipelines/vs2015_autobuild.Jenkinsfile
54 ↗(On Diff #8303)

Might want to delete the PDB as well for consistency. Also the lib files maybe ?

98 ↗(On Diff #8303)

Missing pdb here and above, useful when receiving callstacks.

Angen accepted this revision.Tue, Jun 4, 4:41 PM
This revision is now accepted and ready to land.Tue, Jun 4, 4:41 PM
Vulcan added a comment.Tue, Jun 4, 4:53 PM

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/differential/1598/display/redirect

This revision was automatically updated to reflect the committed changes.