HomeWildfire Games

Always leave/reenter the top-most FSM state when SetNextState is called with…
Needs VerificationrP22023

Description

Always leave/reenter the top-most FSM state when SetNextState is called with the current FSM state.

When SetNextState is called with the current state, it will leave and re-enter so that any relevant cleanup will get called and things work as expected.
It will prevent possibly awkward code issues in UnitAI, which already called SetNextStateAlwaysEntering in a number of places to work around them.

This should most likely have been the default behaviour in the first place.

Reviewed By: Itms

Differential Revision: https://code.wildfiregames.com/D1488

Details

Auditors
elexis
Stan
Committed
wraitiiJan 5 2019, 10:01 AM
Reviewer
Itms
Differential Revision
D1488: Always use SetNextStateAlwaysEntering when processing an order in UnitAI
Parents
rP22022: [Windows] Automated build.
Branches
Unknown
Tags
Unknown
Build Status
Buildable 6711
Build 11039: Post-Commit BuildJenkins

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stan added a comment.Jan 13 2019, 10:39 AM

@Angen and @smiley reported that this commit makes their game segfault. @wraitii, @Itms do you have a clue as to why ?

https://wildfiregames.com/forum/index.php?/topic/25255-a24-crashing-game-hero-related/&do=getNewComment

This commit now has outstanding concerns.Jan 13 2019, 10:39 AM
Silier added a comment.EditedJan 13 2019, 11:16 AM

Adding replays. I had no luck to reproduce bug itself by playing on this revision, but replays keep crashing after pressing No and waiting some time.
What does not uppon 22022

Itms added a comment.Jan 13 2019, 11:26 AM

That's wild! This commit should not be able to create a segfault, so it probably just reveals a bug in the engine. The commands file should be enough to find the bug and fix it.

By the way, the fact that the commands do not create the crash on the previous revision is not a proof that this commit is the issue: since this changes the sim state, replays can be producing a slightly different game on the previous revision.

@Itms no logs no dump files, no errors. Just crash. Btw build date is messed up.

Itms added a comment.Jan 13 2019, 11:31 AM
In rP22023#32208, @Angen wrote:

@Itms no logs no dump files, no errors. Just crash.

A developer should still be able to get a trace or useful information by running the replay in a debugger.

I'll give a go at this in the afternoon I think. Hopefully I should be done with a first working version of my componentManager map>entity map rewrite thingy by then.

elexis added a subscriber: elexis.Jan 13 2019, 12:32 PM

Theres a spidermonkey stack on the forums. Sounds like infinite loop (which could be confirmed by adding some debug output and running the replay.)

(As mentioned in https://code.wildfiregames.com/D1488#69236 the replacement previous SetNextStateAlwaysEntering -> new SetNextState should work, but the previous SetNextState -> new SetNextState triggers new code behavior ("cleanup") in possibly every occurrence that is expected to work but might be verified to be the case.)

Silier added a comment.EditedJan 13 2019, 2:01 PM

It goes -> enter IDLE -> Find Targets -> Order.Attack -> cannot reach -> finishorder -> leave and enter IDLE -> ...

Just small js calls:
7097 enter IDLE
7097 Order.Attack
7097 cant reach
7097 leave IDLE
7097 enter IDLE
7097 Order.Attack
7097 cant reach
7097 leave IDLE
7097 enter IDLE
7097 Order.Attack
7097 cant reach

Silier added inline comments.Jan 13 2019, 2:03 PM
/ps/trunk/binaries/data/mods/public/simulation/components/UnitAI.js
497

this is cant reach in my calls

Stan added inline comments.Apr 16 2019, 6:36 PM
/ps/trunk/binaries/data/mods/public/simulation/components/UnitAI.js
450

This is useless now and should be removed.

wraitii requested verification of this commit.Jun 3 2019, 7:38 PM

@Stan I think this can be marked as fixed.

This commit now requires verification by auditors.Jun 3 2019, 7:38 PM
Stan added a comment.Jun 3 2019, 7:41 PM

@Angen since you were the original reporter do you still experience the bug ? :)

Will close it if not.

Silier added a comment.EditedJun 3 2019, 7:44 PM

no I dont, the problem is fixed for me

Stan accepted this commit.Jun 3 2019, 7:45 PM
All concerns with this commit have now been addressed.Jun 3 2019, 7:45 PM

I want to add more evidence in this ticket about the problem and what get this error!

From the commands.txt I found:

  • There are two entities in this problem, 7584("Bolt Shooter") and 7883("Libyan Spearman")
  • Entity 7584 was moving close to 7883 because an Order.Walk
Executing turn 2164 of 2292
WARNING: ProcessMessage7584-({type:"Order.Stop", data:{force:true}})-1
WARNING: "INDIVIDUAL.WALKING"
WARNING: 7584-"IDLE"-0
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.FinishOrder@simulation/components/UnitAI.js:3603:2
UnitAI.prototype.UnitFsmSpec["Order.Stop"]@simulation/components/UnitAI.js:255:3
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.PushOrder@simulation/components/UnitAI.js:3637:13
UnitAI.prototype.ReplaceOrder@simulation/components/UnitAI.js:3750:3
UnitAI.prototype.AddOrder@simulation/components/UnitAI.js:4830:3
UnitAI.prototype.Stop@simulation/components/UnitAI.js:4959:2
g_Commands.stop/<@simulation/helpers/Commands.js:502:4
g_Commands.stop@simulation/helpers/Commands.js:501:3
ProcessCommand@simulation/helpers/Commands.js:47:3

WARNING: 7584-"INDIVIDUAL.IDLE"-0
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.UnitFsmSpec["Order.Stop"]@simulation/components/UnitAI.js:261:4
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.PushOrder@simulation/components/UnitAI.js:3637:13
UnitAI.prototype.ReplaceOrder@simulation/components/UnitAI.js:3750:3
UnitAI.prototype.AddOrder@simulation/components/UnitAI.js:4830:3
UnitAI.prototype.Stop@simulation/components/UnitAI.js:4959:2
g_Commands.stop/<@simulation/helpers/Commands.js:502:4
g_Commands.stop@simulation/helpers/Commands.js:501:3
ProcessCommand@simulation/helpers/Commands.js:47:3

WARNING: ProcessMessage[RETURN]7584-(void 0)-"INDIVIDUAL.IDLE"
WARNING: ProcessMessage7584-({type:"Order.Walk", data:{x:538.5, z:593.5, min:0, max:15, force:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7584-"INDIVIDUAL.WALKING"-1
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.UnitFsmSpec["Order.Walk"]@simulation/components/UnitAI.js:290:4
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.PushOrder@simulation/components/UnitAI.js:3637:13
UnitAI.prototype.ReplaceOrder@simulation/components/UnitAI.js:3750:3
UnitAI.prototype.AddOrder@simulation/components/UnitAI.js:4830:3
UnitAI.prototype.WalkToPointRange@simulation/components/UnitAI.js:4951:2
g_Commands["walk-to-range"]@simulation/helpers/Commands.js:170:5
ProcessCommand@simulation/helpers/Commands.js:47:3

WARNING: ProcessMessage[RETURN]7584-(void 0)-"INDIVIDUAL.WALKING"
WARNING: ProcessMessage7584-({type:"MoveStarted", data:{starting:true, error:false}})-1
WARNING: "INDIVIDUAL.WALKING"
WARNING: ProcessMessage[RETURN]7584-(void 0)-(void 0)
Executing turn 2165 of 2292

While 7584 was getting its destination the entity 7883 was attacked for another entity

Executing turn 2288 of 2292
WARNING: OnCreate(UnitAI)7883:"Libyan Spearman"
WARNING: ProcessMessage7883-({type:"HealthChanged", from:100, to:158.4})-0
WARNING: "INDIVIDUAL.IDLE"
WARNING: ProcessMessage[RETURN]7883-(void 0)-(void 0)
WARNING: ProcessMessage7883-({type:"HealthChanged", from:158.4, to:5.965006500000037})-0
WARNING: "INDIVIDUAL.IDLE"
WARNING: ProcessMessage[RETURN]7883-(void 0)-(void 0)
WARNING: ProcessMessage7883-({type:"Order.Cheering", data:{force:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7883-"INDIVIDUAL.CHEERING"-1
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.UnitFsmSpec["Order.Cheering"]@simulation/components/UnitAI.js:726:3
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.PushOrder@simulation/components/UnitAI.js:3637:13
UnitAI.prototype.ReplaceOrder@simulation/components/UnitAI.js:3750:3
UnitAI.prototype.AddOrder@simulation/components/UnitAI.js:4830:3
UnitAI.prototype.Cheer@simulation/components/UnitAI.js:5334:2
Promotion.prototype.Promote@simulation/components/Promotion.js:87:3
Promotion.prototype.IncreaseXp@simulation/components/Promotion.js:167:3
Damage.prototype.CauseDamage@simulation/components/Damage.js:256:1
Attack.prototype.PerformAttack@simulation/components/Attack.js:620:1
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.COMBAT.ATTACKING.Timer@simulation/components/UnitAI.js:1951:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.TimerHandler@simulation/components/UnitAI.js:3879:2
Timer.prototype.OnUpdate@simulation/components/Timer.js:120:4

WARNING: ProcessMessage[RETURN]7883-(void 0)-"INDIVIDUAL.CHEERING"
Executing turn 2289 of 2292
WARNING: ProcessMessage7883-({type:"Attacked", data:{attacker:7542, target:7883, type:"Melee", damage:-0, attackerOwner:2}})-2
WARNING: "INDIVIDUAL.CHEERING"
WARNING: ProcessMessage[RETURN]7883-(void 0)-(void 0)
Executing turn 2290 of 2292

And starting to cheering. BTW( Why damage:-0?)

So when the entity 7584 stop and set to state "idle", the INDIVIDUAL.IDLE.ENTER find this target

Executing turn 2292 of 2292
TIMER| common/modern/setup.xml: 221 us
TIMER| common/modern/styles.xml: 247 us
TIMER| common/modern/sprites.xml: 1.002 ms
TIMER| msgbox/msgbox.xml: 14.532 ms
WARNING: ProcessMessage7584-({type:"MoveCompleted", data:{starting:false, error:false}})-1
WARNING: "INDIVIDUAL.WALKING"
WARNING: 7584-"IDLE"-0
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.FinishOrder@simulation/components/UnitAI.js:3603:2
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.WALKING.MoveCompleted@simulation/components/UnitAI.js:1567:5
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.OnMotionChanged@simulation/components/UnitAI.js:3921:3

WARNING: ProcessMessage[RETURN]7584-(void 0)-"IDLE"
WARNING: ProcessMessage7584-({type:"Order.Attack", data:{target:7883, force:false, allowCapture:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7584-"IDLE"-0

So the infinite loop was caused by the method "PushOrderFront" because "Add an order onto the front of the queue, and execute it immediately."

WARNING: ProcessMessage[RETURN]7584-(void 0)-"IDLE"
WARNING: ProcessMessage7584-({type:"Order.Attack", data:{target:7883, force:false, allowCapture:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7584-"IDLE"-0
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.FinishOrder@simulation/components/UnitAI.js:3603:2
UnitAI.prototype.UnitFsmSpec["Order.Attack"]@simulation/components/UnitAI.js:471:4
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
UnitAI.prototype.OnMotionChanged@simulation/components/UnitAI.js:3921:3

WARNING: ProcessMessage[RETURN]7584-(void 0)-"IDLE"
WARNING: ProcessMessage7584-({type:"Order.Attack", data:{target:7883, force:false, allowCapture:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7584-"IDLE"-0

When I was looking at the code of "Order.Attack" I found this lines

// If we can't reach the target, but are standing ground, then abandon this attack order.
// Unless we're hunting, that's a special case where we should continue attacking our target.
if (this.GetStance().respondStandGround && !this.order.data.force && !this.order.data.hunting || this.IsTurret())
{
	this.FinishOrder();
	return;
}

and the entity 7584 in its template has "<DefaultStance>standground</DefaultStance>", and call once again "FinishOrder" and so on and on and on (Infinite Loop)

Now the main reason this didn't happen before its because this line was checking this kind of behavior

if (nextStateName != obj.fsmStateName || obj.fsmReenter)
			this.SwitchToNextState(obj, nextStateName);

So talking about https://code.wildfiregames.com/rP22059, that patch solve infinite loop only for "IDLE" but we currently don't know about another method (state) different from IDLE than cause this same behavior.
I strongly suggest to "check" or find a way to detect we are in a loop inside "FSM.prototype.ProcessMessage", something like "previousState", "CurrentState", "NextState" if all three of them equally just stop, or adding a "deep" property inside FSM. something like

FSM.prototype.ProcessMessage = function(obj, msg)
{
	obj.deepProcessMessage++;
	if (obj.deepProcessMessage >= MAX_CALL_DEEP)
	{
		warn("Infinite Loop detected " + new Error().stack);
		return;
	}
	...
	obj.deepProcessMessage--;
	return ret;
}

In this way we could detected another infinite loops without break the game.

I'm also reading about:
esprima
estraverse
escodegen
esrefactor
esformatter

for code instrumentation, code coverage, check styling, extract graph of the UnitFsmSpec, dependencies and methods call.

I would add more stacktrace and let me know if it useful:

WARNING: ProcessMessage[RETURN]7584-(void 0)-"IDLE"
WARNING: ProcessMessage7584-({type:"Order.Attack", data:{target:7883, force:false, allowCapture:true}})-1
WARNING: "INDIVIDUAL.IDLE"
WARNING: 7584-"IDLE"-0
WARNING: FSM.prototype.SetNextState@simulation/helpers/FSM.js:254:8
UnitAI.prototype.SetNextState@simulation/components/UnitAI.js:3544:2
UnitAI.prototype.FinishOrder@simulation/components/UnitAI.js:3603:2
UnitAI.prototype.UnitFsmSpec["Order.Attack"]@simulation/components/UnitAI.js:471:4
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:274:12
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
**UnitAI.prototype.PushOrderFront@simulation/components/UnitAI.js:3673:13**
UnitAI.prototype.AttackVisibleEntity@simulation/components/UnitAI.js:4559:1
UnitAI.prototype.RespondToTargetedEntities@simulation/components/UnitAI.js:4596:1
UnitAI.prototype.AttackEntitiesByPreference@simulation/components/UnitAI.js:5941:8
UnitAI.prototype.FindNewTargets@simulation/components/UnitAI.js:5430:9
UnitAI.prototype.UnitFsmSpec.INDIVIDUAL.IDLE.enter@simulation/components/UnitAI.js:1509:9
FSM.prototype.SwitchToNextState@simulation/helpers/FSM.js:380:8
FSM.prototype.ProcessMessage@simulation/helpers/FSM.js:288:3
UnitAI.prototype.OnMotionChanged@simulation/components/UnitAI.js:3921:3

@trompetin17 I'm not entirely sure I follow what you are talking about to be honest.
Anyways, this infinite loop with FindNewTarget was also reported, for a different reason, by elexis. I am starting to think that we should Move this call to the 'Timer', to avoid such infinite loops.

elexis added a comment.Jul 2 2019, 2:18 PM

So talking about rP22059, that patch solve infinite loop only for "IDLE" but we currently don't know about another method (state) different from IDLE than cause this same behavior

In #5460 there are one or two ways to reproduce another infinite loop.

The previous code didn't have this class of infinite loops, so there must be a good argument to have the code this way or with the timer, as compared to what it was before when the code abandoned if the state didn't change.

Inline comments denote the SetNextState calls that received behavior changes that were not checked.

/ps/trunk/binaries/data/mods/public/simulation/components/UnitAI.js
233

^

261

^

290

^

315

^

343

^

382

^

396

^

409

^

514

^

537

^

554

^

614

^

675

^

692

^

697

^

711

^

717

^

726

^

733

^

741

^

766

^

773

^

780

^

786

^

793

^

800

^

810

^

840

^

852

^

874

^

879

^

1198

^

1311

^

1330

^

1345

^

1686

^

1714

^

1822

^

1858

^

1975

^

2073

^

2189

^

2381

^

2396

^

2490

^

2555

^

2594

^

2693

^

2718

^

2767

^

2890

^

2960

^

2981

^

3060

^

3103

^

3113

^

3127

^

3170

^

3209

^

elexis raised a concern with this commit.Jul 2 2019, 7:07 PM

The jebel airplane bug in #5460 exists in this revision and later, but not the one before this.

This commit now has outstanding concerns.Jul 2 2019, 7:07 PM

@elexis before reverting, do consider that this fixes D1644 for example - and possibly other similar classes of bugs. Consider also that you're not fixing all currently known infinite loops by doing that.
Also, @Itms agreed with the patch, so perhaps ask for his opinion.

I'll argue more for moving FindNewTargets away from Idle.Enter, but we need a way to not re-run Idle.Enter on any one turn in an infinite loop - because that is going to happen.

elexis added a comment.EditedJul 5 2019, 2:52 PM

Commit incidentally fixed #4132.

wraitii's audit comments at https://code.wildfiregames.com/D1743#70413

wraitii requested verification of this commit.Jul 19 2019, 2:46 PM
This commit now requires verification by auditors.Jul 19 2019, 2:46 PM