Page MenuHomeWildfire Games

AIManager: loads only the used templates when starting (or deserializing) a game
ClosedPublic

Authored by mimo on Nov 20 2017, 10:57 PM.

Details

Reviewers
wraitii
Group Reviewers
Restricted Owners Package(Owns No Changed Paths)
Commits
rP20509: AIManager: loads only the used templates when starting (or deserializing) a game
Summary

As the tithe say.
As now the number of templates loaded is much reduced (factor 10) and because we can no more start to load templates when the AIManager is initialized (we have to wait to know which templates are needed), the patch also removes the use of the ProgressiveLoad messages which was no more useful and only complicates the code.

Test Plan

Check that everything works as expected, either for a new game or when deserializing.

Diff Detail

Repository
rP 0 A.D. Public Repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mimo created this revision.Nov 20 2017, 10:57 PM
Owners added a subscriber: Restricted Owners Package.Nov 20 2017, 10:57 PM
Vulcan added a subscriber: Vulcan.Nov 20 2017, 11:45 PM

Successful build - Chance fights ever on the side of the prudent.

Updating workspaces...
Build (release)...
Build (debug)...
Running release tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Running debug tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Checking XML files...
Relax-NG validity error : Extra element props in interleave
/public/art/actors/structures/carthaginians/stable_elephant.xml:0: Relax-NG validity error : Element variant failed to validate content
/public/art/actors/structures/carthaginians/stable_elephant.xml:0: Relax-NG validity error : Element group has extra content: variant
Relax-NG validity error : Extra element group in interleave
/public/art/actors/structures/carthaginians/stable_elephant.xml:0: Relax-NG validity error : Element actor failed to validate content
Executing section Default...
Executing section Source...
Executing section JS...
leper added a reviewer: Restricted Owners Package.Nov 21 2017, 4:00 AM
leper added a subscriber: leper.
leper added inline comments.
source/simulation2/components/CCmpAIManager.cpp
150 ↗(On Diff #4281)

Do we actually have any AIs that don't use this? And if we don't, is this still worth supporting?

245 ↗(On Diff #4281)

const

1185 ↗(On Diff #4281)

Could be a range-based for loop, no?

With C++20 we could do for (const auto& templateNames = cmp->Bar(), usedTemplates.reserve(templateNames.size()); const auto& template : templateNames), but well.

source/simulation2/components/CCmpTemplateManager.cpp
225 ↗(On Diff #4281)

Depending on the number of templates on maps this might not be optimal. (In the worst case we do about |m_LatestTemplates|*|usedTemplates| - |usedTemplates|^2/2 steps, which could be worse than adding everthing, sorting, calling unique (which would be |m_LatestTemplates|*log(|m_LatestTemplates|)).)

I'm not sure if this is worth worrying about, but would this be nicer if we just kept track of templates and a count? Or possibly returned more templates than needed by just using m_TemplateSchemaValidity? Or just using a map here for the looping (depending on the size of usedTemplates, that might be slower)?

I guess you could just check how long this takes on most maps and point out that it is faster than what we did previously and possibly add a TODO that this could be improved if one really wanted to.

mimo added a comment.Nov 21 2017, 8:17 PM

Results of some timing tests:

starting a new scenario azure coast 3 (8700 entities for 86 templates to load)
current svn = 3.5 ms (FindAlltemplates) + 163 ms (loading the 1130 templates)
modif A = 2.2 ms (FindUsedTemplates using vector) + 8 ms (loading 86 templates)
modif B = 5.3 ms (FindUsedTemplates using sort+unique) + 8 ms
modif C = 2.9 ms (FindUsedTemplates using set) + 8 ms
modif D = 2.8 ms (FindUsedTemplates using unordered_set) + 8 ms

starting a new random giant map with 8 players on schwarzwald (9350 entities for 62 templates to load)
current svn = 3.5 ms (FindAlltemplates) + 200 ms (loading the 1130 templates)
modif A = 2.8 ms (FindUsedTemplates using vector) + 8 ms (loading 62 templates)
modif B = 6.0 ms (FindUsedTemplates using sort+unique) + 8 ms
modif C = 3.5 ms (FindUsedTemplates using set) + 8 ms
modif D = 2.9 ms (FindUsedTemplates using unordered_set) + 8 ms

loading a saved game with 4 players on medium map, (3400 entities for 135 templates to load)
current svn = 4 ms (FindAlltemplates) + 275 ms (loading the 1130 templates)
modif A = 0.5 ms (FindUsedTemplates using vector) + 50 ms (loading 135 templates)
modif B = 2.0 ms (FindUsedTemplates using sort+unique) + 50 ms
modif C = 0.8 ms (FindUsedTemplates using set) + 50 ms
modif D = 1.3 ms (FindUsedTemplates using unordered_set) + 50 ms

Although these numbers do not fully make sense (may depend on the load of my pc at that precise moment), the global pattern is clear: the main gain is from not loading all templates, and the way we recover the used templates does not matter with our current number of entities/templates. Nonetheless using an unordered_set could be a bit better for scaling with more templates.

source/simulation2/components/CCmpAIManager.cpp
150 ↗(On Diff #4281)

Agree that we should get rid of that, I don't think there are any gain to create any new AI not using shared component. But that's not for that patch.

245 ↗(On Diff #4281)

yep

1185 ↗(On Diff #4281)

Sure for the loop (I focused too much at the way the old code worked).

source/simulation2/components/CCmpTemplateManager.cpp
225 ↗(On Diff #4281)

I did some tests (see below), and the main conclusion is that the time is completely negligible with our number of entities and templates, and going with an unordered_set seems the best for scaling with the number of templates.

m_TemplateSchemaValidity can't be used as not set when deserializing (nor when testing without validation), and that would look like a hack. We could nonetheless create a new unordered_set to keep track of templates, but not really worth as used only once.

mimo updated this revision to Diff 4311.Nov 21 2017, 8:23 PM
mimo edited the summary of this revision. (Show Details)

update following leper's comment

Executing section Default...
Executing section Source...
Executing section JS...
leper added inline comments.Nov 21 2017, 9:10 PM
source/simulation2/components/CCmpAIManager.cpp
1184 ↗(On Diff #4311)

&

1188 ↗(On Diff #4311)

Ah, I think that was just broken on gcc 4.6, then 4.7 had proper docs and 4.8 had it implemented. (and we dropped < 4.8).

source/simulation2/components/CCmpTemplateManager.cpp
225 ↗(On Diff #4281)

I had a suspicion that it might just not matter for the amount of starting entities we have. Nice to have that confirmed with some tests.

Given those numbers I'm not sure if an unordered_set is that important, especially since we'd either have to sort it at some point, or run into some risks with differently ordered lists passed to JS.

Oh, this is not using vector anymore. Give the experimental results and the possibility for platform differences this way I do prefer the vector variant.

mimo updated this revision to Diff 4313.Nov 21 2017, 10:12 PM

updated with vectors

Successful build - Chance fights ever on the side of the prudent.

Updating workspaces...
Build (release)...
Build (debug)...
Running release tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Running debug tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Checking XML files...
Executing section Default...
Executing section Source...
Executing section JS...

Successful build - Chance fights ever on the side of the prudent.

Updating workspaces...
Build (release)...
Build (debug)...
Running release tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Running debug tests...
Running cxxtest tests (307 tests)...................................................................................................................................................................................................................................................................................................................OK!
Checking XML files...

Code looks fine to me. Didn't do any testing though.

wraitii added a subscriber: wraitii.

I'll run a few games later today and if it works (which it should) I'll accept this. I just have some small nitpicks that can be fixed before committing.

source/simulation2/components/CCmpAIManager.cpp
1038 ↗(On Diff #4313)

This seems odd

150 ↗(On Diff #4281)

IIRC I kept it around back in the days for the tutorial AI, which may or may not have been removed. Agreed that it should be used for any new AI.

source/simulation2/components/CCmpTemplateManager.cpp
221 ↗(On Diff #4313)

You might want to reserve an arbitrary number here, just to avoid uselessly reallocating too much (as I assume we generally will have more than 8/16 templates). Won't really matter but it's good form.

source/simulation2/components/ICmpTemplateManager.h
107 ↗(On Diff #4313)

I don't really understand what this line means. What's the condition to be "validly passed"?

mimo added inline comments.Nov 22 2017, 7:47 PM
source/simulation2/components/CCmpAIManager.cpp
1038 ↗(On Diff #4313)

I've no idea what it is used for.
But anyway, all this component would be better with a good cleanup (and threading).

source/simulation2/components/CCmpTemplateManager.cpp
221 ↗(On Diff #4313)

I don't know. This function is only called once, with about 100 entries currently (so we don't care of the overhead of reallocation), and we don't know a priori what will be the number of templates in mods. So it would just be a magic number.

source/simulation2/components/ICmpTemplateManager.h
107 ↗(On Diff #4313)

I've just kept the same wording as was used before (for findAllTemplates). These templates are loaded using GetTemplateWithoutValidation, so they are expected to exist and be valid.

This revision was automatically updated to reflect the committed changes.