HomeWildfire Games

Add an interface for Reinforcement Learning.

Description

Add an interface for Reinforcement Learning.

Implement a simple HTTP server to start games, receive the gamestate and pass commands to the simulation.
This is mainly intended for training reinforcement learning agents in 0 AD. As such, a python client and a small example are included.

This option can be enabled using the -rl-interface flag.

Patch by: irishninja

Reviewed By: wraitii, Itms

Fixes #5548

Differential Revision: https://code.wildfiregames.com/D2199

Event Timeline

Itms awarded a token.Aug 1 2020, 1:09 PM
vladislavbelov raised a concern with this commit.Aug 7 2020, 7:57 PM
vladislavbelov added a subscriber: vladislavbelov.

Too many issues.

/ps/trunk/source/rlinterface/RLInterface.h
26

Braces are on new line, the same below.

31

Too common name in the global scope (also might break ODR for future changes).

36

Too common enum names in the global space, it should use enum class or should be renamed.

45

Too common name in the global scope, and without description.

49

Useless empty line.

52

const std::vector<Command>&.

54

const std::vector<std::string>&.

57

const GameMessage&.

61

bool IsGameRunning() const.

68

Why no CC in naming?

This commit now has outstanding concerns.Aug 7 2020, 7:57 PM
vladislavbelov added inline comments.Aug 9 2020, 2:23 AM
/ps/trunk/source/main.cpp
79

WAT?

@irishninja, could you take a look at the issues Vlad mentions? We should have detected them before, sorry ? You can read the IRC logs for a few more details.

@Itms, Yeah, I will check those issues out and make a new revision!

irishninja added a comment.EditedAug 11 2020, 2:04 PM

I have addressed the concerns in D2947. Let me know what you think!

I believe there was also a discussion around the loading of mods in the RL interface. This could be related to my limited experience with mods but I figured that if there are mods that edit units or create new maps, these should also be available when running 0 AD as an RL environment. It looks like there are some mods that add/modify civilizations which would certainly be relevant when using 0 AD as an RL environment. Most GUI mods would likely not be important.

Itms added a comment.Aug 11 2020, 2:53 PM

I believe there was also a discussion around the loading of mods in the RL interface. This could be related to my limited experience with mods but I figured that if there are mods that edit units or create new maps, these should also be available when running 0 AD as an RL environment. It looks like there are some mods that add/modify civilizations which would certainly be relevant when using 0 AD as an RL environment. Most GUI mods would likely not be important.

The question was rather about the installation of mods in the RL interface. If it is acceptable to ask the user to install mods first (by running the game or moving files to the correct place) before starting the RL interface, the installation code in the RL interface should be removed. During the review, I assumed you wanted people to be able to install mods on the fly when starting the server. ? I should have asked whether it was your intent, so that we would have clarified the installation/loading confusion earlier...

Makes sense. I admit that I had been thinking that there was no need to change most of the initial startup for the game when running the RL interface so I just left it in.

That said, I think it would be great for the RL interface (as well as any future features making ML approaches to game balancing possible) to bring in users who are interesting in some of these ML problems and are not necessarily familiar with 0 AD itself. In this case, I think it is preferable to make the setup as easy as possible and installing the mods automatically when spinning up the RL interface seems like it could remove a step in the process. It certainly would be possible to ask users to simply copy files but this can be a little annoying when explaining the setup imo. (I really like having 1 or 2 commands as a "quick start" rather than requiring users to manually edit files when possible.) Given that I could see mods being very useful for things like containing benchmark scenarios, I think making sure mod installation is a smooth process could be pretty rewarding.

Of course, my experience with mods is admittedly limited so I may be overlooking something. My hope is that using 0 AD as an RL environment (setup and all) will be as painless as possible so it can make an even more compelling case for anyone interested :)

Itms added a comment.EditedAug 11 2020, 4:33 PM

In this case, I think it is preferable to make the setup as easy as possible and installing the mods automatically when spinning up the RL interface seems like it could remove a step in the process.

Yes indeed, it was my reasoning, but are you aware that it is not as easy as just specifying a mod name? The name corresponds to a filename of the zipped mod which has to be placed in the current directory. If you want the experience to be oriented towards new users, it is probably preferable to make them start the game normally and use the mod downloader in the GUI. They can have a look around before starting the RL interface. I was more thinking of containerized environments and the need to not start things several times, which does not correspond to what you describe.

Yeah, that makes sense. It is probably also unnecessary for the containerized case since you can just mount the files into the container and the environment can still be spun up with a single command.

I will go ahead and remove the mod installation step from main.cpp when running the RL interface :)

I removed the mod installation from RunRLServer then realized it was pretty easy to decrease the code duplication from that method (as @vladislavbelov suggested over irc, iirc). Removing the code duplication resulted in keeping the mod installation logic. I am happy to revert if that is preferred; let me know what you think!

wraitii requested verification of this commit.Fri, Jan 15, 5:30 PM
This commit now requires verification by auditors.Fri, Jan 15, 5:30 PM
vladislavbelov resigned from this commit.Fri, Jan 15, 9:04 PM
This commit no longer requires audit.Fri, Jan 15, 9:04 PM