Hello and thanks for the patch ! I'm not a programmer so I won't be able to review it fully. I will make some general comments though, I'll have a closer look today :)

After a second look at it RL code should probably be moved to libraries

Hey, thanks for contributing this on Phabricator :)

Few things first:

• yes, this will need to be behind a feature flag I think.
• I wouldn't version control the generated files but rather write a script to generate them when building workspaces. (also it would be easier to read on phabricator)

Implementation-wise, I have one high level comment for now: I don't think we want game configs to be hardcoded in C++ . It's not performance-sensitive, and I would much rather pass it as JSON and keep the engine mostly agnostic of it. Did you have a particular reason to not pass it as a string, like you did template data?
(template data is one thing where a gRPC-specific channel might matter enough...)

What about the test results from the patch, and the feasibility of machine learning in 0ad?
The example AI can move an army but not much more? What is it's incentive (how is it training?)?
What is the prospect of the feature?

Maybe some more insights can be found here https://github.com/0ad/0ad/pull/25

I personally think this can be a great feature, and could maybe be interesting outside the scope of 0 A.D. it's the second time people try to develop such a feature. It would probably make things like Open AI easier to implement.

Thanks for the feedback! I can move it to be behind a feature flag and remove the generated files.

I hardcoded the game configs because I started from the code to start a scenario from the command line and then refactored it so I could use basically the same logic when I receive a protobuf message (and then I wasn't copying code from creating the game config from the CLI but sharing the logic to create the JSON). It also made sure that defaults were the same, too, so there would hopefully be minimal surprises.

I can certainly move this logic to the python side (that is, wrap the game config like I did with actions and send them as a string). On the positive side, other games using pyrogenesis would also be able to use the python wrapper as long as they provided their own ScenarioConfig class. Let me know if you prefer this approach and I can update the diff.

As @Stan said, there is more discussion on the PR on GitHub. The goal of this contribution is to provide the fundamental underlying capabilities for exploring RL/ML agents in 0 AD (such as an OpenAI gym environment which can actually be implemented on top of this quite easily). I am intentionally not adding any action/observation spaces or reward as expected by OpenAI gym as these are not obvious and there are many different candidates for each of these (this is discussed more on the GitHub thread).

That said, defining an OpenAI gym environment is pretty straight-forward and can be done entirely in Python (no need to change 0 AD at all once this is integrated!). A gym environment needs:

• a conversion from the symbolic game state to a fixed size observation tensor
• an action space and mapping to 0 AD JSON commands. For example, there could be a discrete action space with 2 actions which could be (semantically) mapped to "attack" and "retreat" actions. When the RL agent takes the first action, it would need to be converted to a valid JSON command which would make the player's units attack (or whatever we want to action to actually do).
• a function to compute the reward given the interaction (ie, previous state, action, and current state)

It is worth mentioning that there are many different levels of abstraction which could be used for defining both the observation and action spaces. There are also many different types of reward functions which could be used including reward shaping techniques such as using damage dealt, units killed, etc. There has also been some interesting work on companion AI in games that was trained using a reward based on what the other player was able to do (it learned to enable the player to be successful rather than win itself). More information about OpenAI gym spaces can be found at http://gym.openai.com/docs/#spaces.

Anyway, I was hoping to integrate the fundamental (unopinionated) capabilities so that 0 AD could be used to explore the myriad of possibilities that are out there. I am hoping that this will both help researchers as well as make 0 AD more awesome :)

More specific answers to @elexis 's comments are below:

What about the test results from the patch, and the feasibility of machine learning in 0ad?

About the test results, this is likely a mistake on my part due to a lack of familiarity with Phabricator (and the included tools). I am not sure why they were skipped but I can try to find out how to run them.

I am not exactly sure if you are considering feasibility of 0 AD as an environment (ie, can 0 AD be used to train an RL agent) or existing algorithms/compute. As far as feasibility of 0 AD as an environment, 0 AD is great imo as it can be run headlessly and on linux (convenient since linux is such a common platform for ML). This also makes deployment/installation easy as it can be run in a docker container and only requires docker to be installed. Some of the other complex OpenAI gym environments require some tricks with xvfb and other x11 tricks to get it running in a headless linux environment.

As far as feasibility of existing algorithms, there have been some promising results with AlphaStar and OpenAI Five but it is certainly challenging to train an RL agent to play the game completely end-to-end. However, there are simpler integrations of AI that could be fruitful such as learning a high level controller that decides when to attack or what build order to follow.

The example AI can move an army but not much more? What is it's incentive (how is it training?)?

Currently, I have mostly been moving units and attacking others. Commands are sent to the game engine as JSON so anything supported by the game engine should be supported by the RPC interface. That said, I haven't added all of the game engines to the python wrapper (though they certainly can be sent regardless) just yet. As far as the reward signal used during training, this is expected to be implemented on the Python side and is mentioned above.

@wraitii @elexis any changes you'd like to be made for that patch ?

Maybe you could split the gamesetup → gameconfig change to another patch ?

See discussion on http://irclogs.wildfiregames.com/2019-09/2019-09-11-QuakeNet-%230ad-dev.log

Recap:

• There are two tickets about Gamesetup.cpp autostart options needing refactoring
• There exist map specific options, so hardcoding gamesettings in C++ is not so ideal (for example random map biome supported by about 20 maps and not supported by about 60). The idea was that one could specify one JS file with JS GUI logic for that setting. There could be a python handler for this specific setting as well. And then the entire thing could be passed as JSON from python. That would mean the *.proto file doesn't have to hardcode the specific settings and is extensible too. Incidentally making this patch shorter (but Gamesetup.cpp still hardcodes the settings. If you want to clean up that file, it can be done independently in the course of the referred tickets in the irclogs)
• If the ToJSValue function remains, it could become ScriptInterface::ToJSVal (doesn't have to be actually, since the thing is big and grouping it in a class doesnt sound wrong)
• Adding the feature itself seems still ok, even if there wont be a functional AI at first, since it provides an interface for people interested in ML to implement that. There were the other ML people on the forums, perhaps they could be convinced that this approach is what they want as well? (Bundling the forces, or having them point out features that this diff misses)
• Maintaining the library is a mess, distributing it with it even more - it sounded okay if it is excluded by default and only tested that it still works once before the release (refs longer release process).

I wonder if one could hook petra to it as a separate node.js server.

In D2199#99595, @irishninja wrote:

Updates:

• Define ScenarioConfig in Python. Added basic helpers for defining victory conditions as well as more generic setters (set_map_setting, set_game_setting).
• Refactored RL Server main loop to own method
• Changed default build to not include RL interface (uses --with-rlinterface)

Nice I had a chat with Itms and we agreed it's better that way since most people won't use it :)

Next Steps:

• Make another example demonstrating usage with OpenAI gym. I am not sure if this should be included here or in its own repository...

I guess the external code should be provided as a git link or something like all the alternative ais.

• Add windows support (should only be needed for the code generation step in the premake). Any help here would be appreciated as I am a linux user...

Tell me what you need you can ping me on IRc any time or @tell in case I'm not here :)

• Write more tests in python for features not covered by the example script. This includes:
  • setting game speed
  • setting map size
  • configuring built-in AI opponent

As an update, I am currently working on adding tests for the actions and scenario config (using pytest). I have tests for all the actions in zero_ad/actions.py and a number of the different scenario settings. I am currently working on testing random map usage. After finishing the tests, I am planning on training a simple agent as an example demonstrating how it can be used to expose an OpenAI Gym API to 0 AD :) (I will probably do this in an external repo and then just post a link to it.)

Also, thanks again, @Stan for helping out with windows support :)

This adds a couple dependencies when building with the --with-rlinterface (passed to ./update-workspaces.sh):

• GRPC (C++ and Python)
  • Protobuf
  • Python support requires grpcio-tools package
• boost fiber
• boost system

Installation instructions for building on linux (ubuntu):

First, install the required boost libraries:

sudo apt-get install libboost-fiber-dev libboost-system-dev

Next, build GRPC from source (includes protobuf):

git clone --recurse-submodules https://github.com/grpc/grpc
cd grpc && git checkout v1.25.0
sudo make install

Install protobuf (using the submodule used in the GRPC repo)

cd third_party/protobuf
sudo make install

Install python dependencies for generating the python RPC/protobuf files (using pip):

pip install grpcio-tools

Depending on your pip installation and version of ubuntu, you may need to update setuptools (pip install setuptools -U and/or install it for the given user (add the --user flag to the install command).

On other linux distributions, the first command will likely change as the fibers and system boost libraries may be installed with a different command. Additionally, if the distribution (such as arch linux) has a more up-to-date version of grpc in the repos, it may be possible to simply install grpc from the repositories rather than building from source.