Page MenuHomeWildfire Games

i18n scripts - switch from pology to Babel for .POT generation and update to python3
Needs ReviewPublic

Authored by wraitii on Fri, May 22, 12:17 PM.

Details

Reviewers
Itms
Gallaecio
Summary

Related to D2501 and D506.

To generate .POT files, we are currently using the pology library, which has not updated to python3 (at the moment of writing this, there is only a stable branch on their repo).
pology has the further drawback of not being available on pip or usual distributions.

babel is a BSD-licensed python i18n library that also has support for .POT generation, which seems sufficient for our use-case.
It does require some slight tweaking to perform equivalently to our current setup, but nothing too major.

The generated POT messages are not in exactly the same order, but are compatible from what I can tell.


This also updates scripts to run with python3 (see also D2501).
I'm also bundling in a requirements.txt and a simple command-line argument to not-reparse everything.
I am further using Multiprocessing to generate the template files, which is up to [n_cores] faster.


I am unclear if we need to provide a LICENSE for this?


ping @Gallaecio, @Itms

Test Plan

Run the extractors and compare files.

Event Timeline

wraitii created this revision.Fri, May 22, 12:17 PM
wraitii edited the summary of this revision. (Show Details)

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2196/display/redirect

This sounds like a good idea 👍

Right now we also use pology in checkTranslationsForSpam.py and generateLongStringTranslations.py, they should be updated as well.

Our most peculiar use of pology tools is in maintenanceTasks.sh: we use the CLI tool poediff to avoid committing unneeded changes. Does Babel provide this kind of tool?

wraitii planned changes to this revision.Fri, May 22, 1:45 PM
In D2757#116518, @Itms wrote:

Right now we also use pology in checkTranslationsForSpam.py and generateLongStringTranslations.py, they should be updated as well.

Ah, indeed.

In D2757#116518, @Itms wrote:

Our most peculiar use of pology tools is in maintenanceTasks.sh: we use the CLI tool poediff to avoid committing unneeded changes. Does Babel provide this kind of tool?

From what I can see, we merely check if there have been "significant" changes, aka changes to any message?
Babel can sort the POT template, which might be a partial fix for that. For the rest, it sounds vaguely greppable?

wraitii updated this revision to Diff 11975.Fri, May 22, 3:44 PM

Same process for checkTranslationsForSpam.py.

I've had to change the regex, I'm kinda not sure it ever worked correctly?

Haven't done the long-translation thing yet because I don't really understand what it is supposed to be doing.

Itms added a comment.Fri, May 22, 3:48 PM

Haven't done the long-translation thing yet because I don't really understand what it is supposed to be doing.

This one goes through all Po files and for each original string, it uses the longest translation it can find in all languages. It generates a dummy language with all the longest translations, so that by testing this dummy language you can make sure the GUI elements (buttons, etc) are large enough to accommodate for all languages.

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2201/display/redirect

In D2757#116560, @Itms wrote:

Haven't done the long-translation thing yet because I don't really understand what it is supposed to be doing.

This one goes through all Po files and for each original string, it uses the longest translation it can find in all languages. It generates a dummy language with all the longest translations, so that by testing this dummy language you can make sure the GUI elements (buttons, etc) are large enough to accommodate for all languages.

Ah, alright, thanks :) .
Then I'll update it too.

Can you tell me what we're using poediff for exactly too?

wraitii updated this revision to Diff 12028.Tue, May 26, 2:57 PM

Update generateLongStrings (quite a funny output :p ). Also cleanup a a little more in a bunch of places.

I'm still unsure what poediff does, so that hasn't changed.

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2243/display/redirect

Build failure - The Moirai have given mortals hearts that can endure.

Link to build: https://jenkins.wildfiregames.com/job/macos-differential/841/display/redirect

Successful build - Chance fights ever on the side of the prudent.

Link to build: https://jenkins.wildfiregames.com/job/docker-differential/2252/display/redirect