Page MenuHomeWildfire Games

Update LICENSE.txt to include SRTM digital elevation model disclaimer
AbandonedPublic

Authored by elexis on Oct 11 2017, 6:30 PM.

Details

Reviewers
None
Summary

In rP18523, Alpha 21, _kali has created random map scripts (for example Red Sea and Mediterranean) that import the famous SRTM v4 digital elevation model.
Since a free software project needs to include the license of all imported work, we need to state that data source clearly.

The SRTM v4 data was downloaded via a script written by _kali here https://github.com/kali0ad/mappr/blob/master/dstk.go
from the the Data Science Toolkit http://www.datasciencetoolkit.org/coordinates2statistics which just makes the existing data accessible without a prior download of the complete SRTM v4 data.

That DSTK data actually uses the SRTM v4 data can be seen here:
http://www.datasciencetoolkit.org/developerdocs#coordinates2statistics

elevation - The height of the surface above sea level at this point. From the NASA and the CGIAR Consortium for Spatial Information

with a link to http://srtm.csi.cgiar.org/

(Yes, I did check that dstk website at the time, no I didn't commit it properly).

Test Plan

Follow the white rabbit.

Event Timeline

elexis created this revision.Oct 11 2017, 6:30 PM
Vulcan added a subscriber: Vulcan.Oct 11 2017, 7:18 PM

Build is green

Updating workspaces...
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDAnimation]’:
FCollada/FCDocument/FCDLibrary.cpp:149:30:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
  const T* cptr = ((const FCDLibrary<T>*)l1)->GetEntity(0);
           ^
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDAnimationClip]’:
FCollada/FCDocument/FCDLibrary.cpp:150:34:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDCamera]’:
FCollada/FCDocument/FCDLibrary.cpp:151:27:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDController]’:
FCollada/FCDocument/FCDLibrary.cpp:152:31:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDEffect]’:
FCollada/FCDocument/FCDLibrary.cpp:153:27:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDEmitter]’:
FCollada/FCDocument/FCDLibrary.cpp:154:28:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDForceField]’:
FCollada/FCDocument/FCDLibrary.cpp:155:31:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDGeometry]’:
FCollada/FCDocument/FCDLibrary.cpp:156:29:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDImage]’:
FCollada/FCDocument/FCDLibrary.cpp:157:26:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDLight]’:
FCollada/FCDocument/FCDLibrary.cpp:158:26:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDMaterial]’:
FCollada/FCDocument/FCDLibrary.cpp:159:29:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDSceneNode]’:
FCollada/FCDocument/FCDLibrary.cpp:160:30:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDPhysicsModel]’:
FCollada/FCDocument/FCDLibrary.cpp:161:33:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ set but not used [-Wunused-but-set-variable]
FCollada/FCDocument/FCDLibrary.cpp: In instantiation of ‘void LibraryExport() [with T = FCDPhysicsMaterial]’:
FCollada/FCDocument/FCDLibrary.cpp:162:36:   required from here
FCollada/FCDocument/FCDLibrary.cpp:141:11: warning: variable ‘cptr’ s

http://jenkins-master:8080/job/phabricator/2117/ for more details.

Executing section Default...
Executing section Source...
Executing section JS...
Executing section XML GUI...

http://jenkins-master:8080/job/phabricator_lint/588/ for more details.

leper added inline comments.Oct 12 2017, 1:47 AM
binaries/data/mods/public/maps/random/LICENSE_SRTM.txt
11

That prohibited part does conflict with quite a few other things, it's similar to CC(-BY)-NC licenses which we also do not accept for contributions.

So your options are to either try to get another source for your data (NASA should be fine since US government creations are public domain, then again this only applies to things not produced by contractors, but last I checked things produced by NASA were public domain), ask for that to be under CC-BY-SA, or remove those files.

elexis planned changes to this revision.Oct 12 2017, 3:55 PM

At least it will be easy for everyone to create new random maps with a fixed heightmap when they were redone (and they'll become round too then).

License research:
I lost the list of available elevation models and their exact license when my computer froze while converting the data.
But I made the same observation as openstreetmap project noted here that also matches lepers recollection:
http://wiki.openstreetmap.org/wiki/SRTM

i.e. that SRTM v3 and prior do not restict commercial usage and should be compatible to the current 0 A.D license, while v4.1 and the german DLR model are not.

Obtaining & Processing data:
NASA has only one more server running where the SRTM v3 and prior files can be downloaded without creating an Earth Explorer account that is intended for Scientific use.

1. SRTM v3

# See coordinate grid at https://dds.cr.usgs.gov/srtm/version2_1/Documentation/Continent_def.gif
# Download red Sea region (1172 zip files 1.3GB)
CONTINENT="Africa" # "Eurasia"
for LAT in {00..45};
do
	for LON in {020..065};
	do
		wget -nc -P files https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/${CONTINENT}/N${LAT}E${LON}.hgt.zip;
	done
done

# extract to 3.4GB
unzip -d unzipped "files/*.zip"

# convert hgt files to 4GB TIF using the Geospatial Data Abstraction Library, easy install via pip
gdal_merge.py unzipped/*.hgt  -o merged.tif

# scale down to 4MB PNG
convert -monitor  -define registry:temporary-path=temp merged.tif -scale 2048x2048 scaled.png

That image should now be opened in gimp, scaled to 320x320 (Normal mapsize for 0AD), change the levels, brightness and contrast.
Most importantly, the maximum brightness of the output image determines the maximum height of the map (must be scaled down drastically).

Result:

2. SRTM v4.1

# License: http://srtm.csi.cgiar.org/SELECTION/SRT_disclaimer.htm
# Download 41 GeoTIFF files (1.1GB)
for LAT in {42..48};
do
	for LON in {05..10};
	do
		wget -nc -P files http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_${LAT}_${LON}.zip;
	done
done

# same unzip command

gdal_merge.py unzipped/*.tif  -o merged.tif

# same scaling command

# arbitrary brightness adaptation in gimp

Result (do not sell):

There is the issue that water is white instead of black. But other than that this is much more usable as most of the noise was removed.

3. Blue Marble Series:
After running out of SRTM versions to test, I checked the wikipedia DEM article.
Something I should have done much sooner, because the 21600x10800 sized 18MB PNG encompasses the entire world in one file that doesn't need any processing and already had been photoshopped completely:
https://visibleearth.nasa.gov/view_cat.php?categoryID=1484

Terms of Use: https://visibleearth.nasa.gov/useterms.php

After that I found this underappreciated Atlas heightmap import guide recommending the same DEM (so should have RTFM sooner too):
https://wildfiregames.com/forum/index.php?/topic/20786-palaxins-heightmap-guide/

Cutting out that Red Sea region in gimp, adapting the heightlevels and scaling it down, we get this

Result:

Visual comparison:
The "SRTM3 files contain 1201 lines and 1201 samples" [https://dds.cr.usgs.gov/srtm/version2_1/Documentation/Quickstart.pdf | 1]] and
that the red sea region consists of 32*32 files 2,
so there are about 1.477.018.624 data points for red sea.
The SRTM v4.1 red sea region contains 42001x36001 = 1512078001 data points, which is about the same (3% difference).

The red sea map _kali created was first scaled down to 320 * 320 = 102400 data points, so the heightmap uses only 0,0064 percent. of the data.
The brightness manipulation removes about 95% of the remaining data (elevation differences).

The manual editing in gimp and atlas replaced the remaining data, which we can see when opening the red sea terrain in atlas:

blue marble:


  • The Ethopian Highlands were almost completely removed and changed in shape
  • The Nile river was removed
  • The cliffs near the red sea were completely replaced with custom shapes that are more playable than realistic
  • The mainland was replaced with flat terrain
  • Every bump seen in the map cannot be found in the SRTM data
  • Every bump in the SRTM data cannot be seen in the 0ad map
  • There are parts of continents that actually don't exist in the real world and were added on
  • There are seas that don't exist in the real world

_kali could have started out on any of these data sets.
The blue marble version would have saved him time,
the shoreline-only version probably more.
The heightmap is _kalis work, not the one of Mr Jarvis.

So it is still true that users of SRTM v4.1 are prohibited from any commercial, non-free resale, or redistribution without explicit written permission from CIAT and
we actually have no reason to include this disclaimer in 0 A.D.

binaries/data/mods/public/maps/random/LICENSE_SRTM.txt
9

This part is false too

elexis abandoned this revision.Oct 13 2017, 10:42 PM

So it is still true that users of SRTM v4.1 are prohibited from any commercial, non-free resale, or redistribution without explicit written permission from CIAT and
we actually have no reason to include this disclaimer in 0 A.D.

Yet we still do include something that is a derivative work of those. Which means we have to abide by that license.

So there are still the same options as there were previously, remove the maps again (or at least the terrain data), ask for an exception (that is if the few things that were used could be released under a license that is compatible), do nothing about those files but inform every downstream that the result is quite likely not following DFSG (or whatever the equivalent Fedora things is) anymore.

In D957#37563, @leper wrote:

that license

That isn't a license (in particular not a copyright license).

@leper wrote:

That prohibited part does conflict

It doesn't, read it again.
The disclaimer does not state anything about "any commercial, non-free sale, or distribution" of derived products.

What it actually states:

The data distributed here are in ARC GRID, ARC ASCII and Geotiff format, in decimal degrees and datum WGS84.
Users are prohibited from any commercial, non-free resale, or redistribution without explicit written permission from CIAT

Wildfire Games does not distribute or resale that data.

What it actually states about derived products:

Users should acknowledge CIAT as the source used in the creation of any reports, publications, new data sets, derived products, or services resulting from the use of this data set.

"Acknowledging CIAT as the source used in the creation of a derived product or service resulting from the use of this data set" does not restrict commercial distribution of derived products.

@leper wrote:

that is a derivative work

From the United States Copyright Act in 17 U.S.C. § 101:

A “derivative work” is a work based upon one or more preexisting works (...) consisting of (...) modifications

"based upon" and "modifications" are not defined there.

See Litchfield v. Spielberg, 736 F.2d 1352, 1357 (9th Cir. 1984).

A work is not derivative unless it has been substantially copied from a prior work’s expression.

See the prior post for the questioning of the substantial part.

An illustration:

  1. Step: Original data Mona Lisa in 1024px x 1526px:

  1. Step: Scale down by a factor of 115:

  1. Step: Throw out 90% of the elevation differences by changing brightness and contrast to smooth the map:

  1. Step: Replace shorelines, mountains and bodies of water with plain land:

@leper wrote:

at least the terrain data

If we do not know what work "based on prior work" or "modifications" exactly mean,
then we cannot tell wheather it is only the elevation model of the map or wheather further work on the map would also constitute a derivative, for example

  • which texture is painted at which location of the map
  • which objects are placed at which frequency and location of the map
  • which textures would fit to that map in general
  • the reddish dust texture which we created to fit the map

(So there are more options than the mentioned ones, but I'd like to establish consensus on the interpretation of the lawful and legal aspects prior. What the current title of the differential proposal revision proposes is wrong afaics.)

leper added a subscriber: Itms.Oct 21 2017, 7:39 PM
In D957#37648, @elexis wrote:
In D957#37563, @leper wrote:

that license

That isn't a license (in particular not a copyright license).

In which case we have _no_ right to do anything with it. Unless you want to get lawyers involved to make sure that use is actually legal.

What it actually states about derived products:

Users should acknowledge CIAT as the source used in the creation of any reports, publications, new data sets, derived products, or services resulting from the use of this data set.

"Acknowledging CIAT as the source used in the creation of a derived product or service resulting from the use of this data set" does not restrict commercial distribution of derived products.

That does not grant us any rights, unless you want to get some copyright lawyer to confirm that this allows everything when listing the source, which we currently don't do (but we also suck at that for most other things, and last time I pointed that during the whole credits work it was not considered important...).

If we do not know what work "based on prior work" or "modifications" exactly mean,

If you want to know wait until someone gets sued, then a court will figure that out. But since there is a chance that this (given that the way those things were produced quite explicitly derive from the base material) is infringing I for one will get as far away from the whole thing as fast as I can, good luck.

[...] wheather [...]

whether

The point being this is possibly a derivative work, and the license conflicts with what we (last I checked) wanted to release the whole game under, so the choice is either to ignore this liability, remove the offending files, or contact a lawyer to tell us what to do (which most likely will result in spending money and getting something along the lines of contact the authors for a clarification or remove it). But I guess we just don't care about actually producing something that we can distribute without having to worry.

Maybe @Itms might be interested.

In D957#37797, @leper wrote:

lawyers make sure that use is legal

Lawyers have an opinion, only the judge has the ability to declare what legal and lawful means last time I checked.

we just don't care

I don't know if you care, but if I wouldn't care I wouldn't have created the differential revision proposal and the days of research on the data and legalese.

Just because I don't want to act under a false premise doesn't mean I don't want to act.
I have repeated that there are more options than the mentioned ones.
The options you mentioned should be worked out better rather than rushing them.

For once, a more constructive variant of deletion variant would instead be a move to a separate, differently licensed download in the release page, forum or fancy map downloader.

Secondly, I don't know if a similar shape of a continent (which seems to be everything that wasn't replaced but is the same data on all DEMs) can be considered intellectual property.
If it does, then we would at most need to ask to legalize the affected 320px*320px, not the entire SRTM dataset, nor all of the gigabytes within the contested regions.
In that case, we would still not have to ask to legalize the actual SRTM data, but only enable the 0AD terrain data to become licensed (which can't really be used by anyone from the scientific community and most unlikely not be used for artistic purposes either as there is land which doesn't exist and land removed that does exist).

without having to worry

As I have pointed out, just deleting the terrain data won't imply not having to worry, since for example other data could be considered a derived work of the derived work, even if unlikely from a common-sense point of view.

In case it wasn't clear, I won't click on reclaim revision, because the statements herein are wrong, but instead a new differential revision or whatever shall be created once the options were worked out more constructively and concisely.
For instance one part of the investigation stage that is still missing which will require not few time is converting the 0AD terrain data to a bitmap, so that we and everyone down the line can actually see the differences.

For once, a more constructive variant of deletion variant would instead be a move to a separate, differently licensed download in the release page, forum or fancy map downloader.

There is still a difference between shipping something in the default distribution that we are possibly not allowed to ship, and having users able to download whatever from wherever.

without having to worry

As I have pointed out, just deleting the terrain data won't imply not having to worry, since for example other data could be considered a derived work of the derived work, even if unlikely from a common-sense point of view.

So delete all those maps and call it done. Sometimes the simple solution is the best one. If anyone cares enough about those maps they will be able to recreate them without using questionable (in the compatible license sense) source material.

Basically clean up the mess you made.

without having to worry

So delete all those maps and call it done.

My example was the reddish dust actor that isn't a map but something that was made to fit to a possibly derived work, just like the terrain changes were made to fit and perhaps that ad absurdum example could be taken further with some lawyers. Maybe we don't have to worry but we surely can if we want to. Legal certainty can only exist where words are defined legally or otherwise determined by a decision from a judge.

So delete all those maps and call it done

I got the message the first time. No need to repeat that you don't care about preserving the gameplay experience of these maps.
I hope it has become clear enough that I do care about the maps that _kali and me have spent weeks and months on building,
that I won't rush deleting them and move on like nothing happened.

Sometimes the simple solution is the best one.

Because the terrain data is far from SRTM data, it is not simple to recreate the map.
So deletion + recreation is far more complex than other options, such as for example asking a question which was optimized for answering positively.

Basically clean up the mess you made.

I thought I have created the differential revision proposal, the following research and mentioning of the build up of other plans to do that, but I must be wrong.

If anyone cares enough about those maps they will be able to recreate them without using questionable (in the compatible license sense) source material.

As I have stated there are other options. Would be nice if you wouldn't delete the maps before I get a chance to execute them.

and yes I spent some hours today progressing with that (json -> png), but I can't post before I conclude (comparing with the exact srtm data).

In D957#38067, @elexis wrote:

without having to worry

So delete all those maps and call it done.

My example was the reddish dust actor that isn't a map but something that was made to fit to a possibly derived work, just like the terrain changes were made to fit and perhaps that ad absurdum example could be taken further with some lawyers. Maybe we don't have to worry but we surely can if we want to. Legal certainty can only exist where words are defined legally or otherwise determined by a decision from a judge.

Neither of us are lawyers, but if making something to fit something else is considered a derived work then a lot would be wrong. Also maybe you should try to see the issue instead of trying to argue that any concern could also be applied to something not really related.

So delete all those maps and call it done

I got the message the first time. No need to repeat that you don't care about preserving the gameplay experience of these maps.

I do care about not shipping something we most likely cannot legally ship.

I hope it has become clear enough that I do care about the maps that _kali and me have spent weeks and months on building,
that I won't rush deleting them and move on like nothing happened.

Maybe check your sources the next time, then you might not have to try arguing why you doing something wrong is not that wrong in some twisted way instead of dealing with it.

As I have stated there are other options. Would be nice if you wouldn't delete the maps before I get a chance to execute them.

Feel free to include more questionable content if you think that is going to be beneficial in the long run, I stopped caring quite a while ago.

In D957#38073, @leper wrote:

I stopped caring quite a while ago.

So did I

For future reference, here a script that converts the *.hmap terrain data to a lossless png image.

import glob
import PIL.Image
import json
import math

def hmapToPNG(filename):
    print("Converting " + filename + "...")
    heightmap = json.load(open(filename)).get("heightmap")
    heightmapSize = math.sqrt(len(heightmap))
    
    if heightmapSize != round(heightmapSize):
        raise ValueError('heightmap not square')
    heightmapSize = round(heightmapSize)

    maxHeight = -math.inf
    minHeight = +math.inf
    for i in range(heightmapSize ** 2):
        maxHeight = max(heightmap[i], maxHeight)
        minHeight = min(heightmap[i], minHeight)

    img = PIL.Image.new('RGB', (heightmapSize, heightmapSize), "black")
    pixels = img.load()

    for z in range(heightmapSize):
        for x in range(heightmapSize):
            color = round(255 * (heightmap[z + heightmapSize * x] - minHeight) / (maxHeight - minHeight))
            pixels[x, heightmapSize - z - 1] = (color, color, color)

    img.save(filename + ".png", "PNG")


for filename in glob.glob('*.hmap'):
    hmapToPNG(filename)

So here a comparison:

SRTM v4.1 (original unzipped TIFF mosaic 2.8GB), after scaling down to 320px * 320px and setting the levels from (0, 12) to (0, 255):

0AD terrain data (elevation mapped to (0, 255), 662kb JSON data):

Amazon 0AD terrain:

Mediterranean 0AD terrain:

I think it would make a huge difference whether the SRTM v4.1 original data or the 0AD terrain data would be asked to become sellable, so this investigation seems relevant to me.
As I'm not a judge, I can't decide whether that legally counts as a substantial copy of SRTM v4.1 or what kind of recreation would be guaranteed to not be legally controvertible.
The more often I look at it, the easier it becomes for me to create a similar map without any template (but under which circumstances would that count as a derived product if it recreates a derived product? On the other hand, why would I care to invest the weeks to create something new instead of denying the maps to their fans).

After now reading the DSTK downloader code which was uploaded this May, I noticed it also queries land_cover of the DSTK, which pulls data from Global Land Cover 2000 database. I will investigate the way and extent the data was used before taking an action.