Convert layers to YAML #947
Conversation
Requires PyYAML. If this dependency is an issue on a particular
platform, pretty much any language allows you to do this.
A ruby example is
ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < project.yml > project.json
This one-time conversion was done with the json2yaml node module,
and then substantially reformatted by hand, using mappings and alias
to avoid repeating projection and extents information for each layer,
and to get consistent ordering of the properties of each layer. The
latter does not matter semantically, but makes editing with a text
editor cleaner.
By using the scripts/yaml2mml.py script, the YAML can be converted
back into JSON, which is still what TileMill uses.
A JSON-aware diff of the original project.mml and the new generated
one shows that the only differences are the new "_parts" property,
used for aliases and ignored by TileMill, and layer extents.
The script can be invoked with
scripts/yaml2mml.py < project.yaml > project.mml
The project.mml file remains checked in to git, as TileMill requires
it and this allows someone to clone the repo, and immediately have
it work. Merge conflicts in project.mml are trivial to solve, as
the entire file is just regenerated by running the script again.
This does mean that the .MML file should not be edited directly,
but this is fine, as editing the JSON by hand is extremely difficult
anyways! The HOT OSM style has a similar workflow, making use of cartocc.
By using aliases, we can avoid having to specify the same information repeatedly, saving lines, avoiding mistakes, and making it easier to use a differently named database. The last is evident in that the landcover extents changed slightly in this, because they were different than all the other layers.
Block scalars allow you to use newlines, and are far more readable This commit also catches some extents that were missed previously.
A lot of the SQL was badly formatted, with random tabs, caps, and lines of 5000 characters. This commit cleans up about half of them but is not complete.
| * Add indentation if necessary for complex function calls, WHERE parenthesis, and CASE statements | ||
| * One space before and after = etc | ||
| * Name SQL subqueries after the layer name (but use underscores) | ||
|
|
There was a problem hiding this comment.
What about spaces after commas? We are currently inconsistent in that.
There was a problem hiding this comment.
I've been making that consistent
|
Great work, this will certainly make all our work much easier. Did you make any changes to the SQL apart from whitespace? Does this need a review? On coverting the file I get: I think we need to add the package python-yaml to the dependencies. |
| (SELECT | ||
| way | ||
| FROM planet_osm_line | ||
| WHERE man_made='cutline' |
There was a problem hiding this comment.
Here (and at some other places) are spaces missing around the equals sign. Not really important, but I guess we'd better fix it now.
| (SELECT way, highway, name | ||
| FROM planet_osm_line | ||
| WHERE highway in ('bridleway', 'footway', 'cycleway', 'path', 'track', 'steps') | ||
| and name is not null |
There was a problem hiding this comment.
I haven't gotten this far, just some guided search and replaces
|
I think the fact that we can now add comments and do a proper code review is being well demonstrated with the fact that @math1985 has been able to do one :) |
|
I checked the code, and I think the yaml code is equivalent with the old mml code - with the exception of one typo (space in front of 'construction') and a code block called '_parts'. What does this do? |
|
The parts within it are used for YAML aliasing. # Various parts to be included later on
_parts:
# Extents are used for tilemill, and don't actually make it to the generated XML
extents: &extents
extent: *world
srs-name: "900913"
srs: "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over"
osm2pgsql: &osm2pgsql
type: "postgis"
dbname: "gis"
key_field: ""
geometry_field: "way"
extent: "-20037508,-20037508,20037508,20037508"
Layer:
- id: "citywalls"
name: "citywalls"
class: ""
geometry: "linestring"
<<: *extents
Datasource:
<<: *osm2pgsql
table: |-
(SELECT
way
FROM planet_osm_line
WHERE historic = 'citywalls')
AS citywalls
advanced: {}The Fortunately, an understanding of YAML isn't needed to use them when adding a layer - you just copy from existing layers. It's probably possible to use the first occurrence of an osm2pgsql layer as an anchor and copy mappings from that, but this way is more robust and clearer what's being included. The YAML Reference Card has the YAML syntax (and is itself YAML), not that I see any need to use more YAML features. The _parts in JSON is ignored by |
|
Thanks for the corrections. We're still not consistent in the use of spaces after commas: personally I would prefer to add spaces after all commas. What do you think? There are also still a couple of keywords that need capitzalizing, by the way. |
Well, that's largely because I wanted to get this PR in quicker than I could clean up all of the SQL. |
|
project.yaml includes this comment: Is the |
Closes #711
Although requiring a script is not ideal, the JSON MML file couldn't be sanely edited anyways. The advantages are
I think it'd be easiest to merge this then rebase other merges on top of it - the one-time conversion from JSON to YAML is not trivial, as a great deal of work had to be done by hand to clean up the YAML.