Bounding box column updates #202
Conversation
| @@ -227,7 +227,7 @@ Note: This technique to use the bounding box to improve spatial queries does not | |||
|
|
|||
| ### Bounding Box Columns | |||
|
|
|||
| A bounding box column MUST be a Parquet group field with 4 child fields named `xmin`, `xmax`, `ymin`, and `ymax` representing the geometry's coordinate range. As with the top-level [`bbox`](#bbox) column, the values follow the GeoJSON specification (RFC 7946, section 5), which also describes how to represent the bbox for geometries that cross the antimeridian. For three dimensions the additional fields `zmin` and `zmax` MAY be present but are not required. The fields MUST be of Parquet type `FLOAT` or `DOUBLE` and all columns MUST use the same type. The repetition of a bounding box column MUST match the geometry column's [repetition](#repetition). A row MUST contain a bounding box value if and only if the row contains a geometry value. In cases where the geometry is optional and a row does not contain a geometry value, the row MUST NOT contain a bounding box value. | |||
| A bounding box column MUST be a Parquet group field with 4 or 6 child fields representing the geometry's coordinate range. For two-dimensional data, the child fields MUST be named `xmin`, `ymin`, `xmax`, and `ymax` and MUST be ordered in this same way. As with the top-level [`bbox`](#bbox) column, the values follow the GeoJSON specification (RFC 7946, section 5), which also describes how to represent the bbox for geometries that cross the antimeridian. For three dimensions the additional fields `zmin` and `zmax` MAY be present but are not required. If `zmin` is present then `zmax` MUST be present and vice versa. If `zmin` and `zmax` are present, the ordering of the child fields MUST be `xmin`, `ymin`, `zmin`, `xmax`, `ymax`, `zmax`. The fields MUST be of Parquet type `FLOAT` or `DOUBLE` and all columns MUST use the same type. The repetition of a bounding box column MUST match the geometry column's [repetition](#repetition). A row MUST contain a bounding box value if and only if the row contains a geometry value. In cases where the geometry is optional and a row does not contain a geometry value, the row MUST NOT contain a bounding box value. | |||
There was a problem hiding this comment.
I notice that this requirement about the ordering of the fields makes invalid the newly released 2024-04-16-beta.0 Overture Maps data (where the fields in the bbox group happen to be ordered xmin, xmax, ymin, and ymax).
I've missed recent meetings, and maybe this has been covered, but I'm curious about including both the name and the order of the fields in the spec - this feels "overspecified" to me. Are there clients that cannot work with the named fields in a group?
I don't know of one, but it wouldn't surprise me if there were data producers that couldn't control the order of the fields in a group (maybe using a language where unordered struct fields are serialized in a Parquet group field).
There was a problem hiding this comment.
I'm also wondering why the ordering is required? Shouldn't access go through the names that are specified in the array?
There was a problem hiding this comment.
I think the difference here is whether you consider the struct type as a hashmap or as an ordered list of tuples. I see Parquet's struct type more as the latter than the former. E.g. the struct type is really the collection of fields like so
[
("bbox", "xmin"),
("bbox", "ymin"),
("bbox", "xmax"),
("bbox", "ymax")
]
You can loop through the columns to find the column index for your desired name, but it's not a map. You can't directly access a column by name. Instead, you must first find the column index, then access the column by index.
Given that low-level access is done by index, why not strictly specify column order in the spec as well? If there were some writer that was unable to specify column order in a struct type, that might be a good reason. Does that exist?
|
@Maxxen - is there any problems for DuckDB if we did this proposed strict specification of order in the struct? |
|
@cholmes I can't think of any reason why the ordering would matter for us right now, so feel free to go ahead! ( I also kind of like: xmin, xmax, ymin, ymax [zmin, zmax] as the optional axis won't be interleaved) |
We had a discussion today about this on the call and various people supported that. |
|
The main annoyance of using It's of course a different kind of data structure, the latter is a json array, and what we are discussing here is just the order of fields in the schema. But it would still have been nice to use the same order. Although I definitely agree that |
+1 |
|
Alright, well if thats already standardized then maybe it makes more sense to follow the existing convention. |
|
I believe the column metadata (bbox and geometry_types) follows GeoJSON, which makes sense as it is, in fact, JSON. I can see the argument for consistency, but also that the code iterating over both of those are at different levels of abstraction and I am not sure we risk mass confusion by choosing an option for the bounding box column that makes it easier to write scanners. We would like to add a box/bbox/rect type to GeoArrow and we will probably use whatever order is decided on here to make sure our compute functions don't have to reorder anything...not that this spec is responsible for the choices in GeoArrow, but GeoArrow is a place where it would be nice to ensure that a 3D (or 4D) bounding box can share an implementation with something that only cares about 2D bounding boxes. As above, though, I'm +1 on any option that makes the ordering explicit, since the workaround is not particularly verbose ( |
|
Going to merge this one in, as it's got approvals and extensive discussion. Any further alternatives can be discussed as PR's to main. |
zminis included, thenzmaxmust also be included, and vice versa.