Add fast_float header library for fast string->double, and optimize Parquet WKT->WKB conversion #7690
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background for this work: opengeospatial/geoparquet#170
Several related items:
__has_include("fast_float/fast_float.h")
or by building explicitly with -DUSE_SYSTEM_FAST_FLOAT=1), or fallbacks to the vendored oneBenchmarks:
This shows that generation of WKT is very slow compared to WKB (I presume this could be optimized to be at least twice faster)
bench_ogr_c_api
, that is iterating over OGRFeature returned by OGRLayer::GetNextFeature()The timings of the WKT parsing have been improved through the use of the fast_float library (e.g. 16.8 s before for "WKT, 4 decimals" dataset when using strtod())
bench_ogr_batch --stream-opt GEOMETRY_ENCODING=WKB
, that is using the Arrow Array streaming mode, but forcing geometries to be WKB even if it is not their native encodingThe timings with the WKT files have been considerably improved by avoiding the round-trip to OGRFeature and using fast_float as well (e.g 25 s before for "WKT, 4 decimals" dataset when going through OGRFeature and using strtod())
For those read benchmarks, we're probably clause to the optimal performance