OSM: encoding not correct

[pathmapper]

Expected behavior and actual behavior.

OGR uses UTF-8 as encoding for OSM data but instead it uses something else so some chars are not correctly encoded, e.g. ö, example:
Astrid-Lindgren-Förderschule becomes
Astrid-Lindgren-F├Ârderschule.

Prolog:

Steps to reproduce the problem.

ogrinfo test.osm multipolygons (sample data: test.zip)
-> some values for name are not correctly encoded,e.g.

Operating system

Windows 10 Version 1809

GDAL version and provenance

3.6.4 and 3.7.1 from OSGeo4W

Likely related issues

qgis/QGIS#48328
3liz/QuickOSM#369

[jratike80]

I bet that this is not a bug but what you see comes from the locale settings of your computer.

See this:

ogrinfo liechtenstein-latest.osm.pbf  -sql "select * from points where osm_id='32011360'"
name (String) = Vaduz Quäderle

Then check your Windows codepage (my locale is Finnish) and change the codepage into UTF-8 (65001)

chcp
Active code page: 850

chcp 65001
Active code page: 65001

Try again

ogrinfo liechtenstein-latest.osm.pbf  -sql "select * from points where osm_id='32011360'"
name (String) = Vaduz Quäderle

[pathmapper]

change the codepage into UTF-8 (65001)

Thanks, works for me, too.

not a bug but what you see comes from the locale settings of your computer

The XML Prolog in the file has

<?xml version="1.0" encoding="UTF-8"?>

so shouldn't UTF-8 be used regardless which system encoding is set?

[pathmapper]

When the sample file is drag'n'dropped in QGIS, the Data source encoding shows windows-1252:

I assumed this comes from ogr but maybe not...

[jratike80]

so shouldn't UTF-8 be used regardless which system encoding is set?
When is comes to printing text on a console, in this use case the Operating System never reads the XML header. GDAL does read it. If the console is set to use Latin 1 character set it awaits that it gets Latin 1 formulated data. When GDAL sends out UTF8 there is a conflict - UTF8 strings are shown as if they were Latin 1 strings. That can be avoided in two ways: set the console to use UTF8, or convert the output of GDAL from UTF8 into Latin 1 for example with iconv https://en.wikipedia.org/wiki/Iconv.

There may be a bug in QGIS but I could not reproduce qgis/QGIS#48328 with QGIS 3.32.2.

[pathmapper]

Thanks, seems to be related to the used system encoding and how QGIS handles it - not a GDAL issue.