Performance comparison of open source vector tiles server solutions for providing geodata from PostGIS databases

This GitHub repository contains the attachments of Fabian Rechsteiner's master's thesis from May 2024, written in German, titled: Performancevergleich von Open-Source-Vector-Tiles-Serverlösungen zur Bereitstellung von Geodaten aus PostGIS-Datenbanken

This repository can be used as a vector tiles benchmark to compare the performance of various vector tiles servers using Apache JMeter.

The following requirements must be met:

• A Linux server with Docker Compose is available.
• Apache JMeter is installed on the local computer.
• A localhost connection can be established from the computer to the server.

Abstract

The introduction of the new OGC API family marks a milestone in the exchange of geodata over the web. With the OGC API Tiles standard, both raster and vector data can now be provided as tiles, with vector data no longer being transmitted as images, but as geometry to the client. In this master's thesis, six open-source vector tiles servers (BBOX, Ldproxy, Martin, pg_tileserv, Tegola, and TiPg) are set up and configured using Docker in a public cloud. Vector tiles are created for each server from the vector data of the PostGIS database. Various test scenarios with Apache JMeter are used to determine which server can deliver the vector tiles the fastest. The results of the performance test show clear differences in the performance of the servers. One server proved to be by far the fastest, while another was clearly the slowest across all test scenarios. Additionally, a website is created using Maplibre GL JS to visually illustrate and compare the results. This master's thesis underscores the enormous potential and advantages of the new OGC API standards as well as vector tiles and aims to motivate geodata providers to offer their data according to these standards in the future.

Methodology

For the performance testing of vector tile server solutions, I utilized the cantonal geodata sourced from the Amt für Geoinformation Thurgau as our test dataset. This dataset encompasses various tables stored within a PostGIS database.

To ensure a consistent evaluation of performance across different server solutions, I employed the same PostGIS database as the data source for each solution. To achieve this, I established a public cloud infrastructure through the Swiss cloud provider Infomaniak, deploying a Linux instance. A Linux server was then configured to host various applications within separate Docker containers. The foundational data was stored in its dedicated Docker container named 'postgis'. Each vector tile server under scrutiny was encapsulated within its own Docker container, accessible via individual ports.

In addition to the 6 primary applications, I set up two supplementary containers: 'nginx' served as a web server facilitating external access to the applications and accompanying HTML files, while the 'maputnik' container functioned as a vector tile style editor for defining the visual representation of the vector tiles.

Overview vector-tiles-server

Name	BBOX	ldproxy	Martin	pg_tileserv	Tegola	TiPg
Developer	Sourcepole	interactive instruments	MapLibre	CrunchyData	Go Spatial	Development Seed
Programming Language	Rust	Java	Rust	Go	Go	Python
Source Formats	PostGIS, MBTiles, PMTiles	HTTP, MBTiles, PostGIS, GPKG, SQLite, WFS, (GraphQL)	PostGIS, MBTiles, PMTiles	PostGIS	PostGIS, GPKG, SAP HANA Spatial	PostGIS
Output Data Format	MVT, MBTiles, PMTiles	MVT, MBTiles	MVT, MBTiles	MVT	MVT	MVT
Creation Method	ST_AsMVT	Feature Provider	ST_AsMVT	ST_AsMVT	ST_AsMVT	ST_AsMVT
Supported Tile Schemas	WebMercatorQuad (3857) Benutzerdefiniert	WebMercatorQuad (3857) Benutzerdefiniert	WebMercatorQuad (3857)	WebMercatorQuad (3857)	WebMercatorQuad (3857)	WebMercatorQuad (3857) Benutzerdefiniert
Filter Functions	✔️	✔️	✔️	✔️	✔️	✔️
Multi-Layer-Tiles	✔️	❌	✔️	✔️	✔️	❌
Caching	✔️	✔️	✔️	❌	✔️	❌
OGC-API Features	✔️	✔️	❌	❌	❌	✔️
GitHub	BBOX - GitHub	ldproxy - GitHub	Martin - GitHub	pg_tileserv - GitHub	Tegola - GitHub	Tipg - GitHub
Docker Image	sourcepole/bbox-server-qgis - Docker Image	iide/ldproxy - Docker Image	martin - Docker Image	pramsey/pg_tileserv - Docker Image	gospatial/tegola - Docker Image	tipg - Docker Image
Additional Documentation	BBOX - Documentation	ldproxy - Documentation	Martin - Documentation	pg_tileserv - Documentation	Tegola - Documentation	Tipg - Documentation

Implementation of the Vector Tiles Server

vector-tiles-benchmark

• bbox
  • bbox.toml
• ldproxy
  • cfg.yml
• martin
  • config.yaml
• nginx
  • html
  • nginx.conf
• pg_tileserv
  • pg_tileserv.toml
• sql-scripts
• tegola
  • config.toml
• tipg
  • .env
• vector-tiles-benchmark
  • test_plans
  • test_results
• .env
• .gitignore
• README.md
• compose.yaml

The file "compose.yaml" contains the configurations of all Docker containers. The configurations of each container are explained in the following sub-sections. Each container has been assigned a profile, with which the container can be started and stopped alongside the PostGIS database. An example of starting the container with a profile is: docker compose --profile profilename up -d

PostGIS Database

To start only the postgis database without any other server you have to run the follwoing code with root permission (sudo -s) insisde the mvt-benchmark folder:

docker compose --profile postgis up -d

Note: Use the -it paramter instead of -d to show the log-File in the Terminal to show the errors.

The database can now be connetcet with PGAdmin with the followed parameters:

• host: your public IP-Adress oder Serveradress Example: 195.15.345.35
• port: 5432
• username: postgres
• password: your password defined in the compose.yaml or .env File
• database: postgres

BBOX

BBOX is a Webserver created by Sourcepole and is a new Webserver-Solution to create OGC-API Features and OGC-API Tiles. T-REX -> BBOX

Language: RUST

GitHub

Start Server:

docker compose --profile bbox up -d

Stop Server:

docker compose --profile bbox down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open webbrowser: http://localhost:8804

Vector Tiles Request (Example: bo_boflaeche_mv): http://localhost:8804/xyz/bo_boflaeche_mv/{z}/{x}/{y}.pbf

Note: Because the port 8080 is already used for other Servers, the port 8080 got maped to 8804.

ldproxy

ldproxy is a Webserver created by interactive instruments and is a Webserver-Solution to create OGC-API Tiles and other OGC-API's from a PostGIS Database on the fly.

Language:

GitHub

Start Server:

docker compose --profile ldproxy up -d

Stop Server:

docker compose --profile ldproxy down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open webbrowser: http://localhost:7080/rest/services

Vector Tiles Request (Example: bo_boflaeche_mv): http://localhost:7080/rest/services/avprodukt/collections/bo_boflaeche_mv/tiles/WebMercatorQuad/{z}/{y}/{x}

Martin

Martin is an open-source vector tile server that allows generating and serving MVT (Mapbox Vector Tiles) from PostGIS tables, views, as well as from PMTiles or MBTiles. It enables the dynamic combination of multiple data sources within a single vector tile.

Language: Rust

GitHub

Start Server:

docker compose --profile martin up -d

Stop Server:

docker compose --profile martin down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open catalog in the webbrowser (List of all sources): http://localhost:3000/catalog

Vector Tiles Request (Example: bo_boflaeche_mv):

http://localhost:3000/bo_boflaeche_mv/{z}/{x}/{y} http://localhost:3000/bo_boflaeche_mv,bo_projgebaeude_mv/{z}/{x}/{y}

pg_tileserv

pg_tileserv is a Webserver created by CrunchyData and is a Webserver-Solution to create OGC-API Tiles from a PostGIS Database on the fly.

Language: Go

GitHub

Start Server:

docker compose --profile pg_tileserv up -d

Stop Server:

docker compose --profile pg_tileserv down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open webbrowser: http://localhost:7800

Vector Tiles Request (Example: bo_boflaeche_mv):

http://localhost:7800/avprodukt.bo_boflaeche_mv/{z}/{x}/{y}.pbf http://localhost:7800/avprodukt.bo_boflaeche_mv,avprodukt.bo_projgebaeude_mv/{z}/{x}/{y}.pbf

Tegola

Tegola is a Webserver created by the Go Spatial Team and is a Webserver-Solution to create OGC-API Tiles from a PostGIS Database on the fly.

Language: Go

GitHub

Start Server:

docker compose --profile tegola up -d

Stop Server:

docker compose --profile tegola down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open manager in the webbrowser: http://localhost:8090

Vector Tiles Request (Example: bo_boflaeche_mv): http://localhost:8080/maps/av/{z}/{x}/{y}.pbf

TiPg

TiPg is pronounced T[ee]pg, is a Python package that helps create lightweight OGC Features and Tiles API with a PostGIS Database backend. The API has been designed for OGC Features and OGC Tiles specifications.

Note This project is the result of the merge between tifeatures and timvt.

Language: Python

GitHub

Start Server:

docker compose --profile tipg up -d

Stop Server:

docker compose --profile tipg down

Note: -d --detach Detached mode: Run containers in the background Without the parameter the log-File will be shown in the Terminal

Open collections in the webbrowser (List of all sources): http://localhost:8080/collections

Vector Tiles Request (Example: bo_boflaeche_mv): http://localhost:8080/collections/avprodukt.bo_boflaeche_mv/tiles/WebMercatorQuad/{z}/{x}/{y} http://localhost:8080/bo_boflaeche_mv,bo_projgebaeude_mv/{z}/{x}/{y}

Performancetest with Apchae JMeter

Apache JMeter is a powerful open-source tool used for testing the performance of web applications, APIs, and other services. It enables developers and testers to conduct various types of tests to evaluate the performance, scalability, and stability of their applications. With Apache JMeter, users can perform load tests, stress tests, throughput tests, behavioral and functional tests, and more. The tool can simulate a large number of users accessing the application simultaneously and records metrics such as response times, throughput, CPU usage, and errors to assess the application's performance under different conditions. Additionally, JMeter provides a user-friendly graphical interface for configuring test scenarios and is known for its scalability and flexibility.

To use Apache JMeter in graphical mode, the software is downloaded from its official website, and then the 'jmeter.bat' file is launched. In this mode, various test scenarios can be defined and executed. However, it is strongly recommended to perform the performance test not in GUI mode but in CLI mode (also known as Non-GUI mode).

To test all servers under as similar conditions as possible, a separate test plan is created for each server. This test plan contains all configurations and steps executed during the test. Such a test plan consists of various elements, including Thread Group, Logic Controller, Sampler, Listener, Timer, and Configurations.

The Thread Group forms the top-level element in the hierarchical structure of each test plan. Here, various parameters can be set that apply to the subsequent elements within this group. For example, the "Number of Threads (users)" parameter can be defined to specify how many virtual users simultaneously send requests to the server.

Below the Thread Group, Logic Controller elements are defined to control how requests are sent to the server. Within these Logic Controller elements, Samplers can be created to represent the corresponding server requests, such as HTTP requests.

Listeners can be placed at any point in the hierarchical structure to visualize the statistical data collected during the test. This can be done in the form of tables, graphs, or summary reports.

For each server, a test plan has been created and executed sequentially in CLI mode. The test plans are stored in the subdirectory vector-tiles-benchmark/test_plans.

Results

The bar chart illustrates the wide range of response times per server. Across all test scenarios, the Martin server showed the fastest performance in serving the tiles. For all servers, tile requests took the longest in test scenario 2. It is noticeable that the response times of the servers vary significantly in most test scenarios, except for test 3. For example, in test 1, the values ranged from 95 to 4153 milliseconds, while the differences in test 2 were even greater, with values between 122 and 8754 milliseconds.

Throughout all tests, the BBOX and Tegola servers showed a maximum response time difference of 41 milliseconds, indicating very similar response times. The Martin server delivered the tiles two to three times faster than the second fastest server. LdProxy took 4 to 70 times longer than Martin to deliver the tiles.

With the exception of test 3, the following server ranking can be established for all test scenarios:

1. Martin
2. Tegola
3. BBOX
4. Pg_tileserv
5. TiPg
6. Ldproxy

Visual Performancetest

performancetest_visual.mp4

Try it yourself: www.vectormap.ch

The HTML and Javascript files are stored in the directory nginx.