Fetch a GeoIP Database for MaxMind GeoIP Processor

[rogercoll]

Component(s)

processor/geoip

Is your feature request related to a problem? Please describe.

Initial issue: #32663

Currently, a local file path must be provided to the MaxMind GeoIP processor in order to retrieve the associated IP metadata. This approach requires users to manually download and manage the GeoIP database files, ensuring they are kept up-to-date and properly licensed. This manual process can be cumbersome and error-prone. Automating the retrieval of the GeoIP database would significantly streamline the setup and maintenance process.

Describe the solution you'd like

The idea would be to initially download the latest available database from MaxMind's Direct download link: https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases. The following configuration options will be needed:

• permalink: url to fetch the database from.
• account_id: account ID used to fetch.
• license_key: license key used to fetch.
• auto_update: Whether to frequently check for available updates.

Describe alternatives you've considered

TODO: Verify if it would be feasible to use the auth extension instead of the custom account_id and license_key configuration options.

Additional context

Automatic database fetch

If neither a local path nor a license key is specified, the processor should be able to fetch a default GeoIP database automatically, similar to Elasticsearch's geoip processor. This feature would eliminate the need to specify a custom configuration.

Elasticsearch automatically downloads updates for these databases from the Elastic GeoIP endpoint: https://geoip.elastic.co/v1/database. The processor could use the same endpoint to automatically download the latest available database.

Licensing

There are some concerns regarding the implementation of this feature if we use Elastic's https://geoip.elastic.co/v1/database as default download endpoint. Could there be any licensing implications by using the same URL? MaxMind databases are shared under the CC BY-SA 4.0 license.

[github-actions]

Pinging code owners for processor/geoip: @andrzej-stencel @michalpristas @rogercoll. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[rogercoll]

MaxMind changed from releasing the GeoIP database under a Creative Commons (CC) license to a proprietary end-user license agreement (EULA). The MaxMind EULA requires any software bundling their databases to update them within 30 days of a database update.

The automatic database update strategy would be similar to what the GeoIP Logstash plugin does. By default, the Logstash filter bundles Creative Commons (CC) license databases and if the auto-update feature is enabled, it checks for database updates every day. It downloads the latest and replaces the old database while the plugin is running. After Logstash has switched to a EULA licensed database, the geoip filter will stop enriching events in order to maintain compliance if Logstash fails to check for database updates for 30 days. Events will be tagged with _geoip_expired_database tag to facilitate the handling of this situation.

My main concern when porting this functionality into Otel is the dynamic license change of the Logstash filter itself:

If the auto-update feature is enabled, Logstash upgrades from the CC database license to the EULA version on the first download.

[michalpristas]

does this also apply when db is not bundled but just fetched instead?