Dataset Updating Plan

[hacdias]

While planning to move to the new dataset (see #63), I found some problems I need help with!

Field, information, field and more information

First of all, right now, we have the following data for each location:

{
  "country_code": "US",
  "country_name": "United States",
  "region_code": "CA",
  "city": "Mountain View",
  "postal_code": "94040",
  "latitude": 37.3860,
  "longitude": -122.0838,
  "metro_code": "807",
  "area_code": "650",
  "planet": "Earth"
}

The new datasets contain much more than that:

is_anonymous_proxy
is_satellite_provider
postal_code
latitude
longitude
accuracy_radius
locale_code
continent_code
continent_name
country_iso_code
country_name
subdivision_1_iso_code
subdivision_1_name
subdivision_2_iso_code
subdivision_2_name
city_name
metro_code
time_zone
is_in_european_union

I am pretty sure we don't need all of those fields, so the first goal of this issue is to define which informations do we want to provide through this package.

IPv6

The second issue is: how to support IPv6 (#60)? The newest dataset has an IPv6 table too! Just like the IPv4, we are provided with CIDR addresses that allow us to know the range for which to check for IPv6 addresses. However, unlike IPv4, there's no "int long" form of IPv6 so we can't keep the same structure as we have now for IPv4.

Knowing this, how'd you suggest to tackle this issue? How to organize the information in such a way we can fetch it quickly?

Languages?

The new dataset provides translations for just some languages. Are they worth including or shall we keep just the english ones for now?

---

Also, I am thinking about setting up a way of updating the geoip database automatically since they update it every tuesday. It would be great so we wouldn't need to think a lot about this (perhaps just merging a PR with the newer CID).

---

Ping @lidel

[hacdias]

Also cc @Kubuxu

[Kubuxu]

@hacdias how many entries there are in IPv6 dataset?

[hacdias]

@Kubuxu about 431 990 for the IPv6 dataset and 3 252 943 for the IPv4 one.

[hacdias]

Pinging @lidel and @Kubuxu 😅

[Kubuxu]

Sorry (got a lot on my head).
The best structure for it would be ipld based radix tree, but it wouldn't be easy to implement.
We can do away with something simpler (pseudo radix tree). I will write something more on that tomorrow.

[hacdias]

Thanks! Will be awaiting!

[hacdias]

Not accounting for IPv6 addresses, this is my current idea and plan. I am thinking of keeping the same binary tree structure as we have now, but I do have some doubts and questions to be answered. Please let me know what you think:

1. We have locations and IP blocks. There are ~120 000 location and ~3 115 100 IPv4 blocks, which means the relationship between locations and IP blocks is one to many.
2. Taking the last point in account, I thought about separating both things:
   1. Create a tree with the locations. It should be a really simple tree whose main purpose is to be pinned on the gateways and the locations should be CID addressed.
      1. Is there any limit of IPLD Links? Is it possible to just have one object with links to all locations and pin that?
   2. Create a second tree with the blocks. It is a binary tree as we have now. We address the location by their hash so we just need to get the object of the location after getting the rest of the information about a certain IP.
      1. We have 32 children now and I am thinking about increasing to 64.
      2. We have a lot of IPv4 blocks and this is gonna take tons of time to generate.

Feedback? @lidel @Kubuxu

[lidel]

Sadly I had no bandwidth, this needs another set of eyes.

[lidel]

Figuring out new data structures will be time consuming, and given limited bandwidth I propose we simply convert the new CSV dataset to the format of old CVS source files:

ipfs-geoip/src/generate/index.js

Lines 15 to 22 in 00f72f9

	// All data is stored in an ipfs folder called data
	// this is the hash of that folder. It includes three files
	//
	// data
	// |- blocks.csv
	// |- countries.csv
	// |- locations.csv
	const DATA_HASH = 'QmTMh5Q1CnB9jV774aKCvPSqibwDy9sJmo7BCThD5f1oY3'

That way ipfs-geoip does not need to be rewritten, and we can at least update IPv4 info.