Feature/vector tiles training files

[kshitijrajsharma]

Blocked by #289

What does this PR do ?

• Adds base model to models table , Default is RAMP but choices are Yolo and Ramp

New Model table :

• Adds vector tile generation logic for the aois and labels in each training checkpoint

This is how pmtiles will look like with AOI and labels + Queryable attribute

• Adds banner table with its CURD API

Table :

CURD:

• Implements basic authentication of is user staff ? is user admin ? is this object belongs to user ? and is user logged in

• Adds training centroid option when training is finished

• Refactors all user related information from 'created_by' or 'approved_by' to user --- To maintain consistency

• Adds searching by id feature to models endpoint

eg : https://fair-dev.hotosm.org/api/v1/model/?search=12

• Adds feedback count to training object get endpoint directly

eg : https://fair-dev.hotosm.org/api/v1/training/20/

• Improves admin view to change permission of users for admin -: Adds search , sort and update permission
• Adds the kpi stats endpoint

eg: https://fair-dev.hotosm.org/api/v1/kpi/stats/

Consideration

This should be the stable version , centroid which is being stored after each training is done so that it can be fetched when training is published and linked to model . This PR includes heavy refactor , please be cautious !

Deletion is not allowed for anyone until and unless it is not your object , only staffs/ admin are allowed to delete
POST request is allowed if user is not associated to particular request : for eg - you can still create new training

Django authentication is extended and followed the same standard.
Banner message is a textfield which can store raw .md

How to test ?

• Test Pmtiles : https://fair-dev.hotosm.org/api/v1/workspace/download/dataset_286%2Foutput%2Ftraining_638%2Fmeta.pmtiles/ , Provide your OSM authentication

• Try deleting some objects which is not yours

• Try publishing the training of the model which is not yours , you should still be able to submit the training but not publish it !

• Make sure , this model api is searchable with model id : https://fair-dev.hotosm.org/api/v1/model/?search=12

• Make sure total feedback count is being returned from this api https://fair-dev.hotosm.org/api/v1/training/20/

• Test KPI stats https://fair-dev.hotosm.org/api/v1/kpi/stats/

• Try publishing, disabling or deleting banner and making sure it is reflected on the API https://fair-dev.hotosm.org/api/v1/banner/?limit=1

• Try creating banner via API , you should not be allowed to the system until and unless you are a staff / an admin

TODO :

• Publish .pmtiles to s3 and add cache in s3 bucket requests to render pmtiles faster

[omranlm]

@kshitijrajsharma does the permission on https://fair-dev.hotosm.org/api/v1/workspace/download/dataset_286%2Foutput%2Ftraining_638%2Fmeta.pmtiles/

Means that I can't also download other files in the v1/workspace/download/ end point?
for example, download the model h5?

[kshitijrajsharma]

@kshitijrajsharma does the permission on https://fair-dev.hotosm.org/api/v1/workspace/download/dataset_286%2Foutput%2Ftraining_638%2Fmeta.pmtiles/

Means that I can't also download other files in the v1/workspace/download/ end point?

for example, download the model h5?

yes , you should be able to download once you are logged in ! Pass your osm token you should be in , There is no permission to view the file but login is needed to download ! This is to avoid random script pointing to our production server

[omranlm]

Looks great, thanks for your efforts
@jeafreezy over to you upon deployment on the dev env

[jeafreezy]

Looks great, thanks for your efforts @jeafreezy over to you upon deployment on the dev env

Thanks for your great work @kshitijrajsharma .

I have a question: Is the name and route constant i.e I should retrieve the pmtiles from the ...api/v1/workspace/download/dataset_286/output/training_638/meta.pmtiles. endpoint.

[jeafreezy]

• Adds searching by id feature to models endpoint

eg : https://fair-dev.hotosm.org/api/v1/model/?search=12

This works, however, it's returning multiple results. Id should be unique right?

[jeafreezy]

Adds the kpi stats endpoint
eg: https://fair-dev.hotosm.org/api/v1/kpi/stats/

Please cache this endpoint if you haven't already given that it's going to the landing page.

We might need to add some skeleton/defaults there to avoid long page loads since it's the landing page @omranlm.

[kshitijrajsharma]

This works, however, it's returning multiple results. Id should be unique right?

Its a search , it will return all the id that matches the search parameter
@jeafreezy

for unique use this one :
https://fair-dev.hotosm.org/api/v1/model/?id=12

[kshitijrajsharma]

I have a question: Is the name and route constant i.e I should retrieve the pmtiles from the ...api/v1/workspace/download/dataset_286/output/training_638/meta.pmtiles. endpoint.

Not yet , I will give you the s3 link perhaps , We are thinking about it ! @jeafreezy

[jeafreezy]

This works, however, it's returning multiple results. Id should be unique right?

Its a search , it will return all the id that matches the search parameter @jeafreezy

for unique use this one : https://fair-dev.hotosm.org/api/v1/model/?id=12

This is what I need. Thanks!

[kshitijrajsharma]

@jeafreezy Cache is enabled on the endpoint for kpi stats . Global cache is applied for 15 min ! it should be more than okay ! Thanks