Download Meteo France Radar imagery using Kalisio Krawler https://github.com/kalisio/krawler
This sample is based on https://github.com/kalisio/k-meteoradar, adapted to run on Windows for development and training purpose.
It scrapes public Radar rainfall data from Meteo France public server. The downloaded images are stored in PNG format without any georeference information. The job first georeferenced the image and then converts it to COG format and copies it to a store.
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
Download and install : https://git-scm.com
nvm is a version manager for node.js, designed to be installed per-user, and invoked per-shell.
Download and install : https://github.com/coreybutler/nvm-windows
Then install Node 16
nvm install 16
nvm use 16.15.1
Yarn is a package manager that doubles down as project manager.
Download and install : https://classic.yarnpkg.com/lang/en/docs/install/#windows-stable
Add yarn to Windows environment PATH variable
GDAL is a translator library for raster and vector geospatial data formats
Download and install GDAL from https://trac.osgeo.org/osgeo4w/ (or install QGIS https://www.qgis.org/ that contains GDAL)
Set windows environment variables for GDAL according to your GDAL installation
set PATH=%PATH%;c:\Program Files\QGIS 3.24.0\bin
Rclone is a command-line program to manage and transfer files on local or cloud storage.
Download and install Rclone from https://downloads.rclone.org/v1.58.1/rclone-v1.58.1-windows-amd64.zip
Add rclone to Windows environment PATH variable
Use rclone config to set your store (according your needs, S3, FTP, local ...)
rclone config
NB : In this sample, we use a local storage (n°24 in the config list) named "store"
Krawler is a minimalist ETL that make automated process of extracting and processing (geographic) data from heterogeneous sources with ease
In the working directory (e.g. c:\workspace), create a kalisio directory
cd c:\workspace\kalisio
git clone https://github.com/kalisio/krawler
cd krawler
yarn install
yarn link
In the kalisio directory ( ex: C:\workspace\kalisio)
git clone https://github.com/calysteau/krawler-job03
cd krawler-job03
yarn install
yarn link @kalisio/krawler
Set destination storage used by rclone
set STORE_PATH=c:\output
Run the job
krawler jobfile.js
If you need to activate the Krawler DEBUG
set DEBUG= krawler* (pour CMD)
or
$env:DEBUG="krawler*" (pour PowerShell)