Welcome to stream-iot! This project contains an example of an end-to-end workflow for processing streaming data. The workflow consists of mocking sensor data, channeling it through Kafka, and then storing the parsed data in a MongoDB database.
I started this project as a way to learn and get some initial hands-on experience with Kafka. I hope that this project can help others get started on similar projects. At the same time, I am sure I can learn from others viewing this project. If you have feedback or improvement suggestions, please create an issue in this repository.
- Cloud - Azure
- Infrastructure as Code - Terraform
- Containerization - Docker
- Container Orchestration - Kubernetes
- Monitoring - Prometheus, Grafana
- Workflow Orchestration - Airflow
- Messaging & Stream Processing - Kafka
- Data Storage - CosmosDB, MongoDB
- Languages - Python
- You have az cli installed.
- You have kubectl installed with
az aks install-cli.
Follow the installation instructions in the README's of these directories in order:
- terraform
- kafka
- airflow
- python
- monitoring (optional)
When this is done, open the Airflow UI with
kubectl port-forward svc/airflow-webserver 8080:8080 -n airflowand trigger the DAG's manually.