Project fork from deviantony/docker-elk.
This fork show how to implements ELK stack + filebeat on Docker + docker-compose.
We have one (or more) application (python for example) what write some logs on application/logs/.log
, and the filebeat
will read this file and send to logstash
that will do regex filter and after also send to elasticsearch
and finally we'll build some graphics on kibana
.
This simple python
example will write some logs for us.
import os
import uuid
import time
import log
def hello_log():
while True:
log.registry(log.LOGGING_ENUM.INFO, f'Info {uuid.uuid4()}')
log.registry(log.LOGGING_ENUM.WARNING, f'Warning {uuid.uuid4()}')
log.registry(log.LOGGING_ENUM.ERROR, f'Error {uuid.uuid4()}')
time.sleep(15)
if __name__ == '__main__':
if not os.path.exists(os.path.join(os.getcwd(), 'logs', '.log')):
os.mkdir(f'{os.getcwd()}/logs3')
hello_log()
ps: I split the logs in a specific module: log.py
28/05/2021 20:33:19 INFO || 3e860697f14e || my_module || Company Name || Info 2473c107-aff8-4b48-85d6-a7ef6cdf89a6
28/05/2021 20:33:19 WARNING || 3e860697f14e || my_module || Company Name || Warning 9a2d445a-5859-4f6a-8496-059a04c461d3
28/05/2021 20:33:19 ERROR || 3e860697f14e || my_module || Company Name || Error d336fe31-86f9-4abe-b0c1-3738f5965351
28/05/2021 20:33:24 INFO || 3e860697f14e || my_module || Company Name || Info a469d96b-564e-4af9-a874-ef2237ae3e71
28/05/2021 20:33:24 WARNING || 3e860697f14e || my_module || Company Name || Warning a8a123b3-3379-4000-8c8a-83629a4ecd05
28/05/2021 20:33:24 ERROR || 3e860697f14e || my_module || Company Name || Error 6e07dbeb-8969-4b95-97df-c1253c934d19
28/05/2021 20:33:29 INFO || 3e860697f14e || my_module || Company Name || Info 9c6c5e09-2001-48ba-a33a-f2f2729b68a8
Click here to see the regex from this.
$ make build
This step takes a few minutes to start ELK and all services to connect.
$ docker-compose logs -f # all containers
$ docker-compose logs -f <name> # [app, elasticsearch, logstash, filebeat, kibana]
Go to http://localhost:5601/ to access kibana.
username: elastic
password: changeme
We set the index to my_index_patterns
on logstash.conf
ps: don't care about my old tests
Select timestamp or not and
CREATE INDEX PATTERNS
Click to update to see changes