important: this project is currently in pre-alpha... Some parts work, others don't!!! It's useless to open issues at this point since everything is a work in progress!!!
combined logfile from apache parsing, putting all data in sqlite and visualising said data
I had an apache webserver running... All default... Combined log format... And i wanted some stats.
In the open source world, i ended up with 3 choices:
- awstats
- webalizer
- goaccess
all other tools seem to be worthless or paying. And those 3 were either allmost unmaintained, or not really what i wanted.
So i tought to myself: why not add one more 😄
golang
It's a go program, vendoring included... just clone it, verify the sourcecode, build and run... Or use my precompiled binary's.
- go
- linux (eventough it should also be compileable on windows)
- apache combined logs
- space... Usually your logfiles will be rotated and gzipped. I'll be unpacking them and putting them into a relational database (which produces some more overhead). For every ~70 Mb of UNCOMPRESSED logs, you'll need about 100Mb of diskspace
- time. Eventough compiled go runs pretty fast 😄. The initial testload of about ~70 Mb of logfiles took about 40 seconds on my old XEON dedicated server. That's about 1 second per 2 Mb of logfiles. Subsequent loads will mostly skip already loaded files and it'll skip records that are older than the newest timestamp in the db of changed logfiles.
-
Clone the repo
git clone https://github.com/mocacinno/CLFtoSQLite.git
-
build
go build logfileparser.go go build stats.go
-
copy (or directly edit) config.template.ini (you can swap out nano by vi or any other editor)
cp config.template.ini config.ini nano config.ini
-
build the sqlite database and fill it. You need to run this tool every time you want to load the ascii logs into the sqlite database for analysis/graphing
./logparser -config config.ini
-
run the stats
./stats -config config.ini
- stats uses a cursor to read data. I need to use a struct so i can re-use
- I don't want to load all entries... I need to filter some entries out
- stats needs graphs
Distributed under the Apache2.0 License.