This an app I built using Splunk Lab to extract data from AWS S3 Server Access Logs and use that data for graphs and analysis.
You'll need to configure Server Access Logging in AWS S3. Once that's done, you can either pull down many small logfiles directly, or instead perform daily rollup on them with an AWS S3 Rollup app which I built specifically for this purpose.
Either way, you can use the aws
CLI app to download all of your logs into logs/
directory and then concatenate the contents of each directory into a single file for that bucket with something like this:
aws s3 sync s3://my-accesslogs/rollup-day/ logs
cd logs/
for DIR in $(find . -type d); do cat $DIR/* > $DIR.txt; done
for DIR in $(find . -type d); do rm -rfv $DIR/* > $DIR.txt; done
Naturally, this is highly dependent on how you're storing logs.
Next, start up Splunk Lab with this command:
bash <(curl -s https://github.com/raw/dmuth/splunk-aws-s3-server-accesslogs/master/go.sh)
The script will guide you through various settings you can send to Splunk Lab.
From there, you can go to https://localhost:8000, log into Splunk with the credentials you specified when starting it, and you should be able to search for data or view reports in dashbaords.
A: You'll need to increase that value in app/limits.conf
. You can read more about that here.
./bin/devel.sh splunk
./bin/build.sh
./bin/push.sh
- This ASCII Text Generator
- Splunk - For having a fantastic analytics platform
Here's how to get in touch with me: