Splunk for AWS S3 Server Access Logs

This an app I built using Splunk Lab to extract data from AWS S3 Server Access Logs and use that data for graphs and analysis.

Getting your AWS S3 Server Access Logs

You'll need to configure Server Access Logging in AWS S3. Once that's done, you can either pull down many small logfiles directly, or instead perform daily rollup on them with an AWS S3 Rollup app which I built specifically for this purpose.

Either way, you can use the aws CLI app to download all of your logs into logs/ directory and then concatenate the contents of each directory into a single file for that bucket with something like this:

• aws s3 sync s3://my-accesslogs/rollup-day/ logs
• cd logs/
• for DIR in $(find . -type d); do cat $DIR/* > $DIR.txt; done
• for DIR in $(find . -type d); do rm -rfv $DIR/* > $DIR.txt; done

Naturally, this is highly dependent on how you're storing logs.

Starting up Splunk Lab

Next, start up Splunk Lab with this command:

• bash <(curl -s https://github.com/raw/dmuth/splunk-aws-s3-server-accesslogs/master/go.sh)

The script will guide you through various settings you can send to Splunk Lab.

From there, you can go to https://localhost:8000, log into Splunk with the credentials you specified when starting it, and you should be able to search for data or view reports in dashbaords.

Known Issues

Q: I see an error about exceeding "the configured depth_limit"?

A: You'll need to increase that value in app/limits.conf. You can read more about that here.

Development

• ./bin/devel.sh splunk
• ./bin/build.sh
• ./bin/push.sh

Additional Resources

• AWS S3 Server Access Log Format

Credits

• This ASCII Text Generator
• Splunk - For having a fantastic analytics platform

Bugs/Contact

Here's how to get in touch with me:

• Twitter
• Facebook
• Blog
• Email