You should have node package manager, the docker engine, and docker compose installed.
Make sure you are using Node v14.xx.x.
If you do not have an ELK stack already, you can use this docker image to get started.
Additionally, your AWS account must have the following permissions. Best practice is to create a brand new IAM role with these permissions attached, and then to assign this role to your user.
"iam:PassRole",
"iam:DetachRolePolicy",
"iam:CreatePolicy",
"iam:CreateRole",
"iam:AttachRolePolicy",
"iam:DeletePolicy",
"iam:DeleteRole",
"kms:Decrypt",
"kms:CreateGrant",
"kms:RetireGrant",
"kms:Encrypt",
"lambda:CreateEventSourceMapping",
"lambda:CreateFunction",
"lambda:DeleteEventSourceMapping",
"lambda:DeleteFunction",
"s3:DeleteObject",
"s3:DeleteBucket",
"s3:CreateBucket",
"s3:ListBucket",
"s3:PutObject",
"s3:GetObject",
"sqs:SendMessage",
"sqs:GetQueueAttributes",
"sqs:DeleteQueue",
"sqs:CreateQueue"
Also, you must configure the http plug-in for logstash. Here is an example of a Logstash config file that includes configuration for the plugin:
input {
http {
port => destired_port // default port 8080
}
}
filter {
mutate {
// removes axios artifacts
remove_field => ["[event][original]", "[http]", "[user_agent]", "[url]"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "your_elasticsearch_primary_node"
index => "your_desired_index_name" //example: s3logs-%{+YYYY.MM.dd}
}
}
Finally, your logs to be re-ingested must be stored in JSON format in an S3 Bucket. It is recommended that your S3 Bucket has only logs in it.
git clone https://github.com/Team-Arroyo/arroyo-deployment.git
cd arroyo-deployment
npm install
touch .env
AWS_REGION=your-region
AWS_BUCKET_NAME=name-of-your-s3-bucket-containing-logs
AWS_ACCESS_KEY=your-access-key
AWS_SECRET_KEY=your-secret-key
LOGSTASH_HOST=your-logstash-host
npm run deploy
See our Readme here for instructions on how to perform a bulk re-ingest or query based re-ingest using the browser based graphical user interface.
docker compose up
docker compose start
docker compose stop
docker compose down
npm run destroy