-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lambda-promtail: Add ability to ingest logs from S3 #5065
Merged
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
fd80dd0
Add ability to ingest logs from S3 on lambda-promtail
AndreZiviani a2fec89
fix ci
AndreZiviani 774c5e8
fix typo
AndreZiviani 611d604
bump golang and alpine version
AndreZiviani dc1b849
update changelog
AndreZiviani b42243e
add s3 permissions on terraform
AndreZiviani 0ab4647
use for_each instead of count
AndreZiviani 394b6c7
fix typo
AndreZiviani 74f88ac
improve function naming
AndreZiviani ea56137
add documentation and an example of a s3 file path
AndreZiviani 8331cc1
refact logic to identify event type
AndreZiviani 373eb18
add missing iam permission to allow lambda to run inside a vpc
AndreZiviani e57b6f6
fix typo
AndreZiviani bdfda42
allow lambda to access only specified s3 buckets
AndreZiviani 5e75cea
configure a default log retention policy on log group
AndreZiviani df9eceb
add missing depends_on to make sure iam role is created before lambda…
AndreZiviani 5cfd06a
update docs
AndreZiviani 6e5d6a3
fix label naming convention
AndreZiviani 3b025ff
fix merge conflicts
AndreZiviani 227f77c
fix merge conflict
AndreZiviani b99d800
use new backoff lib and update dependencies
AndreZiviani 2c241b7
add option to limit batch size
AndreZiviani b443ebc
cache s3 client
AndreZiviani 7d3c4f8
update docs and terraform
AndreZiviani 370f116
address some feedback on PR
AndreZiviani b31063a
fix typo
AndreZiviani File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| all: build docker | ||
|
|
||
| build: | ||
| GOOS=linux CGO_ENABLED=0 go build lambda-promtail/main.go | ||
| GOOS=linux CGO_ENABLED=0 go build -o ./main lambda-promtail/*.go | ||
|
|
||
| clean: | ||
| rm main |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "time" | ||
|
|
||
| "github.com/aws/aws-lambda-go/events" | ||
| "github.com/grafana/loki/pkg/logproto" | ||
| "github.com/prometheus/common/model" | ||
| ) | ||
|
|
||
| func parseCWEvent(ctx context.Context, b *batch, ev *events.CloudwatchLogsEvent) error { | ||
| data, err := ev.AWSLogs.Parse() | ||
| if err != nil { | ||
| fmt.Println("error parsing log event: ", err) | ||
| return err | ||
| } | ||
|
|
||
| labels := model.LabelSet{ | ||
| model.LabelName("__aws_cloudwatch_log_group"): model.LabelValue(data.LogGroup), | ||
| model.LabelName("__aws_cloudwatch_owner"): model.LabelValue(data.Owner), | ||
| } | ||
| if keepStream { | ||
| labels[model.LabelName("__aws_cloudwatch_log_stream")] = model.LabelValue(data.LogStream) | ||
| } | ||
|
|
||
| for _, event := range data.LogEvents { | ||
| timestamp := time.UnixMilli(event.Timestamp) | ||
|
|
||
| b.add(ctx, entry{labels, logproto.Entry{ | ||
| Line: event.Message, | ||
| Timestamp: timestamp, | ||
| }}) | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| func processCWEvent(ctx context.Context, ev *events.CloudwatchLogsEvent) error { | ||
| batch, _ := newBatch(ctx) | ||
|
|
||
| err := parseCWEvent(ctx, batch, ev) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| err = sendToPromtail(ctx, batch) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| return nil | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not swallow this error. Perhaps
newBatchshouldn't accept entries and thennewBatchwon't need to return an error.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since the batch will always be empty here it should never fail, thats why I ignored it