Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lambda-promtail: Add ability to ingest logs from S3 #5065

Merged
merged 26 commits into from
Feb 4, 2022
Merged

lambda-promtail: Add ability to ingest logs from S3 #5065

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
fd80dd0
Add ability to ingest logs from S3 on lambda-promtail
AndreZiviani Jan 6, 2022
a2fec89
fix ci
AndreZiviani Jan 6, 2022
774c5e8
fix typo
AndreZiviani Jan 6, 2022
611d604
bump golang and alpine version
AndreZiviani Jan 6, 2022
dc1b849
update changelog
AndreZiviani Jan 7, 2022
b42243e
add s3 permissions on terraform
AndreZiviani Jan 7, 2022
0ab4647
use for_each instead of count
AndreZiviani Jan 7, 2022
394b6c7
fix typo
AndreZiviani Jan 7, 2022
74f88ac
improve function naming
AndreZiviani Jan 13, 2022
ea56137
add documentation and an example of a s3 file path
AndreZiviani Jan 24, 2022
8331cc1
refact logic to identify event type
AndreZiviani Jan 24, 2022
373eb18
add missing iam permission to allow lambda to run inside a vpc
AndreZiviani Jan 24, 2022
e57b6f6
fix typo
AndreZiviani Jan 24, 2022
bdfda42
allow lambda to access only specified s3 buckets
AndreZiviani Jan 26, 2022
5e75cea
configure a default log retention policy on log group
AndreZiviani Jan 26, 2022
df9eceb
add missing depends_on to make sure iam role is created before lambda…
AndreZiviani Jan 26, 2022
5cfd06a
update docs
AndreZiviani Jan 31, 2022
6e5d6a3
fix label naming convention
AndreZiviani Jan 31, 2022
3b025ff
fix merge conflicts
AndreZiviani Jan 31, 2022
227f77c
fix merge conflict
AndreZiviani Jan 31, 2022
b99d800
use new backoff lib and update dependencies
AndreZiviani Jan 31, 2022
2c241b7
add option to limit batch size
AndreZiviani Feb 3, 2022
b443ebc
cache s3 client
AndreZiviani Feb 3, 2022
7d3c4f8
update docs and terraform
AndreZiviani Feb 3, 2022
370f116
address some feedback on PR
AndreZiviani Feb 4, 2022
b31063a
fix typo
AndreZiviani Feb 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions tools/lambda-promtail/lambda-promtail/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (

"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/service/s3"
)

const (
Expand All @@ -26,6 +27,7 @@ var (
username, password string
keepStream bool
batchSize int
s3Clients map[string]*s3.Client
)

func init() {
Expand Down Expand Up @@ -63,6 +65,8 @@ func init() {
} else {
batchSize = 131072 // 128kb
}

s3Clients = make(map[string]*s3.Client)
}

func checkEventType(ev map[string]interface{}) (interface{}, error) {
Expand Down
15 changes: 11 additions & 4 deletions tools/lambda-promtail/lambda-promtail/s3.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,18 @@ var (
)

func getS3Object(ctx context.Context, labels map[string]string) (io.ReadCloser, error) {
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(labels["bucket_region"]))
if err != nil {
return nil, err
var s3Client *s3.Client

if c, ok := s3Clients[labels["bucket_region"]]; ok {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we only need a single client per bucket region? there isn't the possibility of different buckets in the same region requiring different clients for auth reasons?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we can ignore this for now as multiple regions/bucket auth can be attached to this lambda.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think multiple buckets with different auth options would increase the complexity, it would be easier to configure another lambda with another role

s3Client = c
} else {
cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion(labels["bucket_region"]))
if err != nil {
return nil, err
}
s3Client = s3.NewFromConfig(cfg)
s3Clients[labels["bucket_region"]] = s3Client
}
s3Client := s3.NewFromConfig(cfg)

obj, err := s3Client.GetObject(ctx,
&s3.GetObjectInput{
Expand Down