![The following logo was created by Dall-E 2 with the following description _3d render of the twitter logo being uploaded to an aluminum bucket_](/having-fun-serverless/backup-twitter/raw/main/images/logo.png)
Backing up Twitter tweets became a hot topic in the last couple of days. I've decided to create a simple POC using Serverless components.
Report Bug
·
Request Feature
Table of Contents
Taking a real-world issue and trying to tackle it using Serverless components is an excellent way to learn about Serverless. The following Serverless application uses a scheduled task that runs once per day and backs up all the Twitter handles that you've configured.
After deployment you can configure the relevant Twitter handles you want to backup by changing the value of a parameter called TwitterAccounts found under the Parameters Store..
Each day the Twitter handles are backed up at 10 AM UTC time. Due to Twitter API restrictions, only the previous day's tweets are being backup up. You can find your tweets under S3.
- We have an evenbridge as a cron scheduler.
- A Lambda is being triggered every day at 10 AM UTC.
- In order to pull the configuration, the AWS Parameters and Secrets Lambda Extension is used.
- Each Twitter handle is being pushed as a separate message into SQS.
- Using the batch processing utility in the Python Lambda Power tools. Each message is processed
- For each handle, we are making a Twitter API call to get the Twitter account id and then the tweets from the last day
- The Twitter bearer token, which is required for authentication, is pulled from the AWS secret manager using AWS Parameters and Secrets Lambda Extension
- Tweets are written into AWS Kinesis Firehose which writes them into S3.
- Dynamic partitioning is used, therefore the files created under S3 have handle prefixes.
- Make sure your machine is ready to work with AWS SAM
- Create a twitter API account
- And save locally your
Bearer Token
, you should receive it at the end of the registration process
- Clone this repository.
- Run
sam build
and thensam deploy --guided
. Accept the default values, except for- Parameter TwitterBearerToken - paste the token value you've recevied from Twitter.
- Parameter TwitterAccountsValues - Choose the Twitter handles you want to back up. Of course, you can use the default here.
After the deployment is complete you can always change the Twitter handles you want to back up by changing the value found under https://us-east-1.console.aws.amazon.com/systems-manager/parameters?region=us-east-1
- You can test the application manully by executing the
ScheduleBackupFunction
Lambda directly from the console.
Monitoring is done by using Lumigo
- IAM Policy for allowing the FH to write into S3 - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L109
- The FH configuration which includes the Dynamic Partitioning configuration - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L138
- Create relevant parms in SAM's yaml - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L167
- Attached extension to the Lambda - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/template.yaml#L40
- Pull parameters via code - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/shared/parameters_utils.py#L7
- Use SQS batch processor - https://github.com/aws-hebrew-book/backup-twitter/blob/de603d2e41826f5511caefbd78a4b2be841847e5/pull_twitter_stream/app.py#L44
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the Apache License Version 2.0 License. See LICENSE
for more information.
Efi Merdler-Kravitz - @TServerless
The project's logo was created by Dall-E 2 with the following description 3d render of the twitter logo being uploaded to an aluminum bucket