This project provides mutliple implementations of a RESTful API to process JSON formatted data from simulated sensor bots.
The design approach is to leverage CloudFormation and serverless templates for easy duplication. The implementations will use both a PostgreSQL backend database, we also include a DynamoDB alternative for a more scalable design.
The simulator/sim.py provides a simulator which will publish events either to HTTP POST or direct to file system. simulator/signalbot.py is a python object model for the bots.
In addition the pyapi.py utility can maintain a list of endpoints for POST/GET operations, the swagger file swagger.yaml is availble for the OpenAPI 2.0 standard.
TODO - make a cloud formation stack that takes a RDS snapshot as input builds the right DB target; for the purpose of being able to have a code source (GitHub or CodeCommit) be able to trigger builds from a specific branch.
The serverless.yml should be 'aware' of the branch; and be able to pull the config
haven't touched this in a while; there is no automated build environment; we have a develop and master branch only.
Goals - create the base Stack (creates roles, build bucket) - create DB stack (RDS) - create SSM param for branch (parameter for CI process) - web name - domain name - each check-in CI process will runs sam deploy on the rest
There are four basic components to the Architecture
- Foundational components - Hosted Zone, and SSL certs must be set up (Route53, and Certificate Manager) independent of any AWS templates or configuration
- Framework components - S3 buckets, Cloud Resources (SNS, Parameter Manager) generated for each environment with bot-cfn-base.json template
- API Layer - Lambda/API Gateway configuration providing REST interface
- Persistence Layer - A DynamoDB and a PostgreSQL RDS implementation are provided
As of June 2018, the foundational components will have a recurring cost of $0.50 a month for running a hosted zone in Route53, all other costs are withing the AWS Free Tier. Beyond the free tier the expenses for the DynamoDB are the lowest, and based on transactional volume (you should be able to get o(10^6) transactions for under a dollar). The RDS expense will be driven off the footprint of the server.
DynamoDB is not intended for a long-term store of the data, it does serve all the needs for a fully functional and scalable run-time API. For a future state we would like to include AWS IOT as an alternative to HTTP POST to the API.
Both implementations utilize S3 and CloudFront to present the REACT based UI.
The static folder contains NodeJS files for the REACT framework to provide static graphs and interface to the API.
This implementation is provided, to help bridge the user to a serverless model. There are many reasons why a relational database may be needed for reporting or archive that the DynamoDB solution would need to supplement.
The DynamoDB implementation is truly serverless, the user does NOT need to have a VPC defined to use this.
The Lambda function connects to a DynamoDB table, the REST interface does not change.
Open this link with swagger UI - swagger.yaml
Explain layout of code.
Use 'develop', 'test', and 'master' branch to reflect environments.
The install path depends on the API implementation, there are several common steps.
Before running any scripts of utilities in this repository, there must be an AWS account, with a hosted zone and two SSL certs created in the us-east-1 region. Two A record will be created.
- a Hosted Zone must exist in the AWS account
- SSL certs for the web host and the api need to be created in the us-east-1 Zone
This cloud formation template takes the DNS, host, and cert information and creates a stack that is leveraged by the serverless templates. The bot-cfn-base.json template creates the following resources
Currently the serverless.yml is hardcoded to look for botbase
- SNS Topic for stack related events
- Cognito User Pool
- S3 Bucket to perform automated builds
- S3 Bucket to host static web Content
- CloudFront distribution
- Route53 DNS Entries for Web
- Stores key parameters in the Parameter Store
- CodeBuild project is set up with GitHub source (which docker image to build with and how to set environment variables)
You can run the Cloud Formation templates either from a CLI, or the AWS console.
In order to deplpy the Serverless code, we need to set up a SLS workstation, there is a template that does this Next, create a SLS workstation, and deploy the serverless components (the workstation is created because need to run the NodeJS commands and need appropriate system permissions).
Can I do all this with a docker image instead?
Use the Cloud Formation template from mcliff1/aws ec2-slsworkstation
Then create a IAM role we will need to install using the right script. TODO we need to clean up thse permissions scripts, there is a different set for deploying static content, api level, and the different DBs.;
the EC2 create should take as optional input a stack name that was used as a base; this sets a parameter in SSM called /{stackname}/iamEc2Role
TODO - this should create the IAM role necessary
Log into the server; pull the code from GitHub and deploy
git config --global --editto set up GIT credentialsgit clone https://github.com/mcliff1/SignalBot
./deploy-web.sh run from the SLS server will do everything necessary (in us-west-2)
Then sync the build directory to the S3 bucket for the web (which can be gotten from SSM)
AWS CodeBuild is set up as part of the template, by default it is connected to this GitHub repository.
If you choose the RDS option, skip this section.
Here is it different based on the implementation, for Dynabot
cd SignalBot/dynabotsls deploy --region us-west-2(or whatever region you are running in, the EC2 instance has no default)
This has the basestack name hard coded as botbase.
This creates the Lambda function, API gateway, API DNS name, and DynamoDB table.
If you choose the DynamoDB option, skip this section
The simplify implementation we will leverage the CloudFormation templates in the mcliff1/aws repository.
You can either set up the RDS database from a snapshot or build from source. In either case you will need to run either rds-postgres.json or rds-postgres-snapshot.json template. These require that a mcliff1/aws/vpc.json stack has already created. The RDS will be non-publicly accessible and created in the private subnets of the VPC, a DB workstation or Bastion host will be required to directly access. The output parameters dbname, dbuser, and db_endpoint can be queried from either stack (TODO this isn't done yet)
% git clone https://github.com/mcliff1/SignalBot
% docker run -it -v /home/ec2-user/SignalBot:/scripts openbridge/ob_pysh-db psql \
-h <db-endpoint-url> \
-p 5432 -U dbuser -d dbname1 -f /scripts/createdb.sql
This pulls the SignalBot repository locally (all we need is the create script) and runs a docker utility to load the database.
These only apply for the RDS solution, and will create a EC2 instance that you can use to connect to or bridge to the RDS instance of PostgreSQL. It is NOT necessary to build this server to run the solution. The RDS server is created in a private VPC subnet with no access to the public internet.
For now use the mcliff1/aws/bastion.json script.
Use the bastion host template in mcliff1/aws and then run the following; enter the DB password at the prompt (in the future once the bastion.json settles down we can branch that and make this one-stop, along with some prebuilt utilities that know the location of the DB, they are available from CloudFormation).
In order to run these serverless stacks you need to create a SLS Workstation.
| CFN Foundation |
|---|
|
Creates static resources for managing the stack, and sets parameters for other stacks
|
| Dynabot (SLS) |
|---|
|
Creates everything
|
| Bot (SLS) |
|---|
|
Creates everything
|
- need to make the VPC subnet's dynamic or variable
- better define the GET operation (allow parameters to get data to plot)
- cloudformation or serverless project to create the postgres DB
- get the static google-charts.js to deploy on S3
- should we also back this to S3 (we could put some version control on there and a Lambda trigger) (for DyanmoDB)
- parameters for the PostGre SQL database string to connect too
This part is still in development, the plan is to use a REACT framework and GraphQL to be able to generate serverless graphing capabilities. We want to see if it makes sense to expose a GraphQL endpoint to leverage teh Google Charts.js or Charts.js.