Send Amazon RDS Enhanced Monitoring process metrics to Google BigQuery
- Enhanced Monitoring must be enabled at the target RDS instance
- AWS credentials
- which is authorized to invoke
logs:getLogEvents
- which is authorized to invoke
- GCP credential file
- which is authorized to create/edit BigQuery table
bundle install --without development test
docker pull quay.io/dtan4/rdm2bq
bundle exec ./rdm2bq.rb \
-l db-THISISANEXAMPLEDATABASEID0 \
-p foobar-1234 \
-c credentials.json \
-d database_metrics \
-t db_process_metrics
will send the latest process metrics...
- from: CloudWatch Log Event
db-THISISANEXAMPLEDATABASEID0
- to: BigQuery table
foobar-1234:database_metrics.db_process_metrics20180110
20180110
is theYYYYmmdd
-formatted current date in local timezone
UTC
.
If you want to change the timezone of BigQuery table suffix, environment variable TZ
must be set.
docker run --rm -e TZ=Asia/Tokyo --name rdm2bq quay.io/dtan4/rdm2bq:latest \
-l db-THISISANEXAMPLEDATABASEID0 \
-p foobar-1234 \
-c credentials.json \
-d database_metrics \
-t db_process_metrics
Sample CronJob manifest is here.
- Invoke every minutes
- Environment variables are read from the Secret
dotenv
- GCP credential file is read from the Secret
gcp-credentials/credentials.json
kubectl create secret generic gcp-credentials --from-file=credentials.json [-n NAMESPACE]
kubectl create -f kubernetes/cronjob.yaml [-n NAMESPACE]
Option | Environment variable | Description | Example |
---|---|---|---|
TZ |
timezone (tz database zone name) | Asia/Tokyo |
|
-c , --credentials VALUE |
RDM2BQ_GCP_CREDENTIALS |
GCP credentials file | /app/credentials.json |
-d , --dataset VALUE |
RDM2BQ_BIGQUERY_DATASET |
BigQuery dataset | database_metrics |
--dry-run |
Dry run (does not insert any metrics to BigQuery) | ||
-l , --log-stream VALUE |
RDM2BQ_LOG_STREAM |
CloudWatch log stream name | db-THISISANEXAMPLEDATABASEID0 |
-p , --project-id VALUE |
RDM2BQ_GCP_PROJECT_ID |
GCP Project ID | foobar-1234 |
-t , --table-prefix VALUE |
RDM2BQ_BIGQUERY_TABLE_PREFIX |
BigQuery table prefix | db_process_metrics |
field | type | description | example |
---|---|---|---|
timestamp |
TIMESTAMP | timestamp of metrics in UTC | 2018-01-10T02:57:55Z |
pid |
INTEGER | process ID | 65502 |
parent_pid |
INTEGER | parent process ID | 1 |
name |
STRING | name of the process | postgres: user dbname 10.0.0.000(12345) idle |
rss |
INTEGER | amount of RAM allocated to the process (kilobytes) | 12205284 |
vss |
INTEGER | amount of virtual memory allocated to the process (kilobytes) | 32727212 |
Daisuke Fujita (@dtan4)