starry

实时流计算任务, 基于flink

数据流程 http-client => nginx => rsyslog => kafka => flink job => es => grafana

任务列表

• NginxAccessLogEtlToEsJob

解析 nginx access log，写入es

source : kafka sink: es

• NginxAccessLogCollectEvtEtlToKafkaJob

解析 nginx access log 解析 request 中的query参数 写入kafka

source : kafka sink: kafka

• CollectEvtEtlToEsJob

读取 collect-evt kafka 写入es

source : kafka sink: es

• CollectEvt1mPvJob

分钟pv聚合数据

读取 collect-evt kafka 写入es

source : kafka sink: es

flink配置

开启 checkpointing execution.checkpointing.interval: 1min