starry

实时流计算任务, 基于flink

数据流程 http-client => nginx => rsyslog => kafka => flink job => es => grafana

任务列表

NginxAccessLogEtlToEsJob

解析 nginx access log，写入es

source : kafka sink: es

NginxAccessLogCollectEvtEtlToKafkaJob

解析 nginx access log 解析 request 中的query参数写入kafka

source : kafka sink: kafka

CollectEvtEtlToEsJob

读取 collect-evt kafka 写入es

source : kafka sink: es

CollectEvt1mPvJob

分钟pv聚合数据

读取 collect-evt kafka 写入es

source : kafka sink: es

flink配置

开启 checkpointing execution.checkpointing.interval: 1min