diff --git a/README.md b/README.md index 320967b..a77551a 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,19 @@ # fluent-plugin-json -[Fluentd](https://fluentd.org/) filter plugin to do something. - -TODO: write description for you plugin. +[Fluentd](https://fluentd.org/) filter plugin for JSON with JSON pointer support +([RFC-6901](https://tools.ietf.org/html/rfc6901)). ## Installation ### RubyGems -``` +```bash $ gem install fluent-plugin-json ``` ### Bundler -Add following line to your Gemfile: +Add the following line to your Gemfile: ```ruby gem "fluent-plugin-json" @@ -22,22 +21,99 @@ gem "fluent-plugin-json" And then execute: -``` +```bash $ bundle ``` ## Configuration -You can generate configuration template: +### `` section (required) (multiple) + +* `pointer` (string) (required): The JSON pointer to an element. +* `pattern` (regexp) (required): The regular expression to match the element. + +The configuration consists of one or more check(s). Each check contains a +`pointer` to a JSON element and a `pattern` (regex) to test it. + +The checks are evaluated sequentially. The failure of a single check results in +rejection of the event. A rejected event is not routed for further processing. + +NOTE: The JSON element pointed to by the `pointer` is always converted to string +for testing with the `pattern` (regular expression). + +For examples of the syntax of: + +- JSON Pointer, see [RFC-6901](https://tools.ietf.org/html/rfc6901#section-5). +- Ruby's Regular Expression, see [Regexp](https://ruby-doc.org/core-2.4.1/Regexp.html). + +### Example + +Here is a configuration with the input plugin +[`forward`](https://docs.fluentd.org/v/1.0/input/forward), `json` filter plugin +with multiple checks and routing to the output plugin +[`stdout`](https://docs.fluentd.org/v/1.0/output/stdout): + +```text + + @type forward + @id forward_input + + + + @type json + @id json_filter + + pointer /log/user # point to { "log": { "user": "test", ... } } + pattern /test/i # check it against username `test` (ignore case) + + + + pointer /log/codes/0 # point to { "log": { "codes": [123, ...] } } + pattern /123/ # check it against 0th index of codes array + + + + pointer /log/level # point to { "log": { "level": ... } } + pattern /.*/ # check it against all log levels + + + + + @type stdout + @id stdout_output + ``` -$ fluent-plugin-config-format filter json + +For a JSON message: + +```json +{ "log": {"user": "test", "codes": [123, 456], "level": "info"} } ``` -You can copy and paste generated documents here. +Sent using `fluent-cat` with tag `debug.test`: + +```bash +$ echo '{ "log": {"user": "test", "codes": [123, 456], "level": "info"} }' | fluent-cat "debug.test" +``` + +After passing all the checks, the routed event to `stdout` would be: + +```bash +2020-07-23 22:36:06.093187459 +0500 debug.test: {"log":{"user":"test","codes":[123,456],"level":"info"}} +``` + +By default, the logs for checks are generated in `debug` mode only: + +```bash +2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/user -> 'test'] (/test/) +2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/codes/0 -> '123'] (/123/) +2020-07-23 22:47:33 +0500 [debug]: #0 [json_filter] check: pass [/log/level -> 'info'] (/.*/) +2020-07-23 22:47:33.577900915 +0500 debug.test: {"log":{"user":"test","codes":[123,456],"level":"info"}} +``` ## Copyright -* Copyright © 2020 Azeem Sajid +* Copyright © 2020 [Azeem Sajid](https://www.linkedin.com/in/az33msajid/) * License * Apache License, Version 2.0 diff --git a/fluent-plugin-json.gemspec b/fluent-plugin-json.gemspec index e5d5cfc..a809a39 100644 --- a/fluent-plugin-json.gemspec +++ b/fluent-plugin-json.gemspec @@ -10,7 +10,7 @@ Gem::Specification.new do |spec| spec.email = ['azeem.sajid@gmail.com'] spec.summary = 'Fluentd filter plugin for JSON events with JSON Pointer Support' - spec.description = 'Fluentd filter plugin for JSON events with JSON Pointer Support' + spec.description = 'Fluentd filter plugin for JSON events with JSON Pointer Support to pinpoint elements.' spec.homepage = 'https://github.com/iamAzeem/fluent-plugin-json' spec.license = 'Apache-2.0' @@ -23,8 +23,9 @@ Gem::Specification.new do |spec| spec.require_paths = ['lib'] spec.add_development_dependency 'bundler', '~> 1.14' - spec.add_development_dependency 'hana', '~> 1.3.6' + spec.add_development_dependency 'hana', '~> 1.3', '>= 1.3.6' spec.add_development_dependency 'rake', '~> 12.0' spec.add_development_dependency 'test-unit', '~> 3.0' spec.add_runtime_dependency 'fluentd', ['>= 0.14.10', '< 2'] + spec.add_runtime_dependency 'hana', '~> 1.3', '>= 1.3.6' end diff --git a/lib/fluent/plugin/filter_json.rb b/lib/fluent/plugin/filter_json.rb index a9d8554..78ff9ab 100644 --- a/lib/fluent/plugin/filter_json.rb +++ b/lib/fluent/plugin/filter_json.rb @@ -1,7 +1,7 @@ # frozen_string_literal: true # -# Copyright 2020- Azeem Sajid +# Copyright 2020 Azeem Sajid # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -16,6 +16,7 @@ # limitations under the License. require 'fluent/plugin/filter' +require 'hana' module Fluent module Plugin @@ -23,7 +24,36 @@ module Plugin class JsonFilter < Fluent::Plugin::Filter Fluent::Plugin.register_filter('json', self) - def filter(tag, time, record); end + desc 'The sub-section to specify one check.' + config_section :check, required: true, multi: true do + desc 'The JSON pointer to an element.' + config_param :pointer, :string + desc 'The regular expression to match the element.' + config_param :pattern, :regexp + end + + def configure(conf) + super + + @check.each do |chk| + begin + Hana::Pointer.parse(chk.pointer) + rescue Hana::Pointer::FormatError => e + raise Fluent::ConfigError, e + end + end + end + + def filter(_tag, _time, record) + @check.each do |chk| + pointer = Hana::Pointer.new(chk.pointer) + pointee = pointer.eval(record).to_s + matched = chk.pattern.match(pointee).nil? ? false : true + log.debug("check: #{matched ? 'pass' : 'fail'} [#{chk.pointer} -> '#{pointee}'] (/#{chk.pattern.source}/)") + return nil unless matched + end + record + end end end end