Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add postgresql csv fileset #23334

Merged
merged 31 commits into from
Feb 17, 2021
Merged

[Filebeat] Add postgresql csv fileset #23334

merged 31 commits into from
Feb 17, 2021

Commits on Jan 4, 2021

  1. [Filebeat] Add postgresql csv fileset

    As a configuration option, create another pipeline inside the module to
    parse CSV log files.
    
    Although it's necessary to configure the database to emit such logs,
    there is some advantages: configured properly, some events like
    `statement timeout` and `lock timeout` will display the query in the
    same event, opposed to multiple lines, one with the error message and
    other with statement.
    azlev committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    684f912 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2021

  1. Configuration menu
    Copy the full SHA
    a6b2ea0 View commit details
    Browse the repository at this point in the history
  2. Update filebeat/module/postgresql/csv/_meta/fields.yml

    Do not generate duplicate field groups.
    
    Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
    azlev and jsoriano committed Jan 9, 2021
    Configuration menu
    Copy the full SHA
    cacde06 View commit details
    Browse the repository at this point in the history
  3. Running 'make update'

    azlev committed Jan 9, 2021
    Configuration menu
    Copy the full SHA
    7ec70d8 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2021

  1. filebeat(module/postgres): Fix CSV ingestion / tests

    Commit cacde06 changed field names.
    
    I commited in github without pay attention to tests. This commit fixes
    all tests and the pipeline.
    azlev committed Jan 16, 2021
    Configuration menu
    Copy the full SHA
    31334cc View commit details
    Browse the repository at this point in the history
  2. Refactor the PostgreSQL CSV Log pipeline

    * Adding postgresql.log.query_name in grok
    
    The query_step was ignored because it's a subset of command_tag
    
    * Separate message and query fields
    
    Message is the main message, and query is what message is referencing
    azlev committed Jan 16, 2021
    Configuration menu
    Copy the full SHA
    606173e View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2021

  1. Fix(filebeat/postgresql): Change core_id to session_line_number

    There is no `core_id` inside PostgreSQL logs. Deprecating field and moving
    it to `session_line_number`.
    azlev committed Jan 24, 2021
    Configuration menu
    Copy the full SHA
    d358b32 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2021

  1. [Filebeat] Refactor PostgreSQL log pipeline

    This commit will open the path to add 2 pipelines:
    
    - The default one, using log files
    - The CSV pipeline
    
    Since both files represent the same information, makes sense to have
    only one fileset.
    azlev committed Feb 13, 2021
    Configuration menu
    Copy the full SHA
    94de6c1 View commit details
    Browse the repository at this point in the history
  2. [Filebeat] PostgreSQL split grok logic

    Creates a new grok pattern to match only the timestamp and the next
    char, the separator.
    
    Based on separator, choose the right pipeline.
    azlev committed Feb 13, 2021
    Configuration menu
    Copy the full SHA
    56f2d54 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2021

  1. [Filebeat] Merge PostgreSQL's log and csv filesets

    Both of then represents the same information, so use the same pipeline
    to do the job.
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    fa308d6 View commit details
    Browse the repository at this point in the history
  2. [Filebeat] PostgreSQL module adjustments

    - remove unused `thread_id` field
    - issue a `make update`
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    7c90863 View commit details
    Browse the repository at this point in the history
  3. [Filebeat] PostgreSQL module: drop error_severity

    Drop `postgresql.log.error_severity` in favor of `log.level`
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    eee16be View commit details
    Browse the repository at this point in the history
  4. [Filebeat] PostgreSQL: append "error" in relevant events

    Based on PostgreSQL Error Codes, everything that not starts with `00`,
    `01` or `02` is an error.
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    17724a0 View commit details
    Browse the repository at this point in the history
  5. [Filebeat] re-generate test results

    Regenerate files to have a consistent field order.
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    c95556f View commit details
    Browse the repository at this point in the history
  6. [Filebeat] deprecate postgresql.log.error.code

    This field is a long, but postgresql errors can have letters. For now
    just the deprecation, a naive change in grok will break the parser.
    azlev committed Feb 14, 2021
    Configuration menu
    Copy the full SHA
    6558076 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2021

  1. Update filebeat/module/postgresql/log/_meta/fields.yml

    Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
    azlev and jsoriano committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    9eed55c View commit details
    Browse the repository at this point in the history
  2. Update filebeat/module/postgresql/log/manifest.yml

    Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
    azlev and jsoriano committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    f570e58 View commit details
    Browse the repository at this point in the history
  3. [Filebeat] change field name in postgresql module

    Use `raw_message` instead of `message2`.
    azlev committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    85ab1e3 View commit details
    Browse the repository at this point in the history
  4. [Filebeat] deprecate postgresql.log.error.code

    According to PostgreSQL docs, this code is not a number:
    
    https://www.postgresql.org/docs/current/errcodes-appendix.html
    
    Stop collecting it and move it `sql_state_code`
    azlev committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    b11a3d4 View commit details
    Browse the repository at this point in the history
  5. [Filebeat] move postgresql.log.error.code to postgresql.log.sql_state…

    …_code
    
    After deprecate the field, stop to populate it.
    azlev committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    7f353b6 View commit details
    Browse the repository at this point in the history
  6. [Filebeat] update postresql log fields

    Give more details to `details` and `hints` and change `error.code` to an
    alias.
    azlev committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    ad53939 View commit details
    Browse the repository at this point in the history
  7. [Filebeat] postgresql: do not overlap event.type

    The previous logic tag all messages as "info" and then append "error" if
    it's the case.
    
    Put a conditional to not have "event.type": ["info", "error"].
    azlev committed Feb 15, 2021
    Configuration menu
    Copy the full SHA
    978708d View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2021

  1. Configuration menu
    Copy the full SHA
    f131b46 View commit details
    Browse the repository at this point in the history
  2. Update filebeat/module/postgresql/log/_meta/fields.yml

    Change part of the description to a comment
    
    Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
    azlev and jsoriano committed Feb 16, 2021
    Configuration menu
    Copy the full SHA
    3671f00 View commit details
    Browse the repository at this point in the history
  3. [Filebeat] PostgreSQL module adjustments

    - Run update since last `fields.yml` was done only in github
    - Use `set` instead of `append` in pipeline to not create an array
    - Capture the `postgresql.log.query_step` in csv pipeline
    azlev committed Feb 16, 2021
    Configuration menu
    Copy the full SHA
    60f064f View commit details
    Browse the repository at this point in the history
  4. [Filebeat] Fix tester

    Now PostgreSQL has 3 files in the ingest directory:
    
    - pipeline.yml
    - pipeline-log.yml
    - pipeline-csv.yml
    
    Change the count to 3 to reflect that.
    azlev committed Feb 16, 2021
    Configuration menu
    Copy the full SHA
    1d23ce9 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2021

  1. [Filebeat] fix a typo in PostgreSQL module

    postresql.log.sql_state_code -> `postgresql.log.sql_state_code`.
    azlev committed Feb 17, 2021
    Configuration menu
    Copy the full SHA
    5fa37c8 View commit details
    Browse the repository at this point in the history
  2. Update filebeat/module/postgresql/log/ingest/pipeline-csv.yml

    Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
    azlev and jsoriano committed Feb 17, 2021
    Configuration menu
    Copy the full SHA
    991a227 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0bfbde6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    633a8cd View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a518be6 View commit details
    Browse the repository at this point in the history