Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csv processor can't parse empty_value option #55643

Closed
martijnvg opened this issue Apr 23, 2020 · 1 comment · Fixed by #55649
Closed

csv processor can't parse empty_value option #55643

martijnvg opened this issue Apr 23, 2020 · 1 comment · Fixed by #55649
Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP

Comments

@martijnvg
Copy link
Member

martijnvg commented Apr 23, 2020

The following put pipeline call fails:

PUT /_ingest/pipeline/test
{
    "description": "My test pipeline",
    "processors": [
        {
            "csv": {
                "field": "name",
                "target_fields": [
                    "targetField1",
                    "targetField2"
                ],
                "trim": true,
                "empty_value": {
                    "x": "y"
                }
            }
        }
    ]
}

With:

{
    "error": {
        "root_cause": [
            {
                "type": "parse_exception",
                "reason": "processor [csv] doesn't support one or more provided configuration parameters [empty_value]",
                "processor_type": "csv"
            }
        ],
        "type": "parse_exception",
        "reason": "processor [csv] doesn't support one or more provided configuration parameters [empty_value]",
        "processor_type": "csv"
    },
    "status": 400
}

However empty_value is a valid option.

This is caused by a bug in the processor parsing code:
https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/CsvProcessor.java#L104

It checks for the existence of emptyValue field and then tries to parse it as empty_value field. So even if this option is specified in both ways it can never be parsed.

This bug was found in elastic/elasticsearch-net#4652

@martijnvg martijnvg added >bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Apr 23, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@martijnvg martijnvg changed the title csv processor can't part empty_value option csv processor can't parse empty_value option Apr 23, 2020
russcam added a commit to elastic/elasticsearch-net that referenced this issue Apr 23, 2020
Relates: elastic/elasticsearch#55643

Mute CsvProcessor tests, until upstream bug is fixed.
probakowski added a commit to probakowski/elasticsearch that referenced this issue Apr 23, 2020
Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes elastic#55643
Mpdreamz pushed a commit to elastic/elasticsearch-net that referenced this issue Apr 23, 2020
Relates: elastic/elasticsearch#55643

Mute CsvProcessor tests, until upstream bug is fixed.
Mpdreamz pushed a commit to elastic/elasticsearch-net that referenced this issue Apr 23, 2020
Relates: elastic/elasticsearch#55643

Mute CsvProcessor tests, until upstream bug is fixed.

(cherry picked from commit 2214855)
Mpdreamz pushed a commit to elastic/elasticsearch-net that referenced this issue Apr 23, 2020
Relates: elastic/elasticsearch#55643

Mute CsvProcessor tests, until upstream bug is fixed.

(cherry picked from commit 2214855)
(cherry picked from commit 4d67fd9)
probakowski added a commit that referenced this issue Apr 29, 2020
* Fix empty_value handling in CsvProcessor

Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes #55643

* assert changed
probakowski added a commit to probakowski/elasticsearch that referenced this issue Apr 29, 2020
* Fix empty_value handling in CsvProcessor

Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes elastic#55643

* assert changed
probakowski added a commit to probakowski/elasticsearch that referenced this issue Apr 29, 2020
* Fix empty_value handling in CsvProcessor

Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes elastic#55643

* assert changed
probakowski added a commit that referenced this issue Apr 29, 2020
* Fix empty_value handling in CsvProcessor

Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes #55643

* assert changed
probakowski added a commit that referenced this issue Apr 29, 2020
* Fix empty_value handling in CsvProcessor

Due to bug in `CsvProcessor.Factory` it was impossible to specify `empty_value`.
This change fixes that and adds relevant test.

Closes #55643

* assert changed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants