Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiple update of one document in one _bulk too slow #24172

Closed
dizzzyroma opened this issue Apr 19, 2017 · 3 comments
Closed

multiple update of one document in one _bulk too slow #24172

dizzzyroma opened this issue Apr 19, 2017 · 3 comments

Comments

@dizzzyroma
Copy link

Hi,

Multiple update of one document in one _bulk is too slow

Example:

curl -XPOST "http://localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":1}}
{"update":{"_index":"cc_36","_type":"job","_id":"960_87786b7c-69df-4d3d-b90b-b9b26f08e2a1"}}
{"doc":{"status":2}}
'

Elasticsearch version: 5.3, 5.2

Plugins installed: []

JVM version: 1.8.0_121-b13

OS version:Ubuntu 16.04.2 LTS

bulk request with 75 updates takes near 6 seconds on our test index that runs on c3.xlarge (4 CPU, 8 GB, documents in inex 14860619, index size 20GB) .
bulk request with 150 updates takes near 12 seconds.

bulk index with 75 documents takes 0.2 seconds

On ES 2.4 - we haven't this problem

Profiler info:
image

image

If it helps, I may open profiling port to cluster where this problem is reproduced
@jimczi
Copy link
Contributor

jimczi commented Apr 19, 2017

In 5.x the global ordinals are built during the refresh:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-parent-field.html#_global_ordinals
This is a trade off to make sure that the first query with parent/child on the new searcher can run fast. You can disable this behavior as explained in the documentation or you can disable refresh when you're bulk updating your index.
Anyhow this discussion is better suited for the forum. I am going to close this issue because we reserve github for verified bugs and new features. If you have any questions we can continue the discussion in the forum:
https://discuss.elastic.co/c/elasticsearch

@jimczi jimczi closed this as completed Apr 19, 2017
@jimczi
Copy link
Contributor

jimczi commented Apr 19, 2017

My previous comment is not entirely true. The refresh is triggered by the updates itself so disabling the refresh would not solve your problem. In 5.x we rely on refresh to update a document that is not flushed yet:
#20102
This change coupled with the fact that we eagerly load the parent/child ordinals on refresh makes your ingestion rate slow. I'd suggest to revisit your ingestion method by maybe grouping updates of a single document in one when you can ?

@dizzzyroma
Copy link
Author

Thanks for answer I create thread for this https://discuss.elastic.co/t/multiple-update-of-one-document-in-one--bulk-too-slow/82885
I research it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dizzzyroma @jimczi