Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use serde_json_borrow to parse JSON #5035

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

use serde_json_borrow to parse JSON #5035

wants to merge 2 commits into from

Conversation

PSeitz
Copy link
Contributor

@PSeitz PSeitz commented May 27, 2024

  • update tantivy
  • use serde_json_borrow instead of serde_json::Value
➜  quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
serde_json_borrow + CompactDoc
 Num docs   46475 Parse errs     0 PublSplits   0 Input size    56MB Thrghput 28.42MB/s Time 00:00:02
 Num docs   94511 Parse errs     0 PublSplits   0 Input size   115MB Thrghput 38.54MB/s Time 00:00:03
 Num docs  138769 Parse errs     0 PublSplits   0 Input size   170MB Thrghput 42.67MB/s Time 00:00:04
 Num docs  184161 Parse errs     0 PublSplits   0 Input size   225MB Thrghput 45.15MB/s Time 00:00:05
 Num docs  227811 Parse errs     0 PublSplits   0 Input size   280MB Thrghput 56.02MB/s Time 00:00:06
 Num docs  273217 Parse errs     0 PublSplits   0 Input size   337MB Thrghput 55.50MB/s Time 00:00:07
 Num docs  320748 Parse errs     0 PublSplits   0 Input size   395MB Thrghput 56.17MB/s Time 00:00:08
 Num docs  366026 Parse errs     0 PublSplits   0 Input size   451MB Thrghput 56.45MB/s Time 00:00:09
 Num docs  410292 Parse errs     0 PublSplits   0 Input size   506MB Thrghput 56.45MB/s Time 00:00:10
 Num docs  456037 Parse errs     0 PublSplits   0 Input size   562MB Thrghput 56.26MB/s Time 00:00:11
 Num docs  501879 Parse errs     0 PublSplits   0 Input size   619MB Thrghput 56.09MB/s Time 00:00:12
 Num docs  549379 Parse errs     0 PublSplits   0 Input size   677MB Thrghput 56.47MB/s Time 00:00:13
 Num docs  594425 Parse errs     0 PublSplits   0 Input size   732MB Thrghput 56.45MB/s Time 00:00:14
 Num docs  638732 Parse errs     0 PublSplits   0 Input size   788MB Thrghput 56.38MB/s Time 00:00:15
➜  quickwit-indices cat mezmo/mezmo-use-stage-2023-01-20-ndjson/lines.njson | quickwit tool local-ingest --index mezmo
MAIN_BRUNCH
 Num docs   39859 Parse errs     0 PublSplits   0 Input size    48MB Thrghput 48.31MB/s Time 00:00:01
 Num docs   39859 Parse errs     0 PublSplits   0 Input size    48MB Thrghput 24.17MB/s Time 00:00:02
 Num docs   77716 Parse errs     0 PublSplits   0 Input size    95MB Thrghput 31.78MB/s Time 00:00:03
 Num docs  117262 Parse errs     0 PublSplits   0 Input size   144MB Thrghput 36.13MB/s Time 00:00:04
 Num docs  156652 Parse errs     0 PublSplits   0 Input size   192MB Thrghput 36.12MB/s Time 00:00:05
 Num docs  197282 Parse errs     0 PublSplits   0 Input size   242MB Thrghput 48.50MB/s Time 00:00:06
 Num docs  237082 Parse errs     0 PublSplits   0 Input size   291MB Thrghput 49.13MB/s Time 00:00:07
 Num docs  276808 Parse errs     0 PublSplits   0 Input size   341MB Thrghput 49.24MB/s Time 00:00:08
 Num docs  314865 Parse errs     0 PublSplits   0 Input size   388MB Thrghput 48.82MB/s Time 00:00:09
 Num docs  353884 Parse errs     0 PublSplits   0 Input size   437MB Thrghput 48.72MB/s Time 00:00:10
 Num docs  393704 Parse errs     0 PublSplits   0 Input size   485MB Thrghput 48.51MB/s Time 00:00:11
 Num docs  432803 Parse errs     0 PublSplits   0 Input size   534MB Thrghput 48.19MB/s Time 00:00:12
 Num docs  473526 Parse errs     0 PublSplits   0 Input size   583MB Thrghput 48.94MB/s Time 00:00:13
 Num docs  512875 Parse errs     0 PublSplits   0 Input size   633MB Thrghput 49.05MB/s Time 00:00:14
 Num docs  553857 Parse errs     0 PublSplits   0 Input size   682MB Thrghput 49.26MB/s Time 00:00:15

@fulmicoton
Copy link
Contributor

MAIN_BRUNCH :)

@fulmicoton fulmicoton marked this pull request as draft May 29, 2024 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants