Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Display elapsed time in datagen log #4469

Merged
merged 2 commits into from
Dec 21, 2023
Merged

Conversation

sffc
Copy link
Member

@sffc sffc commented Dec 19, 2023

Makes the output more useful/actionable.

cargo make testdata:

INFO  [icu_datagen::driver] Datagen configured with fallback mode Hybrid and these locales: ["ar", "ar-EG", "bn", "ccp", "en", "en-001", "en-ZA", "es", "es-AR", "fil", "fr", "ja", "ru", "sr", "sr-Latn", "th", "tr", "und"]
INFO  [icu_datagen::driver] Generated key props/LOE@1
INFO  [icu_datagen::driver] Generated key props/EComp@1
INFO  [icu_datagen::driver] Generated key props/VS@1
INFO  [icu_datagen::driver] Generated key props/Basic_Emoji@1
INFO  [icu_datagen::driver] Generated key locid_transform/script_dir@1
INFO  [icu_datagen::driver] Generated key calendar/japanese@1
INFO  [icu_datagen::driver] Generated key calendar/japanext@1
INFO  [icu_datagen::driver] Generated key props/nfkcinert@1
INFO  [icu_datagen::driver] Generated key propnames/from/ccc@1
INFO  [icu_datagen::driver] Generated key props/Radical@1
INFO  [icu_datagen::driver] Generated key segmenter/lstm/wl_auto@1
INFO  [icu_datagen::driver] Generated key time_zone/bcp47_to_iana@1
INFO  [icu_datagen::driver] Generated key props/Join_C@1
INFO  [icu_datagen::driver] Generated key props/Dia@1
INFO  [icu_datagen::driver] Generated key units/info@1
INFO  [icu_datagen::driver] Generated key props/Upper@1
INFO  [icu_datagen::driver] Generated key props/WSpace@1
INFO  [icu_datagen::driver] Generated key propnames/from/GCB@1
INFO  [icu_datagen::driver] Generated key plurals/cardinal@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/WB@1
INFO  [icu_datagen::driver] Generated key collator/reord@1
INFO  [icu_datagen::driver] Generated key normalizer/nfd@1
INFO  [icu_datagen::driver] Generated key props/WB@1
INFO  [icu_datagen::driver] Generated key list/or@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/minute@1
INFO  [icu_datagen::driver] Generated key relativetime/short/second@1
INFO  [icu_datagen::driver] Generated key datetime/islamic/datelengths@1
INFO  [icu_datagen::driver] Generated key datetime/coptic/datesymbols@1
INFO  [icu_datagen::driver] Generated key datetime/persian/datelengths@1
INFO  [icu_datagen::driver] Generated key normalizer/decomp@1
INFO  [icu_datagen::driver] Generated key props/XIDC@1
INFO  [icu_datagen::driver] Generated key props/ExtPict@1
INFO  [icu_datagen::driver] Generated key datetime/timelengths@1
INFO  [icu_datagen::driver] Generated key props/EBase@1
INFO  [icu_datagen::driver] Generated key props/alnum@1
INFO  [icu_datagen::driver] Generated key propnames/from/SB@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/ea@1
INFO  [icu_datagen::driver] Generated key props/SB@1
INFO  [icu_datagen::driver] Generated key propnames/from/gcm@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/persian/date@1
INFO  [icu_datagen::driver] Generated key fallback/parents@1
INFO  [icu_datagen::driver] Generated key props/exemplarchars/punctuation@1
INFO  [icu_datagen::driver] Generated key relativetime/long/quarter@1
INFO  [icu_datagen::driver] Generated key collator/jamo@1
INFO  [icu_datagen::driver] Generated key propnames/from/lb@1
INFO  [icu_datagen::driver] Generated key propnames/from/ea@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/hebrew/years@1
INFO  [icu_datagen::driver] Generated key datetime/indian/datesymbols@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/islamic/date@1
INFO  [icu_datagen::driver] Generated key datetime/hebrew/datelengths@1
INFO  [icu_datagen::driver] Generated key props/ccc@1
WARN  [icu_datagen::transform::cldr::characters] Data missing for index set exemplar characters
INFO  [icu_datagen::driver] Generated key datetime/symbols/coptic/years@1
INFO  [icu_datagen::driver] Generated key props/Hyphen@1
INFO  [icu_datagen::driver] Generated key props/exemplarchars/index@1
INFO  [icu_datagen::driver] Generated key props/RI@1
INFO  [icu_datagen::driver] Generated key props/Ideo@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/hebrew/date@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/ethiopic/date@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/InSC@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/indian/years@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/coptic/months@1
INFO  [icu_datagen::driver] Generated key props/Hex@1
INFO  [icu_datagen::driver] Generated key props/blank@1
INFO  [icu_datagen::driver] Generated key displaynames/scripts@1
INFO  [icu_datagen::driver] Generated key relativetime/short/month@1
INFO  [icu_datagen::driver] Generated key relativetime/short/hour@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/chinese/years@1
INFO  [icu_datagen::driver] Generated key plurals/ranges@1
INFO  [icu_datagen::driver] Generated key props/Comp_Ex@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/lb@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/japanext/date@1
INFO  [icu_datagen::driver] Generated key props/exemplarchars/numbers@1
INFO  [icu_datagen::driver] Generated key props/SD@1
INFO  [icu_datagen::driver] Generated key props/graph@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/gregory/months@1
INFO  [icu_datagen::driver] Generated key fallback/likelysubtags@1
INFO  [icu_datagen::driver] Generated key normalizer/comp@1
INFO  [icu_datagen::driver] Generated key props/IDS@1
INFO  [icu_datagen::driver] Generated key props/nfdinert@1
INFO  [icu_datagen::driver] Generated key props/IDST@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/weekdays@1
INFO  [icu_datagen::driver] Generated key relativetime/long/day@1
INFO  [icu_datagen::driver] Generated key props/CWCM@1
INFO  [icu_datagen::driver] Generated key time_zone/specific_short@1
INFO  [icu_datagen::driver] Generated key props/Bidi_C@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/hour@1
INFO  [icu_datagen::driver] Generated key normalizer/nfkd@1
INFO  [icu_datagen::driver] Generated key props/scx@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/second@1
INFO  [icu_datagen::driver] Generated key fallback/supplement/co@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/japanese/years@1
INFO  [icu_datagen::driver] Generated key compactdecimal/short@1
INFO  [icu_datagen::driver] Generated key datetime/chinese/datelengths@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/indian/months@1
INFO  [icu_datagen::driver] Generated key time_zone/formats@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/sparse/ccc@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/roc/years@1
INFO  [icu_datagen::driver] Generated key datetime/buddhist/datesymbols@1
INFO  [icu_datagen::driver] Generated key relativetime/long/minute@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/hebrew/months@1
INFO  [icu_datagen::driver] Generated key relativetime/long/second@1
INFO  [icu_datagen::driver] Generated key locid_transform/likelysubtags_sr@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/month@1
INFO  [icu_datagen::driver] Generated key datetime/timesymbols@1
INFO  [icu_datagen::driver] Generated key props/CI@1
INFO  [icu_datagen::driver] Generated key displaynames/locales@1
INFO  [icu_datagen::driver] Generated key props/CWKCF@1
INFO  [icu_datagen::driver] Generated key props/IDC@1
INFO  [icu_datagen::driver] Generated key relativetime/long/month@1
INFO  [icu_datagen::driver] Generated key props/Ext@1
INFO  [icu_datagen::driver] Generated key datetime/japanext/datelengths@1
INFO  [icu_datagen::driver] Generated key datetime/dangi/datelengths@1
INFO  [icu_datagen::driver] Generated key datetime/roc/datelengths@1
INFO  [icu_datagen::driver] Generated key props/Gr_Base@1
INFO  [icu_datagen::driver] Generated key props/CWL@1
INFO  [icu_datagen::driver] Generated key locid_transform/likelysubtags@1
INFO  [icu_datagen::driver] Generated key props/ea@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/buddhist/date@1
INFO  [icu_datagen::driver] Generated key props/CWU@1
INFO  [icu_datagen::driver] Generated key datetime/hebrew/datesymbols@1
INFO  [icu_datagen::driver] Generated key props/segstart@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/gregory/years@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/ea@1
INFO  [icu_datagen::driver] Generated key relativetime/short/week@1
INFO  [icu_datagen::driver] Generated key relativetime/short/day@1
INFO  [icu_datagen::driver] Generated key props/Dep@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/GCB@1
INFO  [icu_datagen::driver] Generated key props/AHex@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/quarter@1
INFO  [icu_datagen::driver] Generated key collator/dia@1
INFO  [icu_datagen::driver] Generated key props/GCB@1
INFO  [icu_datagen::driver] Generated key props/exemplarchars/auxiliary@1
INFO  [icu_datagen::driver] Generated key props/Term@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/japanext/months@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear4/sc@1
INFO  [icu_datagen::driver] Generated key datetime/indian/datelengths@1
INFO  [icu_datagen::driver] Generated key props/CWT@1
INFO  [icu_datagen::driver] Generated key props/Gr_Link@1
INFO  [icu_datagen::driver] Generated key props/casemap_unfold@1
INFO  [icu_datagen::driver] Generated key time_zone/exemplar_cities@1
INFO  [icu_datagen::driver] Generated key datetime/persian/datesymbols@1
INFO  [icu_datagen::driver] Generated key props/InSC@1
INFO  [icu_datagen::driver] Generated key props/Pat_WS@1
INFO  [icu_datagen::driver] Generated key compactdecimal/long@1
INFO  [icu_datagen::driver] Generated key datetime/buddhist/datelengths@1
INFO  [icu_datagen::driver] Generated key normalizer/uts46d@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/persian/years@1
INFO  [icu_datagen::driver] Generated key locid_transform/aliases@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/ethiopic/years@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/week@1
INFO  [icu_datagen::driver] Generated key datetime/ethiopic/datesymbols@1
INFO  [icu_datagen::driver] Generated key datetime/chinese/datesymbols@1
INFO  [icu_datagen::driver] Generated key segmenter/dictionary/w_auto@1
INFO  [icu_datagen::driver] Generated key props/CWCF@1
INFO  [icu_datagen::driver] Generated key props/Gr_Ext@1
INFO  [icu_datagen::driver] Generated key time_zone/metazone_period@1
INFO  [icu_datagen::driver] Generated key normalizer/nfdex@1
INFO  [icu_datagen::driver] Generated key propnames/from/sc@1
INFO  [icu_datagen::driver] Generated key locid_transform/likelysubtags_ext@1
INFO  [icu_datagen::driver] Generated key props/NChar@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/islamic/years@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/chinese/months@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/sc@1
INFO  [icu_datagen::driver] Generated key propnames/from/InSC@1
INFO  [icu_datagen::driver] Generated key datetime/roc/datesymbols@1
INFO  [icu_datagen::driver] Generated key props/exemplarchars/main@1
INFO  [icu_datagen::driver] Generated key props/Lower@1
INFO  [icu_datagen::driver] Generated key props/XIDS@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/dangi/years@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/chinese/date@1
INFO  [icu_datagen::driver] Generated key props/lb@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/gc@1
INFO  [icu_datagen::driver] Generated key relativetime/long/year@1
INFO  [icu_datagen::driver] Generated key relativetime/short/minute@1
INFO  [icu_datagen::driver] Generated key relativetime/short/year@1
INFO  [icu_datagen::driver] Generated key time_zone/generic_long@1
INFO  [icu_datagen::driver] Generated key locid_transform/likelysubtags_l@1
INFO  [icu_datagen::driver] Generated key datetime/japanese/datesymbols@1
INFO  [icu_datagen::driver] Generated key props/gc@1
INFO  [icu_datagen::driver] Generated key datetime/ethiopic/datelengths@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/roc/months@1
INFO  [icu_datagen::driver] Generated key datetime/japanese/datelengths@1
INFO  [icu_datagen::driver] Generated key relativetime/long/hour@1
INFO  [icu_datagen::driver] Generated key props/Alpha@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/roc/date@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/buddhist/months@1
INFO  [icu_datagen::driver] Generated key datetime/islamic/datesymbols@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/japanese/date@1
INFO  [icu_datagen::driver] Generated key collator/meta@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/InSC@1
INFO  [icu_datagen::driver] Generated key plurals/ordinal@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/SB@1
INFO  [icu_datagen::driver] Generated key relativetime/long/week@1
INFO  [icu_datagen::driver] Generated key datetime/gregory/datelengths@1
INFO  [icu_datagen::driver] Generated key props/Dash@1
INFO  [icu_datagen::driver] Generated key props/Math@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/day@1
INFO  [icu_datagen::driver] Generated key props/UIdeo@1
INFO  [icu_datagen::driver] Generated key props/DI@1
INFO  [icu_datagen::driver] Generated key decimal/symbols@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/indian/date@1
INFO  [icu_datagen::driver] Generated key propnames/from/gc@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/GCB@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/japanese/months@1
INFO  [icu_datagen::driver] Generated key propnames/from/bc@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/SB@1
INFO  [icu_datagen::driver] Generated key props/Bidi_M@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/gregory/date@1
INFO  [icu_datagen::driver] Generated key datetime/coptic/datelengths@1
INFO  [icu_datagen::driver] Generated key props/Pat_Syn@1
INFO  [icu_datagen::driver] Generated key displaynames/variants@1
INFO  [icu_datagen::driver] Generated key props/print@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/bc@1
INFO  [icu_datagen::driver] Generated key propnames/from/WB@1
INFO  [icu_datagen::driver] Generated key props/QMark@1
INFO  [icu_datagen::driver] Generated key datetime/gregory/datesymbols@1
INFO  [icu_datagen::driver] Generated key props/xdigit@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/gc@1
INFO  [icu_datagen::driver] Generated key datetime/dangi/datesymbols@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/dangi/months@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/lb@1
INFO  [icu_datagen::driver] Generated key props/sc@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/dangi/date@1
INFO  [icu_datagen::driver] Generated key props/nfcinert@1
INFO  [icu_datagen::driver] Generated key propnames/to/long/linear/bc@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/sparse/ccc@1
INFO  [icu_datagen::driver] Generated key relativetime/short/quarter@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/dayperiods@1
INFO  [icu_datagen::driver] Generated key time_zone/generic_short@1
INFO  [icu_datagen::driver] Generated key normalizer/nfkdex@1
INFO  [icu_datagen::driver] Generated key relativetime/narrow/year@1
INFO  [icu_datagen::driver] Generated key props/Cased@1
INFO  [icu_datagen::driver] Generated key datetime/week_data@1
INFO  [icu_datagen::driver] Generated key time_zone/iana_to_bcp47@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/buddhist/years@1
INFO  [icu_datagen::driver] Generated key props/Sensitive@1
INFO  [icu_datagen::driver] Generated key props/STerm@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/time@1
INFO  [icu_datagen::driver] Generated key props/Emoji@1
INFO  [icu_datagen::driver] Generated key datetime/japanext/datesymbols@1 (0.612s, 'en-ZA' in 39.286ms, flushed in 10.921µs)
INFO  [icu_datagen::driver] Generated key datetime/symbols/ethiopic/months@1
INFO  [icu_datagen::driver] Generated key props/PCM@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/islamic/months@1
INFO  [icu_datagen::driver] Generated key props/nfkdinert@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/coptic/date@1
INFO  [icu_datagen::driver] Generated key props/IDSB@1
INFO  [icu_datagen::driver] Generated key segmenter/dictionary/wl_ext@1 (0.692s, 'th' in 0.691s, flushed in 12.834µs)
INFO  [icu_datagen::driver] Generated key displaynames/regions@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/persian/months@1
INFO  [icu_datagen::driver] Generated key datetime/patterns/datetime@1
INFO  [icu_datagen::driver] Generated key props/EPres@1
INFO  [icu_datagen::driver] Generated key list/and@1
INFO  [icu_datagen::driver] Generated key props/bc@1
INFO  [icu_datagen::driver] Generated key props/EMod@1
INFO  [icu_datagen::driver] Generated key propnames/to/short/linear/WB@1
INFO  [icu_datagen::driver] Generated key collator/prim@1
INFO  [icu_datagen::driver] Generated key datetime/symbols/japanext/years@1
INFO  [icu_datagen::driver] Generated key list/unit@1
INFO  [icu_datagen::driver] Generated key displaynames/languages@1
INFO  [icu_datagen::driver] Generated key props/casemap@1 (0.808s, flushed in 0.409s)
INFO  [icu_datagen::driver] Generated key collator/data@1 (0.657s, 'en' in 0.301s, flushed in 20.809µs)
INFO  [icu_datagen::driver] Generated key transliterator/rules@1 (0.766s, 'ccp' in 0.323s, flushed in 19.918µs)
INFO  [icu_datagen::driver] Generated key time_zone/specific_long@1 (1.941s, 'th' in 25.644ms, flushed in 29.987µs)
INFO  [icu_datagen::driver] Generated key datetime/skeletons@1 (3.161s, 'ja' in 19.241ms, flushed in 32.411µs)
INFO  [icu_datagen::driver] Generated key currency/essentials@1 (3.362s, 'en-ZA' in 3.023s, flushed in 32.952µs)
INFO  [icu_datagen::driver] Generated key segmenter/grapheme@1 (3.554s, flushed in 3.919ms)
INFO  [icu_datagen::driver] Generated key props/bidiauxiliaryprops@1 (3.941s, flushed in 5.467ms)
INFO  [icu_datagen::driver] Generated key segmenter/word@1 (4.475s, flushed in 6.212ms)
INFO  [icu_datagen::driver] Generated key segmenter/sentence@1 (2.845s, flushed in 6.206ms)
INFO  [icu_datagen::driver] Generated key segmenter/line@1 (9.445s, flushed in 10.847ms)
test tests::make_testdata::generate_json_and_verify_postcard ... ok

.into_par_iter()
.map(|locale| {
let instant2 = Instant::now();
load_with_fallback(key, &locale)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do double-check that the old and new behaviors are the same. I switched things around to avoid duplicating the duration logic.

Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature lgtm, small issue around code readability

let instant2 = Instant::now();
load_with_fallback(key, &locale)
.transpose()?
.map(|payload| sink.put_payload(key, &locale, &payload))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'm not convinced the functional style is clearer here, especially with the timing code, where you kinda actually do need to know where it should run. I think we should revert to the older imperative style here and just add timing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I agree but in this case I was confused why we had an expression like

if let Some(payload) = load_with_fallback(key, &locale) {
    sink.put_payload(key, &locale, &payload?)
} else {
    Ok(())
}
// followed by other chaining logic

Also note that the majority of the diff in this branch is due to changing try_for_each to map so that I can call .max() on the result.

I will do another round of making this more clear though. I realized that I might be losing the error context with the new .transpose()?

Copy link
Member

@Manishearth Manishearth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a bit better. still not convinced that having the .elapsed() in the highly functional code is a good idea

@robertbastian
Copy link
Member

Is this changelog worthy?

@Manishearth
Copy link
Member

shrug

when I make a changelog manually I would include this. Otherwise ... eh

@sffc sffc merged commit 5e816b3 into unicode-org:main Dec 21, 2023
29 checks passed
@sffc sffc deleted the datagen-output branch December 21, 2023 19:00
@sffc
Copy link
Member Author

sffc commented Dec 21, 2023

Changelog in #4484

sffc added a commit that referenced this pull request Dec 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants