Number formatting for many languages and sorting (collation) with ICU

[Omikhleia]

Rationale

• SILE should be able to correctly format numbers in appropriate number system and with appropriate format style for as many languages as possible, and not just a few selected ("en", "eo", "tr")
  • in "default" form (en: 1234 or ar: ١٢٣٤ or zh: 1234)
  • in "decimal" form, i.e. usually with thousands separator etc. (1,234 or ١٬٢٣٤ or 1,234)
  • in "ordinal" form, i.e. the "nth's" (1,234th or ١٬٢٣٤. or 第1,234)
  • as an extra in spelt-out form (one thousand two hundred etc., or some stuff I can't copy, or 一千二百三十四)
  • EDIT: (N.B.) I took the most neutral names I could think of, for these... What typographic rules say, for a given language, may be different; e.g. some French typographers distinguish a cardinal use (= "decimal" with separator, in that case a space) and an ordinal use (as in a year date, a page number, etc. = "default" without separator). Terminology is sometimes ambiguous, it seems (because of course, there are also the actual ordinals: with spaces at thousands... and the raised "e" &c.)
• SILE should be able to correctly sort strings with appropriate rules according to the (main) language usages: indexer and packages of that vein may need it or their sorting entries will be garbage.

First part (number formatting) = Closes #1630

Notes

• Deprecations are currently set to 0.14.6 for removal in 0.16.0
• For counters, it is slightly breaking though - if we consider that what was in tests/counters.sil is a "reference" of sorts, though fully undocumented. I would tend to think it is acceptable (for the main intended use of counters, i.e. sectioning and lists). It's unlikely to affect many users anyway; and the benefits are worth it: We get "default" (cardinal), "decimal" (with thousands separator, etc.), "ordinal" (our former "nth"), as well as "string" (spelt-out) support, for most languages (i.e. as long as supported by ICU).

En français, on écrit « en l'an 1984 », mais « dans 4 500 ans ».
Probably English and others have similar rules, when to use 1984 or 1,984

N.B. I haven't considered all format style options from ICU (e.g. "duration", "currency", etc.) because I felt they were unlikely to be needed. That could be added, if someone really wants - But I'm not convinced it would have a real use in SILE (and in my tests at least, the ICU library is somewhat inconsistent in how it honors these).

Second part (string sorting) = Add language-dependent sorting with ICU collation options

As noted, an imperative for indexes (= relates to #1339), etc.

> SILE.call("language", { main = "fr" })
> t = { "Jean100", "Jean2", "Alinoé", "Alinéa" }
> table.sort(t)
> print(table.concat(t, ", "))

--> Alinoé, Alinéa, Jean100, Jean2 = bad indexing order

> t = { "Jean100", "Jean2", "Alinoé", "Alinéa" }
> SU.collatedSort(t)
> print(table.concat(t, ", "))

--> Alinéa, Alinoé, Jean2, Jean100 = good indexing order, yay!

Supports options passed to SU.collatedSort, or also defined in SU.collatedSort.xx for language-specific override of the default values if need be.

[alerque]

Cardinal vs. Ordinal is also very applicable in Turkish, and to a lesser degree English. It's probably a distinction we should make and supply both for rather than assuming one or the other given the most common use case.

Otherwise 💯 to the analysis here.

[alerque]

I'm looking into dropping the bits of our implementations that are now duplicated. Somewhat amusingly our "string" (spellout) implementation in English handles bigger numbers than the ICU one which starts failing at 32 bits. It leaves off at quadrillions, ours goes on to handle quintillion, sextillion, septillion, and octillion!

Similarly for the Turkish implementation, ours is more complete.

The English apostrophe thing is probably wrong according to most style guides though.

[Omikhleia]

(our implementation) is more complete.

Yep, or even better (with tunable settings, in the case of Esperanto). I'm ok with SILE having a mechanism for bypassing ICU when it can do better, I didn't intend to have them necessarily replace. The main point is to address other languages generically with decent defaults.¹

Footnotes

1. That what we already do e.g. for node breakers, defaulting to ICU breakpoints but possibly overriding the logic as e.g. in SILE.nodeMakers.fr, SILE.nodeMakers.ja, etc. (to handle resp. punctuations, some JIS class things, etc.). ↩