Locale not canonicalizing tlangs to lowercase

[jedel1043]

Locale canonicalizes tlangs (the "-t-en-latn" part inside the "en-t-en-latn" locale) incorrectly, preserving the canonicalized case instead of converting all to lowercase.

The RFC6497 from the IETF that describes the extension T (and tlangs) specifies on one of its paragraphs:

2.3.  Canonicalization

   As required by [BCP47], the use of uppercase or lowercase letters is
   not significant in the subtags used in this extension.  The canonical
   form for all subtags in the extension is lowercase, with the fields
   ordered by the separators, alphabetically.  The order of subtags
   within a field is significant, and MUST NOT be changed in the process
   of canonicalizing.

Emphasis on

The canonical form for all subtags in the extension is lowercase ...

Minimized snippet with the bug

let test: Locale = "en-t-en-latn-ca-emodeng".parse().unwrap();
println!("{test}"); // prints "en-t-en-Latn-CA-emodeng" instead of "en-t-en-latn-ca-emodeng"

Haven't checked if it would be necessary to fix only Locale::from_bytes or also Locale::canonicalize and LocaleCanonicalizer::canonicalize.

[sffc]

Good catch.

CC @zbraniecki @dminor

[zbraniecki]

Damn, that's unfortunate. There has been a lot of good honest work put into optimizing all the codepaths to canonicalize to one form.

We can either rewrite quite a bit of helper code to handle "normal" vs "lowercase" scenarios or bit the bullet and allocate tlang, lowercase it, and move on.

I'm leaning toward the latter as much as it pains me.

[sffc]

It's a nice property that I can get a real LanguageIdentifier out of the tlang field. If we make tlang a separate tree, then it should at least have helper functions to perform the conversion.

[zbraniecki]

Let's wait for #1932 before we look into this. I think we may either want to introduce a concept of write_to_lowercase to Region and Script.