Support lots of ligatures #53
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jules-Bertholet
force-pushed
the
ligatures
branch
3 times, most recently
from
714ddc5
to
19dc63f
Compare
Closed
Manishearth
approved these changes
Jun 6, 2024
| class EffectiveWidth(enum.IntEnum): | ||
| """Represents the width of a Unicode character. All East Asian Width classes resolve into | ||
| either `EffectiveWidth.NARROW`, `EffectiveWidth.WIDE`, or `EffectiveWidth.AMBIGUOUS`. | ||
| def to_sorted_ranges(iter: Iterable[Codepoint]) -> list[tuple[Codepoint, Codepoint]]: |
There was a problem hiding this comment.
thought: given the amount of logic that is ending up in Python here I think it may make sense to switch to a rust script for this. unfortunately that will make CI much slower since a rust script would need HTTP deps
Perhaps we can instead put more work into documenting and cleaning up the python. you've been doing pretty well so far!
There was a problem hiding this comment.
a rust script would need HTTP deps
Or we could just shell out to curl…
There was a problem hiding this comment.
(let's not do that in this PR, though)
There was a problem hiding this comment.
Actually we could also just check in the fetched file and have a CI job ensure it's up to date.
…instead of meaning that it has ambiguous width. Instead, we use a (partly) separate table to handle CJK width.
and a bugfix caught by said tests
- Support nonspacing coeng signs - Assign width 2 to KHMER INDEPENDENT VOWEL QAA and 3 to KHMER SIGN BEYYAL (https://unicode.org/charts/nameslist/n_1780.html)
Jules-Bertholet
force-pushed
the
ligatures
branch
from
19dc63f
to
84455bb
Compare
Manishearth
approved these changes
Jun 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Arabic Lam-Alef ligature
Unicode:
In practice, the three monospace Arabic fonts that seem to be most widespread (Courier New, Simplified Arabic Fixed, and Kawkab Mono) all render this ligature with width 1.
Buginese <a, -i> ya ligature
Documented in Unicode.
Hebrew Alef-Lamed ligature
This one is not documented in the Unicode standard, but all the Hebrew monospace fonts I could find have it. And it requires an explicit ZWJ to get, so there is no real risk in including it.
Khmer coeng signs
Documented in Unicode.
Old Turkic ligature
Documented in Unicode.
Tifinagh bi-consonants
Documented in Unicode.
Emoji modifiers and ZWJ sequences
Documented in UTS 51.
U+17D8 KHMER SIGN BEYYAL
According to the charts, this character should be equivalent to U+17D4 U+179B U+17D4, so give it the same width as that sequence.
This PR also fixes a bug with canonical equivalence for the CJK width of certain strings containing
'\u{0338}'COMBINING LONG SOLIDUS OVERLAY. I can try to split that out if you'd prefer.