WIP: New dictionary format #233

mh-northlander · 2024-07-29T00:08:33Z

rebased #200.

wip todo:

fix errors
add tests
remove legacy code and organize
add way to convert from legacy (user) dictionary
- fix DictionaryPrinter #234 should solve this.
- add migration guide
- add script to convert legacy lexicon
add feature: make split-C work.
add feature: allow word reference with pos-id in the lexicon csv.

maybe:

~~Be able to lookup normalized/dictionary entries, not just their string forms #202~~
- new format does not allow easy access for the normalized/dictionary form
Separate POS list as a def file (loaded when build like matrix.def)

Not decided yet:

compatibility check of system/user dict
- we can check if the user dict is built on the system dict on load. we may raise error/warning.

super WIP

…ly, progress)

sonarcloud · 2024-09-03T09:26:19Z

Quality Gate passed

Issues
23 New issues
0 Accepted issues

Measures
0 Security Hotspots
85.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

eiennohito and others added 30 commits July 17, 2024 17:01

non-boxing variation of dictionary lookup

62183a0

prototype of dictionary redo

854410f

super WIP

working encode-decode for string pointers

ef9f821

layout strings with alignment awareness

1d25747

string layout now really works

0cab624

add documentation for WordLayout

3e991d8

binary dictionary wip

8e9b33a

reading raw csv dictionary

7750eb8

add computation of utf8 length without allocations

9692c58

fix line break in when reading POS

db3f5c4

writing new dictionary wip

0c8bb9b

example reformat

1513027

don't modify positions with negative lengths

628b243

first part of writing dictionary header

a06c071

write dictionary header

fda2d48

clean up BufReaderTest

3c4754a

dictionary new format wip

75c2b63

spotless apply

31496e2

better build progress

e376920

load grammar

86306ba

refactor: rename etc.

e3417cd

add constants for magic numbers

c549bc3

add docstrings

93bca39

fix wordinfo parse (word structure ~)

42c620c

fix lexicon wordref parsing

e1cdc0c

iterate over wordIds in the compiled dict

9fa84a4

build user dict (load system dict, resolve wordref, save user data on…

aaf5c37

…ly, progress)

allow missing connection matrix (user dict)

edad56f

split align position method

b24e37b

print dictionary description

c63784d

mh-northlander added 2 commits August 2, 2024 11:06

enable dict builder to load pos from file

1f3f350

let lexcon csv have either (or both) pos-id or pos-parts column.

4c5b3f0

mh-northlander force-pushed the feature/newdic branch from fab4f08 to 4c5b3f0 Compare August 2, 2024 07:56

mh-northlander added 26 commits August 5, 2024 09:55

try to satisfy sonarcloud

e0b9262

allow pos file only once and for system, fix column existance check

763ffa7

default signature uses comment hash instead of random

d303362

reduce memory usage during StringStorage compile

c217631

rm duplicate test resources

37b2d87

rename classes

e9e704b

merge csvfieldexception into inputfileexception

0c474f6

fix sample code indent

a22c243

remove InMemorychannel

35b7cd4

fix lexicon pos column behaviour

2ebe2d7

add tests

a413a98

fix for tests

3f99a44

rewrite double array lexicon test

19d0178

add constant for wordref split char

5602a25

add header to test dict lexicons, add read lexicon test

1411fda

let split-c work and add test for that

f894d93

Add pos-id column for POS csv

b0afe06

allow using posid in the lexicon wordref, and improve dict-printer

73c4311

test contents of rebuilt dictionary

2f1680b

update test dictionary lexicon

463617e

add new user_dict.md

9b72b46

add pos.csv

9bbde23

add getUserData for morpheme

91ddc90

test empty pos id/parts column

c9e98db

copy entry data for phantom entry

6698ed9

try to satisfy sonarcloud

8f1007a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: New dictionary format #233

WIP: New dictionary format #233

mh-northlander commented Jul 29, 2024 •

edited

Loading

sonarcloud bot commented Sep 3, 2024

WIP: New dictionary format #233

Are you sure you want to change the base?

WIP: New dictionary format #233

Conversation

mh-northlander commented Jul 29, 2024 • edited Loading

sonarcloud bot commented Sep 3, 2024

Quality Gate passed

mh-northlander commented Jul 29, 2024 •

edited

Loading