tokenization issues: ' followed by s, m, t, etc

[AngledLuffa]

it's gets tokenized into three tokens, it, ', s

that should be fixed

same with 'm 't etc

[SecroLoL]

Are you saying that it should be tokenized as it, 's?

[AngledLuffa]

yes, those should be it, 's and i, 'm, etc

[AngledLuffa]

lmk if you need or want some assistance scripting changes like that

[SecroLoL]

I think I've got this, thanks! Will let you know if I need help though

[SecroLoL]

How about cases where a noun is followed by 's? Are these annotated properly?
Example:

The	O
opposition	O
's	O
poor	O
election	O
results	O

[SecroLoL]

Here's what I'm seeing when inspecting some processed data:

national	O
truth	O
-	O
telling	O
process	O
would	O
have	O
on	O
Australia	B-Location
,	O
it	O
's	O
remarkable	O
.	O

"	O
One	O
of	O
the	O
things	O
that	O
we	O
're	O
thinking	O
about	O

I	O
'm	O
a	O
non	O
-	O
conformist	O
politician	O
.	O
I	O
'm	O
a	O
revolutionary	O
,	O
'	O
'	O
Bouteflika	B-Person
told	O
The	B-Organization
Associated	I-Organization
Press	I-Organization

Can't find the cases you're talking about. Was that perhaps only for the raw annotated data?

[AngledLuffa]

the possessive 's and the contraction 'm are correct

when i was going through the data myself, i'd occasionally fix them when i came across such errors

cd processed_annotated
grep "^s  O$" * | less        # that's a tab character between s and O

af_afrol_16.txt.tsv:s   O
af_afrol_18.txt.tsv:s   O
af_allaf_15.txt.tsv:s   O
af_allaf_24.txt.tsv:s   O
af_allaf_24.txt.tsv:s   O
af_ips_10.txt.tsv:s     O
af_ips_10.txt.tsv:s     O
af_ips_10.txt.tsv:s     O
etc etc

[AngledLuffa]

i'm fairly certain most of those can be cleaned up via a script...

just look for s on a line by itself, especially after a ' or a curly apostrophe, check that the labels are the same, combine the rows

again, i can take that on ... maybe i should just go ahead and do that

[SecroLoL]

If you could, that would be great. If you have time, of course.

[AngledLuffa]

I'm about half done with checking incorrect ', but am uncovering a whole bunch of other random tokenization errors in the process.

’ (the fancy apostrophe) on a line by itself, followed by s, d, t, etc

Ms .

Jr .

' ' and backticks or curly apostrophes

. . . instead of as a single token

46-41 or other scores / votes

U . S .

and in one file, cuba_diariodecuba_5.txt.tsv, César got cut off many times. I suspect there will be other words like that which need to be cleaned up

[AngledLuffa]

alright, i have taken on the ’ tokenizations and the ' tokenizations

the others are still TODO

[AngledLuffa]

US, titles, and ellipses are now cleared up. Would still like to look for decade+s

[AngledLuffa]

did the decades as well

maybe still need to look for ' ' on two separate lines