-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tokenizer: Fix line continuation after punctuation #12
base: master
Are you sure you want to change the base?
Conversation
I know that this is a somewhat ugly hack because it goes back 3 positions. Feel free to reject or improve if you think this is not good. |
EDIT: I deleted my suggestion since it disregarded the AND operation. Also, my version (and I assume yours as well), does not throw a |
In your solution, wouldn't a Also, even in a simple case like |
I noticed that my solution did break things, yes.
Your version does throw it correctly. Sorry for the for the noise. :) So your version works entirely as intended then, I suppose? The only question would be if the implementation could be improved. |
Yep, my implementation could definitely be improved. I won't do that today any more, though. |
I think it is good enough actually, since it allows the following statements to handle it right afterwards. Just added some documentation and replaced the symbol = skip(punctuation);
% ends with '...':
% The '...' has to be unskipped and handled here in order
% to not cause and error for line endings such as `+...`
% or `&&...`.
if length(symbol) > 3 && strcmp(symbol(end-2:end), '...')
pos = pos - 3;
symbol = symbol(1:end-3);
end
% one operator: |
Actually, this (both your and mine approach) will still break if you write bad but perfectly valid Matlab code like |
18cf851
to
e71780e
Compare
I've updated this with your suggestions. |
Is my preview of the commit wrong after the force-push or did you make a mistake? :) |
Prior to this change, any line ending with [punctuation + '...'], for example `||...`, would cause the tokenizer to fail.
e71780e
to
1b08e08
Compare
I did. Forgot to commit+amend before pushing :-P Now you should see your suggestion in the diff ;-) |
Prior to this change, any line ending with
[punctuation + '...']
, forexample
||...
, would cause the tokenizer to fail.Fixes #9