-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Doc Comments and General Cleanup of Scanner #195
Conversation
I have fixed the lint problems locally but now I'm hunting down the parser fuzzing error. |
The fuzzing error seen in the failure on here was caused by the shift in the install process for tree-sitter-cli from installing the However, there was also a recent update to the |
I looked deeper and make install is the one that no longer installs |
Hey, just wanted to check what the expected timeline on looking at this is. |
Would love to have this added. Not parsing doc comments is very strange |
Hi! I wanted to check back in on this as it's been sitting in my backlog for a couple of month now. @amaanq is this something that you would need to look at? |
For me this seems to only highlight the first two characters of the doc comment, #168 does not have the same problem |
Would you mind posting the code you were using to test it? I am willing to fix it, I just need to know what it is. @TrueDoctor |
I hooked it up to the helix editor by changing the url of the rust grammer to point to your pr and then adding |
Oh, I see the issue I think.
or
Or something like that. I'm not an expert on crafting queries yet. Once I have time again (don't do a PhD, you'll never have your own life) I can try to drum up some better versions. Oh, if you want to differentiate between inner and outer doc comments you can as well. You would do that like this I think:
or
|
Oh, sorry I think made a small error, I think I actually used the |
The |
1a77348
to
5390f15
Compare
I'm sorry for taking so long to get to this, I had a bit of a hiatus November-January and then was just busy afterwards and forgot about this This looks really good FYI, so thanks a ton @ProfDoof! There's a few things I changed around but otherwise I think we can get this in, most notably:
Might've forgotten something, but that's the meat of it. Thanks! |
5390f15
to
7dcab5e
Compare
Would be helpful to add queries for this (makes it easier for downstream to include). |
not much really needed tbh, there's still queries for comments but this enables querying for doc comments ( There's |
7dcab5e
to
efddc8e
Compare
🥳 |
Hi all!
I was looking at some previous PRs for doc comments like #168 and thought I would take a crack at it addressing some of the issues raised by @the-mikedavis in the other PR to fix #147. I scope creeped myself while I was doing it though and also separated each of the processing steps in the scanner.c file out into their own functions to make it easier to read the file. I was struggling quite a bit to read it myself. I also went through and did some cleanup of functions based on what I read in the documentation for tree-sitter.
If I need to split this into 2 separate PRs, let me know, and I will address that as able.
Quick question though, is there a good way for me to benchmark my changes against the original solution? I want to make sure that the extra processing cost we would be incurring is worth it.
Changes Made
Grammar
Scanner
lexer->lookahead == 0
to instead useeof(lexer)
which calls theeof
function on the lexer. This is recommended in the tree-sitter documentation I read.skip
function similar to theadvance
function and switched parts of the code from manually callingadvance
on thelexer
to using either theadvance
function or theskip
function.markEnd
function for the same reason as above.markEnd
during the block comment processing and moved the block comment processing to the beginning of the scanner function to ensure that we don't skip over whitespace by accident.Test