Skip to content

Commit

Permalink
fix: support newline-terminated regular expressions
Browse files Browse the repository at this point in the history
Previously, parsing a snippet like this:

    %r\nfoo\n

would result in tracking the second newline twice, resulting in a
failed runtime assertion.

Fixing that issue reveals another bug, which is that the _first_
newline was not being tracked at all. So we introduce a call to
yp_newline_list right when we construct the REGEXP_BEGIN token.
  • Loading branch information
flavorjones committed Aug 21, 2023
1 parent 613c6ff commit 0d5d759
Show file tree
Hide file tree
Showing 4 changed files with 20 additions and 1 deletion.
11 changes: 10 additions & 1 deletion src/yarp.c
Original file line number Diff line number Diff line change
Expand Up @@ -6215,6 +6215,9 @@ parser_lex(yp_parser_t *parser) {

if (parser->current.end < parser->end) {
lex_mode_push_regexp(parser, lex_mode_incrementor(*parser->current.end), lex_mode_terminator(*parser->current.end));
if (parser->current.end == '\n') {
yp_newline_list_append(&parser->newline_list, parser->current.end);
}
parser->current.end++;
}

Expand Down Expand Up @@ -6526,7 +6529,13 @@ parser_lex(yp_parser_t *parser) {
// If we've hit a newline, then we need to track that in the
// list of newlines.
if (*breakpoint == '\n') {
yp_newline_list_append(&parser->newline_list, breakpoint);
// For the special case of a newline-terminated regular expression, we will pass
// through this branch twice -- once with YP_TOKEN_REGEXP_BEGIN and then again
// with YP_TOKEN_STRING_CONTENT. Let's avoid tracking the newline twice, by
// tracking it only in the REGEXP_BEGIN case.
if (!(lex_mode->as.regexp.terminator == '\n' && parser->current.type != YP_TOKEN_REGEXP_BEGIN)) {
yp_newline_list_append(&parser->newline_list, breakpoint);
}

if (lex_mode->as.regexp.terminator != '\n') {
// If the terminator is not a newline, then we can set
Expand Down
2 changes: 2 additions & 0 deletions test/fixtures/newline-terminated-things.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
%r
foo
2 changes: 2 additions & 0 deletions test/parse_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ def test_parse_takes_file_path
end

Dir["*.txt", base: base].each do |relative|
next if relative == "newline_terminated.txt"

# We test every snippet (separated by \n\n) in isolation
# to ensure the parser does not try to read bytes further than the end of each snippet
define_method "test_individual_snippets_#{relative}" do
Expand Down
6 changes: 6 additions & 0 deletions test/snapshots/newline-terminated-things.txt

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 0d5d759

Please sign in to comment.