forked from PCRE2Project/pcre2
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
2794 lines (2034 loc) · 128 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Change Log for PCRE2 - see also the Git log
-------------------------------------------
Version 10.41 xx-xxx-2022
-------------------------
1. Add fflush() before and after a fork callout in pcre2grep to get its output
to be the same on all systems. (THere were previously ordering differences in
Alpine Linux).
2. Merged patch from @carenas (GitHub #110) for pthreads support in CMake.
3. SSF scorecards grumbled about possible overflow in an expression in
pcre2test. It never would have overflowed in practice, but some casts have been
added and at the some time there's been some tidying of fprints that output
size_t values.
4. PR #94 showed up an unused enum in pcre2_convert.c, which is now removed.
5. Minor code re-arrangement to remove gcc warning about realloc() in
pcre2test.
6. Change a number of int variables that hold buffer and line lengths in
pcre2grep to PCRE2_SIZE (aka size_t).
7. Added an #ifdef to cut out a call to PRIV(jit_free) when JIT is not
supported (even though that function would do nothing in that case) at the
request of a user who doesn't even want to link with pcre_jit_compile.o. Also
tidied up an untidy #ifdef arrangement in pcre2test.
8. Fixed an issue in the backtracking optimization of character repeats in
JIT. Furthermore optimize star repetitions, not just plus repetitions.
9. Removed the use of an initial backtracking frames vector on the system stack
in pcre2_match() so that it now always uses the heap. (In a multi-thread
environment with very small stacks there had been an issue.) This also is
tidier for JIT matching, which didn't need that vector. The heap vector is now
remembered in the match data block and re-used if that block itself is re-used.
It is freed with the match data block.
10. Adjusted the find_limits code in pcre2test to work with change 9 above.
11. Added find_limits_noheap to pcre2test, because the heap limits are now
different in different environments and so cannot be included in the standard
tests.
12. Created a test for pcre2_match() heap processing that is not part of the
tests run by 'make check', but can be run manually. The current output is from
a 64-bit system.
13. Implemented -Z aka --null in pcre2grep.
14. A minor change to pcre2test and the addition of several new pcre2grep tests
have improved LCOV coverage statistics. At the same time, code in pcre2grep and
elsewhere that can never be obeyed in normal testing has been excluded from
coverage.
15. Fixed a bug in pcre2grep that could cause an extra newline to be written
after output generaed by --output.
16. If a file has a .bz2 extension but is not in fact compressed, pcre2grep
should process it as a plain text file. A bug stopped this happening; now fixed
and added to the tests.
17. When pcre2grep was running not in UTF mode, if a string specified by
--output or obtained from a callout in a pattern contained a character (byte)
greater than 127, it was incorrectly output in UTF-8 format.
18. Added some casts after warnings from Clang sanitize.
19. Merged patch from cbouc (GitHub #139): 4 function prototypes were missing
PCRE2_CALL_CONVENTION in src/pcre2posix.h. All function prototypes returning
pointers had out of place PCRE2_CALL_CONVENTION in src/pcre2.h.*. These
produced errors when building for Windows with #define PCRE2_CALL_CONVENTION
__stdcall.
20. A negative repeat value in a pcre2test subject line was not being
diagnosed, leading to infinite looping.
21. Updated RunGrepTest to discard the warning that Bash now gives when setting
LC_CTYPE to an bad value (because older versions didn't).
Version 10.40 15-April-2022
---------------------------
1. Merged patch from @carenas (GitHub #35, 7db87842) to fix pcre2grep incorrect
handling of multiple passes.
2. Merged patch from @carenas (GitHub #36, dae47509) to fix portability issue
in pcre2grep with buffered fseek(stdin).
3. Merged patch from @carenas (GitHub #37, acc520924) to fix tests when -S is
not supported.
4. Revert an unintended change in JIT repeat detection.
5. Merged patch from @carenas (GitHub #52, b037bfa1) to fix build on GNU Hurd.
6. Merged documentation and comments patches from @carenas (GitHub #47).
7. Merged patch from @carenas (GitHub #49) to remove obsolete JFriedl test code
from pcre2grep.
8. Merged patch from @carenas (GitHub #48) to fix CMake install issue #46.
9. Merged patch from @carenas (GitHub #53) fixing NULL checks in matching and
substituting.
10. Add null_subject and null_replacement modifiers to pcre2test.
11. Add check for NULL subject to POSIX regexec() function.
12. Add check for NULL replacement to pcre2_substitute().
13. For the subject arguments of pcre2_match(), pcre2_dfa_match(), and
pcre2_substitute(), and the replacement argument of the latter, if the pointer
is NULL and the length is zero, treat as an empty string. Apparently a number
of applications treat NULL/0 in this way.
14. Added support for Bidi_Class and a number of binary Unicode properties,
including Bidi_Control.
15. Fix some minor issues raised by clang sanitize.
16. Very minor code speed up for maximizing character property matches.
17. A number of changes to script matching for \p and \P:
(a) Script extensions for a character are now coded as a bitmap instead of
a list of script numbers, which should be faster and does not need a
loop.
(b) Added the syntax \p{script:xxx} and \p{script_extensions:xxx} (synonyms
sc and scx).
(c) Changed \p{scriptname} from being the same as \p{sc:scriptname} to being
the same as \p{scx:scriptname} because this change happened in Perl at
release 5.26.
(d) The standard Unicode 4-letter abbreviations for script names are now
recognized.
(e) In accordance with Unicode and Perl's "loose matching" rules, spaces,
hyphens, and underscores are ignored in property names, which are then
matched independent of case.
18. The Python scripts in the maint directory have been refactored. There are
now three scripts that generate pcre2_ucd.c, pcre2_ucp.h, and pcre2_ucptables.c
(which is #included by pcre2_tables.c). The data lists that used to be
duplicated are now held in a single common Python module.
19. On CHERI, and thus Arm's Morello prototype, pointers are represented as
hardware capabilities, which consist of both an integer address and additional
metadata, meaning they are twice the size of the platform's size_t type, i.e.
16 bytes on a 64-bit system. The ovector member of heapframe happens to only be
8 byte aligned, and so computing frame_size ended up with a multiple of 8 but
not 16. Whilst the first frame was always suitably aligned, this then
misaligned the frame that follows, resulting in an alignment fault when storing
a pointer to Fecode at the start of match. Patch to fix this issue by Jessica
Clarke PR#72.
20. Added -LP and -LS listing options to pcre2test.
21. A user discovered that the library names in CMakeLists.txt for MSVC
debugger (PDB) files were incorrect - perhaps never tried for PCRE2?
22. An item such as [Aa] is optimized into a caseless single character match.
When this was quantified (e.g. [Aa]{2}) and was also the last literal item in a
pattern, the optimizing "must be present for a match" character check was not
being flagged as caseless, causing some matches that should have succeeded to
fail.
23. Fixed a unicode property matching issue in JIT. The character was not
fully read in caseless matching.
24. Fixed an issue affecting recursions in JIT caused by duplicated data
transfers.
25. Merged patch from @carenas (GitHub #96) which fixes some problems with
pcre2test and readline/readedit:
* Use the right header for libedit in FreeBSD with autoconf
* Really allow libedit with cmake
* Avoid using readline headers with libedit
Version 10.39 29-October-2021
-----------------------------
1. Fix incorrect detection of alternatives in first character search in JIT.
2. Merged patch from @carenas (GitHub #28):
Visual Studio 2013 includes support for %zu and %td, so let newer
versions of it avoid the fallback, and while at it, make sure that
the first check is for DISABLE_PERCENT_ZT so it will be always
honoured if chosen.
prtdiff_t is signed, so use a signed type instead, and make sure
that an appropriate width is chosen if pointers are 64bit wide and
long is not (ex: Windows 64bit).
IMHO removing the cast (and therefore the possibilty of truncation)
make the code cleaner and the fallback is likely portable enough
with all 64-bit POSIX systems doing LP64 except for Windows.
3. Merged patch from @carenas (GitHub #29) to update to Unicode 14.0.0.
4. Merged patch from @carenas (GitHub #30):
* Cleanup: remove references to no longer used stdint.h
Since 19c50b9d (Unconditionally use inttypes.h instead of trying for stdint.h
(simplification) and remove the now unnecessary inclusion in
pcre2_internal.h., 2018-11-14), stdint.h is no longer used.
Remove checks for it in autotools and CMake and document better the expected
build failures for systems that might have stdint.h (C99) and not inttypes.h
(from POSIX), like old Windows.
* Cleanup: remove detection for inttypes.h which is a hard dependency
CMake checks for standard headers are not meant to be used for hard
dependencies, so will prevent a possible fallback to work.
Alternatively, the header could be checked to make the configuration fail
instead of breaking the build, but that was punted, as it was missing anyway
from autotools.
5. Merged patch from @carenas (GitHub #32):
* jit: allow building with ancient MSVC versions
Visual Studio older than 2013 fails to build with JIT enabled, because it is
unable to parse non C89 compatible syntax, with mixed declarations and code.
While most recent compilers wouldn't even report this as a warning since it
is valid C99, it could be also made visible by adding to gcc/clang the
-Wdeclaration-after-statement flag at build time.
Move the code below the affected definitions.
* pcre2grep: avoid mixing declarations with code
Since d5a61ee8 (Patch to detect (and ignore) symlink loops in pcre2grep,
2021-08-28), code will fail to build in a strict C89 compiler.
Reformat slightly to make it C89 compatible again.
Version 10.38 01-October-2021
-----------------------------
1. Fix invalid single character repetition issues in JIT when the repetition
is inside a capturing bracket and the bracket is preceded by character
literals.
2. Installed revised CMake configuration files provided by Jan-Willem Blokland.
This extends the CMake build system to build both static and shared libraries
in one go, builds the static library with PIC, and exposes PCRE2 libraries
using the CMake config files. JWB provided these notes:
- Introduced CMake variable BUILD_STATIC_LIBS to build the static library.
- Make a small modification to config-cmake.h.in by removing the PCRE2_STATIC
variable. Added PCRE2_STATIC variable to the static build using the
target_compile_definitions() function.
- Extended the CMake config files.
- Introduced CMake variable PCRE2_USE_STATIC_LIBS to easily switch between
the static and shared libraries.
- Added the PCRE_STATIC variable to the target compile definitions for the
import of the static library.
Building static and shared libraries using MSVC results in a name clash of
the libraries. Both static and shared library builds create, for example, the
file pcre2-8.lib. Therefore, I decided to change the static library names by
adding "-static". For example, pcre2-8.lib has become pcre2-8-static.lib.
[Comment by PH: this is MSVC-specific. It doesn't happen on Linux.]
3. Increased the minimum release number for CMake to 3.0.0 because older than
2.8.12 is deprecated (it was set to 2.8.5) and causes warnings. Even 3.0.0 is
quite old; it was released in 2014.
4. Implemented a modified version of Thomas Tempelmann's pcre2grep patch for
detecting symlink loops. This is dependent on the availability of realpath(),
which is now tested for in ./configure and CMakeLists.txt.
5. Implemented a modified version of Thomas Tempelmann's patch for faster
case-independent "first code unit" searches for unanchored patterns in 8-bit
mode in the interpreters. Instead of just remembering whether one case matched
or not, it remembers the position of a previous match so as to avoid
unnecessary repeated searching.
6. Perl now locks out \K in lookarounds, so PCRE2 now does the same by default.
However, just in case anybody was relying on the old behaviour, there is an
option called PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK that enables the old behaviour.
An option has also been added to pcre2grep to enable this.
7. Re-enable a JIT optimization which was unintentionally disabled in 10.35.
8. There is a loop counter to catch excessively crazy patterns when checking
the lengths of lookbehinds at compile time. This was incorrectly getting reset
whenever a lookahead was processed, leading to some fuzzer-generated patterns
taking a very long time to compile when (?|) was present in the pattern,
because (?|) disables caching of group lengths.
Version 10.37 26-May-2021
-------------------------
1. Change RunGrepTest to use tr instead of sed when testing with binary
zero bytes, because sed varies a lot from system to system and has problems
with binary zeros. This is from Bugzilla #2681. Patch from Jeremie
Courreges-Anglas via Nam Nguyen. This fixes RunGrepTest for OpenBSD. Later:
it broke it for at least one version of Solaris, where tr can't handle binary
zeros. However, that system had /usr/xpg4/bin/tr installed, which works OK, so
RunGrepTest now checks for that command and uses it if found.
2. Compiling with gcc 10.2's -fanalyzer option showed up a hypothetical problem
with a NULL dereference. I don't think this case could ever occur in practice,
but I have put in a check in order to get rid of the compiler error.
3. An alternative patch for CMakeLists.txt because 10.36 #4 breaks CMake on
Windows. Patch from email@cs-ware.de fixes bugzilla #2688.
4. Two bugs related to over-large numbers have been fixed so the behaviour is
now the same as Perl.
(a) A pattern such as /\214748364/ gave an overflow error instead of being
treated as the octal number \214 followed by literal digits.
(b) A sequence such as {65536 that has no terminating } so is not a
quantifier was nevertheless complaining that a quantifier number was too big.
5. A run of autoconf suggested that configure.ac was out-of-date with respect
to the lastest autoconf. Running autoupdate made some valid changes, some valid
suggestions, and also some invalid changes, which were fixed by hand. Autoconf
now runs clean and the resulting "configure" seems to work, so I hope nothing
is broken. Later: the requirement for autoconf 2.70 broke some automatic test
robots. It doesn't seem to be necessary: trying a reduction to 2.60.
6. The pattern /a\K.(?0)*/ when matched against "abac" by the interpreter gave
the answer "bac", whereas Perl and JIT both yield "c". This was because the
effect of \K was not propagating back from the full pattern recursion. Other
recursions such as /(a\K.(?1)*)/ did not have this problem.
7. Restore single character repetition optimization in JIT. Currently fewer
character repetitions are optimized than in 10.34.
8. When the names of the functions in the POSIX wrapper were changed to
pcre2_regcomp() etc. (see change 10.33 #4 below), functions with the original
names were left in the library so that pre-compiled programs would still work.
However, this has proved troublesome when programs link with several libraries,
some of which use PCRE2 via the POSIX interface while others use a native POSIX
library. For this reason, the POSIX function names are removed in this release.
The macros in pcre2posix.h should ensure that re-compiling fixes any programs
that haven't been compiled since before 10.33.
Version 10.36 04-December-2020
------------------------------
1. Add CET_CFLAGS so that when Intel CET is enabled, pass -mshstk to
compiler. This fixes https://bugs.exim.org/show_bug.cgi?id=2578. Patch for
Makefile.am and configure.ac by H.J. Lu. Equivalent patch for CMakeLists.txt
invented by PH.
2. Fix inifinite loop when a single byte newline is searched in JIT when
invalid utf8 mode is enabled.
3. Updated CMakeLists.txt with patch from Wolfgang Stöggl (Bugzilla #2584):
- Include GNUInstallDirs and use ${CMAKE_INSTALL_LIBDIR} instead of hardcoded
lib. This allows differentiation between lib and lib64.
CMAKE_INSTALL_LIBDIR is used for installation of libraries and also for
pkgconfig file generation.
- Add the version of PCRE2 to the configuration summary like ./configure
does.
- Fix typo: MACTHED_STRING->MATCHED_STRING
4. Updated CMakeLists.txt with another patch from Wolfgang Stöggl (Bugzilla
#2588):
- Add escaped double quotes around include directory in CMakeLists.txt to
allow spaces in directory names.
- This fixes a cmake error, if the path of the pcre2 source contains a space.
5. Updated CMakeLists.txt with a patch from B. Scott Michel: CMake's
documentation suggests using CHECK_SYMBOL_EXISTS over CHECK_FUNCTION_EXIST.
Moreover, these functions come from specific header files, which need to be
specified (and, thankfully, are the same on both the Linux and WinXX
platforms.)
6. Added a (uint32_t) cast to prevent a compiler warning in pcre2_compile.c.
7. Applied a patch from Wolfgang Stöggl (Bugzilla #2600) to fix postfix for
debug Windows builds using CMake. This also updated configure so that it
generates *.pc files and pcre2-config with the same content, as in the past.
8. If a pattern ended with (?(VERSION=n.d where n is any number but d is just a
single digit, the code unit beyond d was being read (i.e. there was a read
buffer overflow). Fixes ClusterFuzz 23779.
9. After the rework in r1235, certain character ranges were incorrectly
handled by an optimization in JIT. Furthermore a wrong offset was used to
read a value from a buffer which could lead to memory overread.
10. Unnoticed for many years was the fact that delimiters other than / in the
testinput1 and testinput4 files could cause incorrect behaviour when these
files were processed by perltest.sh. There were several tests that used quotes
as delimiters, and it was just luck that they didn't go wrong with perltest.sh.
All the patterns in testinput1 and testinput4 now use / as their delimiter.
This fixes Bugzilla #2641.
11. Perl has started to give an error for \K within lookarounds (though there
are cases where it doesn't). PCRE2 still allows this, so the tests that include
this case have been moved from test 1 to test 2.
12. Further to 10 above, pcre2test has been updated to detect and grumble if a
delimiter other than / is used after #perltest.
13. Fixed a bug with PCRE2_MATCH_INVALID_UTF in 8-bit mode when PCRE2_CASELESS
was set and PCRE2_NO_START_OPTIMIZE was not set. The optimization for finding
the start of a match was not resetting correctly after a failed match on the
first valid fragment of the subject, possibly causing incorrect "no match"
returns on subsequent fragments. For example, the pattern /A/ failed to match
the subject \xe5A. Fixes Bugzilla #2642.
14. Fixed a bug in character set matching when JIT is enabled and both unicode
scripts and unicode classes are present at the same time.
15. Added GNU grep's -m (aka --max-count) option to pcre2grep.
16. Refactored substitution processing in pcre2grep strings, both for the -O
option and when dealing with callouts. There is now a single function that
handles $ expansion in all cases (instead of multiple copies of almost
identical code). This means that the same escape sequences are available
everywhere, which was not previously the case. At the same time, the escape
sequences $x{...} and $o{...} have been introduced, to allow for characters
whose code points are greater than 255 in Unicode mode.
17. Applied the patch from Bugzilla #2628 to RunGrepTest. This does an explicit
test for a version of sed that can handle binary zero, instead of assuming that
any Linux version will work. Later: replaced $(...) by `...` because not all
shells recognize the former.
18. Fixed a word boundary check bug in JIT when partial matching is enabled.
19. Fix ARM64 compilation warning in JIT. Patch by Carlo.
20. A bug in the RunTest script meant that if the first part of test 2 failed,
the failure was not reported.
21. Test 2 was failing when run from a directory other than the source
directory. This failure was previously missed in RunTest because of 20 above.
Fixes added to both RunTest and RunTest.bat.
22. Patch to CMakeLists.txt from Daniel to fix problem with testing under
Windows.
Version 10.35 09-May-2020
---------------------------
1. Use PCRE2_MATCH_EMPTY flag to detect empty matches in JIT.
2. Fix ARMv5 JIT improper handling of labels right after a constant pool.
3. A JIT bug is fixed which allowed to read the fields of the compiled
pattern before its existence is checked.
4. Back in the PCRE1 day, capturing groups that contained recursive back
references to themselves were made atomic (version 8.01, change 18) because
after the end a repeated group, the captured substrings had their values from
the final repetition, not from an earlier repetition that might be the
destination of a backtrack. This feature was documented, and was carried over
into PCRE2. However, it has now been realized that the major refactoring that
was done for 10.30 has made this atomicizing unnecessary, and it is confusing
when users are unaware of it, making some patterns appear not to be working as
expected. Capture values of recursive back references in repeated groups are
now correctly backtracked, so this unnecessary restriction has been removed.
5. Added PCRE2_SUBSTITUTE_LITERAL.
6. Avoid some VS compiler warnings.
7. Added PCRE2_SUBSTITUTE_MATCHED.
8. Added (?* and (?<* as synonyms for (*napla: and (*naplb: to match another
regex engine. The Perl regex folks are aware of this usage and have made a note
about it.
9. When an assertion is repeated, PCRE2 used to limit the maximum repetition to
1, believing that repeating an assertion is pointless. However, if a positive
assertion contains capturing groups, repetition can be useful. In any case, an
assertion could always be wrapped in a repeated group. The only restriction
that is now imposed is that an unlimited maximum is changed to one more than
the minimum.
10. Fix *THEN verbs in lookahead assertions in JIT.
11. Added PCRE2_SUBSTITUTE_REPLACEMENT_ONLY.
12. The JIT stack should be freed when the low-level stack allocation fails.
13. In pcre2grep, if the final line in a scanned file is output but does not
end with a newline sequence, add a newline according to the --newline setting.
14. (?(DEFINE)...) groups were not being handled correctly when checking for
the fixed length of a lookbehind assertion. Such a group within a lookbehind
should be skipped, as it does not contribute to the length of the group.
Instead, the (DEFINE) group was being processed, and if at the end of the
lookbehind, that end was not correctly recognized. Errors such as "lookbehind
assertion is not fixed length" and also "internal error: bad code value in
parsed_skip()" could result.
15. Put a limit of 1000 on recursive calls in pcre2_study() when searching
nested groups for starting code units, in order to avoid stack overflow issues.
If the limit is reached, it just gives up trying for this optimization.
16. The control verb chain list must always be restored when exiting from a
recurse function in JIT.
17. Fix a crash which occurs when the character type of an invalid UTF
character is decoded in JIT.
18. Changes in many areas of the code so that when Unicode is supported and
PCRE2_UCP is set without PCRE2_UTF, Unicode character properties are used for
upper/lower case computations on characters whose code points are greater than
127.
19. The function for checking UTF-16 validity was returning an incorrect offset
for the start of the error when a high surrogate was not followed by a valid
low surrogate. This caused incorrect behaviour, for example when
PCRE2_MATCH_INVALID_UTF was set and a match started immediately following the
invalid high surrogate, such as /aa/ matching "\x{d800}aa".
20. If a DEFINE group immediately preceded a lookbehind assertion, the pattern
could be mis-compiled and therefore not match correctly. This is the example
that found this: /(?(DEFINE)(?<foo>bar))(?<![-a-z0-9])word/ which failed to
match "word" because the "move back" value was set to zero.
21. Following a request from a user, some extensions and tidies to the
character tables handling have been done:
(a) The dftables auxiliary program is renamed pcre2_dftables, but it is still
not installed for public use.
(b) There is now a -b option for pcre2_dftables, which causes the tables to
be written in binary. There is also a -help option.
(c) PCRE2_CONFIG_TABLES_LENGTH is added to pcre2_config() so that an
application that wants to save tables in binary knows how long they are.
22. Changed setting of CMAKE_MODULE_PATH in CMakeLists.txt from SET to
LIST(APPEND...) to allow a setting from the command line to be included.
23. Updated to Unicode 13.0.0.
24. CMake build now checks for secure_getenv() and strerror(). Patch by Carlo.
25. Avoid using [-1] as a suffix in pcre2test because it can provoke a compiler
warning.
26. Added tests for __attribute__((uninitialized)) to both the configure and
CMake build files, and then applied this attribute to the variable called
stack_frames_vector[] in pcre2_match(). When implemented, this disables
automatic initialization (a facility in clang), which can take time on big
variables.
27. Updated CMakeLists.txt (patches by Uwe Korn) to add support for
pcre2-config, the libpcre*.pc files, SOVERSION, VERSION and the
MACHO_*_VERSIONS settings for CMake builds.
28. Another patch to CMakeLists.txt to check for mkostemp (configure already
does). Patch by Carlo Marcelo Arenas Belon.
29. Check for the existence of memfd_create in both CMake and configure
configurations. Patch by Carlo Marcelo Arenas Belon.
30. Restrict the configuration setting for the SELinux compatible execmem
allocator (change 10.30/44) to Linux and NetBSD.
Version 10.34 21-November-2019
------------------------------
1. The maximum number of capturing subpatterns is 65535 (documented), but no
check on this was ever implemented. This omission has been rectified; it fixes
ClusterFuzz 14376.
2. Improved the invalid utf32 support of the JIT compiler. Now it correctly
detects invalid characters in the 0xd800-0xdfff range.
3. Fix minor typo bug in JIT compile when \X is used in a non-UTF string.
4. Add support for matching in invalid UTF strings to the pcre2_match()
interpreter, and integrate with the existing JIT support via the new
PCRE2_MATCH_INVALID_UTF compile-time option.
5. Give more error detail for invalid UTF-8 when detected in pcre2grep.
6. Add support for invalid UTF-8 to pcre2grep.
7. Adjust the limit for "must have" code unit searching, in particular,
increase it substantially for non-anchored patterns.
8. Allow (*ACCEPT) to be quantified, because an ungreedy quantifier with a zero
minimum is potentially useful.
9. Some changes to the way the minimum subject length is handled:
* When PCRE2_NO_START_OPTIMIZE is set, no minimum length is computed;
pcre2test now omits this item instead of showing a value of zero.
* An incorrect minimum length could be calculated for a pattern that
contained (*ACCEPT) inside a qualified group whose minimum repetition was
zero, for example /A(?:(*ACCEPT))?B/, which incorrectly computed a minimum
of 2. The minimum length scan no longer happens for a pattern that
contains (*ACCEPT).
* When no minimum length is set by the normal scan, but a first and/or last
code unit is recorded, set the minimum to 1 or 2 as appropriate.
* When a pattern contains multiple groups with the same number, a back
reference cannot know which one to scan for a minimum length. This used to
cause the minimum length finder to give up with no result. Now it treats
such references as not adding to the minimum length (which it should have
done all along).
* Furthermore, the above action now happens only if the back reference is to
a group that exists more than once in a pattern instead of any back
reference in a pattern with duplicate numbers.
10. A (*MARK) value inside a successful condition was not being returned by the
interpretive matcher (it was returned by JIT). This bug has been mended.
11. A bug in pcre2grep meant that -o without an argument (or -o0) didn't work
if the pattern had more than 32 capturing parentheses. This is fixed. In
addition (a) the default limit for groups requested by -o<n> has been raised to
50, (b) the new --om-capture option changes the limit, (c) an error is raised
if -o asks for a group that is above the limit.
12. The quantifier {1} was always being ignored, but this is incorrect when it
is made possessive and applied to an item in parentheses, because a
parenthesized item may contain multiple branches or other backtracking points,
for example /(a|ab){1}+c/ or /(a+){1}+a/.
13. For partial matches, pcre2test was always showing the maximum lookbehind
characters, flagged with "<", which is misleading when the lookbehind didn't
actually look behind the start (because it was later in the pattern). Showing
all consulted preceding characters for partial matches is now controlled by the
existing "allusedtext" modifier and, as for complete matches, this facility is
available only for non-JIT matching, because JIT does not maintain the first
and last consulted characters.
14. DFA matching (using pcre2_dfa_match()) was not recognising a partial match
if the end of the subject was encountered in a lookahead (conditional or
otherwise), an atomic group, or a recursion.
15. Give error if pcre2test -t, -T, -tm or -TM is given an argument of zero.
16. Check for integer overflow when computing lookbehind lengths. Fixes
Clusterfuzz issue 15636.
17. Implemented non-atomic positive lookaround assertions.
18. If a lookbehind contained a lookahead that contained another lookbehind
within it, the nested lookbehind was not correctly processed. For example, if
/(?<=(?=(?<=a)))b/ was matched to "ab" it gave no match instead of matching
"b".
19. Implemented pcre2_get_match_data_size().
20. Two alterations to partial matching:
(a) The definition of a partial match is slightly changed: if a pattern
contains any lookbehinds, an empty partial match may be given, because this
is another situation where adding characters to the current subject can
lead to a full match. Example: /c*+(?<=[bc])/ with subject "ab".
(b) Similarly, if a pattern could match an empty string, an empty partial
match may be given. Example: /(?![ab]).*/ with subject "ab". This case
applies only to PCRE2_PARTIAL_HARD.
(c) An empty string partial hard match can be returned for \z and \Z as it
is documented that they shouldn't match.
21. A branch that started with (*ACCEPT) was not being recognized as one that
could match an empty string.
22. Corrected pcre2_set_character_tables() tables data type: was const unsigned
char * instead of const uint8_t *, as generated by pcre2_maketables().
23. Upgraded to Unicode 12.1.0.
24. Add -jitfast command line option to pcre2test (to make all the jit options
available directly).
25. Make pcre2test -C show if libreadline or libedit is supported.
26. If the length of one branch of a group exceeded 65535 (the maximum value
that is remembered as a minimum length), the whole group's length was
incorrectly recorded as 65535, leading to incorrect "no match" when start-up
optimizations were in force.
27. The "rightmost consulted character" value was not always correct; in
particular, if a pattern ended with a negative lookahead, characters that were
inspected in that lookahead were not included.
28. Add the pcre2_maketables_free() function.
29. The start-up optimization that looks for a unique initial matching
code unit in the interpretive engines uses memchr() in 8-bit mode. When the
search is caseless, it was doing so inefficiently, which ended up slowing down
the match drastically when the subject was very long. The revised code (a)
remembers if one case is not found, so it never repeats the search for that
case after a bumpalong and (b) when one case has been found, it searches only
up to that position for an earlier occurrence of the other case. This fix
applies to both interpretive pcre2_match() and to pcre2_dfa_match().
30. While scanning to find the minimum length of a group, if any branch has
minimum length zero, there is no need to scan any subsequent branches (a small
compile-time performance improvement).
31. Installed a .gitignore file on a user's suggestion. When using the svn
repository with git (through git svn) this helps keep it tidy.
32. Add underflow check in JIT which may occur when the value of subject
string pointer is close to 0.
33. Arrange for classes such as [Aa] which contain just the two cases of the
same character, to be treated as a single caseless character. This causes the
first and required code unit optimizations to kick in where relevant.
34. Improve the bitmap of starting bytes for positive classes that include wide
characters, but no property types, in UTF-8 mode. Previously, on encountering
such a class, the bits for all bytes greater than \xc4 were set, thus
specifying any character with codepoint >= 0x100. Now the only bits that are
set are for the relevant bytes that start the wide characters. This can give a
noticeable performance improvement.
35. If the bitmap of starting code units contains only 1 or 2 bits, replace it
with a single starting code unit (1 bit) or a caseless single starting code
unit if the two relevant characters are case-partners. This is particularly
relevant to the 8-bit library, though it applies to all. It can give a
performance boost for patterns such as [Ww]ord and (word|WORD). However, this
optimization doesn't happen if there is a "required" code unit of the same
value (because the search for a "required" code unit starts at the match start
for non-unique first code unit patterns, but after a unique first code unit,
and patterns such as a*a need the former action).
36. Small patch to pcre2posix.c to set the erroroffset field to -1 immediately
after a successful compile, instead of at the start of matching to avoid a
sanitizer complaint (regexec is supposed to be thread safe).
37. Add NEON vectorization to JIT to speed up matching of first character and
pairs of characters on ARM64 CPUs.
38. If a non-ASCII character was the first in a starting assertion in a
caseless match, the "first code unit" optimization did not get the casing
right, and the assertion failed to match a character in the other case if it
did not start with the same code unit.
39. Fixed the incorrect computation of jump sizes on x86 CPUs in JIT. A masking
operation was incorrectly removed in r1136. Reported by Ralf Junker.
Version 10.33 16-April-2019
---------------------------
1. Added "allvector" to pcre2test to make it easy to check the part of the
ovector that shouldn't be changed, in particular after substitute and failed or
partial matches.
2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
a greater than 1 fixed quantifier. This issue was found by Yunho Kim.
3. Added support for callouts from pcre2_substitute(). After 10.33-RC1, but
prior to release, fixed a bug that caused a crash if pcre2_substitute() was
called with a NULL match context.
4. The POSIX functions are now all called pcre2_regcomp() etc., with wrapper
functions that use the standard POSIX names. However, in pcre2posix.h the POSIX
names are defined as macros. This should help avoid linking with the wrong
library in some environments while still exporting the POSIX names for
pre-existing programs that use them. (The Debian alternative names are also
defined as macros, but not documented.)
5. Fix an xclass matching issue in JIT.
6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).
7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
characterized by a lower case letter following (* and to simplify coding for
this, the character tables created by pcre2_maketables() were updated to add a
new "is lower case letter" bit. At the same time, the now unused "is
hexadecimal digit" bit was removed. The default tables in
src/pcre2_chartables.c.dist are updated.
8. Implement the new Perl "script run" features (*script_run:...) and
(*atomic_script_run:...) aka (*sr:...) and (*asr:...).
9. Fixed two typos in change 22 for 10.21, which added special handling for
ranges such as a-z in EBCDIC environments. The original code probably never
worked, though there were no bug reports.
10. Implement PCRE2_COPY_MATCHED_SUBJECT for pcre2_match() (including JIT via
pcre2_match()) and pcre2_dfa_match(), but *not* the pcre2_jit_match() fast
path. Also, when a match fails, set the subject field in the match data to NULL
for tidiness - none of the substring extractors should reference this after
match failure.
11. If a pattern started with a subroutine call that had a quantifier with a
minimum of zero, an incorrect "match must start with this character" could be
recorded. Example: /(?&xxx)*ABC(?<xxx>XYZ)/ would (incorrectly) expect 'A' to
be the first character of a match.
12. The heap limit checking code in pcre2_dfa_match() could suffer from
overflow if the heap limit was set very large. This could cause incorrect "heap
limit exceeded" errors.
13. Add "kibibytes" to the heap limit output from pcre2test -C to make the
units clear.
14. Add a call to pcre2_jit_free_unused_memory() in pcre2grep, for tidiness.
15. Updated the VMS-specific code in pcre2test on the advice of a VMS user.
16. Removed the unnecessary inclusion of stdint.h (or inttypes.h) from
pcre2_internal.h as it is now included by pcre2.h. Also, change 17 for 10.32
below was unnecessarily complicated, as inttypes.h is a Standard C header,
which is defined to be a superset of stdint.h. Instead of conditionally
including stdint.h or inttypes.h, pcre2.h now unconditionally includes
inttypes.h. This supports environments that do not have stdint.h but do have
inttypes.h, which are known to exist. A note in the autotools documentation
says (November 2018) that there are none known that are the other way round.
17. Added --disable-percent-zt to "configure" (and equivalent to CMake) to
forcibly disable the use of %zu and %td in formatting strings because there is
at least one version of VMS that claims to be C99 but does not support these
modifiers.
18. Added --disable-pcre2grep-callout-fork, which restricts the callout support
in pcre2grep to the inbuilt echo facility. This may be useful in environments
that do not support fork().
19. Fix two instances of <= 0 being applied to unsigned integers (the VMS
compiler complains).
20. Added "fork" support for VMS to pcre2grep, for running an external program
via a string callout.
21. Improve MAP_JIT flag usage on MacOS. Patch by Rich Siegel.
22. If a pattern started with (*MARK), (*COMMIT), (*PRUNE), (*SKIP), or (*THEN)
followed by ^ it was not recognized as anchored.
23. The RunGrepTest script used to cut out the test of NUL characters for
Solaris and MacOS as printf and sed can't handle them. It seems that the *BSD
systems can't either. I've inverted the test so that only those OS that are
known to work (currently only Linux) try to run this test.
24. Some tests in RunGrepTest appended to testtrygrep from two different file
descriptors instead of redirecting stderr to stdout. This worked on Linux, but
it was reported not to on other systems, causing the tests to fail.
25. In the RunTest script, make the test for stack setting use the same value
for the stack as it needs for -bigstack.
26. Insert a cast in pcre2_dfa_match.c to suppress a compiler warning.
26. With PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL set, escape sequences such as \s
which are valid in character classes, but not as the end of ranges, were being
treated as literals. An example is [_-\s] (but not [\s-_] because that gave an
error at the *start* of a range). Now an "invalid range" error is given
independently of PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
27. Related to 26 above, PCRE2_BAD_ESCAPE_IS_LITERAL was affecting known escape
sequences such as \eX when they appeared invalidly in a character class. Now
the option applies only to unrecognized or malformed escape sequences.
28. Fix word boundary in JIT compiler. Patch by Mike Munday.
29. The pcre2_dfa_match() function was incorrectly handling conditional version
tests such as (?(VERSION>=0)...) when the version test was true. Incorrect
processing or a crash could result.
30. When PCRE2_UTF is set, allow non-ASCII letters and decimal digits in group
names, as Perl does. There was a small bug in this new code, found by
ClusterFuzz 12950, fixed before release.
31. Implemented PCRE2_EXTRA_ALT_BSUX to support ECMAScript 6's \u{hhh}
construct.
32. Compile \p{Any} to be the same as . in DOTALL mode, so that it benefits
from auto-anchoring if \p{Any}* starts a pattern.
33. Compile invalid UTF check in JIT test when only pcre32 is enabled.
34. For some time now, CMake has been warning about the setting of policy
CMP0026 to "OLD" in CmakeLists.txt, and hinting that the feature might be
removed in a future version. A request for CMake expertise on the list produced
no result, so I have now hacked CMakeLists.txt along the lines of some changes
I found on the Internet. The new code no longer needs the policy setting, and
it appears to work fine on Linux.
35. Setting --enable-jit=auto for an out-of-tree build failed because the
source directory wasn't in the search path for AC_TRY_COMPILE always. Patch
from Ross Burton.
36. Disable SSE2 JIT optimizations in x86 CPUs when SSE2 is not available.
Patch by Guillem Jover.
37. Changed expressions such as 1<<10 to 1u<<10 in many places because compiler
warnings were reported.
38. Using the clang compiler with sanitizing options causes runtime complaints
about truncation for statements such as x = ~x when x is an 8-bit value; it
seems to compute ~x as a 32-bit value. Changing such statements to x = 255 ^ x
gets rid of the warnings. There were also two missing casts in pcre2test.
Version 10.32 10-September-2018
-------------------------------
1. When matching using the the REG_STARTEND feature of the POSIX API with a
non-zero starting offset, unset capturing groups with lower numbers than a
group that did capture something were not being correctly returned as "unset"
(that is, with offset values of -1).
2. When matching using the POSIX API, pcre2test used to omit listing unset
groups altogether. Now it shows those that come before any actual captures as
"<unset>", as happens for non-POSIX matching.
3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
whatever the build configuration was. It now correctly says "\R matches all
Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
specified. Similarly, running "pcre2test -C bsr" never produced the result
ANY.
4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
multi-code-unit characters caused bad behaviour and possibly a crash. This
issue was fixed for other kinds of repeat in release 10.20 by change 19, but
repeating character classes were overlooked.
5. pcre2grep now supports the inclusion of binary zeros in patterns that are
read from files via the -f option.
6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.
7. Added --enable-jit=auto support to configure.ac.
8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
modes for the benefit of m68k, where pointers can be 16-bit aligned. The
dummies force 32-bit alignment and this ensures that the structure is a
multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
architectures, alignment requirements take care of this automatically.
9. When returning an error from pcre2_pattern_convert(), ensure the error
offset is set zero for early errors.
10. A number of patches for Windows support from Daniel Richard G:
(a) List of error numbers in Runtest.bat corrected (it was not the same as in
Runtest).
(b) pcre2grep snprintf() workaround as used elsewhere in the tree.
(c) Support for non-C99 snprintf() that returns -1 in the overflow case.
11. Minor tidy of pcre2_dfa_match() code.
12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
use the stack for local workspace and local ovectors. Instead, an initial block
of stack is reserved, but if this is insufficient, heap memory is used. The
heap limit parameter now applies to pcre2_dfa_match().
13. If a "find limits" test of DFA matching in pcre2test resulted in too many
matches for the ovector, no matches were displayed.
14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
EOF. The test looks to have come from a fuzzer.
15. If PCRE2 was built with a default match limit a lot greater than the
default default of 10 000 000, some JIT tests of the match limit no longer
failed. All such tests now set 10 000 000 as the upper limit.
16. Another Windows related patch for pcregrep to ensure that WIN32 is
undefined under Cygwin.
17. Test for the presence of stdint.h and inttypes.h in configure and CMake and