A grep-like tool which understands source code syntax and allows for manipulation in
addition to search.
Like grep, regular expressions are a core primitive. Unlike grep, additional
capabilities allow for higher precision, with options for manipulation. This allows
srgn to operate along dimensions regular expressions and IDE tooling (Rename all,
Find all references, ...) alone cannot, thus complementing them.
srgn is organized around actions to take (if any), acting only within precise,
optionally language grammar-aware scopes. In terms of existing tools, think of it as
a mix of
tr,
sed,
ripgrep and
tree-sitter, with a design goal of
simplicity: if you know regex and the basics of the language you are working with, you
are ready to go.
Tip
All code snippets displayed here are verified as part of unit tests
using the actual srgn binary. What is showcased here is guaranteed to work.
The simplest form works similar to tr:
$ echo 'Hello World!' | srgn '[wW]orld' 'there'
Hello there!Matches for the regular expression pattern '[wW]orld' (the scope) are replaced (the
action) by the second positional argument. Zero or more actions can be specified:
$ echo 'Hello World!' | srgn '[wW]orld' # zero actions: input returned unchanged
Hello World!
$ echo 'Hello World!' | srgn --upper '[wW]orld' 'you' # two actions
Hello YOU!Similarly, more than one scope can be specified: in addition to the regex pattern, a
language grammar-aware scope can be
given, which scopes to syntactical elements of source code (think, for example, "all
bodies of class definitions in Python"). If both are given, the regular expression
pattern is then only applied within that first scope. This enables search and
manipulation at precision not normally possible using plain regular expressions, and
serving a dimension different from tools such as Rename all in IDEs.
For example, consider this (pointless) Python source file:
"""Module for watching birds and their age."""
from dataclasses import dataclass
@dataclass
class Bird:
"""A bird!"""
name: str
age: int
def celebrate_birthday(self):
print("🎉")
self.age += 1
@classmethod
def from_egg(egg):
"""Create a bird from an egg."""
pass # No bird here yet!
def register_bird(bird: Bird, db: Db) -> None:
assert bird.age >= 0
with db.tx() as tx:
tx.insert(bird)which can be searched using:
$ cat birds.py | srgn --python 'class' 'age'
11: age: int
15: self.age += 1The string age was sought and found only within Python class definitions (and not,
for example, in function bodies such as register_bird). By default, this 'search mode'
also prints line numbers. Search mode is entered if no actions are specified, and a
language such as --python is given1—think of it like
'ripgrep but with syntactical language
elements'.
Searching can also be performed across
lines, for example to
find methods (aka def within class) lacking docstrings:
$ cat birds.py | srgn --python 'class' 'def .+:\n\s+[^"\s]{3}' # do not try this pattern at home
13: def celebrate_birthday(self):
14: print("🎉")Note how this does not surface either from_egg (has a docstring) or register_bird
(not a method, def outside class).
Language scopes themselves can be specified multiple times as well. For example, in the Rust snippet
pub enum Genre {
Rock(Subgenre),
Jazz,
}
const MOST_POPULAR_SUBGENRE: Subgenre = Subgenre::Something;
pub struct Musician {
name: String,
genres: Vec<Subgenre>,
}multiple items can be surgically drilled down into as
$ cat music.rs | srgn --rust 'pub-enum' --rust 'type-identifier' 'Subgenre' # AND'ed together
2: Rock(Subgenre),where only lines matching all criteria are returned, acting like a logical and
between all conditions. Note that conditions are evaluated left-to-right, precluding
some combinations from making sense: for example, searching for a class body inside
of doc-strings usually returns nothing. The inverse works as expected however:
$ cat birds.py | srgn --python 'class' --python 'doc-strings' # From earlier example
8: """A bird!"""
19: """Create a bird from an egg."""No docstrings outside class bodies are surfaced!
The -j flag changes this behavior: from intersecting left-to-right, to
running all queries independently and joining their results, allowing you to search
multiple ways at once:
$ cat birds.py | srgn -j --python 'comments' --python 'doc-strings' 'bird[^s]'
8: """A bird!"""
19: """Create a bird from an egg."""
20: pass # No bird here yet!If standard input is not given, srgn knows how to find relevant source files
automatically, for example in this repository:
$ srgn --python 'class' 'age'
docs/samples/birds
11: age: int
15: self.age += 1
docs/samples/birds.py
9: age: int
13: self.age += 1It recursively walks its current directory, finding files based on file
extensions and shebang lines, processing
at very high speed. For example, srgn --go strings '\d+' finds and prints all ~140,000
runs of digits in literal Go strings inside the Kubernetes
codebase
of ~3,000,000 lines of Go code within 3 seconds on 12 cores of M3. For more, see
below.
Scopes and actions can be combined almost arbitrarily. For example, consider this Python snippet (more languages are supported):
"""GNU module."""
def GNU_says_moo():
"""The GNU function -> say moo -> ✅"""
GNU = """
GNU
""" # the GNU...
print(GNU + " says moo") # ...says mooAn invocation of
cat gnu.py | srgn --titlecase --python 'doc-strings' '(?<!The )GNU ([a-z]+)' '$1: GNU 🐂 is not Unix'makes use of multiple scopes (language and regex pattern) and multiple actions (replacement and titlecasing). The result then reads
"""Module: GNU 🐂 Is Not Unix."""
def GNU_says_moo():
"""The GNU function -> say moo -> ✅"""
GNU = """
GNU
""" # the GNU...
print(GNU + " says moo") # ...says moowhere the changes are limited to:
- """GNU module."""
+ """Module: GNU 🐂 Is Not Unix."""
def GNU_says_moo():
"""The GNU -> say moo -> ✅"""which additionally demonstrates...
- fully Unicode-capable operation (🐂),
- advanced regex features (negative
lookbehind, excluding matches
not prefixed by
The), and - variable binding and substitution (
$1, which carries the contents captured by the first capturing regex group).
Warning
While srgn is in beta (major version 0), make sure to only (recursively) process
files you can safely restore.
Search mode does not overwrite files, so is always safe.
See below for the full help output of the tool.
Download a prebuilt binary from the releases.
This crate provides its binaries in a format
compatible
with cargo-binstall:
- Install the Rust toolchain
- Run
cargo install cargo-binstall(might take a while) - Run
cargo binstall srgn(couple seconds, as it downloads prebuilt binaries from GitHub)
These steps are guaranteed to work™, as they are tested in CI. They also work if no prebuilt binaries are available for your platform, as the tool will fall back to compiling from source.
A formula is available via:
brew install srgn
Available via unstable:
nix-shell -p srgn
Available via the AUR.
A port is available:
sudo port install srgn
All GitHub Actions runner
images come with cargo
preinstalled, and cargo-binstall provides a convenient GitHub
Action:
jobs:
srgn:
name: Install srgn in CI
# All three major OSes work
runs-on: ubuntu-latest
steps:
- uses: cargo-bins/cargo-binstall@main
- name: Install binary
run: >
cargo binstall
--no-confirm
srgn
- name: Use binary
run: srgn --versionThe above concludes in just 5 seconds
total, as no
compilation is required. For more context, see cargo-binstall's advise on
CI.
- Install the Rust toolchain
- A C compiler is required:
-
On Linux,
gccworks. -
On macOS, use
clang. -
On Windows, MSVC works.
Select "Desktop development with C++" on installation.
-
- Run
cargo install srgn
cargo add srgn
See here for more.
Various
shells
are supported for shell completion scripts. For example, append eval "$(srgn --completions zsh)" to ~/.zshrc for completions in ZSH.
The tool is designed around scopes and actions. Scopes narrow down the parts of the input to process. Actions then perform the processing. Generally, both scopes and actions are composable, so more than one of each may be passed. Both are optional (but taking no action is pointless); specifying no scope implies the entire input is in scope.
At the same time, there is considerable overlap with plain
tr: the tool is designed to have close correspondence in the most common use
cases, and only go beyond when needed.
The simplest action is replacement. It is specially accessed (as an argument, not an
option) for compatibility with tr, and general ergonomics. All other actions are
given as flags, or options should they take a value.
For example, simple, single-character replacements work as in tr:
$ echo 'Hello, World!' | srgn 'H' 'J'
Jello, World!The first argument is the scope (literal H in this case). Anything matched by it is
subject to processing (replacement by J, the second argument, in this case). However,
there is no direct concept of character classes as in tr. Instead, by
default, the scope is a regular expression pattern, so its
classes can be used to
similar effect:
$ echo 'Hello, World!' | srgn '[a-z]' '_'
H____, W____!The replacement occurs greedily across the entire match by default (note the UTS
character class,
reminiscent of tr's
[:alnum:]):
$ echo 'ghp_oHn0As3cr3T!!' | srgn 'ghp_[[:alnum:]]+' '*' # A GitHub token
*!!Advanced regex features are supported, for example lookarounds:
$ echo 'ghp_oHn0As3cr3T' | srgn '(?<=ghp_)[[:alnum:]]+' '*'
ghp_*Take care in using these safely, as advanced patterns come without certain safety and performance guarantees. If they aren't used, performance is not impacted.
The replacement is not limited to a single character. It can be any string, for example to fix this quote:
$ echo '"Using regex, I now have no issues."' | srgn 'no issues' '2 problems'
"Using regex, I now have 2 problems."The tool is fully Unicode-aware, with useful support for certain advanced character classes:
$ echo 'Mood: 🙂' | srgn '🙂' '😀'
Mood: 😀
$ echo 'Mood: 🤮🤒🤧🦠 :(' | srgn '\p{Emoji_Presentation}' '😷'
Mood: 😷😷😷😷 :(Replacements are aware of variables, which are made accessible for use through regex capture groups. Capture groups can be numbered, or optionally named. The zeroth capture group corresponds to the entire match.
$ echo 'Swap It' | srgn '(\w+) (\w+)' '$2 $1' # Regular, numbered
It Swap
$ echo 'Swap It' | srgn '(\w+) (\w+)' '$2 $1$1$1' # Use as many times as you'd like
It SwapSwapSwap
$ echo 'Call +1-206-555-0100!' | srgn 'Call (\+?\d\-\d{3}\-\d{3}\-\d{4}).+' 'The phone number in "$0" is: $1.' # Variable `0` is the entire match
The phone number in "Call +1-206-555-0100!" is: +1-206-555-0100.A more advanced use case is, for example, code refactoring using named capture groups (perhaps you can come up with a more useful one...):
$ echo 'let x = 3;' | srgn 'let (?<var>[a-z]+) = (?<expr>.+);' 'const $var$var = $expr + $expr;'
const xx = 3 + 3;As in bash, use curly braces to disambiguate variables from immediately adjacent content:
$ echo '12' | srgn '(\d)(\d)' '$2${1}1'
211
$ echo '12' | srgn '(\d)(\d)' '$2$11' # will fail (`11` is unknown)
$ echo '12' | srgn '(\d)(\d)' '$2${11' # will fail (brace was not closed)Seeing how the replacement is merely a static string, its usefulness is limited. This is
where tr's secret sauce
ordinarily comes into play: using its character classes, which are valid in the second
position as well, neatly translating from members of the first to the second. Here,
those classes are instead regexes, and only valid in first position (the scope). A
regular expression being a state machine, it is impossible to match onto a 'list of
characters', which in tr is the second (optional) argument. That concept is out the
window, and its flexibility lost.
Instead, the offered actions, all of them fixed, are used. A peek at the most
common use cases for tr reveals that the provided set of
actions covers virtually all of them! Feel free to file an issue if your use case is not
covered.
Onto the next action.
Removes whatever is found from the input. Same flag name as in tr.
$ echo 'Hello, World!' | srgn -d '(H|W|!)'
ello, orldNote
As the default scope is to match the entire input, it is an error to specify deletion without a scope.
Squeezes repeats of characters matching the scope into single occurrences. Same flag
name as in tr.
$ echo 'Helloooo Woooorld!!!' | srgn -s '(o|!)'
Hello World!If a character class is passed, all members of that class are squeezed into whatever class member was encountered first:
$ echo 'The number is: 3490834' | srgn -s '\d'
The number is: 3Greediness in matching is not modified, so take care:
$ echo 'Winter is coming... 🌞🌞🌞' | srgn -s '🌞+'
Winter is coming... 🌞🌞🌞Note
The pattern matched the entire run of suns, so there's nothing to squeeze. Summer prevails.
Invert greediness if the use case calls for it:
$ echo 'Winter is coming... 🌞🌞🌞' | srgn -s '🌞+?' '☃️'
Winter is coming... ☃️Note
Again, as with deletion, specifying squeezing without an explicit scope is an error. Otherwise, the entire input is squeezed.
A good chunk of tr usage falls into this category. It's
very straightforward.
$ echo 'Hello, World!' | srgn --lower
hello, world!
$ echo 'Hello, World!' | srgn --upper
HELLO, WORLD!
$ echo 'hello, world!' | srgn --titlecase
Hello, World!Decomposes input according to Normalization Form D, and then discards code points of the Mark category (see examples). That roughly means: take fancy character, rip off dangly bits, throw those away.
$ echo 'Naïve jalapeño ärgert mgła' | srgn -d '\P{ASCII}' # Naive approach
Nave jalapeo rgert mga
$ echo 'Naïve jalapeño ärgert mgła' | srgn --normalize # Normalize is smarter
Naive jalapeno argert mgłaNotice how mgła is out of scope for NFD, as it is "atomic" and thus not decomposable
(at least that's what ChatGPT whispers in my ear).
This action replaces multi-character, ASCII symbols with appropriate single-code point, native Unicode counterparts.
$ echo '(A --> B) != C --- obviously' | srgn --symbols
(A ⟶ B) ≠ C — obviouslyAlternatively, if you're only interested in math, make use of scoping:
$ echo 'A <= B --- More is--obviously--possible' | srgn --symbols '<='
A ≤ B --- More is--obviously--possibleAs there is a 1:1 correspondence between an ASCII symbol and its replacement, the effect is reversible2:
$ echo 'A ⇒ B' | srgn --symbols --invert
A => BThere is only a limited set of symbols supported as of right now, but more can be added.
This action replaces alternative spellings of German special characters (ae, oe, ue, ss) with their native versions (ä, ö, ü, ß)3.
$ echo 'Gruess Gott, Neueroeffnungen, Poeten und Abenteuergruetze!' | srgn --german
Grüß Gott, Neueröffnungen, Poeten und Abenteuergrütze!This action is based on a word list (compile without
german feature if this bloats your binary too much). Note the following features about
the above example:
- empty scope and replacement: the entire input will be processed, and no replacement is performed
Poetenremained as-is, instead of being naively and mistakenly converted toPöten- as a (compound) word,
Abenteuergrützeis not going to be found in any reasonable word list, but was handled properly nonetheless - while part of a compound word,
Abenteuerremained as-is as well, instead of being incorrectly converted toAbenteür - lastly,
Neueroeffnungensneakily forms aueelement neither constituent word (neu,Eröffnungen) possesses, but is still processed correctly (despite the mismatched casings as well)
On request, replacements may be forced, as is potentially useful for names:
$ echo 'Frau Loetter steht ueber der Mauer.' | srgn --german-naive '(?<=Frau )\w+'
Frau Lötter steht ueber der Mauer.Through positive lookahead, nothing but the salutation was scoped and therefore changed.
Mauer correctly remained as-is, but ueber was not processed. A second pass fixes
this:
$ echo 'Frau Loetter steht ueber der Mauer.' | srgn --german-naive '(?<=Frau )\w+' | srgn --german
Frau Lötter steht über der Mauer.Note
Options and flags pertaining to some "parent" are prefixed with their parent's name,
and will imply their parent when given, such that the latter does not need to be
passed explicitly. That's why --german-naive is named as it is, and --german
needn't be passed.
This behavior might change once clap supports subcommand
chaining.
Some branches are undecidable for this modest tool, as it operates without language
context. For example, both Busse (busses) and Buße (penance) are legal words. By
default, replacements are greedily performed if legal (that's the whole
point of srgn,
after all), but there's a flag for toggling this behavior:
$ echo 'Busse und Geluebte 🙏' | srgn --german
Buße und Gelübte 🙏
$ echo 'Busse 🚌 und Fussgaenger 🚶♀️' | srgn --german-prefer-original
Busse 🚌 und Fußgänger 🚶♀️Most actions are composable, unless doing so were nonsensical (like for deletion). Their order of application is fixed, so the order of the flags given has no influence (piping multiple runs is an alternative, if needed). Replacements always occur first. Generally, the CLI is designed to prevent misuse and surprises: it prefers crashing to doing something unexpected (which is subjective, of course). Note that lots of combinations are technically possible, but might yield nonsensical results.
Combining actions might look like:
$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu
KOEFFIZIENTEN ≠ BRÜCKEN...A more narrow scope can be specified, and will apply to all actions equally:
$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu '\b\w{1,8}\b'
Koeffizienten != BRÜCKEN...The word boundaries are
required as otherwise Koeffizienten is matched as Koeffizi and enten. Note how the
trailing periods cannot be, for example, squeezed. The required scope of \. would
interfere with the given one. Regular piping solves this:
$ echo 'Koeffizienten != Bruecken...' | srgn -Sgu '\b\w{1,8}\b' | srgn -s '\.'
Koeffizienten != BRÜCKEN.Note: regex escaping (\.) can be circumvent using literal scoping.
The specially treated replacement action is also composable:
$ echo 'Mooood: 🤮🤒🤧🦠!!!' | srgn -s '\p{Emoji}' '😷'
Mooood: 😷!!!Emojis are first all replaced, then squeezed. Notice how nothing else is squeezed.
Scopes are the second driving concept to srgn. In the default case, the main scope is
a regular expression. The actions section showcased this use case in some
detail, so it's not repeated here. It is given as a first positional argument.
srgn extends this through prepared, language grammar-aware scopes, made possible
through the excellent tree-sitter
library. It offers a
queries feature,
which works much like pattern matching against a tree data
structure.
srgn comes bundled with a handful of the most useful of these queries. Through its
discoverable API (either as a library or via CLI, srgn --help), one
can learn of the supported languages and available, prepared queries. Each supported
language comes with an escape hatch, allowing you to run your own, custom ad-hoc
queries. The hatch comes in the form of --lang-query <S EXPRESSION>, where lang is a
language such as python. See below for more on this advanced topic.
Note
Language scopes are applied first, so whatever regex aka main scope you pass, it operates on each matched language construct individually.
This section shows examples for some of the prepared queries.
As part of a large refactor (say, after an acquisition), imagine all imports of a specific package needed renaming:
import math
from pathlib import Path
import good_company.infra
import good_company.aws.auth as aws_auth
from good_company.util.iter import dedupe
from good_company.shopping.cart import * # Ok but don't do this at home!
good_company = "good_company" # good_companyAt the same time, a move to src/
layout
is desired. Achieve this move with:
cat imports.py | srgn --python 'imports' '^good_company' 'src.better_company'which will yield
import math
from pathlib import Path
import src.better_company.infra
import src.better_company.aws.auth as aws_auth
from src.better_company.util.iter import dedupe
from src.better_company.shopping.cart import * # Ok but don't do this at home!
good_company = "good_company" # good_companyNote how the last line remains untouched by this particular operation. To run across
many files, see the files option.
Similar import-related edits are supported for other languages as well, for example Rust:
use std::collections::HashMap;
use good_company::infra;
use good_company::aws::auth as aws_auth;
use good_company::util::iter::dedupe;
use good_company::shopping::cart::*;
good_company = "good_company"; // good_companywhich, using
cat imports.rs | srgn --rust 'uses' '^good_company' 'better_company'becomes
use std::collections::HashMap;
use better_company::infra;
use better_company::aws::auth as aws_auth;
use better_company::util::iter::dedupe;
use better_company::shopping::cart::*;
good_company = "good_company"; // good_companyPerhaps you're using a system of TODO notes in comments:
class TODOApp {
// TODO app for writing TODO lists
addTodo(todo: TODO): void {
// TODO: everything, actually 🤷♀️
}
}and usually assign people to each note. It's possible to automate assigning yourself to every unassigned note (lucky you!) using
cat todo.ts | srgn --typescript 'comments' 'TODO(?=:)' 'TODO(@poorguy)'which in this case gives
class TODOApp {
// TODO app for writing TODO lists
addTodo(todo: TODO): void {
// TODO(@poorguy): everything, actually 🤷♀️
}
}Notice the positive lookahead of
(?=:), ensuring an actual TODO note is hit (TODO:). Otherwise, the other TODOs
mentioned around the comments would be matched as well.
Say there's code making liberal use of print:
def print_money():
"""Let's print money 💸."""
amount = 32
print("Got here.")
print_more = lambda s: print(f"Printed {s}")
print_more(23) # print the stuff
print_money()
print("Done.")and a move to logging is desired.
That's fully automated by a call of
cat money.py | srgn --python 'function-calls' '^print$' 'logging.info'yielding
def print_money():
"""Let's print money 💸."""
amount = 32
logging.info("Got here.")
print_more = lambda s: logging.info(f"Printed {s}")
print_more(23) # print the stuff
print_money()
logging.info("Done.")Note
Note the anchors: print_more is
a function call as well, but ^print$ ensures it's not matched.
The regular expression applies after grammar scoping, so operates entirely within the already-scoped context.
Overdone, comments can turn into smells. If not tended to, they might very well start lying:
using System.Linq;
public class UserService
{
private readonly AppDbContext _dbContext;
/// <summary>
/// Initializes a new instance of the <see cref="FileService"/> class.
/// </summary>
/// <param name="dbContext">The configuration for manipulating text.</param>
public UserService(AppDbContext dbContext)
{
_dbContext /* the logging context */ = dbContext;
}
/// <summary>
/// Uploads a file to the server.
/// </summary>
// Method to log users out of the system
public void DoWork()
{
_dbContext.Database.EnsureCreated(); // Ensure the database schema is deleted
_dbContext.Users.Add(new User /* the car */ { Name = "Alice" });
/* Begin reading file */
_dbContext.SaveChanges();
var user = _dbContext.Users.Where(/* fetch products */ u => u.Name == "Alice").FirstOrDefault();
/// Delete all records before proceeding
if (user /* the product */ != null)
{
System.Console.WriteLine($"Found user with ID: {user.Id}");
}
}
}So, should you count purging comments among your fetishes, more power to you:
cat UserService.cs | srgn --csharp 'comments' -d '.*' | srgn -d '[[:blank:]]+\n'The result is a tidy, yet taciturn:
using System.Linq;
public class UserService
{
private readonly AppDbContext _dbContext;
public UserService(AppDbContext dbContext)
{
_dbContext = dbContext;
}
public void DoWork()
{
_dbContext.Database.EnsureCreated();
_dbContext.Users.Add(new User { Name = "Alice" });
_dbContext.SaveChanges();
var user = _dbContext.Users.Where( u => u.Name == "Alice").FirstOrDefault();
if (user != null)
{
System.Console.WriteLine($"Found user with ID: {user.Id}");
}
}
}Note how all
different
sorts of
comments were identified and removed. The second pass removes all leftover dangling
lines ([:blank:] is tabs and
spaces).
Note
When deleting (-d), for reasons of safety and sanity, a scope is required.
Say you'd like to upgrade the instance size you're using:
data "aws_ec2_instance_type" "tiny" {
instance_type = "t2.micro"
}
resource "aws_instance" "main" {
ami = "ami-022f20bb44daf4c86"
instance_type = data.aws_ec2_instance_type.tiny.instance_type
}with
cat ec2.tf | srgn --hcl 'strings' '^t2\.(\w+)$' 't3.$1' | srgn --hcl 'data-names' 'tiny' 'small'will give
data "aws_ec2_instance_type" "small" {
instance_type = "t3.micro"
}
resource "aws_instance" "main" {
ami = "ami-022f20bb44daf4c86"
instance_type = data.aws_ec2_instance_type.small.instance_type
}Custom queries allow you to create ad-hoc scopes. These might be useful, for example, to create small, ad-hoc, tailor-made linters, for example to catch code such as:
if x:
return left
else:
return rightwith an invocation of
cat cond.py | srgn --python-query '(if_statement consequence: (block (return_statement (identifier))) alternative: (else_clause body: (block (return_statement (identifier))))) @cond' --fail-any # will failto hint that the code can be more idiomatically rewritten as return left if x else right. Another example, this one in Go, is ensuring sensitive fields are not
serialized:
package main
type User struct {
Name string `json:"name"`
Token string `json:"token"`
}which can be caught as:
cat sensitive.go | srgn --go-query '(field_declaration name: (field_identifier) @name tag: (raw_string_literal) @tag (#match? @name "[tT]oken") (#not-eq? @tag "`json:\"-\"`"))' --fail-any # will failOccassionally, parts of a match need to be ignored, for example when no suitable
tree-sitter node type is available. For example, say we'd like to replace the error
with wrong inside the string of the macro body:
fn wrong() {
let wrong = "wrong";
error!("This went error");
}Let's assume there's a node type for matching entire macros (macro_invocation) and
one to match macro names (((macro_invocation macro: (identifier) @name))), but
none to match macro contents (this is wrong, tree-sitter offers this in the form of
token_tree, but let's imagine...). To match just "This went error", the entire macro
would need to be matched, with the name part ignored. Any capture name starting with
_SRGN_IGNORE will provide just that:
cat wrong.rs | srgn --rust-query '((macro_invocation macro: (identifier) @_SRGN_IGNORE_name) @macro)' 'error' 'wrong'fn wrong() {
let wrong = "wrong";
error!("This went wrong");
}If it weren't ignored, the result would read wrong!("This went wrong");.
These matching expressions are a mouthful. A couple resources exist for getting started with your own queries:
-
the great official playground for interactive use, which makes developing queries a breeze. For example, the above Go sample looks like:
-
How to write a linter using tree-sitter in an hour, a great introduction to the topic in general
-
the official
tree-sitterCLI -
using
srgnwith high verbosity (-vvvv) is supposed to grant detailed insights into what's happening to your input, including a representation of the parsed tree
Use the --glob option to run against multiple files, in-place. This option accepts a
glob pattern. The glob is
processed within srgn: it must be quoted to prevent premature shell interpretation.
The --glob option takes precedence over the heuristics of language scoping. For
example,
$ srgn --go 'comments' --glob 'tests/langs/go/fizz*.go' '\w+'
tests/langs/go/fizzbuzz.go
5:// fizzBuzz prints the numbers from 1 to a specified limit.
6:// For multiples of 3, it prints "Fizz" instead of the number,
7:// for multiples of 5, it prints "Buzz", and for multiples of both 3 and 5,
8:// it prints "FizzBuzz".
25: // Run the FizzBuzz function for numbers from 1 to 100finds only what's matched by the (narrow) glob, even though --go queries by themselves
would match much more.
srgn will process results fully parallel, using all available threads. For example,
450k lines of Python are processed in about a second, altering
over 1000 lines across a couple hundred files:
Run the benchmarks too see performance for your own system.
After all scopes are applied, it might turn out no matches were found. The default behavior is to silently succeed:
$ echo 'Some input...' | srgn --delete '\d'
Some input...The output matches the specification: all digits are removed. There just happened to be none. No matter how many actions are applied, the input is returned unprocessed once this situation is detected. Hence, no unnecessary work is done.
One might prefer receiving explicit feedback (exit code other than zero) on failure:
echo 'Some input...' | srgn --delete --fail-none '\d' # will failThe inverse scenario is also supported: failing if anything matched. This is useful for checks (for example, in CI) against "undesirable" content. This works much like a custom, ad-hoc linter.
Take for example "old-style" Python code, where type hints are not yet surfaced to the syntax-level:
def square(a):
"""Squares a number.
:param a: The number (type: int or float)
"""
return a**2This style can be checked against and "forbidden" using:
cat oldtyping.py | srgn --python 'doc-strings' --fail-any 'param.+type' # will failThis causes whatever was passed as the regex scope to be interpreted literally. Useful for scopes containing lots of special characters that otherwise would need to be escaped:
$ echo 'stuff...' | srgn -d --literal-string '.'
stuffFor reference, the full help output with all available options is given below. As with all other snippets, the output is validated for correctness as part of unit tests. Checkout git tags to view help output of specific versions.
$ srgn --help
A grep-like tool which understands source code syntax and allows for manipulation in
addition to search
Usage: srgn [OPTIONS] [SCOPE] [REPLACEMENT]
Arguments:
[SCOPE]
Scope to apply to, as a regular expression pattern.
If string literal mode is requested, will be interpreted as a literal
string.
Actions will apply their transformations within this scope only.
The default is the global scope, matching the entire input. Where that
default is meaningless or dangerous (e.g., deletion), this argument is
required.
[default: .*]
Options:
--completions <SHELL>
Print shell completions for the given shell.
[possible values: bash, elvish, fish, powershell, zsh]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
Composable Actions:
-u, --upper
Uppercase anything in scope.
[env: UPPER=]
-l, --lower
Lowercase anything in scope.
[env: LOWER=]
-t, --titlecase
Titlecase anything in scope.
[env: TITLECASE=]
-n, --normalize
Normalize (Normalization Form D) anything in scope, and throw away marks.
[env: NORMALIZE=]
-g, --german
Perform substitutions on German words, such as 'Abenteuergruesse' to
'Abenteuergrüße', for anything in scope.
ASCII spellings for Umlauts (ae, oe, ue) and Eszett (ss) are replaced by
their respective native Unicode (ä, ö, ü, ß).
Arbitrary compound words are supported.
Words legally containing alternative spellings are not modified.
Words require correct spelling to be detected.
-S, --symbols
Perform substitutions on symbols, such as '!=' to '≠', '->' to '→', on
anything in scope.
Helps translate 'ASCII art' into native Unicode representations.
[REPLACEMENT]
Replace anything in scope with this value.
Variables are supported: if a regex pattern was used for scoping and
captured content in named or numbered capture groups, access these in the
replacement value using `$1` etc. for numbered, `$NAME` etc. for named
capture groups.
This action is specially treated as a positional argument for ergonomics and
compatibility with `tr`.
If given, will run before any other action.
[env: REPLACE=]
Standalone Actions (only usable alone):
-d, --delete
Delete anything in scope.
Cannot be used with any other action: there is no point in deleting and
performing any other processing. Sibling actions would either receive empty
input or have their work wiped.
-s, --squeeze
Squeeze consecutive occurrences of scope into one.
[env: SQUEEZE=]
[aliases: squeeze-repeats]
Options (global):
-G, --glob <GLOB>
Glob of files to work on (instead of reading stdin).
If actions are applied, they overwrite files in-place.
For supported glob syntax, see:
<https://docs.rs/glob/0.3.1/glob/struct.Pattern.html>
Names of processed files are written to stdout.
--fail-no-files
Fail if working on files (e.g. globbing is requested) but none are found.
Processing no files is not an error condition in itself, but might be an
unexpected outcome in some contexts. This flag makes the condition explicit.
-i, --invert
Undo the effects of passed actions, where applicable.
Requires a 1:1 mapping between replacements and original, which is currently
available only for:
- symbols: '≠' <-> '!=' etc.
Other actions:
- german: inverting e.g. 'Ä' is ambiguous (can be 'Ae' or 'AE')
- upper, lower, deletion, squeeze: inversion is impossible as information is
lost
These may still be passed, but will be ignored for inversion and applied
normally.
[env: INVERT=]
-L, --literal-string
Do not interpret the scope as a regex. Instead, interpret it as a literal
string. Will require a scope to be passed.
[env: LITERAL_STRING=]
--fail-any
If anything at all is found to be in scope, fail.
The default is to continue processing normally.
--fail-none
If nothing is found to be in scope, fail.
The default is to return the input unchanged (without failure).
-j, --join-language-scopes
Join (logical 'OR') multiple language scopes, instead of intersecting them.
The default when multiple language scopes are given is to intersect their
scopes, left to right. For example, `--go func --go strings` will first
scope down to `func` bodies, then look for strings only within those. This
flag instead joins (in the set logic sense) all scopes. The example would
then scope any `func` bodies, and any strings, anywhere. Language scopers
can then also be given in any order.
No effect if only a single language scope is given. Also does not affect
non-language scopers (regex pattern etc.), which always intersect.
-H, --hidden
Do not ignore hidden files and directories.
--gitignored
Do not ignore `.gitignore`d files and directories.
--sorted
Process files in lexicographically sorted order, by file path.
In search mode, this emits results in sorted order. Otherwise, it processes
files in sorted order.
Sorted processing disables parallel processing.
--threads <THREADS>
Number of threads to run processing on, when working with files.
If not specified, will default to available parallelism. Set to 1 for
sequential, deterministic (but not sorted) output.
-v, --verbose...
Increase log verbosity level.
The base log level to use is read from the `RUST_LOG` environment variable
(if unspecified, defaults to 'error'), and increased according to the number
of times this flag is given, maxing out at 'trace' verbosity.
Language scopes:
--csharp <CSHARP>
Scope C# code using a prepared query.
[env: CSHARP=]
Possible values:
- comments: Comments (including XML, inline, doc comments)
- strings: Strings (incl. verbatim, interpolated; incl. quotes,
except for interpolated)
- usings: `using` directives (including periods)
- struct: `struct` definitions (in their entirety)
- enum: `enum` definitions (in their entirety)
- interface: `interface` definitions (in their entirety)
- class: `class` definitions (in their entirety)
- method: Method definitions (in their entirety)
- variable-declaration: Variable declarations (in their entirety)
- property: Property definitions (in their entirety)
- constructor: Constructor definitions (in their entirety)
- destructor: Destructor definitions (in their entirety)
- field: Field definitions on types (in their entirety)
- attribute: Attribute names
- identifier: Identifier names
--csharp-query <TREE-SITTER-QUERY>
Scope C# code using a custom tree-sitter query.
[env: CSHARP_QUERY=]
--go <GO>
Scope Go code using a prepared query.
[env: GO=]
Possible values:
- comments: Comments (single- and multi-line)
- strings: Strings (interpreted and raw; excluding struct tags)
- imports: Imports
- type-def: Type definitions
- type-alias: Type alias assignments
- struct: `struct` type definitions
- interface: `interface` type definitions
- const: `const` specifications
- var: `var` specifications
- func: `func` definitions
- method: Method `func` definitions (`func (recv Recv) SomeFunc()`)
- free-func: Free `func` definitions (`func SomeFunc()`)
- init-func: `func init()` definitions
- type-params: Type parameters (generics)
- defer: `defer` blocks
- select: `select` blocks
- go: `go` blocks
- switch: `switch` blocks
- labeled: Labeled statements
- goto: `goto` statements
- struct-tags: Struct tags
--go-query <TREE-SITTER-QUERY>
Scope Go code using a custom tree-sitter query.
[env: GO_QUERY=]
--hcl <HCL>
Scope HashiCorp Configuration Language code using a prepared query.
[env: HCL=]
Possible values:
- variable: `variable` blocks (in their entirety)
- resource: `resource` blocks (in their entirety)
- data: `data` blocks (in their entirety)
- output: `output` blocks (in their entirety)
- provider: `provider` blocks (in their entirety)
- terraform: `terraform` blocks (in their entirety)
- locals: `locals` blocks (in their entirety)
- module: `module` blocks (in their entirety)
- variables: Variable declarations and usages
- resource-names: `resource` name declarations and usages
- resource-types: `resource` type declarations and usages
- data-names: `data` name declarations and usages
- data-sources: `data` source declarations and usages
- comments: Comments
- strings: Literal strings
--hcl-query <TREE-SITTER-QUERY>
Scope HashiCorp Configuration Language code using a custom tree-sitter query.
[env: HCL_QUERY=]
--python <PYTHON>
Scope Python code using a prepared query.
[env: PYTHON=]
Possible values:
- comments: Comments
- strings: Strings (raw, byte, f-strings; interpolation not
included)
- imports: Module names in imports (incl. periods; excl.
`import`/`from`/`as`/`*`)
- doc-strings: Docstrings (not including multi-line strings)
- function-names: Function names, at the definition site
- function-calls: Function calls
- class: Class definitions (in their entirety)
- def: Function definitions (*all* `def` block in their
entirety)
- async-def: Async function definitions (*all* `async def` block in
their entirety)
- methods: Function definitions inside `class` bodies
- class-methods: Function definitions decorated as `classmethod` (excl.
the decorator)
- static-methods: Function definitions decorated as `staticmethod` (excl.
the decorator)
- with: `with` blocks (in their entirety)
- try: `try` blocks (in their entirety)
- lambda: `lambda` statements (in their entirety)
- globals: Global, i.e. module-level variables
- variable-identifiers: Identifiers for variables (left-hand side of
assignments)
- types: Types in type hints
--python-query <TREE-SITTER-QUERY>
Scope Python code using a custom tree-sitter query.
[env: PYTHON_QUERY=]
--rust <RUST>
Scope Rust code using a prepared query.
[env: RUST=]
Possible values:
- comments: Comments (line and block styles; excluding doc comments;
comment chars incl.)
- doc-comments: Doc comments (comment chars included)
- uses: Use statements (paths only; excl. `use`/`as`/`*`)
- strings: Strings (regular, raw, byte; includes interpolation parts in
format strings!)
- attribute: Attributes like `#[attr]`
- struct: `struct` definitions
- priv-struct: `struct` definitions not marked `pub`
- pub-struct: `struct` definitions marked `pub`
- pub-crate-struct: `struct` definitions marked `pub(crate)`
- pub-self-struct: `struct` definitions marked `pub(self)`
- pub-super-struct: `struct` definitions marked `pub(super)`
- enum: `enum` definitions
- priv-enum: `enum` definitions not marked `pub`
- pub-enum: `enum` definitions marked `pub`
- pub-crate-enum: `enum` definitions marked `pub(crate)`
- pub-self-enum: `enum` definitions marked `pub(self)`
- pub-super-enum: `enum` definitions marked `pub(super)`
- enum-variant: Variant members of `enum` definitions
- fn: Function definitions
- impl-fn: Function definitions inside `impl` blocks (associated
functions/methods)
- priv-fn: Function definitions not marked `pub`
- pub-fn: Function definitions marked `pub`
- pub-crate-fn: Function definitions marked `pub(crate)`
- pub-self-fn: Function definitions marked `pub(self)`
- pub-super-fn: Function definitions marked `pub(super)`
- const-fn: Function definitions marked `const`
- async-fn: Function definitions marked `async`
- unsafe-fn: Function definitions marked `unsafe`
- extern-fn: Function definitions marked `extern`
- test-fn: Function definitions with attributes containing `test`
(`#[test]`, `#[rstest]`, ...)
- trait: `trait` definitions
- impl: `impl` blocks
- impl-type: `impl` blocks for types (`impl SomeType {}`)
- impl-trait: `impl` blocks for traits on types (`impl SomeTrait for
SomeType {}`)
- mod: `mod` blocks
- mod-tests: `mod tests` blocks
- type-def: Type definitions (`struct`, `enum`, `union`)
- identifier: Identifiers
- type-identifier: Identifiers for types
- closure: Closure definitions
--rust-query <TREE-SITTER-QUERY>
Scope Rust code using a custom tree-sitter query.
[env: RUST_QUERY=]
--typescript <TYPESCRIPT>
Scope TypeScript code using a prepared query.
[env: TYPESCRIPT=]
Possible values:
- comments: Comments
- strings: Strings (literal, template)
- imports: Imports (module specifiers)
- function: Any `function` definitions
- async-function: `async function` definitions
- sync-function: Non-`async function` definitions
- method: Method definitions
- constructor: `constructor` method definitions
- class: `class` definitions
- enum: `enum` definitions
- interface: `interface` definitions
- try-catch: `try`/`catch`/`finally` blocks
- var-decl: Variable declarations (`let`, `const`, `var`)
- let: `let` variable declarations
- const: `const` variable declarations
- var: `var` variable declarations
- type-params: Type (generic) parameters
- type-alias: Type alias declarations
- namespace: `namespace` blocks
- export: `export` blocks
--typescript-query <TREE-SITTER-QUERY>
Scope TypeScript code using a custom tree-sitter query.
[env: TYPESCRIPT_QUERY=]
Options (german):
--german-prefer-original
When some original version and its replacement are equally legal, prefer the
original and do not modify.
For example, "Busse" (original) and "Buße" (replacement) are equally legal
words: by default, the tool would prefer the latter.
[env: GERMAN_PREFER_ORIGINAL=]
--german-naive
Always perform any possible replacement ('ae' -> 'ä', 'ss' -> 'ß', etc.),
regardless of legality of the resulting word
Useful for names, which are otherwise not modifiable as they do not occur in
dictionaries. Called 'naive' as this does not perform legal checks.
[env: GERMAN_NAIVE=]While this tool is CLI-first, it is library-very-close-second, and library usage is treated as a first-class citizen just the same. See the library documentation for more, library-specific details.
Note that the binary takes precedence though, which with the crate currently being both a library and binary, creates problems. This might be fixed in the future.
Note: these apply to the entire repository, including the binary.
The code is currently structured as (color indicates coverage):
Hover over the rectangles for file names.
To see how to build, refer to compiling from source. Otherwise, refer to the guidelines.
An unordered list of similar tools you might be interested in.
- GritQL (very similar)
ast-grep(very similar)- Semgrep
pssfastmodprefactorgrep-astambersdripgrepripgrep-structured- tree-sitter CLI
gron- Ruleguard (quite different, but useful for custom linting)
- Coccinelle for Rust
srgn is inspired by tr, and in its simplest form behaves similarly, but not
identically. In theory, tr is quite flexible. In practice, it is commonly used mainly
across a couple specific tasks. Next to its two positional arguments ('arrays of
characters'), one finds four flags:
-c,-C,--complement: complement the first array-d,--delete: delete characters in the first first array-s,--squeeze-repeats: squeeze repeats of characters in the first array-t,--truncate-set1: truncate the first array to the length of the second
In srgn, these are implemented as follows:
- is not available directly as an option; instead, negation of regular expression
classes can be used (e.g.,
[^a-z]), to much more potent, flexible and well-known effect - available (via regex)
- available (via regex)
- not available: it's inapplicable to regular expressions, not commonly used and, if used, often misused
To show how uses of tr found in the wild can translate to srgn, consider the
following section.
The following sections are the approximate categories much of tr usage falls into.
They were found using GitHub's code search. The corresponding
queries are given. Results are from the first page of results at the time. The code
samples are links to their respective sources.
As the stdin isn't known (usually dynamic), some representative samples are used and the tool is exercised on those.
Making inputs safe for use as identifiers, for example as variable names.
-
Translates to:
$ echo 'some-variable? 🤔' | srgn '[^[:alnum:]_\n]' '_' some_variable___
Similar examples are:
-
Translates to:
$ echo 'some variablê' | srgn '[^[:alnum:]]' '_' some__variabl_
-
Translates to:
$ echo '🙂 hellö???' | srgn -s '[^[:alnum:]]' '-' -hell-
Translates a single, literal character to another, for example to clean newlines.
-
Translates to:
$ echo 'x86_64 arm64 i386' | srgn ' ' ';' x86_64;arm64;i386
Similar examples are:
-
Translates to:
$ echo '3.12.1' | srgn --literal-string '.' '\n' # Escape sequence works 3 12 1 $ echo '3.12.1' | srgn '\.' '\n' # Escape regex otherwise 3 12 1
-
Translates to:
$ echo -ne 'Some\nMulti\nLine\nText' | srgn --literal-string '\n' ',' Some,Multi,Line,Text
If escape sequences remain uninterpreted (
echo -E, the default), the scope's escape sequence will need to be turned into a literal\andnas well, as it is otherwise interpreted by the tool as a newline:$ echo -nE 'Some\nMulti\nLine\nText' | srgn --literal-string '\\n' ',' Some,Multi,Line,Text
Similar examples are:
Very useful to remove whole categories in one fell swoop.
-
tr -d '[:punct:]'which they describe as:Omit all punctuation characters
translates to:
$ echo 'Lots... of... punctuation, man.' | srgn -d '[[:punct:]]' Lots of punctuation man
Lots of use cases also call for inverting, then removing a character class.
-
Translates to:
$ echo 'i RLY love LOWERCASING everything!' | srgn -d '[^[:lower:]]' iloveeverything
-
Translates to:
$ echo 'All0wed ??? 💥' | srgn -d '[^[:alnum:]]' All0wed
-
Translates to:
$ echo '{"id": 34987, "name": "Harold"}' | srgn -d '[^[:digit:]]' 34987
Identical to replacing them with the empty string.
-
Translates to:
$ echo '1632485561.123456' | srgn -d '\.' # Unix timestamp 1632485561123456
Similar examples are:
-
Translates to:
$ echo -e 'DOS-Style\r\n\r\nLines' | srgn -d '\r\n' DOS-StyleLines
Similar examples are:
Remove repeated whitespace, as it often occurs when slicing and dicing text.
-
Translates to:
$ echo 'Lots of space !' | srgn -s '[[:space:]]' # Single space stays Lots of space !
Similar examples are:
tr -s " "tr -s [:blank:](blankis\tand space)tr -s(no argument: this will error out; presumably space was meant)tr -s ' 'tr -s ' 'tr -s '[:space:]'tr -s ' '
-
tr -s ' ' '\n'(squeeze, then replace)Translates to:
$ echo '1969-12-28 13:37:45Z' | srgn -s ' ' 'T' # ISO8601 1969-12-28T13:37:45Z
-
Translates to:
$ echo -e '/usr/local/sbin \t /usr/local/bin' | srgn -s '[[:blank:]]' ':' /usr/local/sbin:/usr/local/bin
A straightforward use case. Upper- and lowercase are often used.
-
tr A-Z a-z(lowercasing)Translates to:
$ echo 'WHY ARE WE YELLING?' | srgn --lower why are we yelling?
Notice the default scope. It can be refined to lowercase only long words, for example:
$ echo 'WHY ARE WE YELLING?' | srgn --lower '\b\w{,3}\b' why are we YELLING?
Similar examples are:
-
tr '[a-z]' '[A-Z]'(uppercasing)Translates to:
$ echo 'why are we not yelling?' | srgn --upper WHY ARE WE NOT YELLING?
Similar examples are:
This project is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Footnotes
-
With zero actions and no language scoping provided,
srgnbecomes 'useless', and other tools such as ripgrep are much more suitable. That's why an error is emitted and input is returned unchanged. ↩ -
Currently, reversibility is not possible for any other action. For example, lowercasing is not the inverse of uppercasing. Information is lost, so it cannot be undone. Structure (imagine mixed case) was lost. Something something entropy... ↩
-
Why is such a bizzare, unrelated feature included? As usual, historical reasons. The original, core version of
srgnwas merely a Rust rewrite of a previous, existing tool, which was only concerned with the German feature.srgnthen grew from there. ↩