Decide on our long term/ideal goals for macros

[nrc]

Both macro_rules and syntax extensions, and other forms of syntactic fiddling. At the moment I don't have a good sense of what is intentional and what is accidental at the moment. I think these long term goals should inform our choice of what to do for 1.0.

cc @kmcallister @sfackler

[brendanzab]

cc @paulstansifer

[kmcallister]

cc @jbclements

[glaebhoerl]

see also #416

[jbclements]

I strongly agree that we need some consensus here. In particular, it seems like the macro system needs to be an ongoing project for a full-timer....

[paulstansifer]

Off the top of my head:

• Macros ought to be importable and exportable (and able to be referred to with paths, e.g., a::b::c::d!()), and interact with modules in a reasonable way.
• Procedural macros should be refined and made easy-to-use (and, when that happens, the name "syntax extension" can be dropped entirely, we can just call everything a macro)
• Hygiene should be fully implemented (currently, it only applies to let-bound names)
• The macro parser should be able to backtrack out of parsing Rust syntax, and the dreaded local ambiguity: multiple parsing options: errors should go away. macro parser should back out of failed nonterminal parsing rust#3232
• Macro-defining macros should work. macros should be parsed more lazily rust#6994 is one obstacle, but there are others

Less sweeping issues (the line between this and the above is somewhat fuzzy):

• Identifier concatenation should work (I believe that this best way to do this is through a special operator) Can't define a function name using concat_idents!() rust#12249
• macro_rules! foo{() => ()} should become macro_rules!{ foo!() => () }
• We need a way for macros to count the repetitions in a sequence
• We probably need some additional positions to allow macro invocations in (currently, only expression, statement, and item macros are allowed)
• The rule for when to treat a macro invocation as a statement or an expression should be refined.
• The macro parser probably could benefit from a few more features.

[paulstansifer]

I'm currently working on the macro parser issue; unfortunately, I'm also currently working on graduating and at least one other side project, so not much progress has been made recently.

[kmcallister]

currently, only expression, statement, and item macros are allowed

I added patterns a while back.

[paulstansifer]

@kmcallister Awesome; I'm sorry I forgot about that.

[nrc]

A few more (some of which might be plain wrong):

• Syntax extensions allowed in more places
• Separating the macro rules phase from the syntax extensions phase (since syntax extensions can change the module structure, it seems like it would be wrong to rely on module scoping for macros whilst the modules can change)
• allow forward references to macros
• get rid of macro_escape (we should not need it with proper import/export)
• hygiene for crates (name resolution should just work, like it should for modules)
• is name hiding hygienic? Should we support it? Could we have a lint for this?
• macro rules and syntax extensions should be somewhat pluggable - should be possible to switch them for something else or nothing at all, whilst still using the parser parts of libsyntax and the all of librustc

[paulstansifer]

@nick29581

• Syntax extensions in more places is the thing that @kmcallister mentioned having done for patterns.
• I think we should not phase-separate macro_rules macros from procedural macros: instead we should have a macro name resolution system that behaves consistently across all kinds of macros (like Scheme's does).
• I believe that we already have forward references, provided those references are inside a macro definition (i.e., one can define two mutually-recursive macros). We should not allow "top-level" forward references like a!() ; macro_rules a!{ ... }, because this, among other things, would interact poorly with procedural macros (it would reorder their side-effects).
• macro_escape should definitely go away when we have import/export.
• I don't understand exactly what you mean, but I agree that macros be accessible from other crates.
• Is name hiding the same thing as shadowing? I don't think that we need to lint-check whether a macro shadows names, but that's just because I don't have much of an intuition for when that would happen accidentally.
• Pluggable macro_rules (and pluggable procedural macros) should fall out of a proper implementation of procedural macros. In fact, making macro_rules into a procedural macro will be a good testcase for whether we got it right, because it ought to be straightforward when proper procedural (and macro-defining) macros are in place.

[reem]

Also, let the bike shedding begin, but I think macro_rules! should be renamed to macro!. The longer name adds minimal clarity and goes against Rust's short-keyword tendencies.

[kmcallister]

@reem: Me too; it's in the RFC I'm drafting.

[paulstansifer]

@reem I agree, provided that we decide to make macro! be used for defining both procedural and pattern-matching-rules-based macros. If not, we could have macro_rules! and macro_cases! (to mimic Scheme; I like reusing existing naming schemes, even if they're not optimal). I'm not dead set against having macro! and macro_procedural! (or whatever), but it seems less clear.

[reem]

I saw this idea brought up a while back, and it's certainly something we should consider for 1.0:

We could accomplish full macro hygiene by having all bindings other than those generated via input arguments to the macro be resolved in the lexical namespace the macro is defined in, rather than the one it is placed in.

This has certain drawbacks, like using a macro causing you to silently adopt several new dependencies, but is a huge step forward because it would provide total hygiene, whereas today we have all kinds of problems with std/core and things like deriving with Encodable, Decodable, and serialize.

Also, this was brought up on reddit but I'll repeat it here, but a stop-gap for full module namespacing of macros would be to namespace them under crates only and confine them to be define in lib.rs, as that would be forward compatible with full module namespacing in the future.

[kmcallister]

Isn't that really hard to implement, especially cross-crate?

[kmcallister]

@paulstansifer: I'm not sure what procedural macros will look like long-term, but it's plausible that they'll want a macro_rules!-like LHS, where the RHS is Rust code instead of transcription instructions, and that would fit nicely with using macro! for both.

I think macro_rules! and macro_cases! is super confusing. Not enough people use Scheme for it to be worth inheriting their naming mistakes.

[nrc]

@paulstansifer I thought the patterns thing was for macro_rules. Syntax extensions can only be places on items I think, and not all of them (e.g., last time I checked, you could put one on a function, but not a method).

Could you expand on how such a naming scheme would work please? I don't see how you can refer to a macro via a module path, when the set of modules can be arbitrarily changed by a syntax extension. Indeed, before running the syntax extensions, you don't even have a guarantee that the source will parse, only tokenise.

Name hiding is a real footgun when using macros since it violates the usual mutability rules of rust. E.g.,

let x = 5i; //immutable
foo!(x);
println!("{}", x);

name hiding means that x could have any value at all.

[nrc]

@reem that is my expectation for how macros should work. I would like to know why that is not the case - are there advantages to the current system? Or is it just the implementation effort?

[paulstansifer]

@reem That sounds pretty similar to the way that @jbclements made things work for let. The essence of the Dybvig hygiene algorithm is to distinguish syntax that "originates" inside a macro from syntax from the "outside" (or from any other macro).

[paulstansifer]

@kmcallister Yeah, the advantage is that we have a pretty big syntactic space to play with inside macro!. Perhaps a different kind of arrow would suffice? That would also allow the user to mix procedural and pattern-based definitions for different rules, which might be nice.

[paulstansifer]

@nick29581 Are you referring to attributes when you mentioned syntax extensions being placed on items? I haven't thought about attributes much; I probably owe them some study. I usually use "syntax extension" to refer to things like macro_rules! and file! along with Rust macros in general.

The rough shape of what I was imagining was that a::b::c::d!() would work, provided that the definition (or import) of a is above the current line, and thus has been fully expanded (the macro resolver is going to have to operate during macro expansion, but it gets to be a lot simpler than the normal Rust resolver because it doesn't have to deal with circular imports). Part of the reason that macro expansion goes outside-inwards and top-to-bottom is so that the macro expander will always have a coherent environment in which to resolve the next macro invocation (something we haven't fully taken advantage of yet).

As for name hiding, I see what you mean, but I'm worried that the lint might give false positives in cases where the user writes a chain like my_let!(x = ...); my_let!(x = ...);, etc. I dunno how common each possibility is.

[brendanzab]

For compiler plugins, I wonder if we need to have a more stable surface API for libsyntax, rather than forcing folks to delve into internal APIs that could change quite a bit as the compiler evolves.

[nrc]

@bjz we definitely do - that is part of why we can't stabilise them for 1.0

[nrc]

@paulstansifer yes, I have been using 'syntax extension' to mean procedural macro, as opposed to macro_rules macros and built in macros.

[nrc]

@paulstansifer so about the top-down expansion, that sounds like it would work, but also that it could be a bit surprising and unpredictable.It seems that we can either choose to go with top to bottom expansion and do procedural macros and macro_rules macros in the same phase, or allow forward references and separate the phases. The latter sounds better to me, since we should be able to spot any cycles and is more consistent with the rest of Rust. What are the advantages of going for the former arrangement? (I imagine it is more flexible, but I can't think of a use case off the top of my head. Are there expansion issues?)

[huonw]

Also, let the bike shedding begin, but I think macro_rules! should be renamed to macro!. The longer name adds minimal clarity and goes against Rust's short-keyword tendencies.

I disagree; if we are stabilising macro_rules! before it is really ready, we should leave it as macro_rules! and keep the macro! name for some better system that eventually appears. (It may turn out that some future version of macro_rules! is awesome, but we can make the decision about changing the name then.)

[reem]

I think the idea is to make it ready before we stabilize. I'm against stabilizing if we still think it's not a good system by 1.0, that's just inviting a world of pain.

[huonw]

@reem, see this blog post:

For macros, we will likely provide an alternative way to define macros (with better hygiene) at some later date, and will incrementally improve the “macro rules” feature until then. The 1.0 release will stabilize all current macro support, including import/export.

Efforts are to make macro_rules as good as possible before 1.0, but we aren't blocking on perfecting it.

[reem]

@huonw and see my response :P

The thing that most worries me about this and seems to me like the decision most guided by pressure is the stabilization of the macro system for imports and exports, as it is today, for 1.0.

The current system for importing and exporting macros is pretty terrible since it provides very little support for encapsulation, complex hygiene, and other pretty large problems that would be very hard to change after 1.0.

Primarily I'm extremely concerned that this means that macros will always be silently imported through crate boundaries with no name spacing forever, which would be a real shame.

Otherwise, this sounds like a reasonable plan forward given the constraints of today.

There are serious problems with macro_rules as it stands today, and I think it's crucial to the future of Rust that we solve a lot of those before we stabilize it for a very long time. It would also be a shame for rust 1.2 to include a new way to define macros and for us to have this huge macro_rules feature that needs to be maintained, updated, and worked-around even if it's deprecated.

[huonw]

@kmcallister's pending RFC and (partly) this issue are designed to clean up macro_rules into a state where it isn't entirely horrible. We cannot guarantee they will be perfect by 1.0 without risking the 1.0 release being seriously delayed. But some sort of namespacing is definitely one of the minimal required improvements.

The 'current' in the blog post refers to the features of macro_rules at 1.0, not the features now. Also, as the blog post states, there's no reason we cannot have two similar macro definition systems, e.g. the legacy 1.0 macro_rules! and some future macro! which makes all the backwards incompatible tweaks to macro_rules! (the former would be deprecated in favour of the latter, but would be retained to ensure backwards compatibility until a major release).

[kmcallister]

I don't want to pick the ugly name just because we want to use the better name later. The likely outcome is that we're stuck with the ugly name for a long time

I am pondering something like

#[version(2)]
macro! foo ( ... )

[blaenk]

I am definitely for changing the name to macro!, which is much nicer, but I think it would be completely negated by having to carry around an attribute to specify which macro type/version.

[kmcallister]

@blaenk: It's not negated, because it's still easier to keep track of version numbers rather than macro! vs macro_rules! vs good_macro! vs super_macro!.

Anyway I was imagining you would need this only when defining a macro that uses features not stabilized in the 1.x branch.

Another option is to include those in the crate-level feature gates and output metadata so that an importing crate can transcribe macros correctly.

[blaenk]

It's not negated, because it's still easier to keep track of version numbers rather than macro! vs macro_rules! vs good_macro! vs super_macro!.

I strongly disagree that it would be easier for people to have to remember the specific version number of the macro system which contains the features they want as opposed to a change in a perhaps descriptive name.

Anyway I was imagining you would need this only when defining a macro that uses features not stabilized in the 1.x branch.

This is ultimately what everyone would want to be doing though, right? Wouldn't this then be required from that point forward, forever (to avoid breaking compatibility)? Or is this implying that at some point in the future, such as 2.0, the new macro system would become the default?

Another option is to include those in the crate-level feature gates and output metadata so that an importing crate can transcribe macros correctly.

This sounds more reasonable to me from the perspective of someone writing macros. So this would be something like #[feature(new_macros)] and based on that it would know that the macro! used in that crate refers to the new macros?

btw kmc I'm glad this is finally getting looked at/discussed and that you're part of it.

[paulstansifer]

@nick29581 Well, partially it's because that expansion ordering is what we currently do, and it's the "standard" way for macro systems to work (which means that we already have an idea how that expansion strategy cooperates with hygiene and other language features). But it's also more internally predictable: instead of having to remember how a macro was defined to figure out how name resolution will work, all macro name resolution will behave the same way. This also means that defining macros will be fully pluggable: macro_rules! (or whatever it will be called then) will just be a normal macro that a user could have defined, without any magical phase-defying properties.

Another part of the reason is more subjective: macro definitions will be more natural to topographically sort because people use macros and functions differently. It's natural and appropriate for a programmer to write lots and lots of functions that are only invoked in one place: functions provide code organization. But macros have little or no code organization power that functions don't have. What macros are good at is abstraction, which means that there are typically many macro invocations per definition, so there's no drive to put macro definitions right next to the invocation, since there isn't just one invocation.

[ben0x539]

Can we add later phases for macro evaluation eventually so macros can operate on types and typed expressions, not just tokens?

[paulstansifer]

@ben0x539 It's an interesting thought, but I doubt that our typechecker would be able to cope with a not-fully-expanded program, and having a more conservative but totally consistent version of the typechecker seems like a big ask. And then there's the unintuitiveness of having the typechecker behave differently depending on when it's invoked, and having multiple different phases of macro expansion.

Perhaps it would be possible to have a type_of!() macro that calls out to the typechecker to determine the type of its argument, but I'm not sure where it would get its type environment from; it would probably be fairly limited.

[brendanzab]

The likely outcome is that we're stuck with the ugly name for a long time

Why would that be? We could always claim a stub to macro! for use in the future.

[glaebhoerl]

This work on "modular macros" in OCaml may be of interest.

[Diggsey]

Identifier concatenation/generation can be done without significantly complicating the rust parser by extending the existing substitution syntax ($prefexed names) to support macro invocations.

[burdges]

#1738 (comment) already proposes typeof(value) @paulstansifer

[burdges]

Just mentioned in #1738 (comment) that a name for the current function name might be useful, so maybe a macro like current_fn! or something.

[paulstansifer]

I think that typeof would be very useful! But getting that information
(accurately) at macro-expansion time would, I believe, be a major
undertaking, which would at least require redesigning the macro system (in
a non-backwards-compatible way), and also possibly doing major surgery on
the type system. It's a great idea for some other language, but it's way
outside the scope of Rust.

A limited typeof might be possible, but I'm skeptical of the value of
something that gives you approximately the right type.

[glaebhoerl]

It might be possible to provide something like declared_type_of! for module-levels items which already, by law, have a type signature written by the user, and just provides that signature as-is without typechecker involvement? (No idea about usefulness. But at least it seems like a clear separation.)

[tobia]

Procedural macros

I would like to bring to your attention the work of Chicken Scheme in this regard. I think they nailed it with their Explicit Renaming (opt-in hygiene) and Implicit Renaming (opt-out hygiene) procedural macro API. They are very simple to use and right to the point.

Of course, a great part of that simplicity lies in Lisp's quasi-quotation syntax, which is probably not going to work for a language like Rust. (Or is it?) But the idea of having both explicit and implicit hygiene, with resp. opt-in and opt-out method calls, is neat.

We need a way for macros to count the repetitions in a sequence

A specific syntax extension (such as count! in the same vein as stringify!) would probably be the cleanest solution. In the meantime, I posted a workaround with good performance.

The macro parser probably could benefit from a few more features.

$( )? would surely be useful. The number of times $( )* is used to match zero-or-one is embarrassing.

On a more serious note, tail-call optimization would go a long way towards making macro_rules! almost as powerful as procedural macros.

[steveklabnik]

I believe, with the Macros 2.0 RFC accepted, we can give this a close. Please let me know if that's wrong!

[tobia]

@steveklabnik The proliferation of names here is very confusing. Which RFC is the Macros 2.0?

[steveklabnik]

The proliferation of names here is very confusing.

I actually will be posting a blog post tomorrow about exactly this; I agree 100%

The RFC is here: https://github.com/rust-lang/rfcs/blob/master/text/1566-proc-macros.md

[strega-nil]

@steveklabnik proc-macros 2.0 and macros 2.0 are different beasts; I'm not sure that RFC solves this issue. We still (afaik) aren't sure what to do with macro!s 2.0.

[steveklabnik]

That RFC is in FCP right now: #1584

I assume that all the discussion that would have gone on in this issue would be on the actual RFC.