RFC: Macro reform

[kmcallister]

Rendered view

[kmcallister]

cc @jbclements @sfackler @mitsuhiko @paulstansifer @nick29581 @bjz @huonw @reem @blaenk @aturon @glaebhoerl

[huonw]

As I stated on #440, I don't like taking this nice name for this imperfect macro system. It seems unfortunate to me that one would have to explicitly opt-in to some new improved macro system if/when we create a parallel one that incorporates the backwards-incompatible improvements to macro_rules!. (That is, it is extremely unfortunate that the obvious macro! foo { ... } invocation would give the deprecated form.)

[kmcallister]

Let's rename trait to crappy_trait because we don't have HKT yet.

[kmcallister]

Okay, maybe that's an unfair comparison because HKT are supposed to be back-compat. But every macro system we ever have in Rust is going to be "imperfect" in some way. At some point we have to live with the imperfections and improve them in non-breaking ways, which is what 1.0 is all about, but why back away from that idea just in the case of macros?

The practical outcome of your alternative is that Rust programmers are stuck with the ugly name macro_rules! for perhaps years and the unstable alternative will be something not descriptively different, like macro_case!.

[zwarich]

With macros, we already know before 1.0 that the ways we want to improve them in the future are not backwards compatible, and the stabilization of the current system of macros is itself a compromise. I don't know of another Rust language feature in a similar situation, but maybe there is an example?

[kmcallister]

Basically I see it as

• force most people to write macro_rules!, for years
• force some people to write #![feature(improved_macros)] or #[version(2)] macro!, if and when such back-incompat enhancements are developed

[kmcallister]

Long term, improved_macros are what people are going to use,

But long term, they'll be on Rust 2.x or Rust 3.x, which gives us another chance to make breaking changes, especially fairly superficial ones like naming.

I'm not sure how long 1.x is meant to last but it would seem relevant to this discussion.

[paulstansifer]

One thing to note: some of the ways that macros will change, like hygiene, will be (a) a change to the entire system, preventing backwards compatibility from even being possible (i.e., define_a_macro_in_the_old_way! won't recover the old semantics), and (b) make things almost strictly better (i.e., the old semantics mainly gets in people's way).

However, the reason it is called macro_rules! is that it's not able to define all kinds of macros, only rule-based ones. A construct named macro! should be capable of defining procedural macros, and I think that the arrival of procedural macros is an appropriate time to make the name change.

[huonw]

I was expecting that we would have two essentially independent hygiene systems inside the compiler (i.e. identifiers would go from storing just ctxt: SyntaxContext to storing ctxt: SyntaxContext, improved_ctxt: SyntaxContext); it's pretty sad, but it seems to me to be the only way we can possibly aim to improve hygiene without breaking backwards compatibility.

[nrc]

@huonw this is horrifying! I was expecting that we would be able to have a single hygiene system in rustc and any macro system could do what they like - i.e., rustc::resolve would expect all lexical context ids to be 'correct'. macro! would output correct lexical contexts, macro_rules! would output lexical contexts that correspond to the current level of hygiene, etc. I'm not entirely sure if this is possible, because I don't exactly know how hygiene is currently broken.

As far as I see it, being able to do this backwards compatibly after 1.0 is the most important issue for macros pre-1.0.

[huonw]

@nick29581 I agree 100% that it is not nice, but I believe that it is an existence proof that we can have a macro system with improved hygiene post-1.0 without being forced to break what we already have. (Your approach does sound better, if it works.)

[mitchmindtree]

Great, it's awesome to see an RFC on this!

The lack of name-spacing is something that I find particularly jarring about macros - it makes them seem a little wild and pollutant.

One thing I'd love to see is macros abiding by the module system. @huonw explained to me here that macro expansion runs prior to "the module system", which makes sense! However I don't see why we still can't use the same syntax for cohesiveness with the rest of the language.

Perhaps we could add a ! to help the compiler distinguish between the 'macro modules' and the regular modules. i.e.

pub mod! module { ... }

use! module::my_macro;

I feel like this would be a lot cleaner and a lot less ad-hoc than using #[visible] or #[export] or #[macro_escape] and trying to keep track of the purpose of all these attributes.

[ftxqxd]

Although this RFC gives ways of avoiding many of the current problems with macros, the system is still not ideal:

• Ideally, $crate would not be necessary, and instead, name resolution would occur at the time the macro was defined. (This problem is mentioned in the RFC under the ‘Alternatives’ section.) That would mean that if I defined a macro in a module bar of crate foo, and referred to some function baz, I wouldn’t need to write out $crate::bar::baz, I’d just write baz, and if I used the macro from another crate, it would expand to foo::bar::baz. This would make macros work in the same way as everything else, which is what I would expect.
• Preferably, macro scoping would also work in the same way as other items. That would mean that macros would be scoped by modules, not crates as the RFC suggests, or globally as we have today. Ideally, in a similar fashion, use statements would be extended to support macros (e.g., use std::vec::vec!;), and pub would work on macros too (although maybe with a #[pub] attribute instead). These changes would presumably require reshuffling the order of macro expansion and resolution significantly. It could also potentially allow public macros to refer to private macros, like how public functions can refer to private functions. This would probably require such a private macro to be exported in the crate metadata (or wherever, I have no idea where macros are stored in crates), but access to it be disallowed from other crates at compile-time.

Those are some pretty big problems in my opinion, and although this RFC provides workarounds for those problems ($crate, #[visible], etc.), it doesn’t actually give any solutions for these problems (e.g., it’s still possible to create a macro that doesn’t work in the crate it’s defined in). I understand that these problems would be potentially extremely difficult to resolve (no pun intended), but I think that avoiding ever solving these problems on technical grounds is the wrong way to go (unless it turns out to be NP-complete or otherwise impractical).

That said, this RFC is still a huge step forward. If we have to stabilise macros for 1.0, and if solving the problems above is unlikely to happen for the 1.0 timeframe due to implementation difficulty, then I think that this RFC is at the very least a good solution to go with until, if ever, something better is implemented. If something better does come, then we can make them opt-in ‘version 2 macros’ or something until the next breaking release, when the old macros can be removed.

[kmcallister]

pub mod! module { ... }
use! module::my_macro;

Weird, but I like it at a glance.

[kmcallister]

Although this RFC gives ways of avoiding many of the current problems with macros, the system is still not ideal

sigh

The point of this is not to make an "ideal" system, it's to find improvements we can feasibly land in 1 or 2 months. I really don't know how I could have been more clear about that.

I understand that these problems would be potentially extremely difficult to resolve, but I think that avoiding ever solving these problems on technical grounds is the wrong way to go

Noooooooooobody is suggesting ignoring these problems forever. Why are you strawman-ing my RFC?

[blaenk]

While most of us (I think including @P1start) are aware of the compromise that has been chosen of stabilizing the macro system we have now for 1.0, and then improving (perhaps redoing) it afterward, I think that that context is not evident in this RFC, so I think it's understandable that people may see it and come to the conclusion that this is a proposal for the final macro system. From that perspective, I don't think @P1start was intending to "straw man" your RFC.

[ftxqxd]

The point of this is not to make an "ideal" system, it's to find improvements we can feasibly make in 1 or 2 months. I really don't know how I could have been more clear about that.

Yes, I understand that, and as I noted at the end of my comment I agree that this is a good solution for now. I was merely attempting to outline what this RFC does not address. (I understand that this RFC is not supposed to fully address those problems due to the time restrictions on 1.0.) On reflection I see that I did not come across that way.

Noooooooooobody is suggesting ignoring these problems forever. Why are you strawmanning my RFC?

Sorry, I didn’t mean to come across that way. I was merely trying to point out that doing so would be a bad idea (without implying that this RFC stated that), but I suppose that was unnecessary.

[blaenk]

I think it would be useful to make it explicit in the RFC that the macro system is intended to be reformed more comprehensively after 1.0, and that this is intended as a stop-gap measure (perhaps in preparation for that). This is alluded to but not explicitly mentioned.

[kmcallister]

Fair enough, I guess that was more clear in the various discussion threads than in the actual RFC.

[Manishearth]

Bikeshed: I would prefer something like $_crate, or $$crate.

[brendanzab]

Reading quickly over this, these seem like reasonable stop gap measures, but it might be helpful to describe possible post 1.0 transition paths towards the ultimate RustMacroSystem™.

[petrochenkov]

Minor syntactic detail: is it possible to change the delimiter between macro_rules! arms from ; to , to make it more in line with other language constructs, match in particular?

[bill-myers]

How about using the normal namespacing mechanism for macros too?

That means that macro expansion and name resolution need to be done in the same pass, but it avoids all the namespace reinvention in this RFC.

[brendanzab]

@bill-myers I would ultimately love to see something like use std::io::println! and pub macro! panic { ... }, but I don't see it happening by 1.0. We need something ugly to stabilize on so that we can implement the pretty thing later.

[brendanzab]

Could this expand to something like pub use collections::vec!; post 1.0?

[glaebhoerl]

This is basically what I suggested on reddit, but how hard would it be to:

• scope macros under crates (but not modules!), so that one would/could write e.g. std::println! (but not std::io::println!),
• import them likewise, e.g. use std::{ println! };,
• at the same time, restrict exported macros to living at the crate root, so that even after we grow a more general ability to scope macros under modules (not just crates), all existing code will retain the same meaning (i.e. so that the 1.0 functionality is a strict subset of future functionality)? In other words, paths and use could only refer to macros directly under crate roots, but this is not a problem because exported macros could not be defined anywhere else.

Importantly, I think that just because these use the same syntax as name resolution and import for items, that doesn't mean they also necessarily have to be implemented using the same mechanisms. We could do some kind of ugly hacks under the hood where e.g. (I'm 99% oblivious to the current implementation, so I'm probably getting the details wrong) libsyntax and the macro expansion code looks at use statements and grabs all the things that end with ! and "imports" them one way (e.g. like use_macros does in the RFC), and then later on librustc also looks at the use statements and grabs all the things which don't end with ! and imports them another way (i.e. the current item import mechanism). As long as it observably appears to be doing the same thing for the restricted case of crate-level scoping (which is the only case we would allow for macros), then clients (user code) don't really have to care. (We could do the same kind of under-the-hood hackery to have pub macro! foo { ... } denote an exported macro; again, same syntax, different (but observably indistinguishable) underlying mechanism.)

(We could even require use statements to consist of either only-macros or only-non-macros, if this helps, so that e.g. use std::{Option, println!}; would be illegal, and you would have to split it up into use std::Option; use std::println!;, but this is still a strict subset of future functionality, forwards-compatible, and not a separate, parallel syntax.)

The third bullet point above corresponds to an alternative mentioned in the RFC. I personally think that having to "pollute" lib.rs with macro definitions is a price that would be well worth paying in exchange for strict forwards-compatibility. (If all of this is actually feasible.)

[kmcallister]

That means that macro expansion and name resolution need to be done in the same pass

@bill-myers: Yeah, this is a massive change to rustc and not really in scope for 1.0.

[kmcallister]

@glaebhoerl:

scope macros under crates (but not modules!), so that one would/could write e.g. std::println!

I don't think we want to actually support invoking macros that way, because it will involve changes to the parsing of exprs, items, patterns, etc. in ways that probably have unintended negative consequences. My plan was to use macros and always invoke them unqualified. Maybe that's what you had in mind, too.

import them likewise, e.g. use std::{ println! };... just because these use the same syntax as name resolution and import for items, that doesn't mean they also necessarily have to be implemented using the same mechanisms

That syntax would be nice, but I hesitate to introduce two different mechanisms (in two different crates even!) that interpret use for different purposes. Macro use will have all kinds of restrictions that make it confusingly different from regular use. Right now macros are controlled entirely with "meta-level" syntax, which feels natural to me.

I might warm to this approach, though, because the syntax would be really nice and should be back-compat with whatever better approach we take post-1.0.

Also I'm not sure about using ! to distinguish macros from ordinary items in use. It's more part of the invocation syntax for macros than part of the name. Other syntax extensions, such as deriving-like item decorators, don't use it. Perhaps we should change that (I always thought it was weird that synexts aren't distinguished from built-in attributes), or perhaps ignore it for now and don't implement scope for non-macro-like synexts at all, since there's no stable way to define your own. shrug

I personally think that having to "pollute" lib.rs with macro definitions is a price that would be well worth paying in exchange for strict forwards-compatibility.

I see the virtue in this, but I'd like to think of ways to avoid it. If we're already introducing a syntax for macro re-export, perhaps we should make it work on modules within a crate, and then require that exported macros are either defined or re-exported at the crate root.

We could also just say "all exported macros also appear at the crate root, forever" which makes things less nice in the future, but not hugely so.

[glaebhoerl]

Yeah, I guess you could go even further and disallow paths-to-macros entirely, and require use to be able to refer to them. That's still a strict subset of potential future capabilities (an even smaller one).

[huonw]

I don't think we want to actually support invoking macros that way, because it will involve changes to the parsing of exprs, items, patterns, etc. in ways that probably have unintended negative consequences. My plan was to use macros and always invoke them unqualified. Maybe that's what you had in mind, too.

FWIW, we already support parsing x::y!() as an expression, it is just an error in the expander itself (other valid macro locations do not currently support parsing that).

[comex]

(One issue with x::y!() is what happens if, in a macro body, you want to do some_mod:: concat_idents!($a, $b)? i.e. look up the concatenated name in some_mod. At the moment this doesn't work, like most uses of concat_idents!, but it should.)

[paulstansifer]

@bill-myers @bjz I guess that this isn't all that relevant, because this RFC is about near-future stuff, but I'd like to clear something up:

The normal namespacing mechanism requires being able to see the whole module system at once. But the macro resolver must be able to run when only the code above (higher up in the tree or earlier in the file) has been expanded. Therefore, the macro system needs its own resolver with its own resolution rules. To minimize confusion, I think it would be prudent to write something like macro_use or syntax_use instead so that people don't expect impossible semantics.

[paulstansifer]

@comex While concat_idents! ought to go away, if we made a good identifier concatenation mechanism (https://mail.mozilla.org/pipermail/rust-dev/2013-February/003170.html), using it in the middle of a path will probably Just Work.

[paulstansifer]

In general, I like this RFC a lot. It proposes to provide more power and flexibility to the user, and it should soften the transition to the Way Things Will Work In The Beautiful Future.

I have a couple of quibbles, though.

• I don't think macro_rules! should be renamed as long as it's unable to define procedural macros.
• More importantly, while $crate is a good solution for the time being, that I'm very worried that future generations of Rust users will be unable to name a syntax variable $crate for backwards-compatibility reasons. The need for it will eventually go away, but code using it needs to continue to work. I know $_crate is ugly, but it'd really help me sleep at night, as would allowing the user to write something like:

    macro_rules! vec(
      name_current_crate $my_crate
      ($($e:expr),*) => ({
        let mut temp = $my_crate::vec::Vec::new();
        //...

(As a bonus, name_current_crate is also more in keeping with the spirit of hygiene, for what it's worth.)

[Manishearth]

Regarding $crate or $$crate, I'm going to propose a mini rfc to disallow bindings of the $ or $$ type (haven't decided which) so that we can later use them for macro magic without breaking backcompat.

-----Original Message-----
From: "Paul Stansifer" notifications@github.com
Sent: ‎11/‎9/‎2014 9:28 AM
To: "rust-lang/rfcs" rfcs@noreply.github.com
Cc: "Manish Goregaokar" manishsmail@gmail.com
Subject: Re: [rfcs] RFC: Macro reform (#453)

In general, I like this RFC a lot. It proposes to provide more power and flexibility to the user, and it should soften the transition to the Way Things Will Work In The Beautiful Future.
I have a couple of quibbles, though.
I don't think macro_rules! should be renamed as long as it's unable to define procedural macros.
More importantly, while $crate is a good solution for the time being, that I'm very worried that future generations of Rust users will be unable to name a syntax variable $crate for backwards-compatibility reasons. The need for it will eventually go away, but code using it needs to continue to work. I know $_crate is ugly, but it'd really help me sleep at night, as would allowing the user to write something like:
macro_rules! vec(
name_current_crate $my_crate
($($e:expr),*) => ({
let mut temp = $my_crate::vec::Vec::new();
//...
(As a bonus, name_current_crate is also more in keeping with the spirit of hygiene, for what it's worth.)
—
Reply to this email directly or view it on GitHub.=

[paulstansifer]

@kmcallister I think keeping macro_rules! for the time being is nice because it dodges the trickiness of non-exclamation-pointed macro definitions and it minimizes churn. It also pretty much answers the rest of the questions (attributes are a more natural choice, though I guess keywords inside the invocation could also work).

[nikomatsakis]

I think there are a lot of good ideas here, but I think that this proposal is trying to do too much at once. The original vision here was to do a kind of "maximally minimal" set of extensions that can correct issues around macro scoping, import/export, and cross-crate macros. Trying to do more than that for 1.0 doesn't seem right to me; I'd rather tackle bigger redesigns when we have more space to implement and consider more radical changes.

In particular, I think we can separate out macro item sugar and the macro foo syntax for declaring macros. They are both fairly straightforward backwards compatible extensions. (For the record, I've wanted something like the "macro item sugar" you propose for a while, but I hope we can make it more accepting than what you've written out here, which is fairly limited. For example, I'd like to be able to write a for-loop sort of statement using this syntax. This is partly why I say that I think we should revisit this idea in an RFC of its own, so that we can try to suss out the full range of use cases and consider it more deliberately.)

I can see the desire to use use macro syntax for bringing macros into scope, but I do not like the design as written. The reason is that use macro (as well as the use of pub on macro items and so forth) is made to "look and feel" like the import/export/privacy of other Rust items, but in fact it acts very differently. Here are some examples:

1. A pub macro is nothing at all like a pub struct -- it causes the macro name to escape out into the rest of the crate, rather than affecting whether or not a macro can be imported and referenced. There is no equivalent mechanism for structs or other items.
2. The "path" in a view item like use macro foo::bar is not a normal path, and it can't be, since macro resolution occurs before normal name resolution. Presumably (this is not specified in the RFC, but I am guessing) it takes the form of crate_name::macro_name. This is surprising, because I expect to be able to put arbitrary paths there and to use pub use to re-export and so forth.
3. In any case, the idea of "crate name" is not a pre-existing concept in Rust anyhow. We don't have any rules for how to resolve a "crate name" as something different from a normal path; everything builds off of the module hierarchy. So you need to define some rules for how a "crate name" is mapped to a particular "extern crate" declaration -- are these declarations only at the local level?
4. The extern vs pub distinction still feels strange to me. It's not how our other visibility and export rules work, so having these new keywords (extern macro, extern use) is again kind of surprising and contrary to my expectations.

To be clear, I think that the underlying system you proposed is a reasonably good idea and seems like a simple extension to macros that will make them usable and easy to understand. However, I think that packing these semantics in "normal" syntax like pub macro and use macro foo::bar is a bad idea, because macros behave very differently from other things in the language. I find it confusing just reading the RFC -- my brain keeps trying to draw analogies to other things in the language that just do not hold.

I preferred the original attribute version, since that at least kept the macro syntax as clearly different and confined into the "meta world" of macros and attributes. I think this makes sense. If we want something that looks more like a use statement than a crate-level attribute, perhaps we could use macro syntax here, similar to macro_rules!: macro_use!(foo::bar).

That said, I would prefer to avoid the need to define name resolution rules for crate names, and hence think that perhaps we ought to consider attaching attributes to extern crate declarations to control what macros from that crate are brought into scope and how (i.e., module-local, or globally to the crate). Something like:

#[use_macros(foo, bar)] // use macros for this module and all submodules
extern crate some_crate;

#[use_macros(global=(foo, bar))] // use macros for the entire crate
extern crate some_crate;

[kmcallister]

The reason is that use macro (as well as the use of pub on macro items and so forth) is made to "look and feel" like the import/export/privacy of other Rust items, but in fact it acts very differently

I'm counting on people to learn, through documentation and error messages, that use macro is something different from ordinary use. Introducing the macro keyword is what changed my mind about repurposing pub and extern, too. That said, I do share most of your concerns and I think this is a difficult tradeoff.

The "path" in a view item like use macro foo::bar is not a normal path… So you need to define some rules for how a "crate name" is mapped to a particular "extern crate" declaration -- are these declarations only at the local level?

It would make sense if we refuse to import macros from an extern crate that isn't at the crate root, since that also breaks $crate. In that case, each relevant

extern crate "foo" as bar;

is introducing paths of the form ::bar::..., and we take bar as the crate name. There's a single environment of crate names for the whole crate, and it doesn't change during macro expansion. (If a macro expansion produces an extern crate view item, it will be inside another node like an ExprBlock and we can ignore it for these purposes. There's no way to make a macro that expands to a single view item, or the crate root module.)

Because of the difficulties presented by extern crate outside the crate root, I would especially like to avoid

#[use_macros(foo, bar)] // use macros for this module and all submodules
extern crate some_crate;

In particular, I think we can separate out macro item sugar and the macro foo syntax for declaring macros. They are both fairly straightforward backwards compatible extensions.

I'm fine with this, as long as we get another chance to address the macro_rules! name before 1.0. I don't think we lose much by committing to

<quals...> macro <ident> { <tts...> }

because the syntax for the definition body can change in backwards-compatible ways. For example, I described two ways to integrate procedural macros.

For that matter, an improved macro system could be implemented as a library providing an ItemModifier to be used on macro. I want to think more about this case and make sure it's really possible. The implementation of this RFC will include tests. See also rust-lang/rust#19083.

[Manishearth]

Note that use is already with two different behaviors. use Trait and use type/use variant/use function have very different behaviors.

(I'm contemplating an RfC that changes use Trait to import Trait or something to make this distinction clear, though I don't see it getting much support)

[kmcallister]

I removed some inessential parts of the proposal, and clarified the way that crate identifiers are used.

[kmcallister]

I updated this RFC to account for decisions made in the macro meeting last Thursday. Niko's proposal to make macro argument consumption more greedy should be a separate RFC, I think.

[sfackler]

Why would a syntax extension want to link itself in at runtime? It injects a runtime dependency on librustc, so most (all?) syntax extensions have a separate crate for runtime support (e.g. regex and regex_macros, or phf and phf_mac).

[kmcallister]

Yeah, I don't expect it to be a common situation. One example would be quasiquoting as a library. quote_foo!() expands to calls into syntax::ext::quote::rt::.... And there's no need for this to live in a separate crate from the quote macro, because the user of quasiquoting is also going to link librustc, or libsyntax anyway.

Basically it's a capability we have today with #[phase(plugin, link)] and I didn't want to throw it away. Though, the Registry API is unstable and we'd only be adding capabilities to it, so maybe we should throw this out for the time being and see if anyone complains. It might be good to nudge people towards doing thing the two-crates way.

[sfackler]

My gut says that #[phase(plugin, link)] is only ever used for imports of macro_rules plugins, so I'd vote for axing the functionality until people start complaining :).

[sfackler]

The current system's lexical order dependence is also a pain, but probably not worth worrying about before 1.0.

[kmcallister]

Yeah, I thought a bit about how to fix the lexical order thing, and I think it's pretty hard. You'd have to move away from the model where macro_rules! is a macro that defines a macro as a side effect.

It's not a huge annoyance in practice because

• people like putting project-wide macros in macros.rs that can be imported first thing in lib.rs, and also because
• macro foo! which expands to an invocation of bar! can be defined before bar!, as long as bar! is defined by the time foo! is used.

[sfackler]

I think it's doable - possibly by making expand a bit more like resolve, where it repeatedly folds the AST. In any case, it's annoying but not hugely so like you said :)

[paulstansifer]

@sfackler The current side-effect-based system has to be replaced by a proper scope system, but removing order dependence is not possible; see the second paragraph of this comment. If we completely broke evaluation order for the side-effects of procedural macros, we could make it resemble the behavior of resolve more, but I suspect that accurately recreating the behavior of resolve is completely impossible, because resolve was designed to see the entire tree at once, and macro resolution necessarily has to happen while parts of the tree are still unexpanded. (I don't know the exact behavior of resolve, but I remember that getting it right turned out to be surprisingly tricky and subtle!) The best approach (and the only one that I know of being used in any other language's macro system) is to expand macros in lexical order and to resolve macros against only already-expanded parts of the AST.

I recognize that this isn't really relevant at the moment, but judging by the number of times that I've heard this suggestion, I'm worried that macro-names-that-resolve-like-function-names is going wind in the conventional wisdom about the future of Rust's macro system, and stay there until someone tries to implement it.

[kmcallister]

I am more optimistic than @paulstansifer about resolve for macros, but I agree that it's not something we can or should count on ever having.

[nikomatsakis]

This RFC has been accepted and merged as https://github.com/rust-lang/rfcs/blob/master/text/0453-macro-reform.md. Thanks @kmcallister for your hard work in preparing it.