From ddcaeea7657c0bb7e509f3ca6d7cd10362ea729f Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 11:24:46 -0500 Subject: [PATCH 01/60] feat: Add skeleton --- text/0000-frontmatter.md | 97 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 text/0000-frontmatter.md diff --git a/text/0000-frontmatter.md b/text/0000-frontmatter.md new file mode 100644 index 00000000000..a2ab4c4c8a6 --- /dev/null +++ b/text/0000-frontmatter.md @@ -0,0 +1,97 @@ +- Feature Name: (fill me in with a unique ident, `my_awesome_feature`) +- Start Date: (fill me in with today's date, YYYY-MM-DD) +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +One paragraph explanation of the feature. + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected outcome? + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: + +- Introducing new named concepts. +- Explaining the feature largely in terms of examples. +- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. +- If applicable, provide sample error messages, deprecation warnings, or migration guidance. +- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. +- Discuss how this impacts the ability to read, understand, and maintain Rust code. Code is read and modified far more often than written; will the proposed feature make code easier to maintain? + +For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? +- If this is a language proposal, could this be done in a library or macro instead? Does the proposed change make Rust code easier or harder to read, understand, and maintain? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that rust sometimes intentionally diverges from common language features. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From 290cb97fd167ec7393adca852991860fc79c913d Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 12:17:09 -0500 Subject: [PATCH 02/60] feat: Initial frontmatter RFC --- text/0000-frontmatter.md | 387 ++++++++++++++++++++++++++++++++++----- 1 file changed, 337 insertions(+), 50 deletions(-) diff --git a/text/0000-frontmatter.md b/text/0000-frontmatter.md index a2ab4c4c8a6..37d2ede5e63 100644 --- a/text/0000-frontmatter.md +++ b/text/0000-frontmatter.md @@ -1,72 +1,370 @@ -- Feature Name: (fill me in with a unique ident, `my_awesome_feature`) -- Start Date: (fill me in with today's date, YYYY-MM-DD) +- Feature Name: `frontmatter` +- Start Date: 2023-09-26 - RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) # Summary [summary]: #summary -One paragraph explanation of the feature. +Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code](TODO): +````rust +#!/usr/bin/env cargo +```cargo +[dependencies] +clap = { version = "4.2", features = ["derive"] } +``` + +use clap::Parser; + +#[derive(Parser, Debug)] +#[clap(version)] +struct Args { + #[clap(short, long, help = "Path to config")] + config: Option, +} + +fn main() { + let args = Args::parse(); + println!("{:?}", args); +} +```` # Motivation [motivation]: #motivation -Why are we doing this? What use cases does it support? What is the expected outcome? +["cargo script"](TODO) is in need of a syntax for embedding manifests in source. +See that RFC for its motivations. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Explain the proposal as if it was already included in the language and you were teaching it to another Rust programmer. That generally means: +Static site generators use a frontmatter syntax to embed metadata in markdown files: +```md +--- +name: My Blog Post +--- + +## Stuff happened + +Hello world! +``` + +We are carrying this concept over to Rust with a twist: using code fences which +will be familiar to Rust developers who when documenting their code: +````rust +#!/usr/bin/env cargo +```cargo +[dependencies] +clap = { version = "4.2", features = ["derive"] } +``` + +use clap::Parser; + +#[derive(Parser, Debug)] +#[clap(version)] +struct Args { + #[clap(short, long, help = "Path to config")] + config: Option, +} -- Introducing new named concepts. -- Explaining the feature largely in terms of examples. -- Explaining how Rust programmers should *think* about the feature, and how it should impact the way they use Rust. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing Rust programmers and new Rust programmers. -- Discuss how this impacts the ability to read, understand, and maintain Rust code. Code is read and modified far more often than written; will the proposed feature make code easier to maintain? +fn main() { + let args = Args::parse(); + println!("{:?}", args); +} +```` -For implementation-oriented RFCs (e.g. for compiler internals), this section should focus on how compiler contributors should think about the change, and give examples of its concrete impact. For policy RFCs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. +As we work to better understand how tool authors will want to use frontmatter, we are restricting it to just the `cargo` infostring. +This means users will only be exposed to this within the concept of ["cargo script"](TODO). # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -This is the technical portion of the RFC. Explain the design in sufficient detail that: +When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a code fence: +- Must be immediately at the top (after shebang stripping), meaning no blank lines +- Opens with 3+ backticks and "cargo" followed by a newline +- All content is ignored until a matching number of backticks is found at the start of a line. It is an error to have anything come between the backticks and the newline. -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. - -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +As cargo will be the first step in the process to parse this, +the responsibility for high quality error messages will largely fall on cargo. # Drawbacks [drawbacks]: #drawbacks -Why should we *not* do this? +- A new concept for Rust syntax, adding to overall cognitive load +- Requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? -- If this is a language proposal, could this be done in a library or macro instead? Does the proposed change make Rust code easier or harder to read, understand, and maintain? +Within this solution, +we considered starting with only allowing this in the root `mod` (e.g. `main.rs`) +but decided to allow it in any file mostly for ease of implementation. +Like with Python, this allows any file in a package (with the correct deps and `mod`s) to be executed, allowing easier interacting experiences in verifying behavior. + +As for the hard-coded infostring used by cargo, that is a decision for [RFC TODO](TODO). + +## Syntax + +When choosing a syntax for embedding manifests in source files, our care-abouts are +- How obvious it is for new users when they see it +- How easy it is for newer users to remember it and type it out +- How machine editable it is for `cargo add` and friends +- Needs to be valid Rust code for quality of error messages, etc +- Leave the door open in case we want to reuse the syntax for embedded lockfiles +- Leave the door open for single-file `lib`s + +Benefits for the proposed solution are: +- Visually/syntactically lightweight +- Has parallels to ideas outside of Rust, building on external knowledge that might exist +- Easy for cargo to parse and modify + - Cargo needs a simple syntax subset to parse to ensure it can adequately report errors for the part it cares about but does not get in the way of rustc reporting the high quality errors it cares about +- In the future, this can be leveraged by other build systems or tools + +### Alternative 1: Doc-comment + +```rust +#!/usr/bin/env cargo + +//! ```cargo +//! [package] +//! edition = "2018" +//! ``` + +fn main() { +} +``` + +Benefits +- Parsers are available to make this work (e.g. `syn`, `pulldown-cmark`) +- Familiar syntax both to read and write. + - When discussing with a Rust author, it was pointed out many times people preface code with a comment specifying the dependencies ([example](https://github.com/dtolnay/prettyplease#example)), this is the same idea but reusable by cargo + - When discussing on forums, people expressed how they had never seen the syntax but instantly were able to understand it + +Downsides: +- When discussing with a a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch + - Having the explain the overloading of concepts to new users + - Unpredictable location (both the doc comment and the cargo code block within it) + - Visual clutter (where clutter is overwhelming already in Rust) +- Might be a bit complicated to do edits (translating between location within + `toml_edit` spans to the location within `syn` spans especially with different comment styles) +- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` +- Requires pulling in a full markdown parser to extract the manifest + - Incorrectly formatted markdown would lead to a missing manifest and confusing error messages at best or silent incorrect behavior at worse + +### Alternative 2: Macro + +```rust +#!/usr/bin/env cargo + +cargo! { +[package] +edition = "2018" +} + +fn main() { +} +``` +Benefits +- Parsers are available to make this work (e.g. `syn`) + +Downsides +- The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge +- A lot of tools/IDEs have problems in dealing with macros +- Free-form rust code makes it harder for cargo to make edits to the manifest +- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` + +### Alternative 3: Attribute + +```rust +#!/usr/bin/env cargo + +#![cargo(manifest = r#" +[package] +edition = "2018" +"#)] + +fn main() { +} +``` +- `cargo` could register this attribute or `rustc` could get a generic `metadata` attribute +- As an alternative, `manifest` could a less stringly-typed format but that + makes it harder for cargo to parse and edit, makes it harder for users to + migrate between single and multi-file packages, and makes it harder to transfer + knowledge and experience + +Benefits +- Parsers are available to make this work (e.g. `syn`) + +Downsides +- From talking to a teacher, users are more forgiving of not understanding the details for structure data in an unstructured format (doc comments / comments) but something that looks meaningful, they will want to understand it all requiring dealing with all of the concepts + - The attribute approach requires explaining multiple "advanced" topics: One teacher doesn't get to teaching any attributes until the second level in his crash course series and two teachers have found it difficult to teach people raw strings +- Attributes look "scary" (and they are in some respects for the hidden stuff they do) +- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` + + +### Alternative 4: Presentation Streams + +```rust +#!/usr/bin/env cargo + +fn main() { +} + +---Cargo.toml +[package] +edition = "2018" +``` +YAML allows several documents to be concatenated together variant +[presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream) +which might seem familiar as this is frequently used in static-site generators +for adding frontmatter to pages. +What if we extended Rust's syntax to allow something similar? + +Benefits +- Flexible for other content + +Downsides +- Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal +- Being new syntax, there would be a lot of details to work out, including + - How to delineate and label documents + - How to allow escaping to avoid conflicts with content in a documents + - Potentially an API for accessing the document from within Rust +- Unfamiliar, new syntax, unclear how it will work out for newer users + +### Alternative 5: Regular Comment + +Simple header: +```rust +#!/usr/bin/env cargo +/* Cargo.toml: +[package] +edition = "2018" +*/ + +fn main() { +} +``` + +HEREDOC: +```rust +#!/usr/bin/env cargo +/* Cargo.TOML >>> +[package] +edition = "2018" +<<< +*/ + +fn main() { +} +``` +The manifest can be a regular comment with a header. If we were to support +multiple types of content (manifest, lockfile), we could either use multiple +comments or HEREDOC. +This does not prescribe the exact syntax used or supported comments + +Benefits + +Downsides +- Unfamiliar syntax +- New style of structured comment for the ecosystem to support with potential + compatibility issues, likely requiring a new edition +- Assuming it can't be parsed with `syn` and either we need to write a + sufficiently compatible comment parser or pull in a much larger rust parser + to extract and update comments. + - Like with doc comments, this should map to an attribute and then we'd just start the MVP with that attribute + +### Alternative 6: Static-site generator frontmatter + +```rust +#!/usr/bin/env cargo +--- +[package] +edition = "2018" +--- + +fn main() { +} +``` +This is a subset/specialization of YAML presentation streams that mirrors people's experience with static site generators: +- The first line post-shebang-stripping is 3+ dashes, then capture all content until a matching pair of dashes on a dedicated line. This would be captured into a `#![frontmatter = ""]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it) +- Future evolution: Allow a markdown-like infostring on the frontmatter opening dashes to declare the format with `cargo` being the default +- Future evolution: Allow `frontmatter` attribute on any module + +Benefits +- Visually/syntactically lightweight +- Has parallels to ideas outside of Rust, building on external knowledge that might exist +- Easy for cargo to parse and modify +- Can be leveraged by buck2, meson, etc in the future + +Downsides +- Too general that people might abuse it +- We've extended the frontmatter syntax, undoing some of the "familiarity" benefit +- People are used to YAML going in frontmatter (though some systems allow other syntaxes) +- Doesn't feel very rust-like + +### Alternative 7: Extended Shebang + +````rust +#!/usr/bin/env cargo +# ```cargo +# [dependencies] +# foo = "1.2.3" +# ``` + +fn main() {} +```` +This is a variation on other options that ties itself closer to the shebang syntax. +The hope would be that we could get buy-in from other languages. +- The first line post-shebang-stripping is a hash plus 3+ backticks, then capture all content until a matching pair of backticks on a dedicated line. This would be captured into a `#![frontmatter(info = "cargo", content = "..."]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it). Each content line must be indented to at least the same level as the first backtick. +- Future evolution: Allow `cargo` being the default `info` string +- Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` +- Future evolution: Allow `frontmatter` attribute on any module + +Benefits +- Visually connected to the shebang +- Has parallels to ideas outside of Rust, building on external knowledge that might exist +- Easy for cargo to parse and modify +- Can be leveraged by buck2, meson, etc in the future +- Maybe we can get others on board with this syntax + +Downsides +- More syntactically heavy than the frontmatter solution + - Visually + - More work to type it out / copy-paste + - More to get wrong # Prior art [prior-art]: #prior-art -Discuss prior art, both the good and the bad, in relation to this proposal. -A few examples of what this can include are: +See also [Single-file scripts that download their dependencies](https://dbohdan.com/scripts-with-dependencies) +which enumerates the syntax used by different tools. -- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? -- For community proposals: Is this done by some other community and what were their experiences with it? -- For other teams: What lessons can we learn from what other communities have done here? -- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. +## YAML frontmatter -This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. -If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. +As a specialization of [YAML presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream), static site generators use frontmatter to embed YAML at the top of files. +Other systems have extended this for non-YAML use, like [zola using `+++` for TOML](https://www.getzola.org/documentation/content/page/#front-matter). -Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. -Please also take into consideration that rust sometimes intentionally diverges from common language features. +### Proposed Python syntax + +Currently the draft [PEP 723](https://peps.python.org/pep-0723/) proposes allowing begin/end markers inside of source code: + +```python +# /// pyproject +# [run] +# requires-python = ">=3.11" +# dependencies = [ +# "requests<3", +# "rich", +# ] +# /// + +import requests +from rich.pretty import pprint + +resp = requests.get("https://peps.python.org/api/peps.json") +data = resp.json() +pprint([(k, v["title"]) for k, v in data.items()][:10]) +``` # Unresolved questions [unresolved-questions]: #unresolved-questions @@ -78,20 +376,9 @@ Please also take into consideration that rust sometimes intentionally diverges f # Future possibilities [future-possibilities]: #future-possibilities -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how this all fits into the roadmap for the project -and of the relevant sub-team. - -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. - -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. - -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. +- Support more infostring languages + - We need to better understand use cases for how this should be extended +- Support infostring attributes + - We need to better understand use cases for how this should be extended +- Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. + - Since nothing will read this, whether we do it now or in the future will have no affect From c088429c86d4565b009b0e96b2ebd44e166efbbe Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 14:39:39 -0500 Subject: [PATCH 03/60] fix: Update cross-links to cargo-script --- text/0000-frontmatter.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0000-frontmatter.md b/text/0000-frontmatter.md index 37d2ede5e63..1ea386eec21 100644 --- a/text/0000-frontmatter.md +++ b/text/0000-frontmatter.md @@ -6,7 +6,7 @@ # Summary [summary]: #summary -Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code](TODO): +Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code](https://github.com/rust-lang/rfcs/pull/3502): ````rust #!/usr/bin/env cargo ```cargo @@ -32,7 +32,7 @@ fn main() { # Motivation [motivation]: #motivation -["cargo script"](TODO) is in need of a syntax for embedding manifests in source. +["cargo script"](https://github.com/rust-lang/rfcs/pull/3502) is in need of a syntax for embedding manifests in source. See that RFC for its motivations. # Guide-level explanation @@ -74,7 +74,7 @@ fn main() { ```` As we work to better understand how tool authors will want to use frontmatter, we are restricting it to just the `cargo` infostring. -This means users will only be exposed to this within the concept of ["cargo script"](TODO). +This means users will only be exposed to this within the concept of ["cargo script"](https://github.com/rust-lang/rfcs/pull/3502). # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -101,7 +101,7 @@ we considered starting with only allowing this in the root `mod` (e.g. `main.rs` but decided to allow it in any file mostly for ease of implementation. Like with Python, this allows any file in a package (with the correct deps and `mod`s) to be executed, allowing easier interacting experiences in verifying behavior. -As for the hard-coded infostring used by cargo, that is a decision for [RFC TODO](TODO). +As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502](https://github.com/rust-lang/rfcs/pull/3502). ## Syntax From f644b5f18551c1461af65f3110d265882200cc99 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 14:42:27 -0500 Subject: [PATCH 04/60] fix: Update for RFC number --- text/{0000-frontmatter.md => 3503-frontmatter.md} | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) rename text/{0000-frontmatter.md => 3503-frontmatter.md} (98%) diff --git a/text/0000-frontmatter.md b/text/3503-frontmatter.md similarity index 98% rename from text/0000-frontmatter.md rename to text/3503-frontmatter.md index 1ea386eec21..27d948dd3c0 100644 --- a/text/0000-frontmatter.md +++ b/text/3503-frontmatter.md @@ -1,7 +1,8 @@ - Feature Name: `frontmatter` - Start Date: 2023-09-26 -- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) -- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) +- RFC PR: [rust-lang/rfcs#3503](https://github.com/rust-lang/rfcs/pull/3503) +- Rust Issue: [rust-lang/cargo#12207](https://github.com/rust-lang/cargo/issues/12207) + # Summary [summary]: #summary From 29839ae6bb93a956500bdb134ea60e160588a7e2 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 15:07:11 -0500 Subject: [PATCH 05/60] fix: Grammar --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 27d948dd3c0..e4e333ad0b1 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -51,7 +51,7 @@ Hello world! ``` We are carrying this concept over to Rust with a twist: using code fences which -will be familiar to Rust developers who when documenting their code: +will be familiar to Rust developers when documenting their code: ````rust #!/usr/bin/env cargo ```cargo From d97eb11fa6eaa37484f4bf136c371448ce8fe6ae Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 15:07:21 -0500 Subject: [PATCH 06/60] fix: Add multi-frontmatter future possibility --- text/3503-frontmatter.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index e4e333ad0b1..0e1356f673c 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -383,3 +383,28 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - We need to better understand use cases for how this should be extended - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect + +## Multiple frontmatters + +If we decide we need to embed additional metadata, we have a couple of options for extending frontmatter support. + +Distinct blocks, maybe with newlines + +````md +```Cargo.toml +``` + +```Cargo.lock +``` +```` + +Continuous blocks +````md +```Cargo.toml +```Cargo.lock +``` +```` + +Distinct blocks is more like the source inspiration, markdown, +though has more noise, places to get things wrong, and syntax questions +(newlines). From f7b3f4717c0d4d051295d79522b24643be2fdccf Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 15:09:53 -0500 Subject: [PATCH 07/60] styke: Formatting --- text/3503-frontmatter.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 0e1356f673c..799d297ad4c 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -342,8 +342,10 @@ which enumerates the syntax used by different tools. ## YAML frontmatter -As a specialization of [YAML presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream), static site generators use frontmatter to embed YAML at the top of files. -Other systems have extended this for non-YAML use, like [zola using `+++` for TOML](https://www.getzola.org/documentation/content/page/#front-matter). +As a specialization of [YAML presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream), +static site generators use frontmatter to embed YAML at the top of files. +Other systems have extended this for non-YAML use, like +[zola using `+++` for TOML](https://www.getzola.org/documentation/content/page/#front-matter). ### Proposed Python syntax From ea5d05e0e2fac3cff49384b0eb743dcd976c6919 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 15:10:07 -0500 Subject: [PATCH 08/60] fix: Correct PEP 723 description --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 799d297ad4c..a21893db54e 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -349,7 +349,7 @@ Other systems have extended this for non-YAML use, like ### Proposed Python syntax -Currently the draft [PEP 723](https://peps.python.org/pep-0723/) proposes allowing begin/end markers inside of source code: +Currently the draft [PEP 723](https://peps.python.org/pep-0723/) proposes allowing begin/end markers inside of regular comments: ```python # /// pyproject From 8f144ee6e5907972b926a0b8175659a3f70c89af Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 15:10:16 -0500 Subject: [PATCH 09/60] fix: Remove some template boilerplate --- text/3503-frontmatter.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index a21893db54e..b737f31f5b7 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -372,10 +372,6 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) # Unresolved questions [unresolved-questions]: #unresolved-questions -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? - # Future possibilities [future-possibilities]: #future-possibilities From a223b951cfddce24181b67796a00c667dd75ea02 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 19:19:26 -0500 Subject: [PATCH 10/60] fix: Note that cargo being the default infostring for future --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index b737f31f5b7..43dc0fa26cb 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -375,6 +375,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) # Future possibilities [future-possibilities]: #future-possibilities +- Treat `cargo` as the default infostring - Support more infostring languages - We need to better understand use cases for how this should be extended - Support infostring attributes From 1a243fdf7c446272a3603ad4415b7885c68ddd47 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 19:41:13 -0500 Subject: [PATCH 11/60] fix: Add more prior art --- text/3503-frontmatter.md | 75 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 43dc0fa26cb..7745fc50771 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -340,6 +340,81 @@ Downsides See also [Single-file scripts that download their dependencies](https://dbohdan.com/scripts-with-dependencies) which enumerates the syntax used by different tools. +## `cargo-script` family + +There are several forks of [cargo script](https://github.com/DanielKeep/cargo-script). + +doc-comments +```rust +#!/usr/bin/env run-cargo-script +//! This is a regular crate doc comment, but it also contains a partial +//! Cargo manifest. Note the use of a *fenced* code block, and the +//! `cargo` "language". +//! +//! ```cargo +//! [dependencies] +//! time = "0.1.25" +//! ``` +extern crate time; +fn main() { + println!("{}", time::now().rfc822z()); +} +``` +short-hand +```rust +// cargo-deps: time="0.1.25" +// You can also leave off the version number, in which case, it's assumed +// to be "*". Also, the `cargo-deps` comment *must* be a single-line +// comment, and it *must* be the first thing in the file, after the +// hashbang. +extern crate time; +fn main() { + println!("{}", time::now().rfc822z()); +} +``` + +## RustExplorer + +[Rust Explorer](https://users.rust-lang.org/t/rust-playground-with-the-top-10k-crates/75746) +uses a comment syntax for specifying dependencies + +Example: +```rust +/* +[dependencies] +actix-web = "*" +ureq = "*" +tokio = { version = "*", features = ["full"] } +*/ + +use actix_web::App; +use actix_web::get; +use actix_web::HttpResponse; +use actix_web::HttpServer; +use actix_web::post; +use actix_web::Responder; +use actix_web::web; +use tokio::spawn; +use tokio::sync::oneshot; +use tokio::task::spawn_blocking; +``` + +## PL/Rust + +Example: +```sql +CREATE OR REPLACE FUNCTION randint() RETURNS bigint LANGUAGE plrust AS $$ +[dependencies] +rand = "0.8" + +[code] +use rand::Rng; +Ok(Some(rand::thread_rng().gen())) +$$; +``` + +See [External Dependencies](https://github.com/tcdi/plrust/blob/main/doc/src/dependencies.md) + ## YAML frontmatter As a specialization of [YAML presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream), From ddcca018c9f9ed110abede839d5b1cd3b4d78d59 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 19:44:27 -0500 Subject: [PATCH 12/60] fix: Call out why shebang-ext needs backticks --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 7745fc50771..74e3dcb25eb 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -317,6 +317,7 @@ fn main() {} This is a variation on other options that ties itself closer to the shebang syntax. The hope would be that we could get buy-in from other languages. - The first line post-shebang-stripping is a hash plus 3+ backticks, then capture all content until a matching pair of backticks on a dedicated line. This would be captured into a `#![frontmatter(info = "cargo", content = "..."]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it). Each content line must be indented to at least the same level as the first backtick. + - Backticks are needed to know to avoid parsing `#[dependencies]` as an attribute - Future evolution: Allow `cargo` being the default `info` string - Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` - Future evolution: Allow `frontmatter` attribute on any module From 67114fe896b159a5f7b4d83d1524220ebd48774b Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 21:20:59 -0500 Subject: [PATCH 13/60] fix: Expand on a benefit --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 74e3dcb25eb..3e217603b61 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -318,6 +318,7 @@ This is a variation on other options that ties itself closer to the shebang synt The hope would be that we could get buy-in from other languages. - The first line post-shebang-stripping is a hash plus 3+ backticks, then capture all content until a matching pair of backticks on a dedicated line. This would be captured into a `#![frontmatter(info = "cargo", content = "..."]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it). Each content line must be indented to at least the same level as the first backtick. - Backticks are needed to know to avoid parsing `#[dependencies]` as an attribute + - This also allows an infostring so this isn't just a cargo feature - Future evolution: Allow `cargo` being the default `info` string - Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` - Future evolution: Allow `frontmatter` attribute on any module From bcd9c6db91779b38ddce79c1a3db808d628a92c2 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 26 Sep 2023 21:21:06 -0500 Subject: [PATCH 14/60] fix: Adjust formatting --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 3e217603b61..410e5fce8c5 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -424,7 +424,7 @@ static site generators use frontmatter to embed YAML at the top of files. Other systems have extended this for non-YAML use, like [zola using `+++` for TOML](https://www.getzola.org/documentation/content/page/#front-matter). -### Proposed Python syntax +## Proposed Python syntax Currently the draft [PEP 723](https://peps.python.org/pep-0723/) proposes allowing begin/end markers inside of regular comments: From bb26c75bc855de4d3a9b97643fdaf06e353046e1 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 08:48:32 -0500 Subject: [PATCH 15/60] fix: Remove redundant word --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 410e5fce8c5..138f5aa3303 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -142,7 +142,7 @@ Benefits - When discussing on forums, people expressed how they had never seen the syntax but instantly were able to understand it Downsides: -- When discussing with a a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch +- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - Having the explain the overloading of concepts to new users - Unpredictable location (both the doc comment and the cargo code block within it) - Visual clutter (where clutter is overwhelming already in Rust) From 394387b571bf3c080d224a58652cda8103661b3d Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 08:51:00 -0500 Subject: [PATCH 16/60] fix: Clarify cargo-script inside markdown --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 138f5aa3303..8400c4fd44f 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -92,7 +92,7 @@ the responsibility for high quality error messages will largely fall on cargo. [drawbacks]: #drawbacks - A new concept for Rust syntax, adding to overall cognitive load -- Requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) +- When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives From 842f722866e685f9281c30c516f965cf5136b696 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 08:52:51 -0500 Subject: [PATCH 17/60] fix: Clarify why infostring default is deferred --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 8400c4fd44f..78839050a78 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -453,6 +453,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) [future-possibilities]: #future-possibilities - Treat `cargo` as the default infostring + - Since this is a one-way door, we want to make sure we are confident that that is the right default / policy - Support more infostring languages - We need to better understand use cases for how this should be extended - Support infostring attributes From 42077b7f33566db75b4a11af8110259955fa2ac2 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 08:55:33 -0500 Subject: [PATCH 18/60] fix: Highlight meta macro idea --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 78839050a78..5449cd7ee26 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -170,6 +170,7 @@ Benefits Downsides - The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge + - An unexplored direction we could go with this is a `meta!` macro (e.g. we'd need to have a format marker in it) - A lot of tools/IDEs have problems in dealing with macros - Free-form rust code makes it harder for cargo to make edits to the manifest - Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` From f95583824aacb2f7398b07f3eb2fc80dd852be91 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 09:02:05 -0500 Subject: [PATCH 19/60] fix: Be clear on code fence vs fenced code block --- text/3503-frontmatter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 5449cd7ee26..b327379e280 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -50,7 +50,7 @@ name: My Blog Post Hello world! ``` -We are carrying this concept over to Rust with a twist: using code fences which +We are carrying this concept over to Rust with a twist: using fence code blocks which will be familiar to Rust developers when documenting their code: ````rust #!/usr/bin/env cargo @@ -80,7 +80,7 @@ This means users will only be exposed to this within the concept of ["cargo scri # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a code fence: +When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a fenced code block: - Must be immediately at the top (after shebang stripping), meaning no blank lines - Opens with 3+ backticks and "cargo" followed by a newline - All content is ignored until a matching number of backticks is found at the start of a line. It is an error to have anything come between the backticks and the newline. From 0e671bf9136a3c8ed43980f2fd19602dcd091619 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 09:07:46 -0500 Subject: [PATCH 20/60] fix: Update based on commonmark spec See https://spec.commonmark.org/0.30/#fenced-code-blocks --- text/3503-frontmatter.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index b327379e280..f3bcba30302 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -83,7 +83,12 @@ This means users will only be exposed to this within the concept of ["cargo scri When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a fenced code block: - Must be immediately at the top (after shebang stripping), meaning no blank lines - Opens with 3+ backticks and "cargo" followed by a newline -- All content is ignored until a matching number of backticks is found at the start of a line. It is an error to have anything come between the backticks and the newline. + - We are intentionally supporting a subset of commonmark code fences in supporting backticks but not tilde code fences or indented fenced code blocks + - As we aren't supporting an arbitrarily nested file format (though may show up in one), we likely don't need the flexibility + - We are prioritizing on "one right way to do it" to make it easier to learn to write and to read a variety of files. +- All content is ignored until the same number of backticks or more is found at the start of a line. + It is an error to have anything besides spaces and tabs between the backticks and the newline. + - Unlike commonark, it is an error to not close the fenced code block seeing to detect problems earlier in the process seeing as the primary content is what comes after the fenced code block As cargo will be the first step in the process to parse this, the responsibility for high quality error messages will largely fall on cargo. From 1e4f518571266dd793e141de831c3d6caea5bda4 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 13:16:06 -0500 Subject: [PATCH 21/60] fix: Address typo --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index f3bcba30302..aa7461bf85a 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -88,7 +88,7 @@ When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a f - We are prioritizing on "one right way to do it" to make it easier to learn to write and to read a variety of files. - All content is ignored until the same number of backticks or more is found at the start of a line. It is an error to have anything besides spaces and tabs between the backticks and the newline. - - Unlike commonark, it is an error to not close the fenced code block seeing to detect problems earlier in the process seeing as the primary content is what comes after the fenced code block + - Unlike commonmark, it is an error to not close the fenced code block seeing to detect problems earlier in the process seeing as the primary content is what comes after the fenced code block As cargo will be the first step in the process to parse this, the responsibility for high quality error messages will largely fall on cargo. From 97c66584c9df07e23eddaa45c45d841366c5bdef Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 13:18:46 -0500 Subject: [PATCH 22/60] fix: Add heredoc benefits --- text/3503-frontmatter.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index aa7461bf85a..2e2a9f14c2a 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -255,7 +255,7 @@ fn main() { HEREDOC: ```rust #!/usr/bin/env cargo -/* Cargo.TOML >>> +/* Cargo.toml >>> [package] edition = "2018" <<< @@ -270,6 +270,9 @@ comments or HEREDOC. This does not prescribe the exact syntax used or supported comments Benefits +- Natural to support `Cargo.lock` as well +- Without existing baggage, can use file extensions, making a firmer + association in users minds for what is in these (for those used to `Cargo.toml`) Downsides - Unfamiliar syntax From 8ac0bc4c1e67b51f96d192e971f37746481b4310 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 13:21:01 -0500 Subject: [PATCH 23/60] fix: Add qualifier to alt build systems --- text/3503-frontmatter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 2e2a9f14c2a..efb18ccd3f3 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -304,7 +304,7 @@ Benefits - Visually/syntactically lightweight - Has parallels to ideas outside of Rust, building on external knowledge that might exist - Easy for cargo to parse and modify -- Can be leveraged by buck2, meson, etc in the future +- Can easily be leveraged by buck2, meson, etc in the future Downsides - Too general that people might abuse it @@ -336,7 +336,7 @@ Benefits - Visually connected to the shebang - Has parallels to ideas outside of Rust, building on external knowledge that might exist - Easy for cargo to parse and modify -- Can be leveraged by buck2, meson, etc in the future +- Can easily be leveraged by buck2, meson, etc in the future - Maybe we can get others on board with this syntax Downsides From 48c385c74c93fb9268fe95c14d21fd78f75fb9f6 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 27 Sep 2023 13:23:44 -0500 Subject: [PATCH 24/60] fix: Duplicate out downsides --- text/3503-frontmatter.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index efb18ccd3f3..e5d172b96fc 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -174,6 +174,8 @@ Benefits - Parsers are available to make this work (e.g. `syn`) Downsides +- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch + - Unpredictable location (both the doc comment and the cargo code block within it) - The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge - An unexplored direction we could go with this is a `meta!` macro (e.g. we'd need to have a format marker in it) - A lot of tools/IDEs have problems in dealing with macros @@ -203,6 +205,8 @@ Benefits - Parsers are available to make this work (e.g. `syn`) Downsides +- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch + - Unpredictable location (both the doc comment and the cargo code block within it) - From talking to a teacher, users are more forgiving of not understanding the details for structure data in an unstructured format (doc comments / comments) but something that looks meaningful, they will want to understand it all requiring dealing with all of the concepts - The attribute approach requires explaining multiple "advanced" topics: One teacher doesn't get to teaching any attributes until the second level in his crash course series and two teachers have found it difficult to teach people raw strings - Attributes look "scary" (and they are in some respects for the hidden stuff they do) @@ -276,6 +280,10 @@ Benefits Downsides - Unfamiliar syntax +- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch + - Having the explain the overloading of concepts to new users + - Unpredictable location (both the doc comment and the cargo code block within it) + - Visual clutter (where clutter is overwhelming already in Rust) - New style of structured comment for the ecosystem to support with potential compatibility issues, likely requiring a new edition - Assuming it can't be parsed with `syn` and either we need to write a From 421194a122e4cc8d6487014eb1799b0c6df4b2f6 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 28 Sep 2023 08:57:00 -0500 Subject: [PATCH 25/60] fix: Tie this back to rustdoc as precedence --- text/3503-frontmatter.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index e5d172b96fc..b7b1a6d6989 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -119,6 +119,14 @@ When choosing a syntax for embedding manifests in source files, our care-abouts - Leave the door open in case we want to reuse the syntax for embedded lockfiles - Leave the door open for single-file `lib`s +This proposed syntax builds off of the precedence of Rust syntax being specialized for an external tool +(doc-comments for rustdoc). +However, a new syntax is used instead of extending the comment syntax: +- Focused on being parsed by arbitrary tools (cargo, vim plugins, etc) without understanding the full Rust grammar +- Side steps compatibility issues with both + user expectations with the looseness of comment syntax (making it harder to parse) + and any existing comments that may look like a new structured comment syntax + Benefits for the proposed solution are: - Visually/syntactically lightweight - Has parallels to ideas outside of Rust, building on external knowledge that might exist From 346b30128f1cb6db6156ca131d2a157259be0ea5 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 29 Sep 2023 13:19:33 -0500 Subject: [PATCH 26/60] fix: Be more explicit --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index b7b1a6d6989..408381e1219 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -83,7 +83,7 @@ This means users will only be exposed to this within the concept of ["cargo scri When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a fenced code block: - Must be immediately at the top (after shebang stripping), meaning no blank lines - Opens with 3+ backticks and "cargo" followed by a newline - - We are intentionally supporting a subset of commonmark code fences in supporting backticks but not tilde code fences or indented fenced code blocks + - We are intentionally supporting only a subset of commonmark code fences in supporting backticks but not tilde code fences or indented fenced code blocks - As we aren't supporting an arbitrarily nested file format (though may show up in one), we likely don't need the flexibility - We are prioritizing on "one right way to do it" to make it easier to learn to write and to read a variety of files. - All content is ignored until the same number of backticks or more is found at the start of a line. From c237ad852d919c59606e3f39db577c7a3bd3ca66 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 29 Sep 2023 13:20:08 -0500 Subject: [PATCH 27/60] fix: Clarify some rationale for code fences --- text/3503-frontmatter.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 408381e1219..9bb6106fd9e 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -97,6 +97,7 @@ the responsibility for high quality error messages will largely fall on cargo. [drawbacks]: #drawbacks - A new concept for Rust syntax, adding to overall cognitive load +- Ecosystem tooling updates to deal with new syntax - When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) # Rationale and alternatives @@ -119,19 +120,28 @@ When choosing a syntax for embedding manifests in source files, our care-abouts - Leave the door open in case we want to reuse the syntax for embedded lockfiles - Leave the door open for single-file `lib`s -This proposed syntax builds off of the precedence of Rust syntax being specialized for an external tool +### Fenced Code Block Frontmatter + +This proposed syntax builds off of the precedence of Rust having syntax specialized for an external tool (doc-comments for rustdoc). However, a new syntax is used instead of extending the comment syntax: -- Focused on being parsed by arbitrary tools (cargo, vim plugins, etc) without understanding the full Rust grammar +- Simplified for being parsed by arbitrary tools (cargo, vim plugins, etc) without understanding the full Rust grammar - Side steps compatibility issues with both - user expectations with the looseness of comment syntax (making it harder to parse) + user expectations with the looseness of comment syntax (which supporting would make it harder to parse) and any existing comments that may look like a new structured comment syntax -Benefits for the proposed solution are: +This proposal mirrors the location of YAML frontmatter (absolutely first). +As we learn more of its uses and problems people run into in practice, +we can evaluate if we want to loosen any of the rules. + +Benefits: - Visually/syntactically lightweight - Has parallels to ideas outside of Rust, building on external knowledge that might exist -- Easy for cargo to parse and modify - - Cargo needs a simple syntax subset to parse to ensure it can adequately report errors for the part it cares about but does not get in the way of rustc reporting the high quality errors it cares about +- Easy for cargo and any third-party tool to parse and modify + - As cargo will be parsing before rustc, + cargo being able to work off of a reduced syntax is paramount for ensuring + cargo doesn't mask the high quality rustc errors with lower-quality errors of + its own - In the future, this can be leveraged by other build systems or tools ### Alternative 1: Doc-comment From a21b55eee80cb602fc0a2a180ba0fb00b4d58594 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 29 Sep 2023 13:20:26 -0500 Subject: [PATCH 28/60] fix: Clarify downside for yaml presentation streams --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 9bb6106fd9e..dfc77ac6a21 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -254,7 +254,7 @@ Benefits Downsides - Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal -- Being new syntax, there would be a lot of details to work out, including +- Being a new file format (a "text tar" format), there would be a lot of details to work out, including - How to delineate and label documents - How to allow escaping to avoid conflicts with content in a documents - Potentially an API for accessing the document from within Rust From 97ca09a2b05231d4662d780a5406e0112c1e8bd6 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 29 Sep 2023 13:20:38 -0500 Subject: [PATCH 29/60] fix: Clarify the unlikeliness of multiple formats --- text/3503-frontmatter.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index dfc77ac6a21..aa6ad3f5614 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -498,7 +498,11 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) ## Multiple frontmatters -If we decide we need to embed additional metadata, we have a couple of options for extending frontmatter support. +At least for cargo's use cases, the only other file would consier supporting is `Cargo.lock` +and we have other avenues we want to explore as future possibilities before we +even consider the idea of multiple frontmatters. + +So **if** we decide we need to embed additional metadata, we have a couple of options for extending frontmatter support. Distinct blocks, maybe with newlines From 60d05acd8c6a61e0e70dfc0d364645a6a344f6f0 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 29 Sep 2023 13:23:00 -0500 Subject: [PATCH 30/60] fix: Typos --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index aa6ad3f5614..8cf88f348e6 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -498,7 +498,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) ## Multiple frontmatters -At least for cargo's use cases, the only other file would consier supporting is `Cargo.lock` +At least for cargo's use cases, the only other file that we would consider supporting is `Cargo.lock` and we have other avenues we want to explore as future possibilities before we even consider the idea of multiple frontmatters. From 38cf0ee9423736debcab0d7230720c7438ce73b7 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 3 Oct 2023 16:55:12 -0500 Subject: [PATCH 31/60] fix: Bring up newlines --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 8cf88f348e6..4d9a3b17081 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -493,6 +493,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - We need to better understand use cases for how this should be extended - Support infostring attributes - We need to better understand use cases for how this should be extended +- Loosen the code-fence syntax, like allowing newlines - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect From c3625147c24fc277692f1fd61ea13e1c8f76a125 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 3 Oct 2023 17:06:47 -0500 Subject: [PATCH 32/60] fix: Ensure a syntax drawback is also in the comparison --- text/3503-frontmatter.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 4d9a3b17081..444271ebc43 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -144,6 +144,9 @@ Benefits: its own - In the future, this can be leveraged by other build systems or tools +Downsides: +- When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) + ### Alternative 1: Doc-comment ```rust From e0707546acdbf64a2158fb0f479d5daa6044cb1b Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 3 Oct 2023 17:07:08 -0500 Subject: [PATCH 33/60] fix: Add the cost of familiar syntax --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 444271ebc43..bc9e911b5d5 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -146,6 +146,7 @@ Benefits: Downsides: - When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) +- Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). ### Alternative 1: Doc-comment From 5479104a03f599c9866efa1d4cc199ea00c43ff4 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 3 Oct 2023 17:09:20 -0500 Subject: [PATCH 34/60] fix: Call out parallels for rustc/cargo integration --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index bc9e911b5d5..3b5d1afe295 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -495,6 +495,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Since this is a one-way door, we want to make sure we are confident that that is the right default / policy - Support more infostring languages - We need to better understand use cases for how this should be extended + - Hard coding it for now is much like how rustc has hard coded support for `#[rustdoc]` or `clippy`/`rustdoc` lint categories - Support infostring attributes - We need to better understand use cases for how this should be extended - Loosen the code-fence syntax, like allowing newlines From 7bcaca9e84ff1e1aeed3e9dd5da88a06e737142a Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 3 Oct 2023 20:30:19 -0500 Subject: [PATCH 35/60] fix: Cover distinction from this and comments --- text/3503-frontmatter.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 3b5d1afe295..37943ba7a6a 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -130,6 +130,11 @@ However, a new syntax is used instead of extending the comment syntax: user expectations with the looseness of comment syntax (which supporting would make it harder to parse) and any existing comments that may look like a new structured comment syntax +The difference between this syntax and comments is comments are generally +geared towards people, even if a subset (doc-comments) are also able to be +somewhat processed by a program, while this is geared mostly towards machine +processing. + This proposal mirrors the location of YAML frontmatter (absolutely first). As we learn more of its uses and problems people run into in practice, we can evaluate if we want to loosen any of the rules. @@ -147,6 +152,7 @@ Benefits: Downsides: - When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). +- If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments ### Alternative 1: Doc-comment From e431c221c897d03c40cd8e0d86214e8aa4d6f249 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 5 Oct 2023 08:37:07 -0500 Subject: [PATCH 36/60] fix: Highlight edit/copy/paste impact of different options --- text/3503-frontmatter.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 37943ba7a6a..c820d2ceea2 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -141,6 +141,7 @@ we can evaluate if we want to loosen any of the rules. Benefits: - Visually/syntactically lightweight +- Users can edit/copy/paste the manifest without dealing with leading characters - Has parallels to ideas outside of Rust, building on external knowledge that might exist - Easy for cargo and any third-party tool to parse and modify - As cargo will be parsing before rustc, @@ -173,6 +174,7 @@ Benefits - Familiar syntax both to read and write. - When discussing with a Rust author, it was pointed out many times people preface code with a comment specifying the dependencies ([example](https://github.com/dtolnay/prettyplease#example)), this is the same idea but reusable by cargo - When discussing on forums, people expressed how they had never seen the syntax but instantly were able to understand it +- Depending on doc-comment style used, users may be able to edit/copy/paste the manifest without dealing with leading characters Downsides: - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch @@ -200,6 +202,7 @@ fn main() { ``` Benefits - Parsers are available to make this work (e.g. `syn`) +- Users can edit/copy/paste the manifest without dealing with leading characters Downsides - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch @@ -231,6 +234,7 @@ fn main() { Benefits - Parsers are available to make this work (e.g. `syn`) +- Users can edit/copy/paste the manifest without dealing with leading characters Downsides - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch @@ -261,6 +265,7 @@ What if we extended Rust's syntax to allow something similar? Benefits - Flexible for other content +- Users can edit/copy/paste the manifest without dealing with leading characters Downsides - Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal @@ -305,6 +310,7 @@ Benefits - Natural to support `Cargo.lock` as well - Without existing baggage, can use file extensions, making a firmer association in users minds for what is in these (for those used to `Cargo.toml`) +- Depending on the exact syntax decided on, users may be able to edit/copy/paste the manifest without dealing with leading characters Downsides - Unfamiliar syntax @@ -341,6 +347,7 @@ Benefits - Has parallels to ideas outside of Rust, building on external knowledge that might exist - Easy for cargo to parse and modify - Can easily be leveraged by buck2, meson, etc in the future +- Users can edit/copy/paste the manifest without dealing with leading characters Downsides - Too general that people might abuse it @@ -380,6 +387,7 @@ Downsides - Visually - More work to type it out / copy-paste - More to get wrong +- Requires users to deal with leading characters when editing/copying/pasting the manifest # Prior art [prior-art]: #prior-art From a2d78cb6bdbb1709ae7934b952edfad426e358f9 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Sat, 7 Oct 2023 20:57:01 -0500 Subject: [PATCH 37/60] fix: Talk more to design goal difference with markdown --- text/3503-frontmatter.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index c820d2ceea2..6844df14b96 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -83,10 +83,9 @@ This means users will only be exposed to this within the concept of ["cargo scri When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a fenced code block: - Must be immediately at the top (after shebang stripping), meaning no blank lines - Opens with 3+ backticks and "cargo" followed by a newline - - We are intentionally supporting only a subset of commonmark code fences in supporting backticks but not tilde code fences or indented fenced code blocks - As we aren't supporting an arbitrarily nested file format (though may show up in one), we likely don't need the flexibility - We are prioritizing on "one right way to do it" to make it easier to learn to write and to read a variety of files. -- All content is ignored until the same number of backticks or more is found at the start of a line. +- All content is ignored until the same number of backticks is found at the start of a line. It is an error to have anything besides spaces and tabs between the backticks and the newline. - Unlike commonmark, it is an error to not close the fenced code block seeing to detect problems earlier in the process seeing as the primary content is what comes after the fenced code block @@ -139,6 +138,13 @@ This proposal mirrors the location of YAML frontmatter (absolutely first). As we learn more of its uses and problems people run into in practice, we can evaluate if we want to loosen any of the rules. +We are intentionally supporting only a subset of commonmark code fences. +Markdown, like HTML, is meant to always be valid which is different than Rust syntax. +Differences include: +- backticks but not tilde's +- no indenting of the fenced code block +- open/close must be a matching pair, rather than the close having "the same or more" + Benefits: - Visually/syntactically lightweight - Users can edit/copy/paste the manifest without dealing with leading characters From 791f96564483544de44afa9797e7de6e1beed709 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Sat, 7 Oct 2023 21:08:38 -0500 Subject: [PATCH 38/60] fix: Expand on why we delayed some decisions --- text/3503-frontmatter.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 6844df14b96..493c0102380 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -513,11 +513,19 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Treat `cargo` as the default infostring - Since this is a one-way door, we want to make sure we are confident that that is the right default / policy + - If there aren't guard rails (e.g. rustc enforcement) that no-infostring + means `cargo`, it can easily be a free-for-all and we need to decide if we + want that or if there is enough guard rails to discourage it. - Support more infostring languages - - We need to better understand use cases for how this should be extended + - We need to better understand use cases for how this should be extended, particularly what the syntax should be + - For reference, infostrings in commonmark (language + attributes) is just + all non-backtick characters, supports `\` escapes, and entity / numeric + character references with a vague notion of the language being the first "word" + - A safe starting point could be to say that it must be an identifier - Hard coding it for now is much like how rustc has hard coded support for `#[rustdoc]` or `clippy`/`rustdoc` lint categories - Support infostring attributes - - We need to better understand use cases for how this should be extended + - We need to better understand use cases for how this should be extended, particularly what the syntax should be (see infostring language) + - A safe starting point could be to say that a space or comma separates attributes and everything after it is defined as part of the "language" - Loosen the code-fence syntax, like allowing newlines - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect From 7a687c2eaa62fa444820b0d690d0f9c9ab3d2b6f Mon Sep 17 00:00:00 2001 From: Ed Page Date: Sat, 7 Oct 2023 21:22:11 -0500 Subject: [PATCH 39/60] fix: Re-order alternatives with assumed reasonableness of them --- text/3503-frontmatter.md | 201 +++++++++++++++++++-------------------- 1 file changed, 100 insertions(+), 101 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 493c0102380..3ccfc662c83 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -161,7 +161,71 @@ Downsides: - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). - If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments -### Alternative 1: Doc-comment +### Alternative 1: Static-site generator frontmatter + +```rust +#!/usr/bin/env cargo +--- +[package] +edition = "2018" +--- + +fn main() { +} +``` +This is a subset/specialization of YAML presentation streams that mirrors people's experience with static site generators: +- The first line post-shebang-stripping is 3+ dashes, then capture all content until a matching pair of dashes on a dedicated line. This would be captured into a `#![frontmatter = ""]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it) +- Future evolution: Allow a markdown-like infostring on the frontmatter opening dashes to declare the format with `cargo` being the default +- Future evolution: Allow `frontmatter` attribute on any module + +Benefits +- Visually/syntactically lightweight +- Has parallels to ideas outside of Rust, building on external knowledge that might exist +- Easy for cargo to parse and modify +- Can easily be leveraged by buck2, meson, etc in the future +- Users can edit/copy/paste the manifest without dealing with leading characters + +Downsides +- Too general that people might abuse it +- We've extended the frontmatter syntax, undoing some of the "familiarity" benefit +- People are used to YAML going in frontmatter (though some systems allow other syntaxes) +- Doesn't feel very rust-like + +### Alternative 2: Extended Shebang + +````rust +#!/usr/bin/env cargo +# ```cargo +# [dependencies] +# foo = "1.2.3" +# ``` + +fn main() {} +```` +This is a variation on other options that ties itself closer to the shebang syntax. +The hope would be that we could get buy-in from other languages. +- The first line post-shebang-stripping is a hash plus 3+ backticks, then capture all content until a matching pair of backticks on a dedicated line. This would be captured into a `#![frontmatter(info = "cargo", content = "..."]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it). Each content line must be indented to at least the same level as the first backtick. + - Backticks are needed to know to avoid parsing `#[dependencies]` as an attribute + - This also allows an infostring so this isn't just a cargo feature +- Future evolution: Allow `cargo` being the default `info` string +- Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` +- Future evolution: Allow `frontmatter` attribute on any module + +Benefits +- Visually connected to the shebang +- Has parallels to ideas outside of Rust, building on external knowledge that might exist +- Easy for cargo to parse and modify +- Can easily be leveraged by buck2, meson, etc in the future +- Maybe we can get others on board with this syntax + +Downsides +- More syntactically heavy than the frontmatter solution + - Visually + - More work to type it out / copy-paste + - More to get wrong +- Requires users to deal with leading characters when editing/copying/pasting the manifest + +### Alternative 3: Doc-comment ```rust #!/usr/bin/env cargo @@ -193,33 +257,7 @@ Downsides: - Requires pulling in a full markdown parser to extract the manifest - Incorrectly formatted markdown would lead to a missing manifest and confusing error messages at best or silent incorrect behavior at worse -### Alternative 2: Macro - -```rust -#!/usr/bin/env cargo - -cargo! { -[package] -edition = "2018" -} - -fn main() { -} -``` -Benefits -- Parsers are available to make this work (e.g. `syn`) -- Users can edit/copy/paste the manifest without dealing with leading characters - -Downsides -- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - - Unpredictable location (both the doc comment and the cargo code block within it) -- The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge - - An unexplored direction we could go with this is a `meta!` macro (e.g. we'd need to have a format marker in it) -- A lot of tools/IDEs have problems in dealing with macros -- Free-form rust code makes it harder for cargo to make edits to the manifest -- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` - -### Alternative 3: Attribute +### Alternative 4: Attribute ```rust #!/usr/bin/env cargo @@ -250,37 +288,6 @@ Downsides - Attributes look "scary" (and they are in some respects for the hidden stuff they do) - Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` - -### Alternative 4: Presentation Streams - -```rust -#!/usr/bin/env cargo - -fn main() { -} - ----Cargo.toml -[package] -edition = "2018" -``` -YAML allows several documents to be concatenated together variant -[presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream) -which might seem familiar as this is frequently used in static-site generators -for adding frontmatter to pages. -What if we extended Rust's syntax to allow something similar? - -Benefits -- Flexible for other content -- Users can edit/copy/paste the manifest without dealing with leading characters - -Downsides -- Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal -- Being a new file format (a "text tar" format), there would be a lot of details to work out, including - - How to delineate and label documents - - How to allow escaping to avoid conflicts with content in a documents - - Potentially an API for accessing the document from within Rust -- Unfamiliar, new syntax, unclear how it will work out for newer users - ### Alternative 5: Regular Comment Simple header: @@ -331,69 +338,61 @@ Downsides to extract and update comments. - Like with doc comments, this should map to an attribute and then we'd just start the MVP with that attribute -### Alternative 6: Static-site generator frontmatter +### Alternative 6: Macro ```rust #!/usr/bin/env cargo ---- + +cargo! { [package] edition = "2018" ---- +} fn main() { } ``` -This is a subset/specialization of YAML presentation streams that mirrors people's experience with static site generators: -- The first line post-shebang-stripping is 3+ dashes, then capture all content until a matching pair of dashes on a dedicated line. This would be captured into a `#![frontmatter = ""]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it) -- Future evolution: Allow a markdown-like infostring on the frontmatter opening dashes to declare the format with `cargo` being the default -- Future evolution: Allow `frontmatter` attribute on any module - Benefits -- Visually/syntactically lightweight -- Has parallels to ideas outside of Rust, building on external knowledge that might exist -- Easy for cargo to parse and modify -- Can easily be leveraged by buck2, meson, etc in the future +- Parsers are available to make this work (e.g. `syn`) - Users can edit/copy/paste the manifest without dealing with leading characters Downsides -- Too general that people might abuse it -- We've extended the frontmatter syntax, undoing some of the "familiarity" benefit -- People are used to YAML going in frontmatter (though some systems allow other syntaxes) -- Doesn't feel very rust-like +- When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch + - Unpredictable location (both the doc comment and the cargo code block within it) +- The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge + - An unexplored direction we could go with this is a `meta!` macro (e.g. we'd need to have a format marker in it) +- A lot of tools/IDEs have problems in dealing with macros +- Free-form rust code makes it harder for cargo to make edits to the manifest +- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` -### Alternative 7: Extended Shebang +### Alternative 7: Presentation Streams -````rust +```rust #!/usr/bin/env cargo -# ```cargo -# [dependencies] -# foo = "1.2.3" -# ``` -fn main() {} -```` -This is a variation on other options that ties itself closer to the shebang syntax. -The hope would be that we could get buy-in from other languages. -- The first line post-shebang-stripping is a hash plus 3+ backticks, then capture all content until a matching pair of backticks on a dedicated line. This would be captured into a `#![frontmatter(info = "cargo", content = "..."]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it). Each content line must be indented to at least the same level as the first backtick. - - Backticks are needed to know to avoid parsing `#[dependencies]` as an attribute - - This also allows an infostring so this isn't just a cargo feature -- Future evolution: Allow `cargo` being the default `info` string -- Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` -- Future evolution: Allow `frontmatter` attribute on any module +fn main() { +} + +---Cargo.toml +[package] +edition = "2018" +``` +YAML allows several documents to be concatenated together variant +[presentation streams](https://yaml.org/spec/1.2.2/#323-presentation-stream) +which might seem familiar as this is frequently used in static-site generators +for adding frontmatter to pages. +What if we extended Rust's syntax to allow something similar? Benefits -- Visually connected to the shebang -- Has parallels to ideas outside of Rust, building on external knowledge that might exist -- Easy for cargo to parse and modify -- Can easily be leveraged by buck2, meson, etc in the future -- Maybe we can get others on board with this syntax +- Flexible for other content +- Users can edit/copy/paste the manifest without dealing with leading characters Downsides -- More syntactically heavy than the frontmatter solution - - Visually - - More work to type it out / copy-paste - - More to get wrong -- Requires users to deal with leading characters when editing/copying/pasting the manifest +- Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal +- Being a new file format (a "text tar" format), there would be a lot of details to work out, including + - How to delineate and label documents + - How to allow escaping to avoid conflicts with content in a documents + - Potentially an API for accessing the document from within Rust +- Unfamiliar, new syntax, unclear how it will work out for newer users # Prior art [prior-art]: #prior-art From d179824547cb85badf196f13e1ee3d0feaac887c Mon Sep 17 00:00:00 2001 From: Ed Page Date: Sat, 7 Oct 2023 21:27:32 -0500 Subject: [PATCH 40/60] fix: Add a possible markdown workaround --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 3ccfc662c83..0f178f9780c 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -158,6 +158,7 @@ Benefits: Downsides: - When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) + - Maintainers seeding GitHub issue templates with 4 backticks can help - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). - If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments From 1bbeb65f6fe0fb3b40ce16cc7d19fdac1c4127b8 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Sat, 7 Oct 2023 21:37:36 -0500 Subject: [PATCH 41/60] fix: Clarify the relationship between code fence / YAML frontmatter --- text/3503-frontmatter.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 0f178f9780c..18b890326fd 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -164,6 +164,10 @@ Downsides: ### Alternative 1: Static-site generator frontmatter +Note: this alternative was kept close to its source of inspiration, maintaining +no infostring, despite being subject to the same questions that would be +present for fenced code blocks, see [Future possibilities](#future-possibilities). + ```rust #!/usr/bin/env cargo --- From 0b56c9865250b36a757ad2f212656f0028dcf697 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 16 Oct 2023 15:29:18 -0500 Subject: [PATCH 42/60] refactor: Pull out link to 3502 --- text/3503-frontmatter.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 18b890326fd..126a4bc7ffe 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -7,7 +7,7 @@ # Summary [summary]: #summary -Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code](https://github.com/rust-lang/rfcs/pull/3502): +Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code][RFC 3502]: ````rust #!/usr/bin/env cargo ```cargo @@ -33,7 +33,7 @@ fn main() { # Motivation [motivation]: #motivation -["cargo script"](https://github.com/rust-lang/rfcs/pull/3502) is in need of a syntax for embedding manifests in source. +["cargo script"][RFC 3502] is in need of a syntax for embedding manifests in source. See that RFC for its motivations. # Guide-level explanation @@ -75,7 +75,7 @@ fn main() { ```` As we work to better understand how tool authors will want to use frontmatter, we are restricting it to just the `cargo` infostring. -This means users will only be exposed to this within the concept of ["cargo script"](https://github.com/rust-lang/rfcs/pull/3502). +This means users will only be exposed to this within the concept of ["cargo script"][RFC 3502]. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -107,7 +107,7 @@ we considered starting with only allowing this in the root `mod` (e.g. `main.rs` but decided to allow it in any file mostly for ease of implementation. Like with Python, this allows any file in a package (with the correct deps and `mod`s) to be executed, allowing easier interacting experiences in verifying behavior. -As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502](https://github.com/rust-lang/rfcs/pull/3502). +As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502]. ## Syntax @@ -562,3 +562,5 @@ Continuous blocks Distinct blocks is more like the source inspiration, markdown, though has more noise, places to get things wrong, and syntax questions (newlines). + +[RFC 3502]: https://github.com/rust-lang/rfcs/pull/3502 From 6d8782ce407d96de3eaedad3c4e93d0fc086bc47 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 16 Oct 2023 16:24:50 -0500 Subject: [PATCH 43/60] fix: Include design principle from 3502 --- text/3503-frontmatter.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 126a4bc7ffe..e7d28fc8e69 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -111,11 +111,24 @@ As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502 ## Syntax -When choosing a syntax for embedding manifests in source files, our care-abouts are +[RFC 3502] lays out some design principles, including +- Single-file packages should have a first-class experience + - Provides a higher quality of experience (doesn't feel like a hack or tacked on) + - Transferable knowledge, whether experience, stackoverflow answers, etc + - Easier unassisted migration between single-file and multi-file packages + - The more the workflows deviate, the higher the maintenance and support costs for the cargo team + +This led us to wanting to re-use the existing manifest format inside of Rust code. +The question is what that syntax for embedding should be. + +When choosing the syntax, our care-abouts are - How obvious it is for new users when they see it - How easy it is for newer users to remember it and type it out - How machine editable it is for `cargo add` and friends - Needs to be valid Rust code for quality of error messages, etc +- Simple enough syntax for tools to parse without a full Rust parser + - Leave Rust syntax errors to rustc, rather than masking them with lower quality ones + - Ideally we allows random IDE tools (e.g. [`crates.nvim`](https://github.com/Saecki/crates.nvim) to still have easy access to the manifest - Leave the door open in case we want to reuse the syntax for embedded lockfiles - Leave the door open for single-file `lib`s From d2fc29120049344563b417eed6a525534bad1359 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 6 Dec 2023 12:15:40 -0600 Subject: [PATCH 44/60] fix: Better highlight infostring handling --- text/3503-frontmatter.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index e7d28fc8e69..01cf34a378e 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -528,18 +528,6 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) # Future possibilities [future-possibilities]: #future-possibilities -- Treat `cargo` as the default infostring - - Since this is a one-way door, we want to make sure we are confident that that is the right default / policy - - If there aren't guard rails (e.g. rustc enforcement) that no-infostring - means `cargo`, it can easily be a free-for-all and we need to decide if we - want that or if there is enough guard rails to discourage it. -- Support more infostring languages - - We need to better understand use cases for how this should be extended, particularly what the syntax should be - - For reference, infostrings in commonmark (language + attributes) is just - all non-backtick characters, supports `\` escapes, and entity / numeric - character references with a vague notion of the language being the first "word" - - A safe starting point could be to say that it must be an identifier - - Hard coding it for now is much like how rustc has hard coded support for `#[rustdoc]` or `clippy`/`rustdoc` lint categories - Support infostring attributes - We need to better understand use cases for how this should be extended, particularly what the syntax should be (see infostring language) - A safe starting point could be to say that a space or comma separates attributes and everything after it is defined as part of the "language" @@ -547,6 +535,24 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect +## Optional or additional infostrings + +We could support: +- Treat `cargo` as the default infostring +- Support more infostring languages + +The question comes down to whether +- rustc owns the definition of the infostring, allowing us to add additional types of metadata (rustfmt config, static analyzer config, etc) + - This would be similar to our [hard coding of "tool" attributes](https://github.com/rust-lang/rust/issues/44690) +- the shebang tool owns the definition of the infostring + +By us hard coding `cargo` in the infostring in rustc, +we are intentionally deferring the decision for which path we should go down. + +We can add additional infostrings on a case-by-case basis. +In doing so, we can learn more about the use cases involved which can help us +get a better picture for which route we should go down. + ## Multiple frontmatters At least for cargo's use cases, the only other file that we would consider supporting is `Cargo.lock` From de9862a987ef1c4749a53afd23f872d522d32b56 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 6 Dec 2023 12:21:06 -0600 Subject: [PATCH 45/60] fix: Cover required vs optional shebang --- text/3503-frontmatter.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 01cf34a378e..d07fdacfd7c 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -109,6 +109,18 @@ Like with Python, this allows any file in a package (with the correct deps and ` As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502]. +## Required vs Optional Shebang + +We could require the shebang to be present for all cargo-scripts. +This would most negatively impact Windows users as the shebang is a no-op. +We still care about Windows because cargo-scripts can still be used for exploration and prototyping, +even if they can't directly be used as drop-in utilities. + +The main reason to require a shebang is to positively identify the associated "interpreter". +However, statically analyzing a shebang is [complicated](https://stackoverflow.com/questions/38059830/how-does-perl-avoid-shebang-loops) +and we are wanting to avoid it in the core workflow. +This isn't to say that tools like rust-analyzer might choose to require it to help their workflow. + ## Syntax [RFC 3502] lays out some design principles, including From 9f2aec24926b9260f1518401d784c8f08904b53f Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 6 Dec 2023 12:39:02 -0600 Subject: [PATCH 46/60] fix: Expand on alternative syntax --- text/3503-frontmatter.md | 116 ++++++++++++++++++++++++++++++--------- 1 file changed, 91 insertions(+), 25 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index d07fdacfd7c..d5ae4acd72b 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -187,15 +187,19 @@ Downsides: - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). - If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments -### Alternative 1: Static-site generator frontmatter +### Alternative 1: Vary the opening/closing character -Note: this alternative was kept close to its source of inspiration, maintaining -no infostring, despite being subject to the same questions that would be -present for fenced code blocks, see [Future possibilities](#future-possibilities). +Instead of backticks, we could do another character, like +- `-`, making it look like YAML presentation streams, following the pattern of static site generators + - `+` like [zola's frontmatter](https://www.getzola.org/documentation/getting-started/overview/#markdown-content) +- `~`, using a lesser known markdown character +- `=` +- Open with `>>>` and close with `<<<`, like with HEREDOC (or invert it) +In practice: ```rust #!/usr/bin/env cargo ---- +---cargo [package] edition = "2018" --- @@ -203,24 +207,73 @@ edition = "2018" fn main() { } ``` -This is a subset/specialization of YAML presentation streams that mirrors people's experience with static site generators: -- The first line post-shebang-stripping is 3+ dashes, then capture all content until a matching pair of dashes on a dedicated line. This would be captured into a `#![frontmatter = ""]`. `frontmatter` attribute is reserved for crate roots. The 3+ with matching pair is a "just in case" a TOML multi-line string has that syntax in it) -- Future evolution: Allow a markdown-like infostring on the frontmatter opening dashes to declare the format with `cargo` being the default -- Future evolution: Allow `frontmatter` attribute on any module +```rust +#!/usr/bin/env cargo ++++cargo +[package] +edition = "2018" ++++ + +fn main() { +} +``` +```rust +#!/usr/bin/env cargo +~~~cargo +[package] +edition = "2018" +~~~ + +fn main() { +} +``` +```rust +#!/usr/bin/env cargo +===cargo +[package] +edition = "2018" +=== + +fn main() { +} +``` +```rust +#!/usr/bin/env cargo +>>>cargo +[package] +edition = "2018" +<<< + +fn main() { +} +``` +```rust +#!/usr/bin/env cargo +<<>> + +fn main() { +} +``` Benefits -- Visually/syntactically lightweight -- Has parallels to ideas outside of Rust, building on external knowledge that might exist -- Easy for cargo to parse and modify -- Can easily be leveraged by buck2, meson, etc in the future -- Users can edit/copy/paste the manifest without dealing with leading characters +- With `-`, it builds on people's familiarity with static site generators +- People can insert cargo-scripts into markdown (like chat, github issues) + without being familiar enough with markdown to know how to escape backticks + and to actually remember how to do it Downsides -- Too general that people might abuse it -- We've extended the frontmatter syntax, undoing some of the "familiarity" benefit -- People are used to YAML going in frontmatter (though some systems allow other syntaxes) +- With `-` + - We've extended the frontmatter syntax with an infostring, undoing some of the "familiarity" benefit + - Potential congantive disonance as those familiar with frontmatter are used to YAML being there +- With `>>>` it isn't quite like HEREDOC to have less overhead - Doesn't feel very rust-like +Note: +- `"` was not considered because that can feel too familiar and users might carry over their expectations for how strings work + ### Alternative 2: Extended Shebang ````rust @@ -241,6 +294,15 @@ The hope would be that we could get buy-in from other languages. - Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` - Future evolution: Allow `frontmatter` attribute on any module +If we dropped future possibilities for additional content, we could remove the opening/closing syntax and tweak it to avoid attribute ambiguity, +greatly reducing the minimum syntax needed in some cases. +````rust +#!/usr/bin/env cargo +## package.edition = "2018" + +fn main() {} +```` + Benefits - Visually connected to the shebang - Has parallels to ideas outside of Rust, building on external knowledge that might exist @@ -277,13 +339,14 @@ Benefits - Depending on doc-comment style used, users may be able to edit/copy/paste the manifest without dealing with leading characters Downsides: +- **Blocker** Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` + - If we extend additional restrictions to make it more tool friendly, then we break from user expectations for how this syntax works - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - Having the explain the overloading of concepts to new users - Unpredictable location (both the doc comment and the cargo code block within it) - Visual clutter (where clutter is overwhelming already in Rust) - Might be a bit complicated to do edits (translating between location within `toml_edit` spans to the location within `syn` spans especially with different comment styles) -- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` - Requires pulling in a full markdown parser to extract the manifest - Incorrectly formatted markdown would lead to a missing manifest and confusing error messages at best or silent incorrect behavior at worse @@ -311,12 +374,13 @@ Benefits - Users can edit/copy/paste the manifest without dealing with leading characters Downsides +- **Blocker** Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` + - If we extend additional restrictions to make it more tool friendly, then we break from user expectations for how this syntax works - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - Unpredictable location (both the doc comment and the cargo code block within it) - From talking to a teacher, users are more forgiving of not understanding the details for structure data in an unstructured format (doc comments / comments) but something that looks meaningful, they will want to understand it all requiring dealing with all of the concepts - The attribute approach requires explaining multiple "advanced" topics: One teacher doesn't get to teaching any attributes until the second level in his crash course series and two teachers have found it difficult to teach people raw strings - Attributes look "scary" (and they are in some respects for the hidden stuff they do) -- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` ### Alternative 5: Regular Comment @@ -356,6 +420,11 @@ Benefits - Depending on the exact syntax decided on, users may be able to edit/copy/paste the manifest without dealing with leading characters Downsides +- **Blocker** Assuming it can't be parsed with `syn` and either we need to write a + sufficiently compatible comment parser or pull in a much larger rust parser + to extract and update comments. + - If we extend additional restrictions to make it more tool friendly, then we break from user expectations for how this syntax works + - Like with doc comments, this should map to an attribute and then we'd just start the MVP with that attribute - Unfamiliar syntax - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - Having the explain the overloading of concepts to new users @@ -363,10 +432,6 @@ Downsides - Visual clutter (where clutter is overwhelming already in Rust) - New style of structured comment for the ecosystem to support with potential compatibility issues, likely requiring a new edition -- Assuming it can't be parsed with `syn` and either we need to write a - sufficiently compatible comment parser or pull in a much larger rust parser - to extract and update comments. - - Like with doc comments, this should map to an attribute and then we'd just start the MVP with that attribute ### Alternative 6: Macro @@ -386,13 +451,14 @@ Benefits - Users can edit/copy/paste the manifest without dealing with leading characters Downsides +- **Blocker** Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` + - If we extend additional restrictions to make it more tool friendly, then we break from user expectations for how this syntax works - When discussing with a Rust crash course teacher, it was felt their students would have a hard time learning to write these manifests from scratch - Unpredictable location (both the doc comment and the cargo code block within it) - The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge - An unexplored direction we could go with this is a `meta!` macro (e.g. we'd need to have a format marker in it) - A lot of tools/IDEs have problems in dealing with macros - Free-form rust code makes it harder for cargo to make edits to the manifest -- Either we expose `syn`s lesser parse errors or we skip errors, deferring to rustc's, but then have the wrong behavior on commands that don't invoke rustc, like `cargo metadata` ### Alternative 7: Presentation Streams @@ -417,7 +483,7 @@ Benefits - Users can edit/copy/paste the manifest without dealing with leading characters Downsides -- Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal +- **Blocker** Difficult to parse without assistance from something like `syn` as we'd need to distinguish what the start of a stream is vs content of a string literal - Being a new file format (a "text tar" format), there would be a lot of details to work out, including - How to delineate and label documents - How to allow escaping to avoid conflicts with content in a documents From 1b860d481947264dee19a7a44766228354de1db1 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 11 Dec 2023 20:13:07 -0600 Subject: [PATCH 47/60] feat: Make sure people know what they are signing up for --- text/3503-frontmatter.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index d5ae4acd72b..97fc30318f1 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -30,6 +30,11 @@ fn main() { } ```` +Note that to share these in markdown, a priority use case, extra backticks for the markdown code fence to escape the frontmatter code fence. +We expect most users will not be familiar enough with the markdown spec to know this, especially one of our primary target audience: those new to Rust. +This can also be frustrating for experienced users as three backticks is an ingrained habbit and it is common to need to go back and edit a post to properly escape the frontmatter. +However, when weighing out the syntactic needs and the alternatives, we felt this was the least bad option. + # Motivation [motivation]: #motivation @@ -97,7 +102,7 @@ the responsibility for high quality error messages will largely fall on cargo. - A new concept for Rust syntax, adding to overall cognitive load - Ecosystem tooling updates to deal with new syntax -- When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) +- **When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists)** # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives @@ -182,7 +187,7 @@ Benefits: - In the future, this can be leveraged by other build systems or tools Downsides: -- When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists) +- **When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists)** - Maintainers seeding GitHub issue templates with 4 backticks can help - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). - If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments From cdae2a7526f469ad9302f8472ac9ce51b6df0e85 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 11 Dec 2023 20:16:12 -0600 Subject: [PATCH 48/60] fix: Grammar adjustments --- text/3503-frontmatter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 97fc30318f1..c88ec8c7391 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -30,8 +30,8 @@ fn main() { } ```` -Note that to share these in markdown, a priority use case, extra backticks for the markdown code fence to escape the frontmatter code fence. -We expect most users will not be familiar enough with the markdown spec to know this, especially one of our primary target audience: those new to Rust. +Note that to share these in markdown, a priority use case, extra backticks are needed for the markdown code fence to escape the frontmatter code fence. +We expect most users will not be familiar enough with the markdown spec to know this, especially for one of our primary target audiences: those new to Rust. This can also be frustrating for experienced users as three backticks is an ingrained habbit and it is common to need to go back and edit a post to properly escape the frontmatter. However, when weighing out the syntactic needs and the alternatives, we felt this was the least bad option. From 05c44cc738872149742e27ced043a69c65a5ecb1 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 14 Dec 2023 16:14:37 -0600 Subject: [PATCH 49/60] fix: Note Bazel's import macro --- text/3503-frontmatter.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index c88ec8c7391..07d46e52e93 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -465,6 +465,8 @@ Downsides - A lot of tools/IDEs have problems in dealing with macros - Free-form rust code makes it harder for cargo to make edits to the manifest +Bazel has an [import proc-macro](https://github.com/bazelbuild/rules_rust/pull/1142) but its more for simplifying the writing of `extern crate`. + ### Alternative 7: Presentation Streams ```rust From 29554066f756e1ae1d6c95b5ace6a0d8489b2005 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 15 Jan 2024 08:35:31 -0600 Subject: [PATCH 50/60] fix(alt): Note problems with syntax that looks like merge conflicts --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 07d46e52e93..d8b7daff3b1 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -274,6 +274,7 @@ Downsides - We've extended the frontmatter syntax with an infostring, undoing some of the "familiarity" benefit - Potential congantive disonance as those familiar with frontmatter are used to YAML being there - With `>>>` it isn't quite like HEREDOC to have less overhead +- `>>>`, `<<<`, `|||`, `===` at the beginning of lines start to look like merge conflicts which might confuse external tools - Doesn't feel very rust-like Note: From eab230992431288f0076da534560d4b62b3706fb Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 1 Feb 2024 16:48:56 -0600 Subject: [PATCH 51/60] Switch frontmatter syntax - Based on the escaping problem, switch away from markdown-style - Based on my experience, switch to frontmatter-style - Based on observing a new-to-this-syntax user, allow blank lines - Allow infostring to be optional but don't remve it --- text/3503-frontmatter.md | 129 +++++++++++++++++---------------------- 1 file changed, 56 insertions(+), 73 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index d8b7daff3b1..597c7fdc9d9 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -10,10 +10,10 @@ Add a frontmatter syntax to Rust as a way for [cargo to have manifests embedded in source code][RFC 3502]: ````rust #!/usr/bin/env cargo -```cargo +--- [dependencies] clap = { version = "4.2", features = ["derive"] } -``` +--- use clap::Parser; @@ -30,11 +30,6 @@ fn main() { } ```` -Note that to share these in markdown, a priority use case, extra backticks are needed for the markdown code fence to escape the frontmatter code fence. -We expect most users will not be familiar enough with the markdown spec to know this, especially for one of our primary target audiences: those new to Rust. -This can also be frustrating for experienced users as three backticks is an ingrained habbit and it is common to need to go back and edit a post to properly escape the frontmatter. -However, when weighing out the syntactic needs and the alternatives, we felt this was the least bad option. - # Motivation [motivation]: #motivation @@ -55,14 +50,13 @@ name: My Blog Post Hello world! ``` -We are carrying this concept over to Rust with a twist: using fence code blocks which -will be familiar to Rust developers when documenting their code: +We are carrying this concept over to Rust while merging some lessons from commonmark's fenced code blocks: ````rust #!/usr/bin/env cargo -```cargo +--- [dependencies] clap = { version = "4.2", features = ["derive"] } -``` +--- use clap::Parser; @@ -79,20 +73,19 @@ fn main() { } ```` -As we work to better understand how tool authors will want to use frontmatter, we are restricting it to just the `cargo` infostring. -This means users will only be exposed to this within the concept of ["cargo script"][RFC 3502]. +Like with [commonmark code fences](https://spec.commonmark.org/0.30/#info-string), +an info-string is allowed after the opening `---` for use by the command interpreting the block to identify the contents of the block. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -When parsing Rust code, after stripping the shebang (`#!`), rustc will strip a fenced code block: -- Must be immediately at the top (after shebang stripping), meaning no blank lines -- Opens with 3+ backticks and "cargo" followed by a newline - - As we aren't supporting an arbitrarily nested file format (though may show up in one), we likely don't need the flexibility - - We are prioritizing on "one right way to do it" to make it easier to learn to write and to read a variety of files. -- All content is ignored until the same number of backticks is found at the start of a line. - It is an error to have anything besides spaces and tabs between the backticks and the newline. - - Unlike commonmark, it is an error to not close the fenced code block seeing to detect problems earlier in the process seeing as the primary content is what comes after the fenced code block +When parsing Rust code, after stripping the shebang (`#!`), rustc will strip the frontmatter: +- May include 0+ blank lines (whitespace + newline) +- Opens with 3+ dashes followed by 0+ whitespace, an optional identifier, 0+ whitespace, and a newline + - The variable number of dashes is an escaping mechanism in case `---` shows up in the content +- All content is ignored by `rustc` until the same number of dashes is found at the start of a line. + The line must terminate by 0+ whitespace and then a newline. +- Unlike commonmark, it is an error to not close the frontmatter seeing to detect problems earlier in the process seeing as the primary content is what comes after the frontmatter As cargo will be the first step in the process to parse this, the responsibility for high quality error messages will largely fall on cargo. @@ -102,7 +95,6 @@ the responsibility for high quality error messages will largely fall on cargo. - A new concept for Rust syntax, adding to overall cognitive load - Ecosystem tooling updates to deal with new syntax -- **When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists)** # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives @@ -112,8 +104,6 @@ we considered starting with only allowing this in the root `mod` (e.g. `main.rs` but decided to allow it in any file mostly for ease of implementation. Like with Python, this allows any file in a package (with the correct deps and `mod`s) to be executed, allowing easier interacting experiences in verifying behavior. -As for the hard-coded infostring used by cargo, that is a decision for [RFC 3502]. - ## Required vs Optional Shebang We could require the shebang to be present for all cargo-scripts. @@ -126,6 +116,28 @@ However, statically analyzing a shebang is [complicated](https://stackoverflow.c and we are wanting to avoid it in the core workflow. This isn't to say that tools like rust-analyzer might choose to require it to help their workflow. +## Blank lines + +Originally, the proposal viewed the block as being "part of" the shebang and didn't allow them to be separated by blank lines. +However, the shebang is optional and users are likely to assume they can use blanklines +(see https://www.youtube.com/watch?v=S8MLYZv_54w). + +This could cause ordering confusion (doc comments vs attributes vs frontmatter) + +## Infostring + +The main question on infostrings is whether they are tool-defined or rustc-defined. +At one time, we proposed requiring the infostring and requiring it be `cargo` as a way to defer this decision. + +As the design requirements are catered to processing by external tools, as opposed to rustc, +we are instead reserving this syntax for external tools by making the infostrings tool-defined. +The Rust toolchain (rustc, clippy, rustdoc, etc) already have access to attributes for user-provided content. +If they need a more ergonomic way of specifying content, we should solve that more generally for attributes. + +With that decision made, the infostring can be optional. +Can it also be deferred out? +Possibly, but we are leaving them in for unpredictable exception cases and in case users want to make the syntax explicit for their editor (especially if its not `cargo` which more trivial editor implementations will likely assume). + ## Syntax [RFC 3502] lays out some design principles, including @@ -149,7 +161,7 @@ When choosing the syntax, our care-abouts are - Leave the door open in case we want to reuse the syntax for embedded lockfiles - Leave the door open for single-file `lib`s -### Fenced Code Block Frontmatter +### Frontmatter This proposed syntax builds off of the precedence of Rust having syntax specialized for an external tool (doc-comments for rustdoc). @@ -168,10 +180,11 @@ This proposal mirrors the location of YAML frontmatter (absolutely first). As we learn more of its uses and problems people run into in practice, we can evaluate if we want to loosen any of the rules. -We are intentionally supporting only a subset of commonmark code fences. -Markdown, like HTML, is meant to always be valid which is different than Rust syntax. -Differences include: -- backticks but not tilde's +Differences with YAML frontmatter include: +- Variable number of dashes (for escaping) +- Optional frontmatter + +Besides characters, differences with commonmark code fences include: - no indenting of the fenced code block - open/close must be a matching pair, rather than the close having "the same or more" @@ -187,47 +200,45 @@ Benefits: - In the future, this can be leveraged by other build systems or tools Downsides: -- **When sharing in markdown documents (e.g. GitHub issues), requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists)** - - Maintainers seeding GitHub issue templates with 4 backticks can help - Familiar syntax in an unfamiliar use may make users feel unsettled, unsure how to proceed (what works and what doesn't). - If viewed from the lens of a comment, it isn't a variant of comment syntax like doc-comments ### Alternative 1: Vary the opening/closing character -Instead of backticks, we could do another character, like -- `-`, making it look like YAML presentation streams, following the pattern of static site generators - - `+` like [zola's frontmatter](https://www.getzola.org/documentation/getting-started/overview/#markdown-content) -- `~`, using a lesser known markdown character +Instead of dashes, we could do another character, like +- backticks, like in commonmark code fences + - `~`, using a lesser known markdown code fence character +- `+` like [zola and hugo's TOML frontmatter](https://www.getzola.org/documentation/getting-started/overview/#markdown-content) - `=` - Open with `>>>` and close with `<<<`, like with HEREDOC (or invert it) -In practice: -```rust +In practice (with infostrings): +````rust #!/usr/bin/env cargo ----cargo +```cargo [package] edition = "2018" ---- +``` fn main() { } -``` +```` ```rust #!/usr/bin/env cargo -+++cargo +~~~cargo [package] edition = "2018" -+++ +~~~ fn main() { } ``` ```rust #!/usr/bin/env cargo -~~~cargo ++++cargo [package] edition = "2018" -~~~ ++++ fn main() { } @@ -263,19 +274,10 @@ fn main() { } ``` -Benefits -- With `-`, it builds on people's familiarity with static site generators -- People can insert cargo-scripts into markdown (like chat, github issues) - without being familiar enough with markdown to know how to escape backticks - and to actually remember how to do it - Downsides -- With `-` - - We've extended the frontmatter syntax with an infostring, undoing some of the "familiarity" benefit - - Potential congantive disonance as those familiar with frontmatter are used to YAML being there - With `>>>` it isn't quite like HEREDOC to have less overhead - `>>>`, `<<<`, `|||`, `===` at the beginning of lines start to look like merge conflicts which might confuse external tools -- Doesn't feel very rust-like +- Backticks have a problem with users knowing how to and remembering to escape these blocks when sharing them in markdown. Knowing the syntax (only because I've implemented a parser for it), I'm at about 50/50 on whether I properly escape. Note: - `"` was not considered because that can feel too familiar and users might carry over their expectations for how strings work @@ -617,28 +619,9 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Support infostring attributes - We need to better understand use cases for how this should be extended, particularly what the syntax should be (see infostring language) - A safe starting point could be to say that a space or comma separates attributes and everything after it is defined as part of the "language" -- Loosen the code-fence syntax, like allowing newlines - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect -## Optional or additional infostrings - -We could support: -- Treat `cargo` as the default infostring -- Support more infostring languages - -The question comes down to whether -- rustc owns the definition of the infostring, allowing us to add additional types of metadata (rustfmt config, static analyzer config, etc) - - This would be similar to our [hard coding of "tool" attributes](https://github.com/rust-lang/rust/issues/44690) -- the shebang tool owns the definition of the infostring - -By us hard coding `cargo` in the infostring in rustc, -we are intentionally deferring the decision for which path we should go down. - -We can add additional infostrings on a case-by-case basis. -In doing so, we can learn more about the use cases involved which can help us -get a better picture for which route we should go down. - ## Multiple frontmatters At least for cargo's use cases, the only other file that we would consider supporting is `Cargo.lock` From 82352ed14eb0426efc0e57eec718329f33b65f82 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 1 Feb 2024 19:24:43 -0600 Subject: [PATCH 52/60] fix: Cover include macro --- text/3503-frontmatter.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 597c7fdc9d9..5d8689afc44 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -79,7 +79,7 @@ an info-string is allowed after the opening `---` for use by the command interpr # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -When parsing Rust code, after stripping the shebang (`#!`), rustc will strip the frontmatter: +When parsing Rust source, after stripping the shebang (`#!`), rustc will strip the frontmatter: - May include 0+ blank lines (whitespace + newline) - Opens with 3+ dashes followed by 0+ whitespace, an optional identifier, 0+ whitespace, and a newline - The variable number of dashes is an escaping mechanism in case `---` shows up in the content @@ -87,6 +87,9 @@ When parsing Rust code, after stripping the shebang (`#!`), rustc will strip the The line must terminate by 0+ whitespace and then a newline. - Unlike commonmark, it is an error to not close the frontmatter seeing to detect problems earlier in the process seeing as the primary content is what comes after the frontmatter +This applies anywhere shebang stripping is performed. +For example, if [`include!`](https://doc.rust-lang.org/std/macro.include.html) strips shebangs, then it will also frontmatter. + As cargo will be the first step in the process to parse this, the responsibility for high quality error messages will largely fall on cargo. From 40228b8095418a45acb80bd771c377d4afe1adec Mon Sep 17 00:00:00 2001 From: Ed Page Date: Thu, 1 Feb 2024 19:25:44 -0600 Subject: [PATCH 53/60] fix: Talk about infostring stabilization --- text/3503-frontmatter.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 5d8689afc44..d9c12c7a1e9 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -140,6 +140,7 @@ If they need a more ergonomic way of specifying content, we should solve that mo With that decision made, the infostring can be optional. Can it also be deferred out? Possibly, but we are leaving them in for unpredictable exception cases and in case users want to make the syntax explicit for their editor (especially if its not `cargo` which more trivial editor implementations will likely assume). +We may at least defer stabilization of infostrings. ## Syntax From f9be5d49b181e945f56f890815240ce8b9ba397a Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 7 Feb 2024 08:30:59 -0600 Subject: [PATCH 54/60] fix(future): Use latest syntax --- text/3503-frontmatter.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index d9c12c7a1e9..be5a562a6cc 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -637,18 +637,18 @@ So **if** we decide we need to embed additional metadata, we have a couple of op Distinct blocks, maybe with newlines ````md -```Cargo.toml -``` +---Cargo.toml +--- -```Cargo.lock -``` +---Cargo.lock +--- ```` Continuous blocks ````md -```Cargo.toml -```Cargo.lock -``` +---Cargo.toml +---Cargo.lock +--- ```` Distinct blocks is more like the source inspiration, markdown, From 2e68244547f179027b2718658396871d26520abc Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 7 Feb 2024 08:35:37 -0600 Subject: [PATCH 55/60] fix(future): Loosen infostring requirements --- text/3503-frontmatter.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index be5a562a6cc..c804de06b07 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -81,7 +81,7 @@ an info-string is allowed after the opening `---` for use by the command interpr When parsing Rust source, after stripping the shebang (`#!`), rustc will strip the frontmatter: - May include 0+ blank lines (whitespace + newline) -- Opens with 3+ dashes followed by 0+ whitespace, an optional identifier, 0+ whitespace, and a newline +- Opens with 3+ dashes followed by 0+ whitespace, an optional term (one or more characters excluding whitespace and commas), non-comma characters), 0+ whitespace, and a newline - The variable number of dashes is an escaping mechanism in case `---` shows up in the content - All content is ignored by `rustc` until the same number of dashes is found at the start of a line. The line must terminate by 0+ whitespace and then a newline. @@ -142,6 +142,9 @@ Can it also be deferred out? Possibly, but we are leaving them in for unpredictable exception cases and in case users want to make the syntax explicit for their editor (especially if its not `cargo` which more trivial editor implementations will likely assume). We may at least defer stabilization of infostrings. +The infostring syntax was selected to allow file names (e.g. `Cargo.lock`). +Additional attributes are left to a future possibility. + ## Syntax [RFC 3502] lays out some design principles, including From f037dd05c759add6719616c3c8f800536c236bd4 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 7 Feb 2024 08:42:09 -0600 Subject: [PATCH 56/60] fix(future): Call out different infostring attribute syntax --- text/3503-frontmatter.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index c804de06b07..fc15b17bd24 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -625,7 +625,8 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Support infostring attributes - We need to better understand use cases for how this should be extended, particularly what the syntax should be (see infostring language) - - A safe starting point could be to say that a space or comma separates attributes and everything after it is defined as part of the "language" + - Some tools use use comma separated attributes, some use more elaborate syntax wrapped in `{}` + - A safe starting point could be to say that a space or comma separates the identifier from the attributes and everything after it is defined as part of the "language" - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect From 6d8bf534689ecb5a7d2df0315f21976961cc8b2d Mon Sep 17 00:00:00 2001 From: Ed Page Date: Wed, 7 Feb 2024 08:43:25 -0600 Subject: [PATCH 57/60] fix(ref): Bad edit --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index fc15b17bd24..bbc561c4a19 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -81,7 +81,7 @@ an info-string is allowed after the opening `---` for use by the command interpr When parsing Rust source, after stripping the shebang (`#!`), rustc will strip the frontmatter: - May include 0+ blank lines (whitespace + newline) -- Opens with 3+ dashes followed by 0+ whitespace, an optional term (one or more characters excluding whitespace and commas), non-comma characters), 0+ whitespace, and a newline +- Opens with 3+ dashes followed by 0+ whitespace, an optional term (one or more characters excluding whitespace and commas), 0+ whitespace, and a newline - The variable number of dashes is an escaping mechanism in case `---` shows up in the content - All content is ignored by `rustc` until the same number of dashes is found at the start of a line. The line must terminate by 0+ whitespace and then a newline. From ce78d923e91cec86e513a2305a61041efc7c3cad Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 9 Feb 2024 09:04:58 -0600 Subject: [PATCH 58/60] fix(alt): Revamp prefix section --- text/3503-frontmatter.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index bbc561c4a19..a8a98933914 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -309,11 +309,14 @@ The hope would be that we could get buy-in from other languages. - Future evolution: Allow any `info` string with cargo checking for `content.starts_with(["cargo", "cargo,"])` - Future evolution: Allow `frontmatter` attribute on any module -If we dropped future possibilities for additional content, we could remove the opening/closing syntax and tweak it to avoid attribute ambiguity, -greatly reducing the minimum syntax needed in some cases. +Syntactically, this avoids confusion with attributes by being stripped before lexing. +We could make this less ambiguous by using a double hash. ````rust #!/usr/bin/env cargo -## package.edition = "2018" +## ```cargo +## [dependencies] +## foo = "1.2.3" +## ``` fn main() {} ```` @@ -326,11 +329,20 @@ Benefits - Maybe we can get others on board with this syntax Downsides +- `#` prefix plus a TOML `[heading]` looks too much like a Rust `#[attribute]`. - More syntactically heavy than the frontmatter solution - Visually - - More work to type it out / copy-paste + - More work to type it out or copy-paste between cargo scripts and regular manifests - More to get wrong -- Requires users to deal with leading characters when editing/copying/pasting the manifest + +If we dropped future possibilities for additional content, we could remove the opening/closing syntax, +greatly reducing the minimum syntax needed in some cases. +````rust +#!/usr/bin/env cargo +## package.edition = "2018" + +fn main() {} +```` ### Alternative 3: Doc-comment From 717bacd6dc956c1f10c2374017a45856e14ceeee Mon Sep 17 00:00:00 2001 From: Ed Page Date: Fri, 9 Feb 2024 14:03:24 -0600 Subject: [PATCH 59/60] fix: Typo --- text/3503-frontmatter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index a8a98933914..099202a789c 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -637,7 +637,7 @@ pprint([(k, v["title"]) for k, v in data.items()][:10]) - Support infostring attributes - We need to better understand use cases for how this should be extended, particularly what the syntax should be (see infostring language) - - Some tools use use comma separated attributes, some use more elaborate syntax wrapped in `{}` + - Some tools use comma separated attributes, some use more elaborate syntax wrapped in `{}` - A safe starting point could be to say that a space or comma separates the identifier from the attributes and everything after it is defined as part of the "language" - Add support for a `#[frontmatter(info = "", content = "")]` attribute that this syntax maps to. - Since nothing will read this, whether we do it now or in the future will have no affect From 837efd1603dd86805cbfb099549f9fc6bff20c8d Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 20 Feb 2024 10:27:46 -0600 Subject: [PATCH 60/60] feat(rationale): Step through why this needs to be in rustc --- text/3503-frontmatter.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/text/3503-frontmatter.md b/text/3503-frontmatter.md index 099202a789c..e734c02c6f1 100644 --- a/text/3503-frontmatter.md +++ b/text/3503-frontmatter.md @@ -168,6 +168,26 @@ When choosing the syntax, our care-abouts are - Leave the door open in case we want to reuse the syntax for embedded lockfiles - Leave the door open for single-file `lib`s +### Why add to Rust syntax, rather than letting Cargo handle it + +The most naive way for cargo to handle this is for cargo to strip the manifest, write the Rust file to a temporary file, and compile that. +This is what has traditionally been done with various iterations of "cargo script". + +This provides a second-class experience which runs counter to one of the design goals +- Error messages, `cargo metadata`, etc point to the stripped source with an "odd" path, rather than the real source +- Every tool that plans to support it would need to be updated to do stripping (`cargo fmt`, `cargo clippy`, etc) + +A key part in there is "plan to support". +We'd need to build up buy-in for tools to be supporting a Cargo-only syntax. +This becomes more difficult when the tool in question tries to be Cargo-agnostic. +By having Cargo-agnostic external tool syntax in Rust, this mostly becomes transparent. + +We could build a special relationship with rustc to support this. +For example, rustdoc passes code to rustc on stdin and sets `UNSTABLE_RUSTDOC_TEST_PATH` and `UNSTABLE_RUSTDOC_TEST_LINE` to control how errors are rendered. +We could then also special case the messages inside of cargo. +This both adds a major support burden to keep this house of lies standing but still falls short when it comes to tooling support. +Now every tool that wants to support the Cargo-only syntax has to build their own house of lies. + ### Frontmatter This proposed syntax builds off of the precedence of Rust having syntax specialized for an external tool