Clarify that 11.9 ASI statement list is informative

[jmm]

ASI is pretty confusing. In just the past few weeks I've seen quite a bit of uncertainty about it, including on the part of the author of a stage 1 proposal; myself; and a well known figure in ES world. The latter 2 cases specifically involved the list of statements in 11.9 and whether they are normative or just examples (and led to #284 & #285 here). In my case I was interpreting the ASI semantics in the context of a proposal whose grammar productions aren't in the spec yet, and therefore of course not in the list.

How would you feel about adding this language that clarifies that the list is just a hint?

(The preceding list is informative. If it were omitted, the remaining prose and grammar productions would fully describe the semantics.)

Is it intended to keep that list updated so that it's exhaustive (assuming it is currently)?

[michaelficarra]

👍. You probably shouldn't make it a parenthetical, and we could probably make that whole paragraph a non-normative NOTE.

[jmm]

@michaelficarra Thanks for the feedback. I'll take a look at reformatting as you've suggested (basically just need to look up an example of a NOTE so I can follow the existing style).

[michaelficarra]

@jmm notes are defined using <emu-note>.

[allenwb]

In ES2015 (and in previous editions) all of 11.9.1 (or equivalent) starting with the list of restricted grammar productions are formatted to indicate they are informative notes. All informative sections start with a paragraph whose first four characters are "NOTE". In the PDF version of ES2015 the left margin of notes the first and all subsequent note paragraph are inset and use a 1 point font size reduction. The html version of ES2015 also has the left margin inset and the reduced font size. This convention seems to be preserved in the ES2016 draft

So basically, this issue ems like a combination of readers not understanding the conventions for identifying informative notes and the use of typographic cues that are too subtle.

Rather than trying to "fix" this one instance it probably would be better to make the typographic distinction of notes be less subtle. Perhaps boxing or a background color distinction.

[allenwb]

There seems to be another point of confusion in this original issue. It said:

The latter 2 cases specifically involved the list of statements in 11.9 and whether they are normative or just examples

There is no list of statements where ASI can be applied because generally ASI applies to all statements. The only list of production in 11.9 is the list of "restricted productions" in 11.9.1 and it is clearly labeled as such. Restricted productions are productions that contain the _{[No LineTerminator here]} annotation. If a new restricted production is added to the grammar it should be added to the 11.9.1 list (even though the list is only informative).

[jmm]

@allenwb Thanks for the background.

it probably would be better to make the typographic distinction of notes be less subtle. Perhaps boxing or a background color distinction.

That sounds good to me in general.

But...

So basically, this issue ems like a combination of readers not understanding the conventions for identifying informative notes and the use of typographic cues that are too subtle.

Rather than trying to "fix" this one instance

The passage I was suggesting changing is in the paragraph right at the beginning of 11.9 and isn't marked as a NOTE currently. Can that whole paragraph, or else the list of statement types, be marked as a NOTE?

Thanks for the tip @michaelficarra.

[jmm]

@allenwb

There is no list of statements where ASI can be applied because generally ASI applies to all statements. The only list of production in 11.9 is the list of "restricted productions" in 11.9.1 and it is clearly labeled as such. Restricted productions are productions that contain the [No LineTerminator here] annotation. If a new restricted production is added to the grammar it should be added to the 11.9.1 list (even though the list is only informative).

Yeah, I'm referring to the parenthesized list in this sentence that informally refers to families of productions:

Certain ECMAScript statements (empty statement, let, const, import declarations, some forms of export declarations, variable statement, expression statement, debugger statement, continue statement, break statement, return statement, and throw statement) must be terminated with semicolons.

What I was suggesting here is adding text that explains that the list doesn't define where ASI is applied -- the other prose + productions do, like you confirmed for me over in tc39/proposal-class-public-fields#25 (comment).

[jmm]

Well, unless we were talking about different lists at the time 😬. But in any case I think the one I'm talking about is informative.

[allenwb]

@jmm
Ah, I wondered why you kept talking about 11.9 when all the actual normative text is in 11.9.1!

Yes, the entire introductory paragraph of 11.9 could be labelled and formatted as a NOTE. The original Ecma/ISO specification drafting guidelines allow introductory paragraphs like this that don't contain anything normative but yet are not tagged as informative paragraph. Early editions of the ES spec. had these lots of places and we have generally formatted them as NOTEs when we have run into situations where they created normative confusions. For example see 22.1.3.7. However, the standards orgs kind of frown upon starting sections with long explicit NOTEs and would prefer that they be moved to follow the actual normative text. Basically, short non-normative content-free intro paragraph ok, starting with a long descriptive informative NOTE, less ok.

In this particularly case, I think the problem is that statements ending with ; is the normal case rather than the exception and as new "statement" forms are added that list appears to have more significance than was ever intended.

By suggestion, replace the first paragraph of 11.9 with:

Most ECMAScript statements must be terminated with semicolons.

With that change the paragraph is a fairly straight forward introductory paragraphs containing no real normative text that might case confusion.

[jmm]

@allenwb

Yes, the entire introductory paragraph of 11.9 could be labelled and formatted as a NOTE. The original Ecma/ISO specification drafting guidelines allow introductory paragraphs like this that don't contain anything normative but yet are not tagged as informative paragraph. Early editions of the ES spec. had these lots of places and we have generally formatted them as NOTEs when we have run into situations where they created normative confusions. For example see 22.1.3.7. However, the standards orgs kind of frown upon starting sections with long explicit NOTEs and would prefer that they be moved to follow the actual normative text. Basically, short non-normative content-free intro paragraph ok, starting with a long descriptive informative NOTE, less ok.

Thanks! Those are very interesting insights that I was not aware of.

In this particularly case, I think the problem is that statements ending with ; is the normal case rather than the exception and as new "statement" forms are added that list appears to have more significance than was ever intended.

Good point. I think a couple of other factors that are making that seem more significant / scrutinized are that it seems to be increasingly popular to author without semicolons (not with me :)) and proposals for statement forms that aren't yet in the spec are more common and being read and interpreted by more people.

By suggestion, replace the first paragraph of 11.9 with:

Most ECMAScript statements must be terminated with semicolons.

With that change the paragraph is a fairly straight forward introductory paragraphs containing no real normative text that might case confusion.

I just want to make sure I'm on the same page: a) replace the first paragraph, so that it reads what you suggested in its entirety, or b) replace the first sentence so that it reads?:

Most ECMAScript statements must be terminated with semicolons. Such semicolons may always appear explicitly in the source text. For convenience, however, such semicolons may be omitted from the source text in certain situations. These situations are described by saying that semicolons are automatically inserted into the source code token stream in those situations.

(BTW, reading this right now I'd suggest "terminated with a semicolon" either way.)

Statement vs declaration

Re: the language:

[...] ECMAScript statements [...] must be terminated with semicolons

Are declarations implicitly (or explicitly that I'm overlooking) considered to be statements? It's back and forth whether they're called out separately or lumped in with statements. E.g.:

• 4.2 ECMAScript Overview

  sequences of statements and declarations

• 11.9 Automatic Semicolon Insertion

  Certain ECMAScript statements [...] import declarations, some forms of export declarations

  (Emphasis mine.)

• 13 ECMAScript Language: Statements and Declarations

  Statements and Declarations

  And there's a Declaration production that isn't in the Statement production (though it is under StatementList).

• A.3 Statements

  Equivalent section of the grammar summary is just titled Statements.

• A.5 Scripts and Modules

  I don't think import / export are referred to as statements anywhere.

For that matter, why are these productions required to end with a semicolon?:

ImportDeclaration:
  import ImportClause FromClause ;
  import ModuleSpecifier ;

export * FromClause ;
export ExportClause FromClause ;
export ExportClause ;

Maybe I'm trying to do this too quick, but, without a semicolon, it seems unambiguous when they end.

[allenwb]

I just want to make sure I'm on the same page: a) replace the first paragraph, so that it reads what you suggested in its entirety, or b) replace the first sentence so that it reads?:

I meant "first sentence". And "a semicolon" is certainly better.

The distinction between statements and declarations is more semantic than syntactic, but it is probably better to explicitly say "statements and declarations" to avoid causing confusion.

For that matter, why are these productions required to end with a semicolon?:

Because the convention for C-like languages is that the elements of a statement list (statements and declarations) each terminate with a semicolon unless their last token is a } such as FunctionDeclaration and Block (or if the syntactic tail of the statement is another statement). ES6 applied the same convention to ClassElementList.

So the norm is for statements and declaration to end with a semicolon. ASI is a convenience mechanism that allow most such semicolons to be omitted. But it works an a syntactic error recovery mechanism. The grammar says that the semicolon is required, but the ASI error recovery rules trys to insert semicolons that are missing.

[bterlson]

Agree with @allenwb here, we should just remove the list so it's clear it's a non-normative intro. Will also mean we don't need to worry about maintaining the list. I'll commit a fix for this now.

[jmm]

@allenwb

Because the convention for C-like languages is that the elements of a statement list (statements and declarations) each terminate with a semicolon unless their last token is a }

Ok, thanks. Is there any special reason this production departs from that convention?:

export ExportClause ;

👍 to the change that was committed. I agree @allenwb's idea to actually delete the list is even better. Thanks all.

[mathiasbynens]

👍 Thanks for bringing this up, @jmm. Here’s to more clarity and less non-normative confusion! 🍻

[jmm]

Is there any special reason this production departs from that convention?:

Hmm, I guess because there's no positive lookahead and there can be things like:

export {}
from;

export {}
from = "whatever";

[jmm]

Thanks @mathiasbynens, cheers to that! ASI is confusing enough as it is.