cue: use dots as a separator for folding instead of spaces?

[ghost]

This input:

outer middle inner: 3

produces this with CUE

{
   "outer": {
      "middle": {
         "inner": 3
      }
   }
}

and this with YAML:

{
   "outer middle inner": 3
}

this seems like too much sugar to be worth it

https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/fold.md

[jlongtine]

@cup I don't understand why you think this language feature isn't "worth it". Can you elaborate on the cost, so I can better understand why you feel it is too costly?

As an aside, CUE isn't a YAML extension, it is a superset of JSON. As such, I'm not fully understanding your YAML example.

[ghost]

@jlongtine languages dont exist in a vacuum. I would posit that if someone naive
to all sees this:

outer middle inner: 3

they would expect that it represents this structure:

{'outer middle inner': 3}

in which case CUE folding is a clear violation of POLA:

https://wikipedia.org/wiki/Principle_of_least_astonishment

the counter to this would be, having it represent this instead:

{
   "outer": {
      "middle": {
         "inner": 3
      }
   }
}

gives you the benefit of short syntax and/or time saved typing. YAML offers a
similar tactic:

outer:
   middle:
      inner: 3

with one key difference: comprehension and readability are not sacrificed for
terseness. I use this exact syntax in my own repos:

https://github.com/cup/umber/blob/master/radio/flow-M.yaml

but I would never use CUE folding because of the ambiguity. I am all for shorter,
more expressive ways to represent complex data structures. But its a balancing
game. If you push it too far, you end up with something unreadable, or something
only understood by the author. Like Perl, and like CUE folding.

[harrygallagher4]

This seems like more of a personal issue with the syntax, rather than an actual "issue" with cue. I'm not sure, but I'd be willing to bet that the designers of cue recognized that this is a slightly unorthodox feature but had their reasons to explicitly include it.

I agree that it's slightly confusing at first glance, but it isn't exactly a hard concept to grasp. There's a reason the documentation of folding is so short, there's hardly anything to explain.

On top of that, dropping this syntax would be a non-backwards-compatible change and would probably require many people (myself included) to update their cue files. This issue seems like too much of a hassle to be worth it 😉

[ghost]

@harrygallagher4 thanks for the response, but you seem to be trying to hand
wave over some valid concerns.

willing to bet that the designers of cue recognized that this is a slightly
unorthodox feature but had their reasons to explicitly include it.

you say "im sure they had a good reason" is not a reason. if they have a good
reason, i prefer it to be explicitly documented, so that i can understand their
process in weighing the pros and cons. without that, a cynic could assume they
did no such process, and made the decision because its less characters with no
further consideration.

I agree that it's slightly confusing at first glance, but it isn't exactly a
hard concept to grasp. There's a reason the documentation of folding is so
short, there's hardly anything to explain.

the point id argue a majority of people stumbling across this code arent going
to know what it does, or worse assume it does something else. thats not a reason
to kill folding on its own, but should be weighed in decision making.

On top of that, dropping this syntax would be a non-backwards-compatible
change and would probably require many people (myself included) to update
their cue files.

respectfully I am not concerned with whether you have to rewrite files if this
feature is removed. i am more looking at a high level "should this feature have
ever been implemented, if so how do the pros outweigh the cons". and not so much
"hey bro! i dont want to rewrite any scripts!". especially considering that this
software has not even reached 0.1.0, let alone 1.0.0:

https://github.com/cuelang/cue/releases

which means plenty of time and room is available for big removals or additions
to the spec. Finally I am not the only one with reservations, I came here in
response to others voicing similar concerns:

https://news.ycombinator.com/item?id=20363509

[solson]

I would posit that if someone naive to all sees this:

outer middle inner: 3

they would expect that it represents this structure:

{'outer middle inner': 3}

I believe the opposite, that this YAML behavior is the most surprising. There are not many languages that allow unquoted identifiers to include whitespace, so it's natural to parse this as three separate identifiers interacting somehow (and then it just takes one small bit of documentation to resolve what that interaction is).

Additionally, CUE's behavior has precedent in other major configuration languages like Nix. Nix does use dots instead of spaces in its version of this syntax, to be fair, but there's even more precedent for languages using spaces where other languages would use dots (Smalltalk, Obj-C, and many others).

For me, it would reduce the usability of the language to remove this feature from CUE.

[harrygallagher4]

@cup Alright, I'll do my best to address your concerns in this long-winded reply, excuse the digressions here because we seem to be getting into territory of the philosophy of "good features", when something is "worth it", the relationship of different languages, cynicism, and a lot of other things.

you say "im sure they had a good reason" is not a reason. if they have a good
reason, i prefer it to be explicitly documented, so that i can understand their
process in weighing the pros and cons. without that, a cynic could assume they
did no such process, and made the decision because its less characters with no
further consideration.

Okay, I'll accept that saying that isn't a reason in itself, but here's why I personally find this useful:

You can easily set a deeply nested property: this is obviously useful. It's not even sort of non-standard "out there" feature, most languages have some syntax for doing this commonly with dots but using spaces instead isn't much different. In the case of cue, a small difference is that if any of these mid-level structs don't exist they are created which again is useful in cue's overall goal of reducing boilerplate. In addition, folding doesn't interrupt the indentation flow in order to set just one property. When you have a lot of levels of nesting it can become difficult to navigate your eyes/brain back and forth between all of the {'s and }'s (levels of nesting). I suppose using "outer": { "middle": { "inner": 3 } } in one line solves this but that's kinda the point of folding since you're just dropping the quotes and brackets.

the point id argue a majority of people stumbling across this code arent going
to know what it does, or worse assume it does something else. thats not a reason
to kill folding on its own, but should be weighed in decision making.

@solson covered this nicely. I'd like to add that avoiding features that people won't understand properly without reading about them is a bad mindset. If cue was created to add syntax sugar to YAML, sure, changing the behavior here would probably be a pretty bad idea, but cue is its own can of worms. If we never implemented new syntax/features just because other languages were different then nobody would ever push the status quo enough to be innovative. I'm not saying that this super simplistic syntax is some huge game-changer, just that if everybody adapted this reasoning we would be stuck with boring, homogenized languages.

respectfully I am not concerned with whether you have to rewrite files if this
feature is removed. i am more looking at a high level "should this feature have
ever been implemented, if so how do the pros outweigh the cons". and not so much
"hey bro! i dont want to rewrite any scripts!". especially considering that this
software has not even reached 0.1.0, let alone 1.0.0:
...
which means plenty of time and room is available for big removals or additions
to the spec.

Agreed. Perhaps the developers will choose to remove this before 1.0 or change the syntax to use dots or something more approachable. However, I believe I've outlined some reasons why this is useful, and why I believe that dropping something because it's not approachable is a bad idea.

"should this feature have ever been implemented, if so how do the pros outweigh the cons"

My response (and @solson's as it seems-- sorry to drag you into this) to this is yes. Yours seems to be no. It just comes down to a difference in opinion here. As I'm not a maintainer of cue, I'll leave it up to them and stop flooding the replies to this issue with walls of text, I just personally disagree with the reasoning of "this is slightly confusing, and does something different in YAML, so you should get rid of it"

tl;dr: hey bro! i don't wanna rewrite any scripts!

[mpvl]

@cup: thanks for the suggestion. As you mentioned, the time to make suggestions is now. Thanks everyone else as well for contributing to the conversation.

In CUE, whitespace has no semantic meaning. Spaces are only relevant within strings, which are always quoted. The only notable exception is the comma separator, which may be elided at the end of a line (a trick borrowed some Go's semicolon elision). As such, it is a bit surprising that a b c: 3 would be interpreted as "a b c": 3 as it would be highly inconsistent with the language. But if one looks at coming from YAML I see your point.

The "folding" feature has three main uses, though, and is quite crucial. I'll elaborate on that a bit and will explain what could possibly be done differently.

The first reason is the most obvious and most banal one: reducing curly-brace clutter. YAML probably at least partly owes its success to being able to reduce the large number of curly braces as often found in JSON. As whitespace has no meaning in CUE, it cannot take YAML's approach. (This was a deliberate choice. One reason was that CUE is meant to be generated and rewritten by machines as well, something where indentation-sensitive languages don't fare well.) The "folding" approach is quite effective at reducing curly braces as well, just taking a very different approach.

Secondly, this idea is actually not new. BCL (Borg Configuration Language), now roughly 15 years old, has a concept of objects, of which a 'job' is one example. Job specifications can often be long and a service may define many jobs. Before long, one loses sight of what fields relate to. So instead of writing:

job: {
    foo: {
        ...
    }
    bar: {
       ...
    }
    ...
}

one writes

job foo: {
    ...
}
job bar: {
    ...
}
...

This visually makes it easier to place definitions in context and reduces a level of indentation. BCL has its flaws, but it being the third-largest language at Google, we can say by now that this aspect of it works well. Further evidence of this is that this style is also adopted in other languages, like Hashicorp's HCL, which generalizes on this concept. In fact, CUE's interpretation is inspired by HCL's take on this concept.

Thirdly, many people are not very familiar with constraint-based languages or typed-feature structures, the foundations of CUE. One of the easiest ways to explain it is first to realize that any JSON configuration can be represented as a collection of paths in a tree associated with some leaf value. Then CUE is a generalization of this by allowing an expression on the left of the : to select a collection of paths (instead of a single one) and a constraint on the right of the : to be applied to each of these paths. It is helpful if it is actually possible to represent it as explained.

That said, one could argue the name separator should have been a . instead of a space. The choice to use spaces instead of dots clearly has its roots in BCL and HCL, but using dots would be more consistent with the notation for selectors (a b c: 3 can be referred to on the RHS as a.b.c). But from your reasoning, I don't think it would have helped YAML users to use dots, as a.b.c: 3 would equally be interpreted as "a.b.c": 3. The consistency argument would be even more relevant, though, if we add more powerful node selection for both LHS and RHS. But ultimately it is a matter of taste. People coming from BCL, GCL, or HCL would probably be more comfortable with spaces. But if there is a strong preference for dots overall, it could be changed. Introducing dots could be done in a backwards compatible manner (supporting both, at least for a while, and making cue fmt rewrite to using dots).

Anyway, I appreciate the tendency to want to remove features. I deliberately did not add conditional expressions, have removed functions early on, and am thinking of removing the % operator (floating point remainder), making it a library call instead. I would love to get rid of comprehensions, and use a query language instead, but don't see how that could fully replace it.

[ghost]

In CUE, whitespace has no semantic meaning

@mpvl I am confused by this, as it seems the exact opposite of what you said is
true. If you have this markup:

inner:3

obviously it would represent this structure:

{inner:3}

going along with your argument, you can add certain whitespace with no semantic
meaning:

inner:     3

still represents:

{inner:3}

but the confusing part is, the original example I gave:

outer middle inner: 3

directly contradicts what you said, as pretty much every whitespace has
semantic meaning. In this case each space representing a change in depth. I
appreciate your response, but can you clarify your statement?

[mpvl]

Mea culpa, I meant syntactically significant whitespace and, more specifically, the amount and type of a particular whitespace in a sequence is insignificant: it doesn't matter whether there is one whitespace or more and it doesn't matter which kind of whitespace. In other words, spaces are only consumed as is as part of quoted literal strings, but the type and number of spaces between tokens never has any meaning. This is in contrast to languages like YAML and Python, for instance, where indentation influences nesting. YAML in particular has a very complex lexer which determines when spaces are part of a literal token, instead of a token separator, but YAML is somewhat unique in this regard.

The whitespace handling of CUE is similar to many other languages, btw (C, Go, Java, Swift, JSON, ...). YAML will be full of surprises (POLA violations, if you will) to people only familiar to those. For example, converting YAML to JSON, true is true (a boolean), but true! is "true!" (a string, somewhat surprisingly), yet !true is null at least according to some implementations (that one surprised me at least!). To some extent this is unavoidable if one is only familiar with one approach and then switches to the other.

Anyway, CUE clearly falls into the C/Go/Java/JSON/HCL camp when it comes to how whitespace is interpreted. The curly braces are a reminder of that. In that regard there should be no surprise that outer middle inner at the very least cannot mean "outer middle inner", which is the point I and some others were trying to make. But this assumes some familiarity with any of these languages (and specifically their tokenization) and, granted, this point may get lost on people that don't. Just looking at CUE itself, without this background knowledge, one could argue it would be strange to not allow a: b c d (illegal CUE) as a shorthand for "a": "b c d" , but then allow a b c: d for "a b c": d, so that the latter interpretation cannot be right. But other than that it becomes a matter of learning the language.

Is there a notation for folding, though, you would find less confusing?

[ghost]

@mpvl I think you just made my case even stronger. Going off your latest
examples:

1. a: b c d

2. a b c: d

with YAML you get:

1. {"a": "b c d"}

2. {"a b c": "d"}

Consistent behavior. In both cases, the spaces are considered part of a string
token. Using the same markup with CUE, you get:

1. illegal syntax

2. {"a": {"b": {"c": "d"}}}

So you can see how the inconsistency with CUE might be a source of confusion and
frustration for some users.

[mpvl]

Your case was to drop folding as a b c reads as ”a b c”. I argued that’s impossibly a consistent interpretation in CUE, considering the rest of the language,
which this example does not disprove. That’s aside from the fact that people coming from other languages and unfamiliar with CUE, or C-style lexing, may be biased to read it differently.

What this example does show, however, is that a.b.c: val is a more consistent notation for folding, as this same path can be referred to on the RHS as f: a.b.c. So to get the symmetry you have in YAML, CUE should not eliminate folding, as you propose, but rather allow it and use dots instead of spaces on the LHS and spaces should be illegal on the LHS, just as a sequence of identifier or string tokens is illegal on the RHS.

But CUE adopted the current notation as it’s common with its ancestor languages. With some proposed extensions, the space notation will also look more familiar to typescript users, among others. Spaces look a bit cleaner than dots to me, and they are syntactically not necessary on the LHS as they are on the RHS. It also signifies that a path on the LHS is not entirely the same thing as a selector reference on the RHS (just as a LHS string is not the entirely the same thing as a RHS string in YAML). Finally, to stick with your original objection that it is common for people coming from YAML interpreting the notation as a string: flipping this around and viewing it from someone with a C-heritage background

file.cfg: {}

looks less obvious to me then

file cfg: {}

even though having a file.cfg on the RHS is clearly a reference. But this may be again my bias towards C-style languages, that leave no room for allowing file cfg to ever be interpreted as a single token, while a RHS file.cfg is always to be interpreted as a reference of some sort.

So even though dots would be more consistent, they are not obviously better, to me at least. But it may be my historical bias and perhaps dots are indeed clearer.

[ghost]

@mpvl using dots instead of spaces would be an acceptable compromise.

I would say its much clearer, but it seems we just differ on that point.

I dont really have further to add, it seems again the points have been made and
we just disagree

thanks

[mpvl]

@cup, thanks for your input. I’ll leave this issue open to father feedback from others regarding dots vs spaces or perhaps other alternatives.

Thanks!

[mpvl]

I thank @cup again for raising the issue and the link to HackerNews and others again for their input. I'll keep this issue open for a bit to solicit feedback on this issue. I'll change the title to more to focus on the notation issue, as folding is too important for CUE to give up. I'll focus on other features for now, knowing this change can be made in a backwards-compatible way, and allowing to take into account some of the proposed extensions.

[mpvl]

To throw one more data point in the discussion, there is auxiliary advantage of using dots over spaces:
it is possible to break a path across lines. For instance:

foo.bar.baz.qux.quux.quuz: 3

can be split across lines like

foo.bar.baz.
    qux.quux.quuz: 3

Because of the Go-style comma elision CUE uses, this is not possible with space-separators.
In converting some Rego to CUE while analyzing some of the proposals, I noticed one can sometimes convert what is a 10+-line rego program to a single path-value pair in CUE. A nesting level of 6 or more was not uncommon in these examples, which can make such a property rather useful (because nesting levels are deep and yet one will have a single CUE declaration).

[jlongtine]

@mpvl Would this still allow for using constraints in a folding path? I know you've mentioned being able to do things like foo bar [string]: int and similar. I guess the tokenizing and parsing of spaces vs dots wouldn't really impact that too much. Am I correct?

[mpvl]

Yes, that is still the plan. This syntax is needed to allow for associative lists, which are are the only thing missing to allow CUE to operate on a K8s API fully natively.

If anything else, using dots would make parsing easier. Or more precisely, using dots would make it easier to generate nice error messages. Initially there would be a transition period allowing both and then having cue fmt rewrite config files.

The proposal is here btw: https://cue-review.googlesource.com/c/cue/+/2280. Feedback is welcome. There are a few small improvements to be done there, I believe.

[mpvl]

Here is another alternative to consider. Instead of using dots, one could use colons:

foo: bar: baz:
    qux: quux: quuz: 3

This would not be indentation-sensitive, like YAML. Grammar-wise, declarations would be expressions (to some extent). A field without curly braces is then a shorthand for a single-field struct.

The thing against it is that it may be even less conventional and now we have something that looks a bit more like YAML, but yet means something very different, arguably being more confusing than the current situation. Consider, for instance,

foo:
  bar: 3
  baz: 4

which would mean { foo: { bar: 3 }, baz: 4 }. The use of CUE fmt would make that very clear, though, as it would rewrite it to:

foo:
  bar: 3
baz: 4

A thing in favor of it, is that it allows indicating the type of field on the path, allowing a mix of definitions and regular fields:

foo: def:: bar: true

This is not possible with either the space or dot notation, where one would have to write

foo: {
 def :: { bar: true }
}

Another thing in favor, is that it syntactically differentiates the meaning of an LHS vs RHS, while still resulting in a more streamlined syntax (at least so it seems, must investigate).

It has some funky implications, though, like what does [a: 1, b: 2] mean and foo(a: 3). It should probably still only be allowed as a field value, but that is a bit strange, grammatically. Go also has the notion of a key-value expression that is only valid in certain contexts, though, so maybe an acceptable level of irregularity.

[ghost]

Sorry I dont know enough about the syntax to comment intelligently

I would say that if Cue does have some indentation sensitive constructs, that this should not be implemented:

foo: bar: baz:
    qux: quux: quuz: 3

as users would rightfully expect that the indentation means something. if Cue has no indentation constructs, then it would be fair to implement this, as the space doesnt mean anything for Cue in the general case. It might still be confusing as YAML is space sensitive - but thats a weaker argument - similar to comparing Python to Go. In both cases, the latter can use spaces but not required and doesnt have semantic meaning.

[capelio]

To offer a different perspective, when I first came to CUE, I didn't experience any friction using spaces, and I come from years and years of working in the Javascript/JSON/web space. @cup's experience is what it is, but I think this is less of a "principle of least surprise" issue and more of an issue with breaking old habits and giving CUE the space to be CUE. We're talking about an entirely distinct language and I think CUE has the right to define its own boundaries and context when it comes to "CUE principles of least surprise".

If given a vote, I'd keep things the way they are. Whether it's a dot, or a colon, or any other character, the addition feels extraneous and unnecessary after having written thousands of lines of CUE over the past few months.

[mpvl]

@capelio I will be curious if your perspective changes with definitions, especially if they are used to exclude options from the data model, and not just as a "root type". I find having to write

foo: {
 def :: { bar: true }
}

somewhat annoying. There may be a way to relax it, though.

[ukiahsmith]

I agree with @cup. The space notation was a surprise to me. I would be less surprised if dots were used for shorthand notation of nested structs. Though I am open to other, non-space, characters being used. Tools and languages that I have experience with do not use the space as notation, rather it is used as a word (identifier) boundary. Almost any other character could be used in place of the space here and it would be less surprising than the space.

outer middle inner: 3

they would expect that it represents this structure:

{'outer middle inner': 3}

Especially when dots are used for access:

outer middle inner: 9

x: outer.middle.inner

These to things do not seem related to me on first glance. They lack a symmetry of shape.

I do also agree with @capelio that CUE should grow and evolve into it's own thing based on it's needs. However, I would err on the side of similar feeling to existing languages and tools, as a way to foster adoption. What good is it to be the best tool if no one uses it due to complexity and confusion.

[mpvl]

@ukiahsmith @cup @rogpeppe We have done some analysis in the mean time about how future extensions could look like, what kind of issues one would run in to with such extensions etc. Personal preferences aside, using space as separator seems unsustainable.

That said. Dots solve half the problem, but not all. For one possible future construct that makes it easy to deal with shadowing issues, the dots are actually fatally problematic. Finally, I'm looking into expanding the querying capabilities of CUE. One of the main reasons why this is needed is exactly to be more expressive on the LHS (e.g. to support JSON schema's patternProperties). So this would suggest that symmetry is quite beneficial. However, symmetry gets very confusing when some features of a query language are allowed on the RHS, but not on the LHS. For instance, I could imagine that supporting a JMESPath-like multiselect hash/list make sense on the RHS. It would, however, definitely not make sense on the LHS.

Aside from symmetry, I also find the dots don't jive well with the scoping rules, causing syntax and semantics to be misaligned.

So the current thinking is to define query syntax modularly (e.g. a [?<bool expr>] filter), and then allow some components on the LHS, while others not (e.g. allow all filters, but not multiselects). The obvious candidate for the separator would then be :. Interestingly enough, it would mean that

a:
  b:
    c: 2

would be both valid YAML and CUE. In CUE one could write this as a: b: c: 2. In other words, curly braces of structs may be omitted if there is exactly one element in a struct.

Note that the spaces originated from GCL and HCL, especially the latter.

[ukiahsmith]

I really like the syntax of a: b: c: 2 for creating nested structs with a single value. I think it is much better than spaces alone, as it conveys just enough information to show it's nested nature.

I would prefer to not have whitespace significant syntax. I haven't seen evidence of it in other CUE syntax elements, only having it here would be too confusing. Could it also be confusing if it's valid YAML and CUE?

a:
  b:
    c: 2

[rogpeppe]

I would prefer to not have whitespace significant syntax.

As per the spec, the only significant whitespace characters are newlines after some tokens.

This wouldn't change anything in that respect - the significant whitespace in the YAML example is not significant in CUE. This would mean exactly the same thing:

a:
b:
c:
2

[mpvl]

As a consequence of colon separator syntax the <X> syntax becomes somewhat ugly and hard to parse to boot. So adopting it would likely mean phasing out this syntax.

[ukiahsmith]

I find this unappealing. After first learning how to assign to values this would make me think that these are empty or null.

a:
b:
c:
2

I would rather the quick nesting syntax be this. I believe this is less of a surprise as to the outcome. I see it and quickly think this is a two in a c value in a b value in an a value.

a: b: c: 2

[rogpeppe]

I think the point is that the newline separated syntax is possible but that wouldn't be the conventional way of formatting it, and running it through cue fmt would reformat it in a way that was more clear (probably just the way that you wrote it latterly)

[xinau]

As a consequence of colon separator syntax the <X> syntax becomes somewhat ugly and hard to parse to boot. So adopting it would likely mean phasing out this syntax.

@mpvl Just an early idea, but a possible solution could be to do something like the following by replacing the placeholder field with * or so and using the query syntaxt to go upwards. but I'm not sure atm which consequences this might include.
As well as dropping the placeholder syntax, but I'm not sure if this is something you would want to do and if this is even a good idea.

foo:*:baz: {
  bar: // query syntax for go into the object containing `baz` in the tree and get the field name of this object
}

[mpvl]

The latest build of CUE contains an implementation of the : separator. In designing and analyzing the syntactic consequences of various possible extensions, including a possible query syntax, the : syntax proved to be superior to using either spaces or dots as a separator. The two biggest negatives were similarity to YAML (which could also be seen as a benefit) and being the hardest to type of the three options. Note that this does not mean that CUE will be become an indentation-sensitive language. No worries.

I hope to post a design doc of the new features soon. In the mean time, feedback on the new syntax is welcome.

Note that as a consequence of this change, the "template" syntax (a <X>: b, or now a: <X>: { name: X, ... }) will be removed too. It will be replaced with a: [X=string]: { name: X, ... }. In the latter, string can ultimately be substituted for any string type.

[mpvl]

@xinau The * notation is worth considering, especially in querying, but won't be sufficient for all functionality. But it could be considered later as a shorthand for things that turn out annoying to type. Such as foo.*.bar instead of foo.[_].bar when querying.

[mpvl]

@cup #165 contains a bit more analysis of the pros and cons, but by no means complete still. But in the end, dots had serious parsing issues and the benefits of using : were quite overwhelming. It did mean having to get rid of the <Foo>: bar notation.

One more example of why colons worked better: compare defining a map of map of ints using the various approaches:

IntMap :: [string].[string]: int

IntMap :: [string]: [string]: int

IntMap :: <X> <Y>: int

So the decision has been made to go with colons.

[cueckoo]

This issue has been migrated to cue-lang/cue#60.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.