Consideration for moving from objects to arrays.

[kevinswiber]

With a major version bump and an acceptance of breaking changes from OpenAPI 3.x, I'd like to propose moving from JSON object structures to JSON array structures.

For the purposes of this document, JSON will be the format referenced. Assume YAML support as well.

JSON	YAML
object	map
array	sequence
name	key

Motivation

Tooling. That's it. 😅 And the API practitioner experience in using that tooling.

Specifically, running query filters on names in JSON name-value pairs is difficult for lots of tooling and requires deconstructing the definition into an intermediary format. This proposal suggests the OpenAPI document itself could look closer to that intermediary format.

Note
I'm sure there's more that could motivate this. This is just the most immediate one I'm dealing with today. Hoping to hear input from others.

Proposal

With the exception of components, convert all objects with dynamic, user-provided keys to arrays of objects, moving the JSON name to a name property in that object. Modifying the example from the OAI/moonwalk README, it would look something like this:

openapi: 4.0.0
info:
  title: Simplest thing that works
  version: 0.5.0
paths:
  - name: speakers
    requests:
      - name: createSpeaker
        method: post
        contentType: application/json
        contentSchema:
          $ref: "#/components/schemas/speaker"
        responses:
          - name: created
            status: 201
      - name: getSpeakers
        method: get
        responses:
          - name: ok
            status: 200
            contentType: application/json
            contentSchema:
              type: array
              items:
                $ref: "#/components/schemas/speaker"
    pathResponses:
      - name: notFound
        status: 404
        contentType: application/http-problem
apiResponses:
  - name: serverError
    status: 5XX
    contentType: application/http-problem
components:
  schemas:
    Speaker:
      type: object
      properties:
        name: 
          type: string

Note
Using name for every translation might not be ideal, but it's kept here for simplicity of discussion and to avoid bike-shedding on that for now. For example, template might be better than name for a pathItem, but I think that discussion can come only if this proposal sees support.

The reasoning behind leaving components as-is relates to an attempt to stay close to referencing compatibility with JSON Schema, which relies on JSON Pointer and doesn't have advanced filtering capabilities.

Potential Downsides

• Manually editing long sequences in YAML is a known pain. This may have an impact on the design-time experience.
• Dealing with collisions where names should be unique. Do we merge? Last one wins? The spec would need processing instructions.
• Selection by key now becomes a filter operation instead of using a string index. I do believe the trade-off is a good compromise to improve selection and searching by key patterns.

Curious to hear anyone's thoughts on this!

[kevinswiber]

Related: #25

[darrelmiller]

Specifically, running query filters on names in JSON name-value pairs is difficult

Can you show an example of this? I'm not sure I'm aware of this difficulty.

[kevinswiber]

To express a query like, "Which OpenAPI documents use a path that contains /pets?", an extra operation has to occur to isolate the object keys.

Even for jq, this requires a conversion on filtering.

jq '.paths | with_entries(select(.key | contains("/pets")))' < petstore.json

vs.

jq '.paths[] | select(.name | contains("/pets"))' < petstore2.json

This manifests in every query scenario. I'm analyzing hundreds of thousands of OpenAPI files from around the Internet. This extra conversion step is true for just about every JSON-based document store I used, including MongoDB and Couchbase, except the performance hit is much worse at that scale. (Just illustrating my latest encounter. Querying at this scale won't be a common practice.)

In JavaScript:

const pathsWithPets = Object.fromEntries(Object.entries(openapi.paths)
    .filter(([key, val]) => key.includes("/pets")));

vs.

const pathsWithPets = openapi.paths.filter((path) => path.name.includes("/pets"));

[ralfhandl]

• Dealing with collisions where names should be unique. Do we merge? Last one wins? The spec would need processing instructions.

The AJV validator already supports a uniqueItemProperties keyword for detecting/preventing this kind of collision.

Maybe this could become a standard keyword in a future JSON Schema version, or part of a vocabulary?

[kevinswiber]

This is interesting! @handrews, does JSON Schema have a way to achieve this today? I can't think of one off the top of my head.

[handrews]

The best way to get this kind of keyword adopted is for someone to implement it as a properly interoperable extension vocabulary first (which AJV does not support, sadly, but other JavaScript, Python, Perl, Java, .NET, and other implementations do). There's an endless stream of keyword proposals like this and we can't just put them all in the spec. Our focus is more on making it easier to do extension vocabularies so these things can be solved outside of the spec, but still in an interoperable way (because unlike AJV keywords, you can require extension vocabularies in a meta-schema and implementations have to either support it or refuse to process the schema).

[bpedro]

I believe this proposal is interesting, however it creates a tradeoff between the importance of running query filters on user-provided keys and accessing individual elements.

Let's start by analyzing the motivation behind this proposal. According to the author, the goal of the proposed change is to make it easier for API practitioners to work with the data represented by OpenAPI definitions.

Specifically, running query filters on names in JSON name-value pairs is difficult for lots of tooling (...)

The examples provided to obtain all the paths that contain the word "pets" make sense. You can clearly see that using JSON object structures makes it harder to obtain all the elements where their keys contain a given string.

On the other hand, accessing individual elements becomes more difficult after the proposed changes. Let's take the /pets path as an example and see how accessing it directly looks like with JSON object structures versus JSON array structures, using different methods.

Method	JSON object structures	JSON array structures
M1: jq, using api.json as the API definition document	jq '.paths | .["/pets"]' < api.json	jq '.paths[] | select(.name=="/pets")'< api.json
M2: JavaScript, using api as the object holding the API definition	api.paths['/pets']	api.paths.find(path => path.name == '/pets')
M3: Python, using api as the	api['paths']['/pets']	[path for path in api['paths'] if path['name']=='/pets'][0]

While M1 doesn't convey the big difference in effort when using jq, if you look at the other methods, you will immediately see that there is a degree of difficulty introduced when looking up elements on array structures. Object structures can be easily referenced by their keys in most programming languages, while array structures can't be referenced by the value of arbitrary keys.

The value of this proposal is, in my opinion, affected by the importance of how you want to read information from an OpenAPI definition. If you feel it's more important to easily filter name-value pairs, then JSON array structures seem to be the appropriate solution. If, on the other hand, you want to access individual elements as easily as possible, then JSON object structures are the preferred solution.

[kevinswiber]

Thanks for bringing up this perspective, @bpedro. This is a classic debate I've revisited with multiple formats over my career, and I've been on both sides of the fence.

My current position is this: While converting from an object to an array does remove direct access with an indexer key that's a string, it also provides a nice compromise for both use cases. In other words, the reduction of complexity in searching is higher than the addition of complexity in selection. Therefore, I think this is a good trade-off to make.

I think it also opens opportunity for composition, but that debate might end up looking similar to this one, and I don't want to bloat the proposal.

[kevinswiber]

@bpedro I added this to the Potential Downsides in the proposal. /cc @darrelmiller who had the same concern.

[whitlockjc]

My initial concern with this is authoring/reading documents would become much harder. For example, oas.paths['/pets'] is somewhat human readable because I can tell that whatever this thing is, it's referencing a Path Object for the /pets collection of operations. How am I to reason about this using oas.paths[2]? Sure, I can tell I'm working with a Path Object but I now have to peer into the OAS document to make any sense of things. This will also have an impact on any references we have.

Then you have to think about how changes will propagate. Let's say I have a collection of paths and I want to insert one into the middle. I now have to find all places that reference any paths (or children) that are at the inserted index or after and update their values. The authoring experience here is heavily impacted by such a change.

I've never been a fan of changes to OAS that were purely for tooling because it often made the authoring/reading of OAS documents more complex. To me, that is the case here. Tooling is exactly where the complexities should be hidden in my opinion, not making it harder for me to author/read documents.

[MikeRalphson]

Just a couple of notes on the plus side of the argument (though I'm not discounting the many cons listed above):

• arrays impose explicit ordering of the contents where maps do not. This allows semantics such as "last one wins" as mentioned above and provide for predictable error reporting where constraints are broken
• arrays provide for composite keys (like name and in in the current parameterObject) without fudges for maps such as concatenation which limit character sets or require escaping rules

[stickfigure]

IMO the main benefit of using arrays is flexibility. If requests had already been an array of objects, adding a name property is a relatively nonbreaking change and most existing tooling continues to work. Likewise, if something else becomes a more relevant primary key in the future, some degree of backwards compatibility is built in.

If you keep using map keys, v5, v6, v7, etc are likely to see more structural changes that send everyone scrambling.

Consumers can transform the in-memory structure to anything they want on read. IMO it's better to keep the persisted format consistent, and arrays make that easier.

[rattrayalex]

One benefit of moving to arrays is that naming things is hard, and it could enable optional names. Requiring names may also make it harder to automatically migrate specs from OAIv3 to v4, as operationId is not always used today.

(Personally I'm ambivalent about this proposal, just noting something I hadn't seen previously mentioned)

[handrews]

From @arno-di-loreto in #25, consolidated here:

---

Thinking aloud about responses being a map vs a list.

Unless I missed something, the reusable part of a response is what is inside the key.
That means :

• The response key is redefined each time a reusable response is used
• It may request users one info unnecessarily (what's the use of this information vs a summary or a description stored in the response? do all responses would benefit of it?)
• It may lead to inconsistency (different keys used for the same kind of response)
• But that also allows to locally state something (a kind of name/summary specific to the use of the response)
• Also (maybe?) when reading the file in a code editor, a map is more readable than a list, all sub-level can be closed leaving the keys visible. (but is that consideration to take into account?)

If responses is a list:

• There's one info less to provide
• If it's valuable in some use cases, it can be stored (or not) in a summary (or a name if really needed) inside the response object and then be overridden (or defined) by providing the adequate property near the ref.
• The responses property may be less readable in a code editor when all its elements are closed (no info visible)

In that case (list), I wonder if sets of responses couldn't be defined. When a $ref is used, it could be a reference to a single response or a set of responses (defined in components.responsesSets) (but that's another story).

This question is also a general concern: Are some guidelines defined to help choose one or another in OpenAPI v4 design?
For instance, having headers as a map in v3 led to some issues (losing header names in components.headers). The same goes for examples being a map, the extra key level was finally redundant with the info of the example.

Responses as a map:

paths:
  "/path":
        responses:
          ok:
            $ref: "#/components/responses/FooCreated"
  "/another/path":
        responses:
          created:
            $ref: "#/components/responses/FooCreated"
components:
  responses:
     FooCreated:
        status: 201
        contentType: application/json
        contentSchema:
          $ref: "#/components/schemas/foo"

Responses as a list

paths:
  "/path":
        responses:
            - $ref: "#/components/responses/FooCreated"
  "/another/path":
        responses:
            - $ref: "#/components/responses/FooCreated"
              summary: created
components:
  responses:
     FooCreated:
        status: 201
        summary: ok # (or a specific name property if absolutely needed)
        contentType: application/json
        contentSchema:
          $ref: "#/components/schemas/foo"

[handrews]

Response #25 (comment) from @ralfhandl

---

One of the design goals is

• Reduce nested structures to improve readability and editability

so I'd prefer lists over maps.

@darrelmiller writes

I encourage people to explore complete API design languages that will optimize for the types of APIs being built but compile into an interoperable standard like OpenAPI

Compiling into Moonwalk is easier if the compiler doesn't have to invent map keys.

[handrews]

From @spacether at #90 (comment)

---

How about limiting tag names to the same set of characters that component names are limited to (All the fixed fields declared above are objects that MUST use keys that match the regular expression: ^[a-zA-Z0-9\.\-_]+$). In code generation contexts, this will allow easier file (module) name generation and reduce the likelihood of filename collisions.

Also have tags be a map where the key is the tagName. That structurally prevents tag collisions.

[rafalkrupinski]

If there were resources in OAS, and perhaps resource groups, code generators could probably leave tags alone, at least as long we're talking about name spaces. For filtering tags with spaces are OK.

[rattrayalex]

FWIW I quite like this concept. Not saying I like it better than resources, but even resources can be inadequately flexible. For example, as someone working on an integrated SDK + (nascent) docs generator, there are some situations where the hierarchy users want in docs may differ from SDKs. Tags with some description/purpose/other seems like a nice way to give a lot more flexibility.

(I'm personally less worried about the casing, though I could also imagine this being extended to add casing hints, eg FooBar: {description: 'Stuff', purpose: 'nav', casing: {sentence: 'Foo bar'}} (I often want to case things differently in different contexts.

[spacether]

Docs tools want to use them for navigation, so they should be human readable Title Case.

Github markdown needs kebabCase to link to section anchors, so not just title case is needed for docs

[philsturgeon]

Most docs tools will show the human readable version of the tag then if they need to produce something like an anchor link that gets run through a case conversion, which is far more likely to be correct than turning a whatEver_Case tag into something good for humans. But yes we definitely need different types of tags for different use cases and they may have human readable names, sumamry, title, etc.

[andrecedik]

This discussion sounds like you would need a separate attribute for displaying the tag name, but it's probably not as easy as it seems, since you can't just add a new attribute like this:

tags: 

  Foo: 
    description: Stuff
    purpose: navigation
    displayName: Stuff

  CodeStuff: 
    description: Stuff
    purpose: code
    displayName: Code Stuff

because what happens if you have multiple purposes for the CodeStuff tag? Which is the initial problem. Multiple ways of using a tag for different use cases. So the only thing that might work is if there are only the 2 use cases. Code generators use the tag name for namespacing, but the tage name can only be defined using the allowed set of characters mentioned and tooling that needs whitespaces and other "special characters" can use a new attribute which allows for them to be used.

Of course this would defeat the idea of the purpose attribute as it is mentioned until now. It would only work if purpose(s) is an array of objects that hold the name of the purpose and the way the tag should be displayed when using the tag based on this purpose.

[handrews]

Coming back to this, I feel like there are a couple of things to resolve:

How do we identify things within OADs?

This is critical as many of the arguments on this topic have to do with whether things work with JSON Pointers. But JSON Pointers are not our only options, and have downsides beyond difficulty with arrays. This is why JSON Schema offers a location-independent alternative for referencing.

OAS 3.x uses several different mechanisms:

1. URIs with JSON Pointer fragments, which work for anything in the OAD but are fragile with respect to internal refactorings, and are awkward to use with arrays
2. URIs with manually-assigned plain-name fragments (Schema Object only, $anchor and `$dynamicAnchor)); the idea of allowing similar manually-defined fragments in all Objects has recently been discussed.
3. Manually-assigned non-URI identifiers (operationId) that have some ambiguity of scope (per-document? per-OAD? OAS 3.1 just advises avoiding the ambiguous cases)
4. Component names, which are typed and more robust as only the name has to match, and could be paired with per-document namespaces as suggested in the imports proposal to fix the ambiguities that occur with component names in 3.x, but obviously only work for components

It's important to note that we could define URI fragment syntax based on component types and names, which combines aspects of several of the above.

How do tags fit in?

Tags are different from unique identification (you can tag many things with the same tag). Can we handle all ordering or other organizational/presentation metadata through tags regardless of the underlying structure?

Do we want mandatory or optional names?

As @rattrayalex noted, naming things is hard, and it might be nicer to not have to do that unless there's a need for a name. That would point towards something like identification option 2 above.

Of course some of our objects have natural names (status codes, media types), although in some cases those "natural" mappings have proven problematic when we would ideally have multiple different values for a supposedly "natural" key.

Do we want mandatory or optional ordering?

Arrays have inherent ordering, and sometimes we might not want that. We could allow a way to indicate that something is unordered.

Alternatively, we could add a field to various map entries that sets their relative order. I'm thinking a non-negative integer field where the values need not be contiguous. If we want to get fancy we could allow negative integers to set order from the end as well as the beginnign (off the top of my head, I'd say all negative-ordered entries go at the end, all positive at the beginning, and any missing an ordering go in arbitrary order between the ordered blocks- we would not want to have to compute some sort of overlap between negative and positive relative ordered).

Do we want re-use to include names or not?

Re-using a map entry divorces it from its name in the map, while re-using an array entry that sets its own name keeps them together. Either way can be either a disadvantage or an advantage. Does this impact what we do? Do we want the same approach or would different use cases be better served by different approaches?

---

I think that if we delve into these questions, it will become more clear what the answer is. Choosing objects or arrays should be about solving some sort of problem, and we first need to figure out what problem(s) it might solve, and then figure out whether there are already better alternatives that we can use.