commonmark with sourcepos doesn't work right with header nodes

[chapterjason]

Explain the problem.

If I use commonmark all the header in the ast (abstract syntax tree (json)) will get the id in the attributes.
If I use commonmark+sourcepos the upper div will get that id.

After I read the documentation it seams to be incorrect:

[...]For elements that accept attributes, a data-pos attribute is added; [...]
A header actually does, this doesn't work.

File: file.md

# Example

## Example 1 {#example-123}

Actual (With sourcepos) (pandoc --from commonmark_x+sourcepos --to json ./file.md | jq)

{
  "pandoc-api-version": [
    1,
    22,
    1
  ],
  "meta": {},
  "blocks": [
    {
      "t": "Div",
      "c": [
        [
          "example",
          [],
          [
            [
              "data-pos",
              "./file.md@2:1-3:1"
            ]
          ]
        ],
        [
          {
            "t": "Header",
            "c": [
              1,
              [
                "",
                [],
                []
              ],
              [
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@2:3-2:10"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "Example"
                      }
                    ]
                  ]
                }
              ]
            ]
          }
        ]
      ]
    },
    {
      "t": "Div",
      "c": [
        [
          "example-123",
          [],
          [
            [
              "data-pos",
              "./file.md@4:1-5:1"
            ]
          ]
        ],
        [
          {
            "t": "Header",
            "c": [
              2,
              [
                "",
                [],
                []
              ],
              [
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:4-4:11"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "Example"
                      }
                    ]
                  ]
                },
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:11-4:12"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Space"
                      }
                    ]
                  ]
                },
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:12-4:13"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "1"
                      }
                    ]
                  ]
                }
              ]
            ]
          }
        ]
      ]
    }
  ]
}

Actual (Without sourcepos) (pandoc --from commonmark_x --to json ./file.md | jq)

{
  "pandoc-api-version": [
    1,
    22,
    1
  ],
  "meta": {},
  "blocks": [
    {
      "t": "Header",
      "c": [
        1,
        [
          "example",
          [],
          []
        ],
        [
          {
            "t": "Str",
            "c": "Example"
          }
        ]
      ]
    },
    {
      "t": "Header",
      "c": [
        2,
        [
          "example-123",
          [],
          []
        ],
        [
          {
            "t": "Str",
            "c": "Example"
          },
          {
            "t": "Space"
          },
          {
            "t": "Str",
            "c": "1"
          }
        ]
      ]
    }
  ]
}

Expected (With sourcepos) (pandoc --from commonmark_x+sourcepos --to json ./file.md | jq)

{
  "pandoc-api-version": [
    1,
    22,
    1
  ],
  "meta": {},
  "blocks": [
    {
      "t": "Div",
      "c": [
        [
          "",
          [],
          [
            [
              "data-pos",
              "./file.md@2:1-3:1"
            ]
          ]
        ],
        [
          {
            "t": "Header",
            "c": [
              1,
              [
                "example",
                [],
                []
              ],
              [
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@2:3-2:10"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "Example"
                      }
                    ]
                  ]
                }
              ]
            ]
          }
        ]
      ]
    },
    {
      "t": "Div",
      "c": [
        [
          "",
          [],
          [
            [
              "data-pos",
              "./file.md@4:1-5:1"
            ]
          ]
        ],
        [
          {
            "t": "Header",
            "c": [
              2,
              [
                "example-123",
                [],
                []
              ],
              [
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:4-4:11"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "Example"
                      }
                    ]
                  ]
                },
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:11-4:12"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Space"
                      }
                    ]
                  ]
                },
                {
                  "t": "Span",
                  "c": [
                    [
                      "",
                      [],
                      [
                        [
                          "data-pos",
                          "./file.md@4:12-4:13"
                        ]
                      ]
                    ],
                    [
                      {
                        "t": "Str",
                        "c": "1"
                      }
                    ]
                  ]
                }
              ]
            ]
          }
        ]
      ]
    }
  ]
}

Diff

@@ -10,7 +10,7 @@
       "t": "Div",
       "c": [
         [
-          "example",
+          "",
           [],
           [
             [
@@ -25,7 +25,7 @@
             "c": [
               1,
               [
-                "",
+                "example",
                 [],
                 []
               ],
@@ -61,7 +61,7 @@
       "t": "Div",
       "c": [
         [
-          "example-123",
+          "",
           [],
           [
             [
@@ -76,7 +76,7 @@
             "c": [
               2,
               [
-                "",
+                "example-123",
                 [],
                 []
               ],

Pandoc version?
What version of pandoc are you using, on what OS?

$ pandoc -v
pandoc.exe 2.16.2
Compiled with pandoc-types 1.22.1, texmath 0.12.3.3, skylighting 0.12.1,
citeproc 0.6, ipynb 0.1.0.2
User data directory: C:\Users\[...]\AppData\Roaming\pandoc
Copyright (C) 2006-2021 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

[chapterjason]

@jgm Tested nightly, works great, thanks a lot!

[chapterjason]

@jgm I think something went wrong.

I tried the new 2.17 version and got the following output:

File: file.md

# one

# two

Nightly: https://github.com/jgm/pandoc/actions/runs/1610358032

Nightly works as expected: file.md

{
    "pandoc-api-version": [
        1,
        22,
        1
    ],
    "meta": {},
    "blocks": [
        {
            "t": "Header",
            "c": [
                1,
                [
                    "one",
                    [],
                    [
                        [
                            "data-pos",
                            "./index.md@2:1-3:1"
                        ]
                    ]
                ],
                [
                    {
                        "t": "Span",
                        "c": [
                            [
                                "",
                                [],
                                [
                                    [
                                        "data-pos",
                                        "./index.md@2:3-2:6"
                                    ]
                                ]
                            ],
                            [
                                {
                                    "t": "Str",
                                    "c": "one"
                                }
                            ]
                        ]
                    }
                ]
            ]
        },
        {
            "t": "Header",
            "c": [
                1,
                [
                    "two",
                    [],
                    [
                        [
                            "data-pos",
                            "./index.md@4:1-5:1"
                        ]
                    ]
                ],
                [
                    {
                        "t": "Span",
                        "c": [
                            [
                                "",
                                [],
                                [
                                    [
                                        "data-pos",
                                        "./index.md@4:3-4:6"
                                    ]
                                ]
                            ],
                            [
                                {
                                    "t": "Str",
                                    "c": "two"
                                }
                            ]
                        ]
                    }
                ]
            ]
        }
    ]
}

2.17 doesn't work: file.md

{
    "pandoc-api-version": [
        1,
        22,
        1
    ],
    "meta": {},
    "blocks": [
        {
            "t": "Div",
            "c": [
                [
                    "one",
                    [],
                    [
                        [
                            "data-pos",
                            "./index.md@2:1-3:1"
                        ]
                    ]
                ],
                [
                    {
                        "t": "Header",
                        "c": [
                            1,
                            [
                                "",
                                [],
                                []
                            ],
                            [
                                {
                                    "t": "Span",
                                    "c": [
                                        [
                                            "",
                                            [],
                                            [
                                                [
                                                    "data-pos",
                                                    "./index.md@2:3-2:6"
                                                ]
                                            ]
                                        ],
                                        [
                                            {
                                                "t": "Str",
                                                "c": "one"
                                            }
                                        ]
                                    ]
                                }
                            ]
                        ]
                    }
                ]
            ]
        },
        {
            "t": "Div",
            "c": [
                [
                    "two",
                    [],
                    [
                        [
                            "data-pos",
                            "./index.md@4:1-5:1"
                        ]
                    ]
                ],
                [
                    {
                        "t": "Header",
                        "c": [
                            1,
                            [
                                "",
                                [],
                                []
                            ],
                            [
                                {
                                    "t": "Span",
                                    "c": [
                                        [
                                            "",
                                            [],
                                            [
                                                [
                                                    "data-pos",
                                                    "./index.md@4:3-4:6"
                                                ]
                                            ]
                                        ],
                                        [
                                            {
                                                "t": "Str",
                                                "c": "two"
                                            }
                                        ]
                                    ]
                                }
                            ]
                        ]
                    }
                ]
            ]
        }
    ]
}

[jgm]

Here it what I'm seeing with 2.17:

% /usr/local/bin/pandoc -f commonmark_x+sourcepos -t native
# one

# two
[ Div
    ( "one" , [] , [ ( "data-pos" , "1:1-2:1" ) ] )
    [ Header
        1
        ( "" , [] , [] )
        [ Span
            ( "" , [] , [ ( "data-pos" , "1:3-1:6" ) ] ) [ Str "one" ]
        ]
    ]
, Div
    ( "two" , [] , [ ( "data-pos" , "3:1-4:1" ) ] )
    [ Header
        1
        ( "" , [] , [] )
        [ Span
            ( "" , [] , [ ( "data-pos" , "3:3-3:6" ) ] ) [ Str "two" ]
        ]
    ]
]

(Use native output instead of json for better readability.)

I was confused until I realized that the nightly you meant was 23 days ago.

% ~/Downloads/pandoc-nightly-macos-2021-12-22/pandoc -f commonmark_x+sourcepos -t native
# one

# two
^D
[ Header
    1
    ( "one" , [] , [ ( "data-pos" , "1:1-2:1" ) ] )
    [ Span
        ( "" , [] , [ ( "data-pos" , "1:3-1:6" ) ] ) [ Str "one" ]
    ]
, Header
    1
    ( "two" , [] , [ ( "data-pos" , "3:1-4:1" ) ] )
    [ Span
        ( "" , [] , [ ( "data-pos" , "3:3-3:6" ) ] ) [ Str "two" ]
    ]
]

[jgm]

This is quite bizarre; I can't see any changes in the last 23 days that would be relevant.

[jgm]

Oh! Looks like we need a new commonmark-pandoc release.

[chapterjason]

@jgm Sorry I didn't mentioned that this was an old nightly, didn't thought about that, I was literally confused too cause I didn't saw any changes between the 23 days. Thanks a lot! 🚀