sentry-cli debug-files check doesn't seem to work for wasm

[bkotsopoulossc]

Environment

How do you use Sentry?
Can repro with both

Which SDK and version?
Using sentry-cli v2.0.3

Steps to Reproduce

% sentry-cli debug-files check some.symbols.wasm

Expected Result

Something like:

2022-04-21T23:45:08.581564629Z   Type: wasm library
2022-04-21T23:45:08.581600474Z   Contained debug identifiers:
2022-04-21T23:45:09.483088356Z     > Debug ID: b60862f6-cb25-4c8d-8f41-c9de44d4593e
2022-04-21T23:45:09.483131463Z       Code ID:  b60862f6cb254c8d8f41c9de44d4593e
2022-04-21T23:45:09.483134713Z       Arch:     wasm32
2022-04-21T23:45:09.483137020Z   Contained debug information:
2022-04-21T23:45:10.377935057Z     > symtab, debug

This seems to be a regression as this used to work around v1.70.1

I can verify the build_id field is present in the wasm binary:

% wasm-objdump -h some.symbols.wasm | grep build_id
   Custom start=0x1969c1e0 end=0x1969c1f9 (size=0x00000019) "build_id"

Actual Result

Debug Info File Check
  Type: wasm
  Contained debug identifiers:
  Contained debug information:
    > none
  Usable: no (missing debug identifier, likely stripped)

[kamilogorek]

Hey, thanks for reporting the issue. Do you have any concrete wasm file that we could use to debug this?
If not, could you try using different releases and dissect where the regression happened?

https://github.com/getsentry/sentry-cli/releases

We bumped symbolic in 1.74.3 and it had some updates to wasm handling, see: https://github.com/getsentry/symbolic/blob/master/CHANGELOG.md#870

[bkotsopoulossc]

@kamilogorek I can provide a wasm file, what's the best way to share it? It seems to be reproducible with any wasm file that has a build_id inserted by the wasm-split tool https://github.com/getsentry/symbolicator/tree/master/crates/wasm-split.

[bkotsopoulossc]

I see you have some binaries in the test folder here https://github.com/getsentry/sentry-cli/tree/master/tests/integration/_fixtures, how about we do some test-driven development, I'll put up a PR with a wasm binary with debug symbols and a build_id, plus a test case with the wrong expected output, and then we'll go from there? Would be awesome to get some wasm tests in here so we avoid regressions

[kamilogorek]

Sounds great!

[bkotsopoulossc]

And sure enough it seems to pass... getsentry/sentry-cli#1205, will have to dig in deeper

[bkotsopoulossc]

I wonder if this is potentially a hint:

For the trivial binary where this works (the test from the PR above), we get:

Debug Info File Check
  Type: wasm library
  Contained debug identifiers:
    > Debug ID: 67ac9b57-d3ab-40df-9a88-7f0e8fb7babb
      Code ID:  67ac9b57d3ab40df9a887f0e8fb7babb
      Arch:     wasm32
  Contained debug information:
    > symtab, debug
  Usable: yes

From a real-world binary I see:

Debug Info File Check
  Type: wasm
  Contained debug identifiers:
  Contained debug information:
    > none
  Usable: no (missing debug identifier, likely stripped)

Is there a difference between type wasm and type wasm library?

[kamilogorek]

Is there a difference between type wasm and type wasm library?

The latter one is matched by this branch - https://github.com/getsentry/sentry-cli/blob/512d98349008584e7c27dcde1eacf402a1c67dd6/src/utils/dif.rs#L260-L263 which allows it to print not only FileFormat, but also ObjectKind. However, I'm not sure why would symbolic not extract it from the archive correctly.

I believe we still need the repro case for broken scenario to debug this.

cc @Swatinem

[bkotsopoulossc]

I believe we still need the repro case for broken scenario to debug this.

I have a repro case that I can't easily share as the debug symbols contain references to internal code. I will work to create a simpler one, but not sure how hard it will be to do so

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[bkotzz]

still reproducible FYI

[kamilogorek]

cc @szokeasaurusrex

[szokeasaurusrex]

@bkotzz Can you share the steps/file you used to reproduce this issue?

[bkotzz]

Hey @szokeasaurusrex I've been trying to get a repro but so far I'm only able to reproduce it with a debug symbols wasm binary when it has a lot of debug information that I can't share externally.

I basically have a 200MB debug symbols file, that I know has a build_id custom section:

% ./wasm-tools objdump ./symbols.wasm             
  types                                  |        0xb -      0xc71 |      3174 bytes | 247 count
  imports                                |      0xc74 -     0x1d40 |      4300 bytes | 174 count
  functions                              |     0x1d44 -    0x1481e |     76506 bytes | 76162 count
  tables                                 |    0x14820 -    0x14827 |         7 bytes | 1 count
  memories                               |    0x14829 -    0x14830 |         7 bytes | 1 count
  globals                                |    0x14832 -    0x14853 |        33 bytes | 5 count
  exports                                |    0x14856 -    0x14a8e |       568 bytes | 27 count
  elements                               |    0x14a92 -    0x1e9d4 |     40770 bytes | 1 count
  code                                   |    0x1e9d9 -   0xfe1ef0 |  16528663 bytes | 76162 count
  data                                   |   0xfe1ef4 -  0x103cbcc |    371928 bytes | 3 count
  custom ".debug_info"                   |  0x103cbdd -  0x317d7cb |  34868206 bytes | 1 count
  custom ".debug_loc"                    |  0x317d7da -  0x31d1f36 |    345948 bytes | 1 count
  custom ".debug_ranges"                 |  0x31d1f48 -  0x3352e20 |   1576664 bytes | 1 count
  custom ".debug_abbrev"                 |  0x3352e32 -  0x344ccf9 |   1023687 bytes | 1 count
  custom ".debug_line"                   |  0x344cd0a -  0x3f90bbe |  11812532 bytes | 1 count
  custom ".debug_str"                    |  0x3f90bce -  0x7cdf37c |  64284590 bytes | 1 count
  custom "name"                          |  0x7cdf386 -  0xde25161 | 101998043 bytes | 1 count
  custom "build_id"                      |  0xde2516c -  0xde2517d |        17 bytes | 1 count

Where the sentry CLI says no debug ID exists:

% npx @sentry/cli debug-files check ./symbols.wasm
Debug Info File Check
  Type: wasm
  Contained debug identifiers:
  Contained debug information:
    > none
  Usable: no (missing debug identifier, likely stripped)

I narrowed the problem down to the "name" custom section. If I remove the name custom section:

$ ./wasm-tools strip ./symbols.wasm -d "name" -o out.wasm
% ./wasm-tools objdump out.wasm                                        
  types                                  |        0xb -      0xc71 |      3174 bytes | 247 count
  imports                                |      0xc74 -     0x1d40 |      4300 bytes | 174 count
  functions                              |     0x1d44 -    0x1481e |     76506 bytes | 76162 count
  tables                                 |    0x14820 -    0x14827 |         7 bytes | 1 count
  memories                               |    0x14829 -    0x14830 |         7 bytes | 1 count
  globals                                |    0x14832 -    0x14853 |        33 bytes | 5 count
  exports                                |    0x14856 -    0x14a8e |       568 bytes | 27 count
  elements                               |    0x14a92 -    0x1e9d4 |     40770 bytes | 1 count
  code                                   |    0x1e9d9 -   0xfe1ef0 |  16528663 bytes | 76162 count
  data                                   |   0xfe1ef4 -  0x103cbcc |    371928 bytes | 3 count
  custom ".debug_info"                   |  0x103cbdd -  0x317d7cb |  34868206 bytes | 1 count
  custom ".debug_loc"                    |  0x317d7da -  0x31d1f36 |    345948 bytes | 1 count
  custom ".debug_ranges"                 |  0x31d1f48 -  0x3352e20 |   1576664 bytes | 1 count
  custom ".debug_abbrev"                 |  0x3352e32 -  0x344ccf9 |   1023687 bytes | 1 count
  custom ".debug_line"                   |  0x344cd0a -  0x3f90bbe |  11812532 bytes | 1 count
  custom ".debug_str"                    |  0x3f90bce -  0x7cdf37c |  64284590 bytes | 1 count
  custom "build_id"                      |  0x7cdf387 -  0x7cdf398 |        17 bytes | 1 count

Then check is able to find the build_id

% npx @sentry/cli debug-files check  out.wasm                                              
Debug Info File Check
  Type: wasm library
  Contained debug identifiers:
    > Debug ID: 10207ee3-1519-9e4b-4e94-1edf89bb6942
      Code ID:  10207ee315199e4b4e941edf89bb69425f
      Arch:     wasm32
  Contained debug information:
    > symtab, debug
  Usable: yes

So I am wondering if there is some code parsing the custom sections in the order they appear in the binary, and something gets tripped up when it hits the "name" section, and doesn't continue on the the build_id section?

[szokeasaurusrex]

@bkotzz to clarify, is this issue blocking you from achieving something in your workflow? Does the problem only affect debug-files check, or does it also affect debug-files upload?

[bkotzz]

Hi @szokeasaurusrex, it's not just debug-files check that has the problem, it's also debug-files upload. If I have a wasm symbol file in this state, I can call debug-files upload <path to file> and the CLI is not able to find a debug symbol file to upload so just skips it.

This is only a problem in recent versions of sentry CLI, like 2.31.0. Uploading works fine with 1.71.0, as the old CLI is able to find the build ID

[szokeasaurusrex]

recent versions of sentry CLI

By "recent versions," do you mean all 2.x versions? What is the latest version that the command still works?

[bkotzz]

yeah all V2 versions can repro the problem. 1.71.0 works

[trzeciak]

Hi all, I also came across this problem and with your tips I went a step further.

@bkotzz that was a very good lead with the 'name' section 💪

Long story short: The problem is when the name custom section is too large(1), then the code that parse custom section LINK, (after read "build_id" section) try to read "name" section, but under-the-hoot method BinaryReader::read_string() throw an exception. Then we can't get name and build_id.

Just handling those exception is not a solution, because we need a fully "name" section. The "name" section for wasm file may contain lot of debug informations, see documentation: module name, function names and local names.

(1) It seems that easiest solution is just grow limits::MAX_WASM_STRING_SIZE (now: 100_000) in wasmparse module, but I don't know what value to choose because my name section is 100MB. I have the impression that this may cause problems on the backend side, therefore I think that the way of parsing this section should be changed to a more streaming like.

It's also possible that I'm a little wrong, due to my little experience.

---

To get to what I wrote above, I switched between different committees in the sentry-cli repo.

• I saw that, in version 1.71.0 (exacly-commit: e2a25cb595c877f5c2a13e9b4e6a75e36e1c7138) sentry-cli upload-dif seems that works, because in logs I see: (2).

> Found 1 debug information file
> Preparing for upload...

• Version 1.72.0 (exacly-commit: c789723a569253bdec474f36fc5b44e0c55a3fe3) prints:

  WARN    2024-06-27 20:00:10.813920 +02:00 failed to parse `name` custom section string size out of bounds 33 (at offset 93709111)  (from walrus)
> Found 1 debug information file
> Prepared debug information file for upload

because log was added LINK:

• Last commit with those message: 5e4badb2a426be90936e079b21f0be23ca84becf
• Version from commit 37ac899c8deb52194ff631ea5ddf04ab1543ca36 change code from walrus to symbolic-debuginfo, and prints:

> Found 0 debug information files
> No debug information files found

(2) Even though it looks like it works in version 1.71.0, it didn't have to be that way because an exception was thrown, and the code catch a exception looks like:

---

Ending this long message:

• I can try doing a PR with raising limits::MAX_WASM_STRING_SIZE to 100MB+ (this sounds like the beginning of other problems),
• I don't feel up to doing the 'steam like' refactor I mentioned (I have too little knowledge about rust and even less about the sentry-cli product),

Nevertheless, I hope that I helped somehow and that this problem can be solved.
Let me know if I can help in any other way.

PS: I also have an additional question: is this code also used on the cloud server side? I wonder if even if I fixed it locally (increasing limit size), would the symbols be recognized correctly on the sentry cloud side?

[szokeasaurusrex]

Hey @trzeciak, thank you for the detailed investigation! Based on this information, it seems that the underlying problem is originating from Symbolic, specifically from one of Symbolic's dependencies (wasmparser).

I am going to transfer this issue over to the getsentry/symbolic repository, so that you can get help directly from the team responsible for maintaining Symbolic.

[trzeciak]

I'm spending extra time on this issue and it looks like the size of 100_000 is for a single function-symbols entry containing vector of all symbols used in function (not the entire name section as I previously thought) https://webassembly.github.io/spec/core/appendix/custom.html.

I checked it and it seems that in my case I exceed this value by a few percent, so I proposed to slightly increase this limit for now, but ultimately I think that the implementation in wasmpaarser should be improved bytecodealliance/wasm-tools#1650

The workaround for this is probably to split a specific function that exceeds this limit, I'll try to check that.

[trzeciak]

The workaround for this is probably to split a specific function that exceeds this limit, I'll try to check that.

Unfortunately I wasn't able to find it. I tried to help myself with the tools: wasm-tools addr2line or ./symbolic/addr2line but they did not give any helpful hints (under the hood it has the same code base but returns slightly different results). Unfortunately, I have too large a code base and finding it manually is also too time-consuming.

Next workaround is just use sentry cloud to aggregate and group crashes, and make demangling manualy on local PC 😢

[trzeciak]

Okay, after update wasm-tools/wasmparser and symbolic dependencies, it seems that sentry-cli works:

# Tests with latest release (2.33.1)
❯ sentry-cli --version
sentry-cli 2.33.1

❯ sentry-cli debug-files check web-app.wasm
Debug Info File Check
  Type: wasm
  Contained debug identifiers:
  Contained debug information:
    > none
  Usable: no (missing debug identifier, likely stripped)
# ^^ missing, as expected

On getsentry/sentry-cli#2120:

❯ ./sentry-cli debug-files check web-app.wasm
Debug Info File Check
  Type: wasm library
  Contained debug identifiers:
    > Debug ID: 100abcf2-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      Code ID:  100abcf2xxxxxxxxxxxxxxxxxxxxxxxxb9
      Arch:     wasm32
  Contained debug information:
    > symtab, debug
  Usable: yes
# ^^ now detected

My web-app.wasm file is little big: 500MB+, has big name section, and include -fwasm-exceptions (legacy wasm exceptions purpose).

However, I end up getting a different error when trying to upload the symbols, and the second time it claims it's on the server.

❯ ./sentry-cli debug-files upload --project web-app --wait web-app.wasm
> Found 1 debug information file
> Prepared debug information file for upload
> Uploading completed in 420.506s
> Uploaded 1 missing debug information file
> File processing complete:

    ERROR web-app.wasm
        internal server error

Error: some symbols did not process correctly

❯ ./sentry-cli debug-files upload --project web-app --wait web-app.wasm
> Found 1 debug information file
> Prepared debug information file for upload
> Nothing to upload, all files are on the server

I think I'll report this as a separate bug, and this ticket can be closed once the aforementioned PR has been merged.

[szokeasaurusrex]

Fixed by #853 and getsentry/sentry-cli#2120