Store debug symbols

[LukasPaczos]

At the moment, we are unable to symbolicate stack traces that weren't reproduced on a local machine. We need to look into ways of storing symbols when building and releasing a new version and whether it's possible to symbolicate provided, external stack traces.

[tobrun]

afaik and not an expert on this topic, when we build a release build,
the symbols are stripped and we aren't able to symbolicate them anymore.

[ChrisLoer]

Maybe when @kkaefer gets back he can take a look at this? I don't know how or even where Android build configuration happens, but I can't think of a reason we wouldn't be able to build release with symbols.

[kkaefer]

What kind of builds do you want to symbolicate?

[anandthakker]

@kkaefer One example: the failing Firebase test from #12192.

[jfirebaugh]

I've been tracing through the NDK build process as part of binary size optimization. Here's what I've found:

• Release builds currently use -g by default, although there is movement towards changing this (Debug flag is enabled for release build type. android/ndk#243)
• Accordingly, the initial libmapbox-gl.so (at MapboxGLAndroidSDK/build/intermediates/cmake/release/obj/<abi>/libmapbox-gl.so) has debugging info
• The build produces multiple additional libmapbox-gl.so's:
  • In MapboxGLAndroidSDK/build/intermediates/intermediate-jars/release/jni/<abi>, stripped of debug info. This is what our binary size metrics look at.
  • In MapboxGLAndroidSDK/build/intermediates/library_and_local_jars_jni/release/<abi>, stripped of debug info.
  • In MapboxGLAndroidSDK/build/intermediates/transforms/mergeJniLibs/release/0/lib/<abi>, includes debug info.
  • In MapboxGLAndroidSDK/build/intermediates/transforms/stripDebugSymbol/release/0/lib/<abi>, stripped of debug info.

What we probably want to do is archive one of the unstripped intermediates.

Bloaty has a command to read symbol sizes from a stripped binary and names from another, and this works with these files, e.g.:

bloaty -n 0 -d symbols MapboxGLAndroidSDK/build/intermediates/intermediate-jars/release/jni/arm64-v8a/libmapbox-gl.so --debug-file MapboxGLAndroidSDK/build/intermediates/cmake/release/obj/arm64-v8a/libmapbox-gl.so

[tobrun]

Picking this up before the final 6.4.0 release tomorrow so we are able to use these .so files to look into bugreports. In #12628, I'm storing the .so files as circle-ci artefact. At some point I would love to move them to AWS S3 for ease of access.

[tobrun]

Need to relook into which .so files we are using for running ndk-stack on as the ones currently stored do not result in useable traces: eg. below a trace that lists local_glyph_rasterizer but wasn't actually used by the client.

Nurbot:sofiles-6.4.0 Nurbot$ ndk-stack -sym armeabi-v7a/ -dump crash.txt
********** Crash dump: **********
Build fingerprint: 'samsung/j7y17ltexxm/j7y17lte:7.0/NRD90M/J730FMXXU3ARF2:user/release-keys'
pid: 6231, tid: 6492, name: GLThread 40796  >>> com.*.android.debug <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Stack frame #00 pc 0004ad30  /system/lib/libc.so (tgkill+12)
Stack frame #01 pc 000484c3  /system/lib/libc.so (pthread_kill+34)
Stack frame #02 pc 0001dd99  /system/lib/libc.so (raise+10)
Stack frame #03 pc 00019521  /system/lib/libc.so (__libc_android_abort+34)
Stack frame #04 pc 00017160  /system/lib/libc.so (abort+4)
Stack frame #05 pc 0031c069  /system/lib/libart.so (_ZN3art7Runtime5AbortEv+252)
Stack frame #06 pc 000b4dcb  /system/lib/libart.so (_ZN3art10LogMessageD2Ev+866)
Stack frame #07 pc 0023a439  /system/lib/libart.so (_ZN3art9JavaVMExt8JniAbortEPKcS2_+1584)
Stack frame #08 pc 0023a68f  /system/lib/libart.so (_ZN3art9JavaVMExt9JniAbortVEPKcS2_St9__va_list+58)
Stack frame #09 pc 000ca893  /system/lib/libart.so (_ZN3art11ScopedCheck6AbortFEPKcz+42)
Stack frame #10 pc 000ca381  /system/lib/libart.so (_ZN3art11ScopedCheck11CheckThreadEP7_JNIEnv+104)
Stack frame #11 pc 000c9497  /system/lib/libart.so (_ZN3art11ScopedCheck22CheckPossibleHeapValueERNS_18ScopedObjectAccessEcNS_12JniValueTypeE+26)
Stack frame #12 pc 000c8969  /system/lib/libart.so (_ZN3art11ScopedCheck5CheckERNS_18ScopedObjectAccessEbPKcPNS_12JniValueTypeE+800)
Stack frame #13 pc 000cb981  /system/lib/libart.so (_ZN3art8CheckJNI9DeleteRefEPKcP7_JNIEnvP8_jobjectNS_15IndirectRefKindE+456)
Stack frame #14 pc 001379bb  /data/app/com.*.android.debug-1/lib/arm/libmapbox-gl.so: Routine BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
_JNIEnv::DeleteGlobalRef(_jobject*) at /android/sdk/ndk-bundle/sysroot/usr/include/jni.h:553
Stack frame #15 pc 001379e5  /data/app/com.*.android.debug-1/lib/arm/libmapbox-gl.so: Routine BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
~LocalGlyphRasterizer at /src/platform/android/src/text/local_glyph_rasterizer.cpp:104
Stack frame #16 pc 00127073  /data/app/com.*.android.debug-1/lib/arm/libmapbox-gl.so: Routine BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: Offset (3956952321) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Could not find abbrev number 5345.
BFD: Dwarf Error: Offset (1057024920) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (184549915) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Could not find abbrev number 5345.
BFD: Dwarf Error: Offset (1057024920) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (184549915) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (1737884063) greater than or equal to .debug_str size (40664131).
std::__ndk1::default_delete<mbgl::LocalGlyphRasterizer>::operator()(mbgl::LocalGlyphRasterizer*) const at /android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/memory:2233
Stack frame #17 pc 0009f41d  /data/app/com.*.android.debug-1/lib/arm/libmapbox-gl.so: Routine BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: mangled line number section (bad file number).
BFD: Dwarf Error: Offset (2239384914) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (1448477067) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (1208402688) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Offset (2348896257) greater than or equal to .debug_str size (40664131).
BFD: Dwarf Error: Could not find abbrev number 8902.
BFD: Dwarf Error: Could not find abbrev number 1516.
BFD: Dwarf Error: Could not find abbrev number 1561504.
....

[stale]

This issue has been automatically detected as stale because it has not had recent activity and will be archived. Thank you for your contributions.