Aarch64 (ARM64) Linux builds fail to launch when using a kernel with 16 KB PAGESIZE (please add support for other kernel page sizes?)

[rxhfcy]

Issue type

Crash or freeze

Bug description

Aarch64/ARM64 Linux builds seem to basically crash on startup if kernel PAGESIZE is 16 KB (not 4 KB).

Suggestion: please also support other page sizes than 4 KB?
16 KB support is required for at least these:

• Asahi Linux on Macs with ARM processors ("Apple Silicon") (no 4 KB support)
• Raspberry Pi 5 with the default 16KB kernel of 64-bit Raspberry Pi OS (blog post that talks about 16KB)

More information: on ARM, 4 KB is not the only PAGESIZE possible (some systems use 16 KB or 64 KB).

I tested with Aarch64 (ARM64) Linux on an "Apple Silicon" (ARM64) M1 MacBook Air, (re)booting straight to bare Asahi Linux / Fedora Asahi Remix "on the metal" (without using macOS), and the Asahi Linux kernel will only have support for 16 KB page size, because that's what the hardware expects.

Here is some more information about "broken" software that doesn't yet work on ARM64 Asahi Linux (includes links to details on how several previously unsupported applications have already been fixed to support 16 KB page size!):
https://github.com/AsahiLinux/docs/wiki/Broken-Software#if-package-supports-aarch64-why-doesnt-it-work
(random example link from that page about Chromium (fixed in 2022): "ARM64 supports 4kb, 16kb, and 64kb page sizes. Previously, only 4kb was supported by Chromium. This patch adds 16kb support, as is used for example by Asahi Linux on M1 Macs")

PS. Why I'm reporting this: The 4.2 Beta 2 announcement added experimental builds for ARM Linux and asked to "Please let us know if you encounter any issues."

Also, I was able to quickly test (on macOS, using QEMM in UTM) that the same build did work in a similar Fedora 39 ARM64 VM that was using 4K pages, so I'm pretty sure 16K support is the issue here (i.e. not a Fedora bug).

Steps to reproduce

0. Use a kernel with 16 KB page size (e.g. Asahi Linux on ARM MacBook Air M1 or Raspberry Pi 5 with the default 16KB kernel)
1. Try to run the Linux Aarch64 / ARM64 build

What happens (doesn't open at all, only shows an error message):

$ ./MuseScore-4.2.0.233381325-experimental-aarch64.AppImage 
/lib64/libjack.so.0
/lib64/libnss3.so
/tmp/.mount_MuseScsp9iiA/bin/mscore4portabletesting: error while loading shared libraries: libsndfile.so.1: ELF load command address/offset not page-aligned
$
$ getconf PAGESIZE
16384

(the same problem also happens with the latest arm64 Linux nightly)

Screenshots/Screen recordings

No response

MuseScore Version

MuseScore 4.2 Beta 2 (Pre-release)

Regression

No.

Operating system

Fedora Asahi Remix 39 (ARM64 Linux)

Additional context

No response

[marcan]

This is a build configuration problem. Basically every ARM64 Linux distro has defaulted to 64K segment alignment for years, to support all page sizes correctly. It looks like that libsndfile.so.1 is built for 4K segment alignment. I assume that comes from the AppImage - is it some really old build, or was it built with a niche compiler/build environment with bad defaults or something like that?

(Needless to say, distribution builds of MuseScore from e.g. Fedora work perfectly fine on 16K page systems, so this is something specific about the official ARM64 builds - OP: you should be able to just dnf install musescore to use the packaged version.)

[cbjeukendrup]

It seems libsndfile from apt is used; see

MuseScore/build/ci/linux/setup-arm.sh

Line 85 in efbc260

libsndfile1-dev

. If that isn't working, I guess we'd have to find a different place to get it from, or build it (and all its dependencies?) from source...
@theofficialgman What are your thoughts about this issue?

[theofficialgman]

I can't comment more now as I am short on time (will have more next week). But yes I am aware that the ubuntu provided libsndfile binary fails on 16K pagesizes within the appimage (we saw this on Pi5 on PiOS Bookworm as well at pi-apps Botspot/pi-apps@e4a484c).

Not sure whats wrong with it because I did check the elfheader three weeks ago of the libsndfile.so.1 and all LOAD are 64K aligned (0x10000).

readelf -l usr/lib/aarch64-linux-gnu/libsndfile.so.1

Elf file type is DYN (Shared object file)
Entry point 0x69a0
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000070b28 0x0000000000070b28  R E    0x10000
  LOAD           0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002830 0x0000000000004e20  RW     0x10000
  DYNAMIC        0x0000000000072cc0 0x0000000000082cc0 0x0000000000082cc0
                 0x0000000000000250 0x0000000000000250  RW     0x8
  NOTE           0x00000000000001c8 0x00000000000001c8 0x00000000000001c8
                 0x0000000000000024 0x0000000000000024  R      0x4
  GNU_EH_FRAME   0x0000000000061058 0x0000000000061058 0x0000000000061058
                 0x00000000000018ec 0x00000000000018ec  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002060 0x0000000000002060  R      0x1

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   01     .init_array .fini_array .data.rel.ro .dynamic .got .got.plt .data .bss 
   02     .dynamic 
   03     .note.gnu.build-id 
   04     .eh_frame_hdr 
   05     
   06     .init_array .fini_array .data.rel.ro .dynamic .got

CC @marcan the above disproves your theory I think. Feel free to certify yourself though, the binary comes from ubuntu focal libsndfile1 package http://ports.ubuntu.com/pool/main/libs/libsndfile/libsndfile1_1.0.28-7ubuntu0.2_arm64.deb
I would gladly take input from someone who can do binary analysis on this and show why that error appears.

[theofficialgman]

oh huh... interesting
the binary got rewritten with a new load address when copied into the appimage that is only 4K aligned
investigation needed

/tmp/musescore/squashfs-root/lib$ readelf -l libsndfile.so.1 

Elf file type is DYN (Shared object file)
Entry point 0x69a0
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000070b28 0x0000000000070b28  R E    0x10000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  NOTE           0x00000000000001c8 0x00000000000001c8 0x00000000000001c8
                 0x0000000000000024 0x0000000000000024  R      0x4
  GNU_EH_FRAME   0x0000000000061058 0x0000000000061058 0x0000000000061058
                 0x00000000000018ec 0x00000000000018ec  R      0x4
  LOAD           0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002830 0x0000000000004e20  RW     0x10000
  GNU_RELRO      0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002060 0x0000000000002060  R      0x1
  DYNAMIC        0x0000000000074000 0x0000000000086000 0x0000000000086000
                 0x0000000000000260 0x0000000000000260  RW     0x8
  LOAD           0x0000000000074000 0x0000000000086000 0x0000000000086000
                 0x0000000000001048 0x0000000000001048  RW     0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .dynsym .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   01     
   02     
   03     .eh_frame_hdr 
   04     .init_array .fini_array .data.rel.ro .got .got.plt .data .bss 
   05     .init_array .fini_array .data.rel.ro .got 
   06     .dynamic 
   07     .dynamic .dynstr .gnu.hash .note.gnu.build-id

[marcan]

Is there some weird ELF mangling going on as part of the AppImage build process? In the original file GNU_RELRO and DYNAMIC sections are part of the data pheader too, while in the new file they are not.

It sounds to me like something is patching the DYNAMIC section, and in the process moving it out of the existing LOAD pheader and adding a new one... and getting the section alignment wrong.

[marcan]

Yup, this is patchelf's fault: NixOS/patchelf#474

[marcan]

Actually, not so sure. It looks like patchelf correctly considers the base page size for ARM64 to be 64K: https://github.com/NixOS/patchelf/blame/917ea45b79de04f69059f42a8e2621f7caeae1c9/src/patchelf.cc#L361

Is this build of patchelf built using --with-page-size=4096? If so, that is the problem. The default autodetect behavior should be correct as far as I can tell, on ARM64.

[theofficialgman]

I looks to me like linuxdeploy handles copying, stripping, and setting the RPATH in the binaries as part of the appimage build process
https://github.com/musescore/MuseScore/actions/runs/7109599307/job/19354779005#step:7:17747
maybe stripping causes the EFL mangling? maybe its the RPATH setting?

[theofficialgman]

yup, bug has been found.
setting of the RPATH with patchelf breaks it

ubuntu@instance-20220801-1619:/tmp/sndfile/usr/lib/aarch64-linux-gnu$ readelf -l libsndfile.so.1

Elf file type is DYN (Shared object file)
Entry point 0x69a0
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000070b28 0x0000000000070b28  R E    0x10000
  LOAD           0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002830 0x0000000000004e20  RW     0x10000
  DYNAMIC        0x0000000000072cc0 0x0000000000082cc0 0x0000000000082cc0
                 0x0000000000000250 0x0000000000000250  RW     0x8
  NOTE           0x00000000000001c8 0x00000000000001c8 0x00000000000001c8
                 0x0000000000000024 0x0000000000000024  R      0x4
  GNU_EH_FRAME   0x0000000000061058 0x0000000000061058 0x0000000000061058
                 0x00000000000018ec 0x00000000000018ec  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002060 0x0000000000002060  R      0x1

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   01     .init_array .fini_array .data.rel.ro .dynamic .got .got.plt .data .bss 
   02     .dynamic 
   03     .note.gnu.build-id 
   04     .eh_frame_hdr 
   05     
   06     .init_array .fini_array .data.rel.ro .dynamic .got 
ubuntu@instance-20220801-1619:/tmp/sndfile/usr/lib/aarch64-linux-gnu$ patchelf --set-rpath '/tmp' libsndfile.so.1
ubuntu@instance-20220801-1619:/tmp/sndfile/usr/lib/aarch64-linux-gnu$ readelf -l libsndfile.so.1

Elf file type is DYN (Shared object file)
Entry point 0x69a0
There are 8 program headers, starting at offset 475136

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000070b28 0x0000000000070b28  R E    0x10000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  NOTE           0x00000000000001c8 0x00000000000001c8 0x00000000000001c8
                 0x0000000000000024 0x0000000000000024  R      0x4
  GNU_EH_FRAME   0x0000000000061058 0x0000000000061058 0x0000000000061058
                 0x00000000000018ec 0x00000000000018ec  R      0x4
  LOAD           0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002830 0x0000000000004e20  RW     0x10000
  GNU_RELRO      0x0000000000070fa0 0x0000000000080fa0 0x0000000000080fa0
                 0x0000000000002060 0x0000000000002060  R      0x1
  LOAD           0x0000000000074000 0x0000000000086000 0x0000000000086000
                 0x0000000000001078 0x0000000000001078  RW     0x1000
  DYNAMIC        0x00000000000741c0 0x00000000000861c0 0x00000000000861c0
                 0x0000000000000260 0x0000000000000260  RW     0x8

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id .gnu.hash .dynsym .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   01     
   02     .note.gnu.build-id 
   03     .eh_frame_hdr 
   04     .init_array .fini_array .data.rel.ro .got .got.plt .data .bss 
   05     .init_array .fini_array .data.rel.ro .got 
   06     .dynamic .dynstr 
   07     .dynamic

[theofficialgman]

and the patchelf bug that was fixed in 0.12 that causes this NixOS/patchelf@0470d69 .... focal has 0.10

Currently patchelf uses the host system's page size (determined at build
time) as the default section load memory alignment. This causes multiple
issues

[theofficialgman]

patchelf 0.14.3 from jammy builds fine on ubuntu bionic in my testing (that just what I had easily available). I'll add it to a ppa or the setup script

edit: ppa made https://launchpad.net/~theofficialgman/+archive/ubuntu/patchelf waiting on launchpad to publish
I will PR once published and tested

[marcan]

3+-year-old software strikes again... (also hi @delroth).

FWIW, this is a theme in ARM64 land. If you are using any frameworks, builds, tools, etc. that are more than 2 years old, they are highly likely to be buggy or broken in some way. ARM64 Linux is only just taking off, and things were very broken just a few years ago. Even if you wouldn't do so otherwise on x86-64 for e.g. better host system compat, you're much better off going for newer stuff on ARM64. It's not just stuff like this; e.g. I got major Qt (QML javascript crashing) and GCC (miscompiling) bugs fixed and very recently a glibc bug fixed (TLS segfaults with dynamic loading).