An Unlinked Adventure

cargo-dist

cargo dist init will generate a github action to handle automatic release creation on Github when tags update. For maru, I was missing repository = ... in my Cargo.toml, which cargo-dist noticed and prompted me to fix.

After adding:

repository = "https://github.com/erikareads/maru"

to my Cargo.toml, cargo dist init helpfully generated a github action workflow and updated the Cargo.toml with configuration metadata. I chose to use the shell installer, to see how it worked for maru.

Previously, I had run

cargo install --git "https://github.com/erikareads/maru"

which compiled maru locally, which worked on my NixOS machine.

After generating the Github action yaml with cargo-dist, I committed the changes and pushed a tag. To my delight, the Github action automatically compiled maru for three different architectures and generated a release.

However, when I downloaded the maru-x86_64-unknown-linux-gnu.tar.xz, unpacked it, and ran ./maru I got this error:

$ ./maru
bash: ./maru: cannot execute: required file not found

Uh oh, what required file isn’t working? This is where the linking adventure really began.

The adventure begins - strings

Jeff suggested using strings to see the difference between my working locally compiled version and the broken downloaded version.

diff <(strings ~/.cargo/bin/maru) <(strings ~/Downloads/maru-x86_64-unknown-linux-gnu/maru ) | less

From which we found:

< /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/ld-linux-x86-64.so.2
---
> /lib64/ld-linux-x86-64.so.2
> libgcc_s.so.1
...

Oh, non-standard NixOS strikes again. I don’t yet understand what that difference means, but it’s clear that it’s at the heart of what’s breaking the binary.

Now that we know that an interaction with NixOS is probably the source of the issue. A searchI used Duck Duck Go for my searches during this adventure. of

nixos rust cannot execute: required file not found

Led to a matklad article about ld on NixOS. There he suggests readelf as a path forward:

$ readelf -d ~/Downloads/maru-x86_64-unknown-linux-gnu/maru
Dynamic section at offset 0x16b560 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 ...

Comparing that with the locally built working version:

$ readelf -d ~/.cargo/bin/maru
Dynamic section at offset 0x175ea8 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/...]
 ...

The locally built one has an extra shared library -> the ld linker that we found different in the strings output. In addition, NixOS sets a RUNPATH that allows the binary to find the shared libraries on NixOS.

Knowing that ld is the problem on NixOS, I searched:

compiled binary doesn't work on nixos ld

Which led to an article on nix-ld. Binaries assume the ld is a fixed location for glibc, but NixOS breaks that convention. nix-ld addresses the problem by overriding the RUNPATH that was set when we compiled locally, even on binaries that where we can’t recompile them ourselves.

Which is great, but I have the source code available, maru is my project! Is there a way that I can compile maru such that this linking path problem isn’t an issue?

How does ripgrep do it?

My first thought was to look at ripgrep, a known well-distributed Rust CLI. I installed ripgrep using NixOS, so maybe that handled this issue for me, but I wanted to check. Looking in ripgrep‘s releases, I see that it *doesn’t* compile for unknown-linux-gnu, but instead for unknown-linux-musl.

Downloading this binary of ripgrep, the invocation works without issue.

musl?

This led me to search for:

rust compile without dynamic linking

Which led to this stack overflow question which led me to this answer:

Rust statically links everything but glibc (and libgcc, iirc) by default.

If you want to get a 100% statically linked binary, you can use MUSL with 1.1. https://github.com/rust-lang/rust/pull/24777 is the initial support, we hope to make it much easier to use in the future.

EDIT: It’s distributed via rustup now, so you can add x86_64-unknown-linux-musl as a target : rustup target add x86_64-unknown-linux-musl

And then build to this target : cargo build --target=x86_64-unknown-linux-musl

Linking against musl will allow maru to be statically compiled, which should avoid the dynamic linking problem on NixOS.

After running the relevant rustup and cargo build commands, I confirmed that a locally musl compiled binary worked on my NixOS machine.

Patching the generated github actions

Now, I don’t want to locally compile every binary of maru for my project, so I needed to update the Github actions in order to do the musl compile.

An initial test, just switching out the build target failed, because it didn’t have the target environment installed via rustup.

The fix was to add a conditional to my Github action workflow:

- name: Maybe install target
  if: ${{ contains(matrix.dist-args, 'musl') }}
  run: rustup target add x86_64-unknown-linux-musl

The Github actions API exposes a function contains, which allows us to only run this command on the musl compilation in the matrix of operating systems. rustup was already being used to install Rust in the Continuous Integration, so we expected that the rustup target would work.

After a few minutes of eagerly waiting for the new version to compile, we confirmed that the musl binary worked both on NixOS and on Debian.

This was great, but the shell installer provided by cargo-dist no longer worked, since it was looking for the gnu binary.Turns out this was a known problem and was mentioned in the limitations/caveats in the documentation. Check out this issue for discussion on the problem. The shell installer checks to see whether glibc exists on the installing system, which isn’t sufficient for NixOS to avoid the problem, but it doesn’t switch to musl if the gnu binary isn’t found in the release.

Results

Thanks to this adventure, maru is now statically linked for linux systems, and runs without issues straight after download on both NixOS and Debian.

Jeff and I learned about excellent debugging tools such as readelf.Also strace and file which I didn’t mention in the story, since readelf provided the correct information. file did share: interpreter /lib64/ld-linux-x86-64.so.2, which succinctly showed the path.