Hi, I’m Erika Rowland (a.k.a. erikareads).
Hi, I’m Erika. I’m an Ops-shaped Software Engineer, Toolmaker, and Resilience Engineering fan.
I like Elixir, Reading, and Design. She/Her.
If you're looking for experienced talent, I would love to chat.
An Unlinked Adventure
I’ve been working on my new tool maru, to get it polished for release. To that end, I decided to look at cargo-dist a tool that aims to help with release engineering for Rust projects.
Since maru is a Rust CLI, it seemed like a great fit.
This is the story of how the interaction between cross-compilation and NixOS led to a rabbit hole of dynamic linking on linux.Thank you to Jeff for pairing with me as we ventured into this rabbit hole.
cargo dist init will generate a github action to handle automatic release creation on Github when tags update. For maru, I was missing repository = ... in my Cargo.toml, which cargo-dist noticed and prompted me to fix.
repository = "https://github.com/erikareads/maru"
to my Cargo.toml, cargo dist init helpfully generated a github action workflow and updated the Cargo.toml with configuration metadata. I chose to use the shell installer, to see how it worked for maru.
which compiled maru locally, which worked on my NixOS machine.
After generating the Github action yaml with cargo-dist, I committed the changes and pushed a tag.
To my delight, the Github action automatically compiled maru for three different architectures and generated a release.
However, when I downloaded the maru-x86_64-unknown-linux-gnu.tar.xz, unpacked it, and ran ./maru I got this error:
bash: ./maru: cannot execute: required file not found
Uh oh, what required file isn’t working? This is where the linking adventure really began.
The adventure begins - strings
Jeff suggested using strings to see the difference between my working locally compiled version and the broken downloaded version.
diff <(strings ~/.cargo/bin/maru) <(strings ~/Downloads/maru-x86_64-unknown-linux-gnu/maru ) | less
The locally built one has an extra shared library -> the ld linker that we found different in the strings output.
In addition, NixOS sets a RUNPATH that allows the binary to find the shared libraries on NixOS.
Knowing that ld is the problem on NixOS, I searched:
compiled binary doesn't work on nixos ld
Which led to an article on nix-ld. Binaries assume the ld is a fixed location for glibc, but NixOS breaks that convention. nix-ld addresses the problem by overriding the RUNPATH that was set when we compiled locally, even on binaries that where we can’t recompile them ourselves.
Which is great, but I have the source code available, maru is my project! Is there a way that I can compile maru such that this linking path problem isn’t an issue?
How does ripgrep do it?
My first thought was to look at ripgrep, a known well-distributed Rust CLI. I installed ripgrep using NixOS, so maybe that handled this issue for me, but I wanted to check. Looking in ripgrep‘s releases, I see that it *doesn’t* compile for unknown-linux-gnu, but instead for unknown-linux-musl.
Downloading this binary of ripgrep, the invocation works without issue.
This led me to search for:
rust compile without dynamic linking
Which led to this stack overflow question which led me to this answer:
Rust statically links everything but glibc (and libgcc, iirc) by default.
The Github actions API exposes a function contains, which allows us to only run this command on the musl compilation in the matrix of operating systems. rustup was already being used to install Rust in the Continuous Integration, so we expected that the rustup target would work.
After a few minutes of eagerly waiting for the new version to compile, we confirmed that the musl binary worked both on NixOS and on Debian.
This was great, but the shell installer provided by cargo-dist no longer worked, since it was looking for the gnu binary.Turns out this was a known problem and was mentioned in the limitations/caveats in the documentation. Check out this issue for discussion on the problem. The shell installer checks to see whether glibc exists on the installing system, which isn’t sufficient for NixOS to avoid the problem, but it doesn’t switch to musl if the gnu binary isn’t found in the release.
Thanks to this adventure, maru is now statically linked for linux systems, and runs without issues straight after download on both NixOS and Debian.
Jeff and I learned about excellent debugging tools such as readelf.Also strace and file which I didn’t mention in the story, since readelf provided the correct information. file did share: interpreter /lib64/ld-linux-x86-64.so.2, which succinctly showed the path.