For a while now I've been running Broadcaster.VC's github actions on a spare machine I have in the office. Building things for both x64 and arm64 could take some time and if I let those builds happen on Github's hosted runners I ran the risk of spending money I didn't need to - I had spare compute going in the office so why not take advantage of it?
When I was running things on Github's runners - things weren't slow but they also weren't blazingly fast either and so I didn't really notice how slow things truly were. When I moved to a self hosted runner in the office, I did notice how slow things were... compiling things on an AMD Ryzen 9 5900X with 24 threads meant my X64 builds were lightning fast - taking advantage of every thread available meant those builds were super quick - I couldn't have been happier but things were a totally different story for my arm64 builds where it felt like the QEMU virtualisation Docker buildx handled for me, meant those builds were barely making any use of the 24 threads available to it.
For a while I wondered how I could make it quicker... could I add another native arm64 runner to the github actions pool? No I couldn't - because then the multi-arch docker images I was trying to create wouldn't be multi-arch at all. Ultimately the way to do it was so simple I'm annoyed it took me so long to figure out. You need an arm64 host with docker installed, and for that docker server to be available from within the same network as where you're running your self hosted runner - you might choose to run an AWS Graviton host or just run a Pi4 within the same network... that's what I did. I'd have preferred to run the Github runner on an M1 mac mini.... but that was an expense I didn't want to cover just yet.
So how do you do it? Like I said, first things first, install Docker on your arm64 host and then make it available over the network - you can follow the instructions in this gist. Once you've got that... it's as simple as adding a few lines to your Github Action yaml file.
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
id: builder
uses: docker/setup-buildx-action@v1
- name: "Append ARM buildx builder"
uses: baschny/append-buildx-action@v1
with:
builder: ${{ steps.builder.outputs.name }}
endpoint: "tcp://the-fqdn-or-ip-of-arm64-host:2375"
You've probably already got the "Set up QEMU" step and the "Set up Docker Buildx" step but there's an important addition you need in that step... adding an "id" to the buildx builder.
The next section is the important one... you want to add an arm64 host to your "buildx" setup so that it won't try and build arm64 builds in a virtualised builx builder. And that's how simple it is... you don't need to try and add more github action runners etc etc.... you just need to give that Docker Buildx environment the right kind of host where it can ask for a build.
The way things are setup today; I commit to Github, that asks my local linux X64 machine to build the project, and in turn, that x64 runner adds a local Pi 4 to it's docker buildx environment and both the Pi and the Ryzen machine build things in parallel - obviously the Ryzen machine finishes much earlier but the Pi4 is faster than letting the Ryzen emulate arm64 and build using QEMU - which feels crazy but it's definitely true.
Now I could definitely get even faster builds than the Pi4 offers... and I did wonder if setting up a "arm64" emulated host using Proxmox would be even quicker than a Pi 4. But its unlikely worth the time to investigate too much further - the biggest win would be to run on dedicated arm64 hardware that had more horsepower than a Pi4 and as far as I'm aware... the only thing that fits the bill would be an M1 Mac - I have an M1 mac but it's my laptop and won't always be available within the network and so a poor choice for builds.
Know a better way or a cheaper arm64 host I could use? Let me know over on twitter - @dan_jenkins
(Sorry for lack of links.... this was a very quick write up!)