I recently had to verify why our application was behaving differently when running on a different operating system. We used to run all our applications on a CentOS 7-flavored OS, but due to its EOL on June 30, company policy was to use RHEL 8/9.
When our application was running in our CICD solution on an AlmaLinux 8 OS (based on RHEL 8), the go-git-v5 library stopped working with a nice:
Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
After having checked pretty much the entire config on the host, the next thing to try was to spin up an AlmaLinux 8 container locally and debug step-by-step the old-fashioned way. Here is how to do it (because it took me some time to get this working).
Prerequisites (really don’t skip this)
I have a reputation for quickly skimming guides, and I had semi-missed that the project path was important.
The wording of the guide I was following simply said, “Oh, if your local project path is different, update the WORKSPACE env,” nonchalantly, without saying that if they don’t match, breakpointing won’t work. I thought they specified that only for people who were blindly copying and pasting their provided Dockerfile.
So yea, note your pwd, mine was
/Users/fferro/Documents/workspace/fferro/poc
You need to have Go > 1.18 installed and working, and Docker fully running locally.
Simple application
Let’s start with a simple application where we can add a breakpoint.
package main
import "fmt"
func main() {
fmt.Println("Hello")
}
Container
Here is an example of a Dockerfile that will build the above app and run it with delve (the Go debugging tool):
FROM docker.io/golang:1.22 AS build-env
WORKDIR /debug/
ADD . /debug/
# I like to enable this, especially when testing, because it will pack static libs into your binary and won't rely on using external host-dependent libraries (hello glibc)
ENV CGO_ENABLED=0
RUN GOOS=linux GOARCH=arm64 go build -o /debuggingTutorial/app .
RUN go install github.com/go-delve/delve/cmd/dlv@latest
FROM almalinux:8
WORKDIR /Users/fferro/Documents/workspace/fferro/poc
COPY --from=build-env /debug/app /Users/fferro/Documents/workspace/fferro/poc/
COPY --from=build-env /go/bin/dlv /Users/fferro/Documents/workspace/fferro/poc/
EXPOSE 5001
CMD ["./dlv", "--listen=:5001", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./app", "--log"]
You don’t have to follow this word for word; the important thing is that you have a Docker container running with this command:
./dlv", "--listen=:XXXX", "--headless=true", "--api-version=2", "exec", "./app", "--log"
The provided Dockerfile is just to have something self-working. If you have these binaries on your machine, to cut down on build time, you can just COPY the bins directly without building anything.
This is the face of my Dockerfile, for example. The important thing is that you build your bin for the right ARCH…
FROM almalinux:8
WORKDIR /Users/fferro/Documents/workspace/poc
COPY ./poc
COPY ./dlv
EXPOSE 5001
CMD ["./dlv", "--listen=:5001", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./poc", "--log"]
Setup IntelliJ
I’m going to set up all the runs with IntelliJ rather than manually docker build and docker run myself in the terminal. IntelliJ has Docker integration and is very neat.
Create a new “Docker” Run Configuration like this:
It is important to specify these run options:
--security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE
Basically, the two options enable Linux debugging (cap-add=SYS_PTRACE) and disable AppArmor to bypass its restrictions. Without these, your app will successfully attach to delve, but breakpoints won’t work.
Click on “run” and IntelliJ will start its Docker management services. You can see the logs in the service tab. If you don’t have it, check your IntelliJ view settings.
Click on the log tab to get:
API server listening at: [::]:5001
2024-07-18T10:56:04.572210269Z 2024-07-18T10:56:04Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2024-07-18T10:56:04.572213311Z 2024-07-18T10:56:04Z info layer=debugger launching process with args: [./srv]
2024-07-18T10:56:04.654346811Z 2024-07-18T10:56:04Z debug layer=debugger Adding target 11 "/Users/fferro/Documents/workspace/fferro/poc/app"
Run the application
Create a Go remote Run Configuration and leave it as is. If you have modified the listening port in the dlv command, update it here to match (it doesn’t matter which port you choose, just make sure they match). Keep the host as localhost.
Click on ok and add a breakpoint on the fmt.Println line of your application. Then run it. Look into the container log and you should see this:
API server listening at: [::]:5001
2024-07-18T11:05:16.084929366Z 2024-07-18T11:05:16Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2024-07-18T11:05:16.084934074Z 2024-07-18T11:05:16Z info layer=debugger launching process with args: [./srv]
2024-07-18T11:05:16.344759199Z 2024-07-18T11:05:16Z debug layer=debugger Adding target 11 "/Users/fferro/Documents/workspace/fferro/poc/app"
2024-07-18T11:05:20.463589035Z 2024-07-18T11:05:20Z info layer=debugger created breakpoint: &api.Breakpoint{ID:1, Name:"", Addr:.....
2024-07-18T11:05:20.467686493Z 2024-07-18T11:05:20Z debug layer=debugger continuing
2024-07-18T11:05:20.467782160Z 2024-07-18T11:05:20Z debug layer=debugger ContinueOnce
2024-07-18T11:05:20.472648160Z 2024-07-18T11:05:20Z debug layer=debugger callInjection protocol on:
2024-07-18T11:05:20.472685451Z 2024-07-18T11:05:20Z debug layer=debugger 11 PC=0x910d8
2024-07-18T11:05:20.472690618Z 2024-07-18T11:05:20Z debug layer=debugger 16 PC=0x74888
2024-07-18T11:05:20.472994618Z 2024-07-18T11:05:20Z debug layer=debugger 17 PC=0x74eec
2024-07-18T11:05:20.473008451Z 2024-07-18T11:05:20Z debug layer=debugger 18 PC=0x74ef0
If you don’t see these, and especially this created breakpoint: &api. it means something is not right.
Troubleshooting
For me, the first time, nothing worked 😅. This is because I didn’t straight copy-paste the Dockerfile from a guide I was reading and was building my own, so I messed up some stuff. Here are some points that can help you debug.
Path Matching
Yea, this is an important one. At first, my path didn’t match at all with what my WORKSPACE was in the container, and I didn’t know it was going to be an issue.
So note where your project is with pwd and use
the exact same path in your Dockerfile WORKSPACE.
Using the Right Delve ARCH
Basically, locally, I was building multiple arch versions of the dlv, but I messed up a copy-paste and was adding the wrong binary to the container. When debugging wasn’t working, I didn’t immediately think this could be an issue. I didn’t expect the ./dlv bin to work if the arch didn’t match! Usually, you get an error saying the bin arch is wrong and the bin cannot start, but (I think??) because the bins were all built using static linking, no error was thrown.
/Users/fferro/Documents/workspace/fferro/poc # file dlvamd64
dlvamd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=AJqjfgnVS71H08xE_5BT/QVGYKmeViXHArEDoPUdh/v-mVjs15nNAnSv_RRghU/Rs6Ds2Jsk0vfO9lt1iEM, stripped
/Users/fferro/Documents/workspace/fferro/gitpoc # file dlvarm64
dlvarm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=QX3G9j18iEBQI7QR1P4t/a0q72PTRC3yc5SicMqk0/Tyi_xZbekb_G1oV-IbH8/VSkxoBw_bIZSMLKnEefG, stripped
/Users/fferro/Documents/workspace/fferro/gitpoc # uname -a
Linux d846197fd4d2 6.6.30-0-virt #1-Alpine SMP Mon, 06 May 2024 07:55:42 +0000 aarch64 Linux
/Users/fferro/Documents/workspace/fferro/gitpoc # ./dlvamd64
Delve is a source level debugger for Go programs.
I was able to execute /.dlvamd64 and dlvarm64 without issue even though my AlmaLinux 8 container was an ARM64…
I hope this helps! If you need any more assistance, feel free to ask.