Debugging apps on Kubernetes: The future is remocal

Some of the most painful memories from my 15-year career building enterprise Java applications have involved remote debugging. But this isn’t just a Java-specific problem. Many of my .NET and Ruby developer colleagues have shared the same challenges. 

Through my time working with monolithic apps running on large application servers to building microservices that ran as standalone executables on VMs, I’ve lost track of how many times I had to ask the operations or platform team for help configuring realistic pre-production development environments and opening remote ports and enabling debugging protocol support. 

It’s reasonable to ask if the move towards cloud technologies has helped. The answer is not really. Container and cluster management technology like Kubernetes has offered a standard form factor to package, deploy and rapidly scale applications. However, the age-old problem of remotely debugging applications still remains. And in some cases, the addition of container and cloud technology has made this more difficult. 

I believe that we must move away from traditional remote debugging towards remocal debugging.

More layers, more problems

First, let’s take a step back in time. The arrival of containers and the improved UX that Docker brought to our development machines led to the massive adoption of this technology. Around the same time, a shift towards the next iteration of service-oriented architecture was emerging – the move to microservices. Both trends began to feed off each other. The separation of (operational) concerns and specialization enabled by container technology, combined with the new software architecture paradigms, provided a lot of flexibility to those of us writing software. But this also introduced additional layers of infrastructure to manage and a larger number of software components for developers to orchestrate when building and debugging.

Gone were the days when we started our development journey by checking out the entire codebase and running the application as a debuggable local process. With service-oriented architectures, we often have to check out selective parts of the codebase. We must then build, package, and execute this within container runtimes. Once we encounter a situation where we can’t run all of the components we need locally, debugging a service (with arbitrary remote dependencies) becomes challenging. 

Here we often look for inspiration within the past days of testing monolithic applications running in realistic environments (or with realistic) data, and hence we reach for the remote debugger. Naturally, the easy solution appears to be deploying everything into a remote Kubernetes cluster and debug there.

Remote debugging in the cloud

The majority of language-specific remote debuggers will work as advertised if you can connect to a Kubernetes cluster via kubectl and get access to the remote Kubernetes API. Providing you have RBAC permissions and appropriate access with any policies set, you connect your local machine with the corresponding code base and debugger to a remotely running process. You can step through functions, set breakpoints, and inspect variables locally, and everything runs remotely.

You can use ‘kubectl port-forward’ to target the service under test and set up a proxy to connect to the remote debugging ports via your localhost ports. Tools like kubefwd can provide a better developer experience, and if you’re looking to automate this process and are working on a greenfield project or a codebase that is relatively modern, you can use tools like Skaffold and the corresponding `skaffold debug` command.

Remote debugging is still inherently more complicated, as everything goes over the wire, and the extra layers of networking infrastructure in Kubernetes can provide a challenge, e.g. cloud-based SDN security groups and network access control lists, container networking interfaces (CNI) and Network Policies, and service meshes and intentions. If you’ve connected via a corporate VPN or have a slow internet connection, stepping through functions and breakpoints can be painfully slow.

Take a different approach: Remocal debugging

I believe the time for remote debugging when working with Kubernetes is over. Instead, we should adopt “remocal” debugging. This involves “making the remote, local”, hence the portmanteau. 

Tools like ktunnel and Telepresence allow you to establish a two-way networking proxy connection between your remote cluster and local development machine. This approach makes the standard remote debugging process mentioned above easier, as it effectively “puts your dev machine in the cluster”. There is no longer a need to map ports with kubectl port-forward, as you can now connect your local debugger to a remote service using the Kubernetes FQDN syntax of service-name.namespace-name:debug-port. 

Given that these tools often deploy a proxy component into the remote cluster, if the RBAC and policies permissions are configured correctly, debugging should work with every service running in the cluster. You may have to recruit your operations or platform team to get you up and running, but you won’t need to contact them or create a ticket with every new service you want to debug remotely.

The real change of perspective these tools provide is the ability to run a service locally in debug mode, while testing from the entry point of the remote application (just as a user would interact with the app) and still connect to remote dependencies as you step through the local process. These speeds up the code-test-verify loop dramatically. Debugging a process locally is inherently faster, as there is less configuration and data going over the wire. And if you want to modify the application being debugged, you no longer have to rebuild and redeploy or use a bespoke “hotswap” or remote hot reloading tool. Now you can make a change on your local machine, perform a local reload if required, and test the results against the remote dependencies running in the cluster.

While running the application being debugged locally, you can selectively swap one or a subset of the dependent data stores, middleware, API sandboxes, or other services, while still interacting with the remaining services running remotely. For example, you can debug your application with one of the data stores running locally using something like TestContainers. This lets you rapidly modify the data or load new datasets as required. Once you are happy with your local debugging (which can still interact with other remote dependencies), you can reload your application and point this to the remote version of the datastore to verify correct behavior.

Conclusion

The time for remote debugging is over. Opening ports and enabling the correct debugging protocols can be a laborious process for both developers and operations teams, especially when debugging microservice applications running in a container or Kubernetes-based environment. New tools are emerging that enable a “remote to local” debugging experience, which can provide faster feedback without the configuration headache. Once these tools are installed and running within a remote cluster, developers can self-serve by debugging code locally while initiating the tests remotely (just as a user would interact with the application) and with all the services and data store dependencies deployed remotely. 

To learn more about the transformative nature of cloud native applications and open source software, join us at KubeCon + CloudNativeCon Europe 2023,  hosted by the Cloud Native Computing Foundation, which takes place from April 18-21.