“ChatGPT, write me some code”

Recently, ChatGPT passed a Google coding challenge. So, AI practically got a job at one of the tech giants. Copywriters, teachers, and lawyers are afraid to lose their jobs. Now, programmers are scared, too. 

We know ChatGPT can write code, but is it enough to become at least a satisfactory junior Java developer in real life? Should we fear ChatGPT replacing developers shortly? Or should we happily embrace this new technology due to its valuable features? I’m a mentor at CodeGym Java University, and my students who learn Java are worried about their perspectives. So, I decided to test ChatGPT’s coding abilities myself, and I’m ready to share my conclusions.

Round One: A Snake Game

ChatGPT isn’t very humble when describing its coding style (see the screenshot). But any developer knows that bragging is one thing, and delivering is entirely different. So, let’s see if AI tells the truth!

To make the process more comprehensive, I’ve been experimenting with both free and paid versions of ChatGPT. I started with quite a simple task: I asked ChatGPT to write a simple Snake game. It’s beginner level, and any junior Java developer can solve it. ChatGPT generated a solution, too, but I noticed a problem – quite often, it didn’t finish the code. Even after upgrading to the paid version, the problem remained (although AI started typing much faster).

Whenever it stopped, I asked it to proceed, and ChatGPT would resume writing the game. Still, some small parts of the code (a few lines or half of a line) were missing anyway. If you’re not an experienced programmer, you may not notice it – and will get code that doesn’t work.

At this point, I started doubting that ChatGPT can write a big program. If it has trouble writing 200 lines of code (as in the Snake game), how can it handle thousands of lines, like in a typical program?

Also, during the first attempt, ChatGPT wrote the Snake game using just the method main()  without dividing the code into several methods. It’s a bad approach even for junior developers. I checked the code manually and with the static code analyzer (Sonar Lint) and found some other minor flaws.

I decided to ask ChatGPT why it chose such an unfortunate approach. It turned out, it had an answer! It explained its logic and suggested another solution (see the screenshot). But what if I didn’t ask, as a junior developer probably wouldn’t do?

So, I believe it’s naïve to expect ChatGPT to replace even a junior developer anytime soon. If you’re not a programmer, you won’t be able to understand if the resulting quality is good enough. And nobody wants to buy a pig in a poke.

Let’s imagine that a company decided to cut back on several Java trainees or juniors and “hired” ChatGPT instead. Still, it will need a senior developer to do code refactoring, i.e., clean and improve the code. It’s hard to say if it makes sense in terms of cost optimization.

We can safely say that ChatGPT loses the first round. It can’t replace a good Java junior developer, and who needs another bad one?  

Round Two: Autotesting

In the second part of my experiment, I decided to see if ChatGPT could help with testing the code or at least do the basic testing. I asked it to write unit tests for the Snake game solution it provided before, and it did it.

For an untrained eye, the tests ChatGPT created looked okay. But for a senior developer – not so much. Again, don’t expect anything above the junior level here. The code that ChatGPT generated wouldn’t go into production without serious improvement. And, frankly speaking, a junior who wrote it wouldn’t last in my team for long.

Round Three: Interview Preparation

My third hypothesis was that ChatGPT could help you prepare for job interviews. For example, you can ask it to generate technical and non-technical questions for junior/middle/senior developer interviews. I did that, and some questions were pretty good (although some weren’t). Still, AI suggested only four blocks of questions, and it’s definitely not enough to conduct an interview properly.

CAPTION: Some of the technical questions for a middle Java dev interview (number 4 in the Web Services section is outdated)

So, in the third round, ChatGPT didn’t impress me, too. You may use it to generate some interview questions, but you need to know what parts are essential for a good job interview.

On the Bright Side

At this point, I was a bit disappointed (“give me back my $20” even crossed my mind). But if ChatGPT can’t replace programmers yet, it doesn’t mean they can’t benefit from this cutting-edge technology. I continued experimenting and eventually found it to be quite useful. What can ChatGPT help you with?

First, it can teach you programming basics and tell you what to learn. For example, I asked ChatGPT to teach me Java, and it generated a list of necessary topics (lambda expressions, functions, data structures, etc.) and examples. So, it can be your Java (or other programming language) manual. But unlike a book, it can answer your questions. And unlike some teachers, it can explain a concept to you as many times as you need to grasp it without getting annoyed.

Second, it can provide samples. For instance, if a junior developer doesn’t know how to do the unit testing of the code, they can ask ChatGPT to show them simple examples and explain the basic ideas (like I did on a screenshot).

My everyday work is tightly connected to architectural solutions. That’s why I decided to use ChatGPT’s help here. For example, I asked if it knows what AWS (Amazon Web Service) is, and yes, it does. Then, I told it that I needed to create an EC2 (Elastic Compute Cloud) in China, and ChatGPT suggested a relevant list of steps for it (see the screenshot).

It offered me examples when I needed them and even warned me about the limitations of AWS in China. When I asked ChatGPT for more details, it provided adequate information. I was pleasantly surprised and concluded that ChatGPT’s great for finding the answers to architectural, high-level questions. For instance, in my case, about microservices, sending messages between services, etc.

Moreover, it can answer specific questions like “What’s the better message broker?” and offer information about each option (see the screenshot). Finding such information online would take quite a long time. If I googled the same question, I’d get dozens of links and spend hours browsing them.

I noticed some of the answers weren’t complete, so I had to ask for additional details. To do it, you need to deeply understand what you’re talking about. That’s why I see ChatGPT as an assistant for a person who’s already a professional, who can assess the answers, get into more profound matters, etc.

Frankly speaking, for me, ChatGPT is like Stack Overflow (or Google) on steroids. After doing my research and experimenting, I’m sure that ChatGPT isn’t a tool for code-generating. But as a Google replacement that saves time, it serves pretty well. Developers, we are safe (for now).

Also, I think we shouldn’t neglect ChatGPT’s help for learning and other types of assistance. If it can’t replace us, it still can be a useful tool for developers.