Teach the AI to Unit Test

The Gemini AI will make some pretty good guesses about how a 3rd-party API may work. It is good at searching the internet, but when APIs have changed across versions, the old and new docs and examples it’ll find can confuse it. In a dynamic language and environment you’ll not spot these errors until runtime.

To combat the ambiguity and to give the AI agent more power to solve its own problems, ask it to add some tests around the code that uses the API. (In my case, the API is the XTDB client API.) Once it has a way to execute the code through tests, it’ll quickly start figuring out where it’s made mistakes and start running its own experiments to observe errors, search for fixes, and applying those fixes around the codebase. I exhibit the same pattern when I’m doing it by hand.

The tests also give you, the human, an easier entry point to evaluate the code the AI generated. If the tests look gnarly, you know to suggest refactorings to improve the architecture and make it easier to test. When the AI has the tests passing, and the test code is easy enough to read, then you can have a closer look at the application code to refine and keep that maintainable too.