Let’s Talk: Conversational Software Development

Here’s the prompt that kicked off what turned into a very successful interaction with ChatGPT.

A Passing Observation

The prompt was an intentionally naive version of the bash script I envisioned. I had intended to also provide sample output in the initial prompt but forgot to do so. In hindsight, it was probably a good idea to withhold the output and let ChatGPT focus first on the script. Among its observations:

This was a nice example of Learning while coding: How LLMs teach you implicitly. I’d had some understanding of what I think of as strict mode in bash, but it was incomplete. So we paused to explore that topic for a while, I solidified my knowledge, and made a note to consider using other -o arguments — notably -x (xtrace) and -n (noexec) for debugging. We had barely gotten started on the actual task at hand, but this small detour was intrinsically valuable.

Let’s Talk About the JSON

I now circled back to provide the sample output:

In another context, I might have taken the time to pretty-print this JSON output. But LLMs — like compilers and interpreters — don’t care about that. Even better, they tolerate syntax that chokes conventional JSON formatters. If you paste the above example into a conventional formatter you’ll run into two problems. First, it won’t be valid JSON until you wrap it in curly braces. Second, the value of the tickets_list won’t be a JSON object, it’ll be a string, because the value is a stringified representation of JSON. LLMs, quite wonderfully, “know” your intent and do the right thing:

This is trivial stuff, but it’s the kind of thing that can kill your flow and waste cognitive resources best reserved for higher-order functions. At this point I also conducted what I’ll call an explainability test: Did the LLM explain that 1) the fragment needs to be wrapped in curlies, and 2) the nested value needs to be parsed? Here are the results.

This interests me because, although I know that Copilot Chat uses OpenAI and Cody uses Anthropic, that’s not always obvious. The respective chat wrappers augment prompts in ways that make responses differ from those you get from the underlying LLMs. In this example, though, the tools seem to align with the behavior of their wrappers.

Let’s Talk About the Test Strategy

ChatGPT’s observation about set -euo pipefail was one of a half-dozen avenues we could have explored, but I steered it back to the task at hand: refining the test script I included in the initial prompt. The output it tests is from the Freshdesk ticketing system, and the naive test just checked to see that the tool we were testing could call Freshdesk’s API and return more than zero tickets. What would be a better test? ChatGPT’s recommendations included checking to see that priorities and statuses match expected values, that null or long lists of tickets are handled gracefully, and that all date fields exist and contain valid dates.

I opted for the last approach, and it revised the test accordingly. This entailed writing a regex to match dates, and looping through tickets applying the regex to each of a set of date fields in each ticket. That felt too complex, so I asked to limit the scope to just a single date field: created_at. That still felt too complex, so we ditched the loop and whittled the test down to just check for a valid date in a single ticket. Then we moved onto the next topic of conversation.

Let’s Talk About How to Generalize to a Suite of Tests

These were the cases to test:

• List Freshdesk tickets.
• Create a new Freshdesk ticket.
• Delete a Freshdesk ticket.
• Retrieve details of a specific Freshdesk ticket.
• Update a Freshdesk ticket.
• Add a note to a Freshdesk ticket.

I’ve rarely used bash functions, but I figured they might be useful here, so I was pleased by a response that structured the bash script as a set of six functions corresponding to the six test cases. “Looks good,” I said, “please give me a complete version that is structured this way and includes the first test.”

At this point a coworker, looking over my shoulder, wondered if using “please” and “thank you” with LLMs yields better results. Beats me! I just do it because it feels good. Is that a dangerous bit of anthropomorphism? Maybe. But I can’t see any upside to bullying my assistants even when they are nonhuman.

The same coworker then asked about an alternative strategy that would drive the tests from a table. We asked:

The response mentioned pros (maintainability, reusability, parallel execution, consistent logging, clarity) and cons (overhead, complexity, migration time, learning curve). It concluded with:

As we were in a phase of rapid prototyping, and not yet prepared to make a long-term investment in this test strategy, we opted to stick with loose functions. But we both felt it had been valuable to spend a few minutes considering the alternative approach.

Let’s Talk About Persisting the Ticket ID

The next test to write was for creating a ticket. ChatGPT proposed storing the ticket’s id for use in other tests of the ability to read, update, and add a note to a ticket. And it wrote a function that persisted the id to a file. I wondered about that, and asked for an assessment of the tradeoffs between saving it in a file versus a variable. Here’s a summary of the response, revised (courtesy of ChatGPT) into tabular form.

And it added this conclusion:

The alternative file-based approach hadn’t even occurred to me. When ChatGPT took that approach it prompted me to consider, and then discuss, the tradeoffs. Since I wound up doing exactly what I would have done pre-LLM, was this detour a pointless distraction? I don’t think so. It’s always valuable to consider alternative approaches. The detour cost very little time and, while it didn’t change what I ultimately did, it felt valuable.

We had a similar interaction when writing the test for the get_ticket function. Would it be sufficient to check that the ID of a retrieved ticket matched the one that we stored? ChatGPT drew a distinction between efficiency and thoroughness, laid out options for a more thorough check, and suggested that for basic smoke-testing it makes sense to prioritize efficiency. Again, I’d already have done the simpler thing, so this didn’t change anything. But even though I knew that of course we were in smoke-testing mode, saying it out loud seemed helpful.

Let’s Talk About Bash

Here’s an interaction that did very much change the outcome. We added a third test to the suite, but only the first two ran. What could be wrong? The debugging proceeded as it would have pre-LLM, but much faster because ChatGPT can sprinkle print statements into code much faster than I can. After a few iterations of this brute-force approach, we still couldn’t get past the second test.

It then occurred to me to ask: “Would it make sense to turn off set -euo pipefail?” When I did that the script then ran to completion — that is, all six tests ran — but the second test was failing when it should have succeeded. Then the penny dropped. I asked: “Should we be returning $TICKET_ID from test_create_ticket?” ChatGPT reminded me of a behavior I’d read about but never encountered:

Facepalm! Of course! We needed the function to set the global variable with the value of the created ticket, but return zero to avoid a premature exit while enabling strict mode to remain in effect. That was the final breakthrough; after that, it was smooth sailing.

When the Rubber Duck Talks Back

I keep coming back to the theme of the first article in this series: When the rubber duck talks back. Thinking out loud always helps. Ideally, you get to do that with a human partner. A rubber duck, though a poor substitute, is far better than nothing.

Conversing with LLMs isn’t like either of these options, it’s something else entirely; and we’re all in the midst of figuring out how it can work. Asking an LLM to write code, and having it magically appear? That’s an obvious life-changer. Talking with an LLM about the code you’re partnering with it to write? I think that’s a less obvious but equally profound life-changer.

Jon Udell is an author, information architect, software developer, and new media innovator. His 1999 book, Practical Internet Groupware, helped lay the foundation for what we now call social software. He has been a software developer at Lotus; BYTE Magazine's...

Read more from Jon Udell