‘Hiring’ an AI Tool to Code? What to Ask at the Interview
Ever since ChatGPT was released in late 2022, the internet has been abuzz with equal parts doom and optimism. Love it or hate it, artificial intelligence (AI) is coming to your development organization. Even if you don’t plan on developing an AI product or plan to hire an AI development bot to write code for you, it might still be integrated into the tools and platforms used to build, test and run your artisanal source code.
We’ve seen that AI tools will have unique risks that potentially offset the huge gains in productivity offered by automating tasks that once required human brain cells. These risks stem from how the AI is trained, built, hosted and used differently than other software tools that developers trust. Understanding risk is the first step in managing it, and in order to help you understand potential risks associated with your incoming AI tooling, we’ve written some interview questions that should be part of the onboarding process.
There are some generalities associated with AI tools of any level, and these questions should be asked regardless of the AI’s type or purpose:
- Where will it be hosted? Modern AI currently requires dedicated and expensive hardware to do the kind of astounding tasks we’re seeing make headlines today. Unless you’re going to acquire a new data center, your AI bot will work remotely and require the same security considerations associated with human workers using remote access and off-site data storage.
- What kind of safeguards are in place to prevent IP loss as code leaves the boundary? Everything from smart TVs to cars are reporting usage data to their manufacturer. Some are using that data to improve their software, but others are selling that data to advertisers. Understand exactly how your AI tool will use or dispose of source code or other private data it uses for its primary task.
- Will your inputs be used in future training for the AI? Ongoing training of the AI models will be an increasing area of concern both for owners and for those whose data is used to train the model. Owners, for example, might want to keep advertisers from influencing the AI bot in a direction that benefits their clients. Artists who shared works online have seen AI image-generation bots able to replicate their styles wholesale and are very concerned about loss or theft of creative identity.
- What is the fidelity of its results? ChatGPT’s most well-known drawback is the inaccuracy of its results. It will confidently assert falsehoods as easily as truths. This has been referred to as AI hallucinating. Understanding how and where an AI may hallucinate can help manage it when it does.
On top of that, AI owners and developers will have their own host of security concerns. These new concerns include threats to the AI’s training model that can corrupt its results and disclose proprietary information about how the model operates, and threats that could disrupt the model’s quality of results. Additionally, the AI model will have to interface with the traditional world through APIs, web access, mobile applications and other applications that need to be built securely.
Beyond the general questions, developers will have to ask additional questions when using AI tools, such as an AI security scanner to manage risks introduced during software development.
- Is an AI tool a best fit for this use case? Understanding what AI is and isn’t good at is key. The further a task can be broken down into “making a decision according to learned rules” or “writing content that passes learned rules,” the better the AI will be at it. As the problem changes beyond that, the AI will get worse at it.
- What safeguards are in place if the tool doesn’t catch something or hallucinates something that’s not there? Never introduce a single point of failure into your processes, especially one that can hallucinate. Rely on traditional practices associated with defense in depth, or the “Swiss Cheese” method of managing risk where even if one layer misses a problem, the next will catch it.
- What oversight is required to vet the tool results? This is an old problem made new; the traditional logging guidance breaks down into two parts. The first is capturing data about important events; the second is auditing the logs. Until AI matures further and its drawbacks are understood or mitigated, humans will still need to be kept in the loop.
More and more developers are “hiring” ChatGPT to write source code. Initial reports are that ChatGPT is capable of writing source code in many programming languages and is fluent in all of the common and publicly discussed languages. Due to limitations in this beta version’s training and model, the code it produces isn’t always perfect. It often contains business logic flaws that can change the way the software operates, syntax errors where it may blend different versions of software and other problems that appear human in origin.
Basically, ChatGPT is a junior developer. Who will be its manager?
Put another way, ChatGPT is a junior developer. When using code written by this junior developer, one must consider how it will be managed:
- Who will be its manager ensuring that the code written is functional, optimized, quality and up to security standards? Junior developers need senior developers. Every line of code will have to be tested, and some will have to be fixed. However, initial reports are that this proof-reading process is faster and easier than writing code from scratch.
- Is it injecting or remixing training code into your code base? A more insidious threat is that sometimes AI bots like GitHub Copilot produce source code that perfectly replicates blocks of code from its training data. Antiplagiarism tools will be needed to ensure license risk is managed.
- Where does the bot get its training data? An AI model will only be as good as its training data. If the bot is trained on old or incorrect code, it will produce old and incorrect results.
- Where is the engine hosted? An AI bot that analyzes source code will need to bring the source code to its processing facility. Give extra thought to how the data is secured, used and disposed of after leaving your company’s boundary.
December’s release of ChatGPT heralded a new age in software development. It’s important to lean into the changes rather than get knocked out by them. When adopting these new tools, understand that the more things change, the more they stay the same: It’s always better to prevent a security incident than it is to be caught unawares by one.
