AI for Work
Using an AI code assistant without shipping vulnerabilities
The uncomfortable part of the research on AI code assistants is not that the code was worse. It is that the developers were more confident it was better. That combination, lower security and higher confidence, is what quietly ships holes. Here is what the study found, what the vendors themselves document, and the review routine that puts the gap back under control.
What the study actually measured
In the first large-scale user study of its kind, Perry, Srivastava, Kumar and Boneh had participants solve security-related tasks across several programming languages, with and without an AI assistant based on OpenAI's codex-davinci-002 model. It was published at the ACM SIGSAC Conference on Computer and Communications Security in 2023.
Three findings, in the authors' terms:
- Participants with access to the assistant wrote significantly less secure code than those without.
- They were more likely to believe they had written secure code.
- Participants who trusted the AI less and engaged more with the language and format of their prompts, for example by rephrasing them or adjusting temperature, produced code with fewer vulnerabilities.
The third point is the actionable one, and it is easy to miss under the headline. The study does not say "do not use an assistant". It says that the posture of the person using it changed the outcome. Scepticism and iteration were associated with safer code.
Two honest caveats: the study used a model from that period, and a user study measures the people as much as the tool. We have not run our own benchmark, so we do not claim a figure for today's assistants.
What the vendor puts in writing
GitHub's application card for Copilot inline suggestions is unusually explicit, and worth reading once in full. It documents real safeguards: a content filtering system that detects and blocks harmful, offensive or insecure content before it is shown, and public code matching that, depending on your settings, blocks suggestions matching public code or annotates them with the source repository and licence information.
It also documents the limits, in its own words:
- Security risks: suggestions are generated from the surrounding context and can expose sensitive information or vulnerabilities if not used carefully, so you should be careful in security-sensitive applications and always review and test thoroughly.
- Inaccurate code: the tool may generate code that appears valid but is not semantically or syntactically correct, or does not reflect the developer's intent.
- Matches with public code: generation is probabilistic, and although the probability is low, suggestions may match code in the training set.
- Overreliance: accepting incorrect or incomplete output, mainly because mistakes can be hard to detect.
And the sentence to quote in any internal discussion about responsibility: users assume all risks associated with generated code including security vulnerabilities, bugs, and IP infringement. The same documentation asks you to use secure coding and code review practices, giving hard-coded passwords and SQL injection as the examples to avoid.
The review routine, in five habits
None of this is new practice. What changes with an assistant is the volume of code you did not write arriving in your editor, so the habits have to be deliberate.
- Read before you accept, not after. The cost of reading a suggestion is seconds; the cost of finding it in production is not. If a suggestion is too long to read, it is too long to accept in one go.
- Review your own diff as if it were someone else's. Before committing, read the whole diff. This is the single habit that catches the most, because it is the moment where generated code stops being a suggestion and becomes yours.
- Write the test yourself. If the assistant writes both the code and the test, the test encodes the same misunderstanding. Write the test from the requirement, by hand, then let the assistant make it pass.
- Check every boundary. Anything that touches input from the outside world, authentication, file paths, SQL, shell commands, or serialisation deserves a second read regardless of how clean it looks.
- Check what it imported. Verify that every package it added exists, is the one you meant, and is actually needed. A dependency is a far bigger decision than a function.
Prompts that make review possible
Most prompts ask for code. These ask for the information you need in order to review it, and they follow the same structure as our six-part prompt method.
Write this function, then list separately: every assumption you made about the input, every error case you did not handle, and every security consideration a reviewer should check. Do not reassure me that the code is secure.
And the one that does the most work on existing code:
Review this code as a hostile reviewer. For each problem: the line, what an attacker or a bad input could do, and the smallest fix. Do not rewrite the whole file. If you find nothing, say so instead of inventing issues.
A third, before any commit of generated code: "explain what this code does, line by line, as if I had to defend it in review". If the explanation does not match what you wanted, you just found the bug without running anything.
If you are building with an LLM, not just coding with one
Writing code with an assistant and shipping a product that calls a model are different threat models. For the second, the OWASP GenAI Security Project maintains a Top 10 for LLM Applications, the reference list of the most critical vulnerabilities in LLM applications. If your application takes untrusted text and feeds it to a model that can act, read that list before you design anything, not after.
What to take away
The research does not support banning these tools, and it does not support trusting them. It supports a posture: lower trust, more iteration, and review discipline proportional to the volume of code you did not write. The vendor's own documentation agrees, including on who carries the risk. It is you.
We have not benchmarked any assistant ourselves, so this guide contains no ranking, no price and no productivity claim. The study, the vendor documentation and the review habits are the parts that hold.
Tools in this guide
- Vendor information
GitHub Copilot
Code suggestions inside your editor.
CodePrice: to verify
- Vendor information
Cursor
Code editor built around an AI assistant.
CodePrice: to verify
- Vendor information
Claude
Assistant for long documents and careful writing.
WritePrice: to verify
Questions
Does an AI assistant actually make code less secure?
In the study by Perry, Srivastava, Kumar and Boneh, published at CCS 2023, participants who had access to an AI assistant based on OpenAI's codex-davinci-002 wrote significantly less secure code than those without access, and were more likely to believe they had written secure code.
Do the built-in filters protect me?
Only partly. GitHub documents a content filtering system that detects and blocks insecure code, and public code matching that blocks or annotates matches. The same documentation states that users are responsible for reviewing and validating suggestions, and that users assume all risks associated with generated code including security vulnerabilities, bugs and IP infringement.
What behaviour correlated with safer code in the study?
Participants who trusted the AI less and engaged more with the language and format of their prompts, for example by rephrasing them, produced code with fewer security vulnerabilities.
Sources
- Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh, Do Users Write More Insecure Code with AI Assistants?, CCS '23, ACM SIGSAC, pages 2785-2799 (arXiv:2211.03622)
- GitHub Docs, Application card: GitHub Copilot inline suggestions
- OWASP GenAI Security Project, Top 10 for LLM Applications
One AI email a week. Only what changes something for you.