Here’s a good use case as to why developers are needed to monitor code generated by AI.
Using Amazon Q, I set up a project that is a 100% unit test project to be 100% AI generated, for another barely tested project. (More about this later; was not a smooth creation.) This is grails — and I wanted to use its modularity to separate out the tests to isolate them as I finish them up before moving into the target project.
Running a few tests, I noted the same failure — a missing property. I dug into the classes the tests were calling and couldn’t find any reference to (in this case) log property.
Below is my querying of Q:
Basically Q had made an assumption about the project setup, didn’t even check the class for reference of an annotation, but still made the test.
I’ve noticed this kind of behavior especially when I give Q a large task (create 50 test cases) vs small (create 2 test cases). There might be something in the algorithm to save time, credits, whatever and you get incorrect extrapolations causing errors in the code.
This exact reason is why a lot of my current team will not use Q on legacy code: it might do a refactor if you aren’t paying attention, it might produce bogus code, or might make a code assumption.
The amount of time to figure out and correct AI issues in code can easily exceed just writing thecodeitself.
If any more proof needed that real coders are different than AI code generator, this is SonarQube’s current price list:
Note that there are AI code options: detect AI code, AI code assurance, and AI code fixing. The code assurance has me doubtful in that popup description, but I’ll test it at some point. I think part of the detection happens when commits are marked by say a GitHub Copilot or Q generator, which would be easier than detecting from code.
But the point being — the difference between human generated code and AI generated code is being recognized by the industry.
Also, kind of chuckling — if you’ve ever used IntelliJ’s markers @formatter:off/@formatter:on (usually for SQL in my experience), we’ll probably start seeing more markups to indicate areas of legacy, human, and AI code areas.
