10,000 Foot Developer

Here’s a good use case as to why developers are needed to monitor code generated by AI.

Using Amazon Q, I set up a project that is a 100% unit test project to be 100% AI generated, for another barely tested project. (More about this later; was not a smooth creation.) This is grails — and I wanted to use its modularity to separate out the tests to isolate them as I finish them up before moving into the target project.

Running a few tests, I noted the same failure — a missing property. I dug into the classes the tests were calling and couldn’t find any reference to (in this case) log property.

Below is my querying of Q:

Basically Q had made an assumption about the project setup, didn’t even check the class for reference of an annotation, but still made the test.

I’ve noticed this kind of behavior especially when I give Q a large task (create 50 test cases) vs small (create 2 test cases). There might be something in the algorithm to save time, credits, whatever and you get incorrect extrapolations causing errors in the code.

This exact reason is why a lot of my current team will not use Q on legacy code: it might do a refactor if you aren’t paying attention, it might produce bogus code, or might make a code assumption.

The amount of time to figure out and correct AI issues in code can easily exceed just writing the code itself.

If any more proof needed that real coders are different than AI code generator, this is SonarQube’s current price list:

Note that there are AI code options: detect AI code, AI code assurance, and AI code fixing. The code assurance has me doubtful in that popup description, but I’ll test it at some point. I think part of the detection happens when commits are marked by say a GitHub Copilot or Q generator, which would be easier than detecting from code.

But the point being — the difference between human generated code and AI generated code is being recognized by the industry.

Also, kind of chuckling — if you’ve ever used IntelliJ’s markers @formatter:off/@formatter:on (usually for SQL in my experience), we’ll probably start seeing more markups to indicate areas of legacy, human, and AI code areas.

AI often misfires the more I work with it.

I think the big issue is, you cannot forget that you continually need to keep AI guided and boxed into your task, or you will get hallucinations and errors.

Here’s an example of an agent example using Gemini.  It asks for a complex math question, and I asked it for 42 factorial using the proper 

There’s an option to “fix the error” — which I ask it to do. Then we get this:

A fixed result. But why didn’t it have a fixed result in the first place? This kind of thing is frustrating — and in a complex coding environment could be very troublesome, with generated bugs not discovered until production deployment.

Have had two recent experiences with general AI querying I found interesting.

1. I was asking Co-pilot about a famous documentary. There also happened to be a court case with like names to the documentary. Co-pilot *only* told me about the case and not the documentary. It also said I was wrong when I tried to query further. This one I finished on Gemini.
2. Was also asking the AI for Duck Duck Go (the search assist) about a documentary series of an American touring Japanese food — ramen, sushi, tea house etc. The AI only told me of some other documentaries, denying that there was even such a thing with an American. Well the series exists “Pride Japan: Nihon no kokoroni deau.” I finished this one on Grok.

The interesting part of this wasn’t that it couldn’t find the items, it’s that AI told me I was wrong, or denied there even being anything. Also, recently, Claude was getting out of hand calling some of my input “delusional utterances.”

During all this I’ve asked these applications to not use my name, nor use human style interaction. A lot of bias can enter this way, imo.

Here’s some extra reading that is interesting on AI wrongness as well.

https://itrevolution.com/articles/when-ai-cuts-corners-hijacking-the-reward-function/

Also — maybe you should ask several AIs at once. Well, funny enough, I was wataching a Rick Beato guitar video on YouTube, he mentioned an app that can ask several at once. (Honestly, I was about to fabricate a javascript page to do just this). Then I found out about another multi-AI -asker. Here they are:

https://openrouter.ai/

https://www.maxai.co/features/web-ai-chat/

They can cost some money, but worth a try.

Here’ an issue I’ve run into repeatedly: AI having old versions of APIs for services, in this case JIRA, who recently updated theirs and didn’t leave any backwards compatibility (and also hosed my dev data lake).

I have a python script I AI’d out of Claude to extract JIRA data for a BERT model training. But, when running, get the 410 error for the JIRA endpoint:

I am also using Claude in VS Code for this, so having it fix it. Run the update and it works….

This AI coding requires a lot of practice to get better with it, still generating this script even with the api error saved some time.

Let’s talk about using AI in this current age of 2025.  Here’s a stock photo to help out:

As obvious as it sounds, writing prompts for an AI chat session is a lot like golf.  Basically, honing down until you get a good answer OR run out of chat memory and have to move to a new session.

1. Use the driver off the tee.
   -Post your first prompt: “I’d like all the recent JIRA tickets that have work logged on them.”
2. On the fairway.  Now you select an iron to get you close to the hole.
   -“I mean the tickets that have worked logged in the last two weeks.”
3. Now you are on the green.  You decide to putt.
   -“For team members Sally and Frank.”

Sometimes though, AI completely misses it, for instance step two (real scenario with Rovo, Atlassians’s new AI assistant) – “Here you go, a JQL that lists all the tickets with time on them” — but since getting work time requires iteration through the worklog api data, AI didn’t do that correctly and just gave you tickets with recent activity that had work logged at any time, even three months ago.  In this case 2A., need that pitching wedge to get onto the tee.

I’ve been working with Claude, Gemini, Copilot, ChatGPT, Ollama, and a lot of minor AI llms — they all do this as you can imagine, some just get you there faster.  I am finding that smaller less known models need more time to get what you need.  For instance, Junie in IntelliJ.  Or some of the GitHub AI things, and definitely Rovo for Atlassian are less mature.

A big thing is learning how to summarize a session so as to continue on again at some point — have AI summarize, or save the chat context ported off for reloading in a new session.

This is just an open thought/notes post — using agents specifications files in an AI development environment.

These kind of files are written in Markdown, and can do some of these kind of things:

1. Project documentation
   • Anything from testing, code narration, code structure, dependencies that an AI tool can read to assist in code generation or help.
2. Structured input to LLMs
   • Agent guidance — i.e. infor and context about your development environment, project settings

There’s an emerging standard, open source, called AGENTS.md. Here’s their first page sample:

# AGENTS.md

## Setup commands
- Install deps: `pnpm install`
- Start dev server: `pnpm dev`
- Run tests: `pnpm test`

## Code style
- TypeScript strict mode
- Single quotes, no semicolons
- Use functional patterns where possible

These can become very complicated. I’ve ran some tests developing a desktop image environment for VM Ware, and have specified a ton of applications, OS, etc. in these files.

Here is some of my experiment. It’s quite easy to generate such a file, and I was asking it to create a packer instance for VMWare”

# DVM Virtual Dev Environment

## Project Overview
This project creates a Ubuntu 24.04.3 desktop virtual machine using HashiCorp Packer and VMware. The VM is pre-configured with development tools including Visual Studio Code and IntelliJ IDEA Community Edition.

## Purpose
- Build standardized development environments
- Automate VM provisioning with consistent tooling
- Provide reproducible Ubuntu desktop environments for development work

## Key Components

### Packer Configuration (`ubuntu-desktop.pkr.hcl`)
- **Base Image**: Ubuntu 24.04.3 Desktop AMD64 ISO
- **Platform**: VMware (vmware-iso builder)
- **Resources**: 4GB RAM, 2 CPUs, 40GB disk
- **Authentication**: SSH with ubuntu/ubuntu credentials

### Provisioning
The build process installs:

**Development Tools:**
- Microsoft Visual Studio Code
- IntelliJ IDEA Community Edition
- Docker Desktop
- Postman
- MySQL Workbench Community

But also — this remembers a lot of context.

Amazon has the same ides — rules for their “spec driven development” approach. Will probably discuss later as I have questioned them about the lifecycle of biz rules in a development cycle, but, generally stores all the info of your dev environment for your project. Generally, is stored in .amazonq/rules, though can be in a file like amazonq.rules.

Also Anthropic has the CLAUDE.md file, which is like AGENTS. It’s loaded at startup to give your agent context, among other things. There is also use of a json file for settings.

From CLAUDE site:

Key points about the configuration system

• Memory files (CLAUDE.md): Contain instructions and context that Claude loads at startup
• Settings files (JSON): Configure permissions, environment variables, and tool behavior
• Slash commands: Custom commands that can be invoked during a session with /command-name
• MCP servers: Extend Claude Code with additional tools and integrations
• Precedence: Higher-level configurations (Enterprise) override lower-level ones (User/Project)
• Inheritance: Settings are merged, with more specific settings adding to or overriding broader ones

​

Note that these other AI agents will in fact read AGENTS.md, even if they put up a fight against you. Also, be aware that in general you’ll eventually be checking in and maintaining these files for your projects.

Thoughts. Well, after working with these, it’s nice to have a context on loadup mechanism. It would be nice to share among other developers. That said, I wonder how this is going to play into the specs in maven, ivy, ant, gradle etc. files. Why not just read those files, instead of in some cases having a double entry?

And since this is now emerging, there are several standards — am guessing if you ever switch AI providers it could convert the standards (from say Q rules to CLAUDE) with no fuss. It had better, being that conversions seems to be one of the AI strongpoints.

Another really nice related mechism is experimenting with saving chat contexts from the IDE. Amazon has something like this, and Claude can remember as well — but you could technically store prompt history offline, even incorporate the best answers into the md files.

Best thing is to try all of this out.

References:

MARKDOWN

AGENTS.md

CLAUDE Settings

Amazon Q Rules

Further Reading:

There are mentions of Jupyter notebooks in much of the MD config documentation. I didn’t discuss this here, but you can of course explore it more.

JUPYTER