Six years into federal contracting, the pace is slow, the paycheck is good, and none of it was building toward the AI work I actually want to be doing. So a few weeks ago I started a side project: an autonomous agent that watches my AWS account, decides for itself which tools to call, and answers questions about cost and infrastructure. This is the first in a planned weekly series documenting the build — bugs, bad decisions, and small wins included.
The account I'm testing against is essentially empty. Free tier, no production traffic, a couple of S3 buckets. Which turned out to be the perfect environment for the agent to have its first real incident.
The Agent's First Incident Report
I asked it a simple question: "What's driving my AWS spend this week?" It called two tools, cross-referenced the results, and came back with a strikingly formal breakdown of exactly one service trending upward. Here's the summary, presented with the gravity it clearly felt it deserved:
The joke writes itself, but the underlying finding is real and, honestly, kind of great: the biggest line item in my account most weeks isn't S3, isn't EC2 — it's AWS Cost Explorer itself. Every call to GetCostAndUsage costs a flat $0.01, and my agent calls it constantly while I stress-test it. Which means the act of investigating my AWS spend is, itself, my AWS spend. I did not expect the agent's first genuinely useful insight to be a joke about its own existence, but here we are.
The agent's first real insight was that asking questions about your AWS bill is, itself, a line item on your AWS bill.
The Credential Bug That Ate an Afternoon
Not everything was this charming. Early on, every single API call — CLI, boto3, didn't matter — failed with the same error: InvalidClientTokenId: The security token included in the request is invalid. Credentials looked right. Environment variables were clean. aws sts get-caller-identity even worked from one terminal and not another, which is exactly the kind of inconsistency that eats an entire afternoon before you find the actual cause.
The actual cause: I'd pasted the wrong field out of the IAM console. Not a typo — the wrong ID entirely. Turns out AWS's identifiers all have prefixes that quietly tell you what they are, and I'd grabbed the one that looks the most like an access key without being one:
| Prefix | What it actually is |
|---|---|
| AKIA | A real, long-term access key (what you want) |
| ASIA | A temporary/STS session key |
| AIDA | An IAM user's internal ID — not a credential at all |
| AROA | An IAM role's internal ID — also not a credential |
I had an AIDA sitting in the access-key-id field of my credentials file. Every layer of the stack correctly rejected it, every error message correctly said "invalid," and it still took an embarrassing amount of debugging — checking stale environment variables, restarting Jupyter kernels, re-typing the secret twice — before I actually looked closely enough at the string itself to notice it was the wrong kind of thing, not just the wrong value.
A Bug That Actually Mattered
The more useful bug showed up once the agent was working. I asked it the same underlying question two different ways. First: "Is anything running that I forgot about?" — it checked EC2, found nothing, and said so plainly: no instances running. Then: "Are my EC2 costs matching the number of instances actually running?" — same tool, same empty result, same account. But this time it hedged, speculating vaguely about "billing configuration issues" instead of just saying zero equals zero.
Same data. Two different levels of confidence. That's not a data bug or a code bug — I confirmed the tool returned the identical empty result both times. It's a synthesis problem: when the agent had to reconcile two different tool results in one turn instead of reading one cleanly, it got less certain about a fact it already had. The fix ended up being a single line in the system prompt telling it that a service absent from a cost breakdown means zero cost, not missing data. But finding that took building an actual eval log and treating "the agent sounds unsure" as a bug worth root-causing, rather than something to shrug off because the account has no real stakes in it.
That's the part I didn't expect going in: even a completely inconsequential AWS account — free tier, twelve cents a week, nothing that matters — surfaces the exact same reliability questions that a production agent has to answer. The stakes are fake. The failure modes are not.
What's Next
Phase 1 (the agent choosing between multiple tools on its own) is done and logged. Next up is memory — giving the agent a vector store of past incidents so it can answer "has this happened before," instead of investigating every question from zero. I'm aiming for about ten hours a week on this on top of a full-time job, and so far that pace is holding. More next week, incidents permitting.