When AI forgets your key application security controls...
Blink and you’ll miss it - your AI pair programmer just erased a critical web app security control.
I discovered this shocking omission late in a coding session, all thanks to context window overflow causing LLM-amnesia.
It’s not solely about forgetting; sometimes the LLM makes an unrequested tweak to Feature Y while you’re focused on Feature X (but both changes are in the same diff block).
Here’s my streamlined approach to keep your AI assistant from developing selective amnesia:
TODO mastery: Have the AI generate and maintain a TODO file for refactors. It brings sanity and predictability to the chaos. Here’s a real example, fully generated and maintained by Claude Sonnet:
[ ] Update Flask app logging
[ ] Ensure all necessary events are captured in Gunicorn logs
[ ] Remove any direct logging to SQLite
[ ] Implement new logging for app-specific events
[ ] Newsletter Signups
[ ] Search Queries
[ ] Post Views
[ ] Comment Submissions
[ ] Cache Hit/Miss
[ ] Error Events
[ ] Application Startup/Shutdown
[ ] Gunicorn Worker Events
[ ] Update database schema and migrations
[ ] Remove stats-related tables
[ ] Create new migration file for schema changes
[ ] Update requirements.txt
[ ] Add statsd library
[ ] Remove any unnecessary dependencies
[ ] Write tests for new StatsD integration and logging
Divide and conquer: Break changes into smaller, manageable tasks (see the above list!). It’s harder for the AI to “forget” when working on bite-sized pieces, plus it’s easier for you to double-check its homework.
Constant vigilance: Regularly review changes, especially when switching between features. Don’t trust, verify. If you’re a regular reader, you’ll have noticed this is a recurring theme!
This method has dramatically improved my confidence levels when juggling AI generated code diffs. Plus, it’s been a forcing function to level up my git history interrogation skills - an unexpected but welcome benefit (I can now fake it as a pro-coder ;-)).
Remember, the goal isn’t to constrain your AI assistant, but to create a workflow where its power enhances rather than undermines your code integrity.
Struggling to get started with AI pair programming? What’s your biggest hurdle in taking that first step?
Cheers, Craig