Part 1 covered memory: a second brain that keeps portfolio reasoning somewhere trustworthy. Part 2 covered execution: surfaced context, a planned week, and one source of truth for shipping. This post closes the series with continuous improvement: how the system gets better with evidence, and why most of that work turns out to be rejection rather than adoption.
Executive summary
This year I shipped a performance fix on memeinstants.com and believed the work was done. The first round of checks agreed with me. The second round, run across more devices and more pages, did not: mobile was still failing, and the same defect was live in other parts of the codebase. I had improved one instance of the problem, not the system.
That gap, between changing an instance and changing the system, is what this essay is about. Continuous improvement here is not only a meta loop on how I work, in the same spirit as an agile retrospective; it is also an ethos applied to specific products and efforts, to the portfolio as a whole, and to relationships. I now measure the state of a product before a deployment and again after it, so every change carries its own before-and-after evidence. I trial tools against the workflow they want to replace, and I reject most of them. I also run production checks on a schedule, so health does not depend on a complaint arriving first. Part 1 described a system that remembers. Part 2 described a system that executes. This one describes a system that learns: not by accumulating tools, but by running experiments against the current workflow and keeping only what wins.
Business insights
Working this way this year has changed a few beliefs I would have defended confidently when I left corporate life.
- Continuous improvement applies at more than one altitude: as a meta loop on ways of working (much as a retrospective does in agile practice), as an ethos inside specific products and efforts, across the portfolio those efforts form, and in relationships where the current state can also be re-measured and improved.
- A shipped change is a hypothesis, not a result; the second measurement is what decides whether the hypothesis held.
- Commits and product changes remain the main source of regressions I catch, but the world also moves around systems: dependencies change, data drifts underneath the application, and time itself expires certificates, ages caches, and invalidates assumptions. Systems can regress simply because the world around them has changed.
- A healthy improvement process rejects most of what it tries, especially once the baseline is already in decent shape. If you rarely see failure, you are probably playing too safe.
- AI multiplies the need for this discipline rather than removing it. It executes local intent brilliantly and misses systemic patterns, so the evidence loop is what makes that speed usable.
- The output of these loops is not velocity. It is confidence: that a deployment improved the system, that an agent did the right thing, and that a rejected experiment can be revisited later with evidence instead of memory.
The fix that only fixed one place
On memeinstants.com, the marketing site for Meme Instants (a consumer app with about 1.3M monthly active users), page performance affects whether search engines treat the site as healthy. Google scores that through Core Web Vitals: a small set of metrics for how quickly a page becomes usable, how stable the layout stays while it loads, and how soon it responds to interaction. I spent time improving those scores, pointed an AI assistant at the slow code, got a sensible fix, watched the checks go green, and moved on.
The second pass told a different story. On mobile devices the scores were still poor, and the same implementation pattern existed elsewhere in the repository, untouched. The assistant had optimised exactly what I pointed it at. It had not recognised that the fix belonged to a pattern, or that the pattern repeated across the codebase.
I have come to think of AI as a spray gun. It can paint close up or cover a wider area, but the useful question is how closely you stand to the canvas, and whether you have named the target with enough specificity that the paint lands where you intend. Getting paint on the canvas is easy; almost anyone can achieve that. The more valuable the system already is, and the more complex the existing state of the canvas, the greater the risk of a scattergun approach with the wrong context, because you start painting over things you did not mean to touch. Effective use still depends on a deep understanding of where you are standing and what already exists.
That reframed what shipping means to me. Deployment is not the finish line; it is the point where the interesting question becomes answerable. Did I change the system, or did I change one place the system happened to be visible?
Measuring the delta, not just the outcome
The practical consequence is a habit I now apply to every project: the same checks before a deployment and again after it.
Integration tests answer a narrow question, whether the current build passes, which is not the same as knowing what a deployment changed. Before I ship, checks establish the baseline: what works now, what is already broken, and what the scores are. After I ship, the same checks run again, and the comparison between the two is the deliverable. Confidence comes from that delta, not from a green tick on the second run alone.
This matters more than it used to because I develop with AI assistance at a pace that would have been reckless five years ago, and confidence is the limiting resource in that way of working. The pre and post pair also catches something a single test run never can: defects that were already present before I started, which would otherwise be blamed on my change.
Commit hashes surfaced on every deployed version tighten the same discipline. When the version string is visible on the live surface, and the pre-deployment checks, post-deployment checks, and scheduled monitors for a project all write into one shared log, regression attribution can collapse into a single place. Against a known commit pair and a known window of time, the gap between what broke and when it broke becomes much smaller than a feeling reconstructed afterwards.
The sources of regression above are also why the loop keeps running in weeks when I change nothing. The clearest public proof in my stack is the SEO audit loop on Meme Instants: baseline audit, ship fixes, re-run the same audit, and compare evidence rather than impressions. Anyone can run an audit once. Improvement is the second pass.
Most experiments should lose
The other half of continuous improvement is deciding what gets into the system in the first place, and here my conclusion is blunt: the process is working when most experiments fail.
Every experiment answers one question. Is this better than what I already do? Not whether it is interesting, widely discussed, or promising, but whether it is better, measurably, today. The bar is the current workflow, and the current workflow is harder to beat than it looks, because it has already survived this filter many times. Sometimes an experiment passes every functional requirement and still loses, because the non-functional costs (uncertainty, operational risk, performance feel, cognitive load) make it worse than the status quo. Those costs often arrive first as a feeling rather than as a clean argument, until you dig far enough to name why the new way of working or the new tool still failed the comparison.
A recent week from my log, with verdicts:
- An agent automation platform I wanted to run my triggers through. An hour of friction before anything worked: repository recognition, branch discovery, tool configuration. I got it working, and rejected it anyway. The friction told me it would not be stable enough to carry my whole workflow, and it gave me too narrow a window into what it was doing. The result is written down; I can revisit when the product matures.
- Cloud coding agents linked to my project tracker. These earned a narrow acceptance for mature, well-structured repositories where the agent can produce artefacts that prove its work. For fast work where I steer as I go, local agents in parallel sessions beat them, and the review overhead of remote pull requests erased the gains. I kept that one narrow case and put the rest aside.
- A self-hosted workflow tool and, later, local language models as the engine for scheduled jobs. The hosting cost was defensible; the output quality against frontier models was not. Rejected, with one useful residue: I now know I have a worse but working fallback if I ever lose access to frontier models.
Each of these ends the same way: a bounded try, a measurement, a written verdict, and usually a return to where I started. I think of it as gradient descent on my own workflow. Optimise until progress plateaus, then deliberately step out in some direction, measure, and either adopt or walk back. With the state of the art moving this quickly, it is always plausible that my setup is suboptimal, so the stepping-out never stops. The discipline is in the walking back. I judge experiments by what they replace, not by what they demonstrate.
There is a related pressure on the system that holds those verdicts. My second brain stores ways of working, experiment outcomes, and the current best methods, and left alone it will bloat until the useful context is hard to retrieve in the right cases. I am not particularly worried about raw scale yet: running that vault through an editor with repository-wide embeddings keeps tens of thousands of lines inside a comfortable small-to-medium working set. What I do worry about is clarity of retrieval. So I compact and compress with a vault janitor skill, and I keep turning stable ways of working into portable skills the agents can follow. Skills are one of the best ways I have found to codify a verdict so it survives the next busy fortnight.
Watching production the way a person would
Part 2 ended with a reliable pipeline for shipping work. What that pipeline could not tell me was whether the things I had already shipped were still healthy, and "no errors in the logs" is a weak answer to that question.
Being able to stand up a web service that runs exactly the checks I care about, talks to the systems I need, and twists into whatever shape a given product requires, used to be an expensive capital project. That setup cost has fallen hard. The more interesting constraint now is operations: keeping the thing reliable, performant, cheap enough to run, and easy to operate once the novelty of building it has worn off.
So a large part of continuous improvement, for me, is taking reliability back through effort on that operating side. The checks cover front-end services, back-end services, and the systems behind them, hosted apart from the things they watch so an outage does not take the witness down with it. They answer what a person would actually look for, not only whether a port responds: is the data flowing, is the output sane, is the thing doing its job. When something fails, I want the reason, not only a red light. The cadence is time-based, because the sources of regression above do not wait for my next deployment, and the results are surfaced in a form my agents can read, so investigation starts from the aggregated state of production rather than from my description of it. The loops are what make the pace sustainable, not a tax on it.
Tuning instead of maximising
There is a version of continuous improvement that becomes its own pathology: endless experimentation, every hour optimised, exploration as the work itself. I have tried to land somewhere quieter.
Improvement starts before any experiment, with what enters the system. Ingesting new ideas with an open mind, capturing them, and acting on a small fraction is the intake for everything else; without it the loop eventually stagnates, however good the measurement is. That is the same capture habit Part 1 was built on, pointed at methodology instead of tasks.
But intake needs a boundary too. I am not particularly extroverted, and I noticed that networking only produces anything if I do enough of it for luck to have somewhere to land, and little enough that it does not consume the week. Some optimal window, not a maximum. I now try to carry that shape into most things: enough experimentation that better workflows can emerge, enough automation that confidence increases rather than decreases, enough space for something unexpected to occur without forcing it. I have become less interested in maximising activities and more interested in tuning them.
Where learning finishes
The last question continuous improvement has to answer is where a lesson ends up. Understanding something is not the finish; a lesson that lives only in my head is one busy fortnight away from being lost, and Part 1 explained why I stopped trusting that arrangement.
Learning, for me, is complete when it changes how the system operates. In practice each lesson lands in one of four places. It becomes a note in my second brain, durable and findable. It becomes writing like this essay, where explaining it in public tests the reasoning. It becomes a skill or a workflow, a runbook the agent follows, so future me does not need to remember it because the system already embodies it. Or, hardest of all, it changes my behaviour: how I decide, communicate, prioritise, and judge quality.
Those four homes close the series loop. Memory preserves the learning, execution applies it, and continuous improvement captures the result and feeds it back into memory: observe, capture, think, ship, measure, learn, codify, and repeat.
Closing
Part 1 built a system that remembers. Part 2 built a system that executes. This part built a system that learns, mostly by measuring twice and saying no.
If a system is already efficient, you rarely need theatrical change. The better pattern is consistent improvement of around ten percent each cycle, each year: early enough adoption of the right things that you do not leave a large amount on the table, without chasing enormous deltas that inject volatility and then quietly reduce efficacy. Continuous improvement, for me, is less about spectacular swings than about compounding small, durable gains and lowering the noise around them.
Every lesson still has to end up somewhere permanent: a note in the second brain, an article like this one, a deployment checklist, a skill, or a different decision the next time the same problem appears. If learning does not change the system, it remains an interesting thought. The goal was never a better collection of tools. It was a system that gets a little better each cycle because every lesson leaves a permanent trace.
Agent skills
Codifying a verdict as a skill is how lessons survive the next busy fortnight. Skills are short runbooks an agent can follow: how to take in capture, how to keep memory compressible, how to mine for insight, and how to force Accept or Reject on the next experiment. Kept as portable files, those ways of working travel with the agent instead of depending on whatever happened to be in context that week.
The skills from this series are collected on rowanarcher.com/writing/skills/:
If you want them adapted to your stack, get in touch.
Series complete. More at rowanarcher.com. Follow Medium for future craft posts, including one on a deliberately uncomfortable tool that watches where my hours actually go.
Previous: Part 1 - A Second Brain That Survives a Solo Founder Week · Part 2 - An Execution Layer That Survives a Solo Founder Week · All articles