an m5stick breathwork timer I made without learning much blub about m5stick development
I started a new job in July after a little over a year sabbatical. Following is a some rambling about how working as a software engineer has changed for me over the last year.
Maybe not surprising, but it’s possible to move a lot faster in a new environment with agentic help than it previously was.
There was an interesting echo of my first job in this one; in both cases I ended up working on a cost efficiency effort in my first couple of weeks.
At my first job, we stored builds in S3 and never deleted them. This was obviously Bad and Expensive, but deleting old builds was not trivial because there was some security edict floating around about never deleting a build which had been deployed to production. 1
It took me probably 10 weeks and being paired up with a new grad who turned out to be one of the most brilliant engineers I’ve ever worked with in order to come up with a solution. I had to learn the ins and outs of S3 to turn a hand wavey proposal into an actual viable plan of how to do this on paper, utilizing CloudTrail logs, S3 object tagging and lifecycle policies. But then there was the issue of how to do this with billions of rows of data.
It was only by a stroke of luck doing some random scut work that I’d learned about Spark, which solved the issue of making the plan work at scale. But then I had to learn Scala and Spark up to the level required to write an actual two-step pipeline. I spent days hand-verifying every little step of this, dry running the jobs and confirming they behaved exactly the way I’d expect. Actually turning on the pipeline was nerve wracking too.
At my new job I was asked to find some infrastructure cost-savings. Actually I was asked to downsize a chonker of an Elasticache instance, but that proved to be quite challenging for various reasons and, good little paperclip maximizer that I am, I just pointed agents at our codebase and AWS Cost Explorer and started ripping through things I found to generate more savings.
Compared to the first effort which required arriving at a pretty deep handle on how S3 works and enough Scala/Spark/Airflow/other things to even begin to solve the problem, this time I was doing surgery on EKS/Karpenter, MongoDB, EBS volumes within a few days.
I got to a similar amount of savings but in much less time and across a much wider range of infra.
Which brings me to the topic of blub studies. Ben Kuhn’s In defense of blub studies is a wonderful essay from the pre-LLM era that argues you should learn about the fiddly bits and minor details of the code and tooling you use (“blub”) because that compounds into being incredibly effective at a higher level.
It’d be interesting to hear a followup from Ben, who I believe works at Anthropic now. But the essay has been bouncing around in my head for a while because it occurs to me that with the rise of agents, I may have stopped accreting blub knowledge.
In large unfamiliar codebase I’m hacking on with agents I’ve found my blub knowledge comes substantially in handy. A lot of my discussions with agents simply looks like:
Having a lot of useEffects here feels gross, isn’t there a better way to do it?
or
Shouldn’t we use a Terraform module for this?
And that yields good results. But here, I’m leaning on my frozen-in-time blub knowledge, the kind of stuff I got by spending weekends reading AWS S3 docs. I’m fairly deep on certain topics but I may not always be a full-stacky web/API oriented developer, and even there things may move on (is React getting a little long in the tooth at this point?).
On personal projects, I deploy agents fairly guiltlessly to avoid having to learn even a little bit more about how C/C++ build systems work (good riddance).
I’m not sure what the solution is here. Do I keep trad coding on the side to keep my chops up? Do I trust that I’ll still pick up blub filtered through the haze of agentic coding? It may require occasionally going slower to go faster.
Maybe a further step function change in AI will obviate the need for any of this. But for now I still find my blub knowledge valuable, and so will try to figure out how to keep gaining it.
One lovely thing about a small workplace full of talented and thoughtful people has been that I haven’t encountered the walls of slop I was afraid of when re-entering the working world. They still exist in some places (e.g. when pull requests are agent-generated), but it’s considered rude to just hand someone one of these and expect them to actually read and engage with it.
Someone summed this up with a “no meat proxy” document suggesting that any AI-generated walls of text should be clearly marked as such, and things we expect others to read should generally be written by hand first, which I wholeheartedly agree with.
I’ve been trying to implement this myself by always writing an explanation of my own on anything agent-generated. This is useful for me because it confirms my understanding of what’s going on. On PRs I always include at least a small handwritten description of what the PR is doing, and if it involves a larger change this might be a longer explanation of how I arrived at what it does, tested it and so on. I have a personal goal of never speaking in the agentic passive when it comes to a change I’ve made.
I’ve also found it useful to keep writing little one-pager documents or Linear issues for larger changes. These aren’t going to be read by many people if at all, but doing so also crystallizes an understanding of what’s happening with my changes and what might go wrong with them in my mind which previously I arrived at just by coding on the problem for hours/days.
As promised some disjoint thoughts on working with agents with other people. Making software is still a lot of fun, that hasn’t changed.