Hacker News Personal Blogs 2025 | Top 100 Blogs

I canceled my book deal

Austin Z. Henley's Blog — Wed, 31 Dec 2025 22:00:01 GMT

https://austinhenley.com/blog/canceledbookdeal.html

2025 Year in Review

Ben Overmyer's Site — Wed, 31 Dec 2025 02:00:00 GMT

Well then. 2025 did not go as originally planned. Not even close.

A year of challenges

We attempted to foster a puppy in January. She was a cute little rescue pit mix named Clover. Unfortunately, I discovered that I'm allergic to dogs. I don't know if this is new or if it's just something I never noticed before. While it seems to be a relatively mild allergy, combined with other factors, it meant we had to stop fostering. Luckily, there was another foster family able to take Clover right away.

The next event of consequence was Trump taking office. We knew it was going to be bad, but even my worst fears paled compared to what he did. In response, we decided I would attempt a different route to immigration. I started applying to as many universities in Canada and Estonia as I could.

In March, one of them accepted my application: Memorial University of Newfoundland. So, we immediately began the planning process to move to Newfoundland. In April, we flew up to St. John's to explore the area and see if we could find a rental that would accept cats. We were shocked to discover that rentals mostly disallow pets, with no possibility of pet rent or anything like that. However, we got lucky, and found a three-bedroom house whose landlord loves cats. So, we signed the lease and started paying for two places at once. While it meant the new house would stand empty for several months before we could move in, we guaranteed that we'd have a place.

It was around this time that I completed the most complex painting I've ever done. It was a painting of my sister's eleven cats, and originally meant as a Christmas gift. Still, a nice achievement, and she ended up loving it.

The painting of Tasha's cats

For the next few months, we watched as Trump ran roughshod over all the rules and rights that normally guide the government and began systematically destroying the United States. Then it came time to give notice at our jobs and prepare for the two-week drive it would take to get to our new home in St. John's, Newfoundland.

The trek northward

That drive took us to some amazing places with great views. Under other circumstances, it would have been a wonderful experience. However, I had our two cats and a ton of possessions crammed into my Kia Seltos, and the cats hated that drive. I will never put them through that again.

Mojave and Kalahari a few days before we took the trip

Still, we got to our destination safely, with everyone and everything intact. We moved into our new place in mid August. The UHaul U-Box arrived more or less on time, and we got everything that we'd brought with us sorted out. The majority of our stuff is still back in Wilmington, North Carolina, though, in a big storage unit. We can't bring all that up until we get permanent resident status. That won't happen until after I've graduated and we've both spent a year living and working here.

In order to retain access to our US bank accounts and a few other services, we have to retain our US phone numbers. We're using US Mobile for that purpose; it's about $15 USD per month for the two lines on the lowest service level. We have new Canadian numbers at the same time. Surprisingly, it was easy to get Canadian bank accounts and credit cards. Sarah was able to get a remote job to pay the bills, and for the first semester of university, I even got an internship that paid a reasonable amount of money.

Starting grad school

My first semester was stressful. I had to reacclimate to the university life, and between three classes, an entrepreneurship program, and my internship, I was extremely busy. One of my classes, Applied Algorithms, was really difficult for me. After a ton of studying, consulting my dad, and working with ChatGPT (and, later, Gemini) to understand the concepts, I managed to finish the class with a B, but it was a struggle. If you get less than a B in a class at the graduate level, you fail the class and have to retake it. If you fail two classes (or the same class twice) this way, then you're kicked out of the program. If you're an international student, that immediately invalidates your study permit, and you are required to leave the country. So, you can understand why I was stressed.

In the end, though, it worked out. I got As or Bs in all of my classes, and passed the entrepreneurship program. I finished my internship successfully, with an invitation to come back if I ever want to.

So, about those planned new habits

With regards to the new habits I originally intended to set this year, the events of 2025 completely derailed everything. I couldn't bring my guitar with me, so the guitar habit died around about May. I was improving quickly, though, and apparently I can borrow a guitar from the local library here in St. John's, so maybe I'll pick that back up now.

The painting habit never took, though I have painted more this year than any other year since art school. I'm improving at that, too, and I now have a basic painting setup in the new place. There's promise here for 2026.

Alcohol... well, quitting alcohol didn't happen. While I didn't drink as much this year as previous years, I still drank way more than I planned to. This will be another thing I address in 2026.

The daily walking habit never formed. However, I did start walking more in St. John's compared to Wilmington, since there are a number of amazing trails here that are really close to our house. It should be easy to establish a walking habit going forward. I can even walk to campus in about an hour and fifteen minutes, which is ambitious but doable.

One of the trails near our house

Books read

My boss Dwayne at Apiture introduced me to the novel series Dungeon Crawler Carl early in the year, and I adored it. I read everything that was available, and for Christmas I got the latest book in the series, so I intend to start on that early next year.

I also got into the Dune novels after playing the survival MMO Dune: Awakening, and read up to Chapterhouse: Dune, which is the last Dune novel written by Frank Herbert. They were a strange and unusual series of novels. I have no idea how they could make a third Dune movie in the current series without touching on some very weird concepts from the books.

Total books read this year: 20

Music discovered

A little over a month ago, I started a new habit of discovering and buying a brand new album every Friday. I wanted to expand the collection of music that I actually own, as opposed to just streaming everything. My record player is still in storage in North Carolina along with all of my records, so I'm just buying digital albums. Still, it's led me to discover some fascinating new music that I otherwise would never have heard.

I've discovered that I enjoy thrash metal and black metal. One unusual standout was Bloodywood, a nu metal band from New Delhi with Hindu and English lyrics. They even did a collaboration with Babymetal, a unique Japanese metal group that I used to follow years ago.

Hmm. Maybe I should start an "albums discovered" page in the same vein as my "books read" page.

Yoga progress

Another unexpected development this year is that I started getting into yoga. I start most mornings now with a basic routine that takes about 5 minutes. My skill level is still "absolute beginner," though. There are classes here in St. John's, but I don't have a lot of spare money for such things. I enjoy the practice, and use yoga workouts on Apple Fitness+ to learn new asanas.

I am interested in the other aspects of yoga, too, and not just the stretching. The world view is worth exploring.

In closing

That's some of our adventures from 2025. I skipped mentioning a few things, but this post is already quite long enough. 2025 was a year of big changes. One thing is certain: no one will ever say 2025 was boring.

Previous Years

I'm sick and tired of AI coders

ブラック — Wed, 31 Dec 2025 02:00:00 GMT

This has been a weird year for me, with the biggest thing being that I suddenly found myself without a job. This was a bit of a new experience for me, since I hadn’t been unemployed ever since I graduated. Had to fill all kinds of government paperwork about it. Not exactly the happiest of circumstances to find yourself in, but I came into it from a much better starting position than many of my fellow layabouts, so I’ll refrain from crying about it too much.

Now I’m back in the workforce again, still making computers do things for us. Those things are just a bit different, and I’m doing them in a different environment, but I didn’t suddenly become a bricklayer or anything. I’m still even a remote employee, so I didn’t even have to go into the office like many of my peers did. The biggest difference though? I am now in a company where my coworkers, or at least the most vocal of them, have taken the AI pill – hard. And I really don’t care for it.

I’m a proponent of the idea that you get better at things by doing them. That’s why I have a goal of playing my guitar for at least 15 minutes every day, no matter what. I’m currently ~1700 days into this experiment, so that should have given me 425+ hours to improve my playing. According to Malcolm Gladwell, that’s over 4% of mastery. It’s also part of why I started a blog, since even if nobody were to read anything that I wrote here, I’d at least give myself a chance to improve my writing. Not that I write here that often, but I feel like it has still helped.

This is one of the reasons why I’m skeptical of "agentic coding": I don’t think I will become a better developer by letting an AI model write code for me. And I’m almost certain that I won’t become a better developer by doing it like some of my coworkers are doing it. Some of them do not seem to do any kind of validation on the code beyond the fact that the unit tests (which the AI wrote) pass, the integration tests (which the AI wrote) work, and that the documentation (which the AI wrote) is accurate (as evaluated by the AI).

Supposedly the easiest way of catching students using AI to cheat on their homework is simply by asking them questions about their work. The chances are that they will not be able to actually tell you what is inside whatever it is that they’ve just turned in. I can’t vouch for the veracity of this method, but I have noticed that the people who are using agents to code don’t really seem to know what’s inside the commits they’re shipping out. Presumably there are teams where AI is used in a responsible manner and people make sure to understand what they are shipping, but at least where I work, velocity seems to come before reason or understanding.

I’ve seen the LinkedIn post of the tech executive praising their AI-native chief vibist, who is pushing out 250,000+ lines of code every month. That comes out to around 12,000 lines of code generated each working day. Obviously I can’t write 12,000 lines of code in a single day, but I don’t think I can review 12,000 lines of code in a day either. Is the price of our glorious AI-generated future that we just don’t look at the code that we ship?

It really boggles my mind how some people can put so much trust in these language models. I don’t even trust my own code this much; I always go through my own code again before I submit it to be reviewed by another person. Either I’m the odd one here for double-checking my work, or LLMs have blinded developers into thinking they’re near-infallible 10x coders and reviewing their changes serves only to unnecessarily slow down your business. But from my experience working with a codebase that has a significant amount of AI-generated code in it, I think mistrust is the right call.

The general quality of our AI-generated code is quite poor. Claude does very little in terms of adhering to DRY, it’s writing CRUD tests against completely barren databases, and if a test doesn’t pass, it might solve the issue not by changing the core business logic, but rather by changing the test until it passes. And I can't even really get into the truly important specifics, like for example what Claude is doing with access control. We probably wouldn't pass a security audit even if the audit was done by Claude itself.

It feels like every day I discover a new problem that can be described as "developer prompted the ticket, didn’t check it, didn’t run it". There’s only so much code that I’ve had time to look at, or run, so I don’t even know how far the rot has spread. Most of these are in a greenfield project that has yet to land on the monitors of end-users, so the customer experience thankfully hasn’t been impacted. And most importantly, I have never had to interact with a customer, so even if they were interacting with all of that vibe, they couldn’t associate my face with it. If I had to meet with the clients, I'd prefer to hide my face with a paper bag or something.

Unfortunately these terrors are often committed by the same AI evangelists that are always telling us to "not trust the AI blindly" in our many agentic coding meetings, with Claude's --dangerously-skip-permissions always enabled. Why have I even had to attend multiple of these meetings anyway? Surely vibe coding hasn’t changed that much in a month that I needed a refresher course.

The non-coding parts are also not that great in my opinion. We’re churning out so much AI-generated documentation that there’s zero chance that anyone with a beating heart will ever read them. Probably why the text reads like it’s written for other robots. I think the only place where most of the text is written by humans at this point is Slack, and it also has a bunch of AI transcriptions too.

Jira tickets are also hallucinated by LLMs. You know what’s a right laugh? Getting assigned a third-party integration ticket where all of the details on the external service have been made up by a GPU. Someone just had an AI fill out a whole document out of a single-line "Allow fetching data from [CRM service]". The only usable part was the name of the service. None of the use cases or API endpoints existed. It's fairy dust; it doesn't exist, it's not fucking real.

I'm not even that opposed to maintaining documentation, but I did try to use Claude to update the CHANGELOG.md file on my behalf. Unfortunately it managed to generate excruciatingly detailed technical descriptions of all of the changes, so I just deleted all of it, and rewrote it by hand. Hopefully this way the next developer can actually read through the changelog and not ask their AI agent to summarise the changes for them.

I’ve also noticed that trying to ask Claude about our codebases is also not that great. While debugging a problem that Claude had sidestepped by rigging the tests, I asked Claude to identify if we were validating the data we’re entering into our database. Claude then told me that we don’t have database constraints to validate anything, but we’re checking them in the code before it hits the database. I then looked and there was a function to validate the data, but absolutely nothing called it. And when I unrigged the tests, broken and unvalidated data was entered into the database. Shocking. Thankfully I had already determined the root cause before asking Claude, so I wasn’t led astray and only wasted some time looking up that unused code.

The single best use case for Claude that I've seen so far is code reviews. Before, I'd first review my changes myself, then I'd push it to remote and ask someone else to review it for me. Now I can first review it myself, then ask Claude what it thinks of my changes, and only then waste another human's time on it. The reason why I like it is that it has good upsides and minor downsides; if it notices something important, great, I can change it, and if it gets an absolutely abhorrent idea for a change, I can just ignore it.

And since LLMs can understand natural language and code, I can ask one to check if my documentation is technically accurate, which I can then fix myself. As much as I'm shitting on AI, I do actually find this useful. Although not useful enough that I'd actually pay Anthropic for a subscription, and I presume the current prices are heavily subsidized.

As far as the coding tasks that I entrust Claude, I'm currently keeping it for advanced copy-paste tasks only. By this, I mean that I might write five unit tests for a single model by hand, and then ask it to write tests for another model based on those previous handwritten tests. The way I'd have normally gone about it would've been to copy-paste the original tests anyway, so Claude can do it a bit smarter. Unfortunately, even when I've given it an explicit template for what it needs to write, it still has managed to conjure other test cases from outside my specific example, so even this use case is far from perfect.

I think code reviews also become quite interesting in terms of development team dynamics. I’ve seen pull requests co-authored by Claude with thousands upon thousands of lines of changes, and I don’t know if I can bring myself to review them. How much of my day am I going to devote to reviewing this change? And will I end up spending more time looking at this code than the guy who supposedly co-authored it with Claude?

I mean, I’ve definitely had to submit large changesets for review too, but at least I was not asking other people to work harder on reviewing the code than I worked writing it, since those large pull requests took real effort. And I definitely couldn’t open several of them per week.

So far I’ve been flying under the radar, trying to only snipe the smaller code reviews, and leaving the ones with tens of thousands of lines to other developers. I did start reading one larger change too, but someone approved it before I could really get going. Judging by how fast my coworkers get them approved, I can only assume that they’re checking the changes as rigorously as they are checking ~~their own~~ Claude’s work.

Unfortunately, I think AI has come to stay when it comes to software development. No idea what kind of a place the industry has for me in the future. The Internet keeps telling me that I am one of the ones who will be replaced. But I just can't imagine getting up in the morning and have the motivation to just prompt a model all day long. Personally, I’ll try to keep writing code by hand as long as I can, just like I write documentation and blog posts by hand, and enjoy the process and struggle.

I imagine my time at my current place of work is going to be relatively short. My tenure is measured in months, and I'm already pretty sick and tired of the culture. My feelings towards work are so bad that I'd probably sleep better if my employment was terminated tomorrow. Presumably for shipping code too slowly or something, although I don't think anyone has really made a note of how organic my code still is. The only thing that keeps me going is receiving a steady salary, and I've still considered just outright quitting before finding my next job.

The lesson learned from the current job? Figure out how any prospective employer is using AI before continuing, as nobody brought it up before my hiring, and I only discovered having joined a slophouse when I was already in it, and got told that I am expected to code using AI. Probably would've been better for both parties if they'd just told me ahead of time.

Year in community

Notes on software development — Wed, 31 Dec 2025 02:00:00 GMT

This year I ran three book club readings over email with 1,230 unique attendees. I ran 12 coffee club meetups in midtown Manhattan with 170 unique attendees. Angelo and I ran 6 NYC Systems meetups with 12 different speakers and 281 unique attendees. I took 3 visiting PhD students out for Banh Mi.

I raised $6,915 for educational non-profits, offering chats in return. I got coffee, lunch, or took 30 minute calls with 55 people I'd never spoken to before in person or on video. (Most, but not all, were in return for fundraising receipts.) This list included women and men based in the USA, Germany, Canada, Nigeria, Nepal, India, the United Kingdom, Brazil, New Zealand, Israel, and Australia. (I think I'm forgetting one or two.)

Thank you to every person who has been a part of these efforts, making them so special and so valuable. See you in the new year!

Year in community pic.twitter.com/n7jrmsZiKN
— Phil Eaton (@eatonphil) December 31, 2025

What Ilya Sutskever, Satya Nadella, and Sam Altman think about AI progress in 2025

Bytepawn - Marton Trencseni — Wed, 31 Dec 2025 01:00:00 GMT

This article contrasts how Ilya Sutskever, Satya Nadella, and Sam Altman think about AI progress, from research and generalization, through infrastructure and economics, to agents, scaling, and societal impact.

The Pro-life Position Leads to Dark Places

Rondam Ramblings — Tue, 30 Dec 2025 20:21:12 GMT

One of the things I did this year was to engage in one of the most ridiculously long Reddit discussion threads ever, and that is really saying something. That link doesn't even go to the beginning of the thread because the discussion chain was broken by a post that was deleted by my interlocutor, who goes by the pseudonym "uniformist". The discussion branched and ebbed and

Linux Dev / Productivity Environment on Windows with WSL 2 and Docker

From Development to Production on Nick Janetakis — Tue, 30 Dec 2025 02:00:00 GMT

We'll balance using Linux through WSL 2 as well as run native Windows apps to get stuff done related to software development.

If you care about security you might want to move the iPhone Camera app

John Graham-Cumming's blog — Mon, 29 Dec 2025 13:59:06 GMT

There's a quirk in the iPhone Camera app that can drive a security conscious iPhone owner crazy: if you touch your finger on the Camera app icon without actually opening the app, the camera starts operating. That causes the little green dot indicator on the iPhone to turn on and then after a few seconds disappear. This can easily make a security-conscious iPhone user worry that some app is nefariously using the camera.

Because touching the Camera app icon without opening the app is enough, it's easy to activate the green dot while swiping between screens or even just holding your phone. This was driving me nuts and I thought some app was using the camera when I didn't want it to.

I enabled Apple's App Privacy Report (under Privacy and Security) in Settings and, sure enough, it confirmed that the little green dot was happening because the camera was being used by the Camera app. Merely swiping while accidentally touching the Camera app's icon was enough to add an entry in the App Privacy Report.

The solution was moving Camera app away from where I was liable to brush it with my fingers or thumb. That reduced the errant green dots to almost zero and reassured me that nothing untoward was happening.

One of the big problems in security is people get used to odd behaviour they can't explain and end up overlooking a real problem masked be the things that "always happen". The accidental green dots could have been hiding actual nefarious use of the camera. Better to be tidy and eliminate the false positives.

(I previously used the term "hover" for what I was experiencing and that was causing confusion. This happens if you touch the Camera app icon without actually opening the app.)

Some flexibility with Go’s sumdb

ENOSUCHBLOG — Mon, 29 Dec 2025 02:00:00 GMT

I noticed this a year or two ago, but forgot to write it up back then.

My Vibe Coded Projects in 2025

Beetle Space — Sun, 28 Dec 2025 10:00:00 GMT

This was my first year using AI coding tools. I’m often asked what I use AI coding for outside of work. Here I document what I’ve built with AI.

When I say “AI coding”, I’m excluding asking a chatbot for code and copy/pasting it into a …

A Proclamation Regarding the Restoration of the Dash

Beetle Space — Fri, 26 Dec 2025 10:00:00 GMT

WHEREAS, the em-dash (—) has long served as the elegant scaffolding of the English sentence, providing the necessary breadth for parenthetical thought, sudden turns of phrase, and rhythmic pause;

WHEREAS, a modern and unfounded prejudice has arisen, wherein the presence of the em-dash is viewed with suspicion and cited as the …

doubly dual shuffles

Tony Finch's blog — Fri, 26 Dec 2025 01:45:02 GMT

Here’s a pearlescent winter holiday gift for you!

There are four variants of the algorithm for shuffling an array, arising from two independent choices:

whether to swap elements in the higher or lower parts of the array
whether the boundary between the parts moves upwards or downwards

The variants are perfectly symmetrical, but they work in two fundamentally different ways: sampling or permutation.

The most common variant is Richard Durstenfeld’s shuffle algorithm, which moves the boundary downwards and swaps elements in the lower part of the array. Knuth describes it in TAOCP vol. 2 sect. 3.4.2; TAOCP doesn’t discuss the other variants.

(Obeying Stigler’s law, it is often called a “Fisher-Yates” shuffle, but their pre-computer algorithm is arguably different from the modern algorithm.)

the four variants

In the pseudocode below, min and max are the inclusive bounds on the array to be shuffled; the arguments to rand() are the inclusive bounds on its return value; and the loop bounds are inclusive too. I chose this style to make the symmetries more obvious.

In all variants, it’s possible for the indexes this (the boundary between the parts of the array) and that (chosen at random) to be the same, in which case the swap is a no-op.

I could have written the loop bounds as min and max instead of min+1 and max-1 to make the variants look as similar as possible, but it’s more realistic to omit the loop iterations when this and that are guaranteed to be equal.

It should be clear that rand() is invoked for spans of each size between 2 and N (where N = max - min + 1) so the algorithms produce N! possible permutations as expected.

boundary moves down, pick from lower

shuffle(a, min, max)
    for this = max to min+1 step -1
        that = rand(min, this)
        swap a[this] and a[that]

boundary moves up, pick from higher

shuffle(a, min, max)
    for this = min to max-1 step +1
        that = rand(this, max)
        swap a[this] and a[that]

boundary moves down, pick from higher

shuffle(a, min, max)
    for this = max-1 to min step -1
        that = rand(this, max)
        swap a[this] and a[that]

boundary moves up, pick from lower

shuffle(a, min, max)
    for this = min+1 to max step +1
        that = rand(min, this)
        swap a[this] and a[that]

the two kinds of shuffle

To merge four variants into two kinds, I’ll introduce some names that allow me to describe them independently of the direction of iteration. The todo part of the array is ahead of the iterator, i.e. lower if we are moving the boundary downwards, higher if we are moving it upwards. The done part of the array is behind the iterator, vice versa.

Earlier I called this the boundary between the parts, but that was a deliberate fencepost error which allowed me to gloss over an important difference. In fact the boundary is next to this, and which side it’s on depends on the kind of shuffle.

shuffle by sampling

The first two variants above work by sampling.

In this kind of shuffle, the indexes this and that are on the todo side of the boundary. We pick an element at random from the todo part of the array, and swap it into position next to the boundary. Then the boundary is moved so that the picked element is appended to the done part of the array.

The loop invariant or inductive hypothesis is that the done part of the array is a random sample of all the elements from the original array.

shuffle by permutation

The last two variants above work by permutation.

In this kind of shuffle, the indexes this and that are on the done side of the boundary. First we move the boundary to append an element from todo to done. Then we pick a random index in done, and swap the element next to the boundary into that position.

The loop invariant or inductive hypothesis is that the done part of the array is a random permutation of the elements that were originally in the same prefix or suffix of the array.

coda

I was inspired to write this by “The Fisher-Yates shuffle is backward” and the examples that were added to Wikipedia’s article on the Fisher-Yates shuffle a few months ago.

The article by “Possibly Wrong” correctly points out that there are two kinds of shuffle, but it misses out one of the four variants. Wikipedia’s examples show all four variants but they are cluttered by conventional array indexing, and it doesn’t explain there are actually two kinds of shuffle.

There’s a classic off-by-one mistake that turns a shuffle into Sattolo’s algorithm for cyclic permutations. It happens when the random bounds are too narrow, so that is never equal to this, and an element can never land in the same place that it started. This change has the same effect on both kinds of shuffle.

I enjoyed revisiting this old classic and trying to beautify it. I hope you like the results!

I made another little bedside clock

Stavros' Stuff — Thu, 25 Dec 2025 20:04:52 GMT

I call it 'Bedtime'

The other day I saw a video by someone who bought a $16 electronic clock, and it looked interesting, because that little clock had been taunting me for months by showing up constantly in my AliExpress recommendations. I held off on buying it because what am I going to do with yet another clock, but the video said that it has an ESP8266 inside, and that, with a little soldering and programming, you could run ESPhome on it!

Obviously, I didn’t need any more convincing, though I remembered this clock being listed for $6 for a while, and balked at the $16 the video mentioned. I ordered a different listing, which I found for $12, hoping it would be the same as the one from the video.

Fairly serendipitously for this purchase, the last bedside clock I made was showing its age a bit. Mainly, the dimmest setting on its screen was bright enough to be annoying when I’m in bed, and the screen has been burned-in quite a bit. A new monochrome OLED screen is an easy fix, but I’d prefer a color one (especially if it can be dimmed more), so hopefully I’d be able to replace the Do Not Be Alarmed clock with this new one.

The new clock arrived promptly, and I

The Best Things and Stuff of 2025

Send More Paramedics — Tue, 23 Dec 2025 10:17:49 GMT

Whereby I talk about my favorite media/people/ideas that I encountered for the first time in 2025.

New 3d Printer, A new BITX* Release, and it’s almost 2026 already!

MiscDotGeek — Tue, 23 Dec 2025 09:05:53 GMT

Hi. Remember Me? I’m the guy that hasn’t blogged in all of 2025. Yeah, that guy. 2025 has been a huge year. I now own my own home. I have a shed to tear down, another shed to move. I fixed up one of my trucks. I bought a newer (2012) car. I got health …

The post New 3d Printer, A new BITX* Release, and it’s almost 2026 already! appeared first on MiscDotGeek.

Fix Open Terminal Failed Error with tmux

From Development to Production on Nick Janetakis — Tue, 23 Dec 2025 02:00:00 GMT

It's often caused by version incompatiblities from version upgrades.

Year in books

Notes on software development — Tue, 23 Dec 2025 02:00:00 GMT

Among the 50 books I read in 2025, I recommend the following 11 non-fiction and 7 fiction works (complete list here). These were the 18 books that I rated a four or five out of five stars.

Non-fiction

On Writing Well by William Zinsser

This is the third or fourth time I've read this book and it has stood the test of time. It's been a few years since I last read it so it was a good reminder that a lot of the things I believe and tell people about writing actually just came from this book. The last 25% is a bit of a slog but nonetheless it remains one of the single books I think every professional should read. 5/5

I. Asimov by Isaac Asimov

I really like reading about how writers make their living. I've also been a modest fan of Asimov's works (I've loved what I've read I just haven't read that much). I also love to hear stories of first-generation immigrants to the US and also he lived in New York his whole life so it was quite enjoyable. 4/5

Hollywood by Jeanine Basinger and Sam Wasson

This was 100 years of the evolution of the film industry told basically entirely in disparate interviews edited together. 5/5

The Chief by David Nasaw

I love the history and business of newspapers and media. Moreover it's the guy that Citizen Kane was based on. 4/5

Brazil by Lilia Moritz Schwarcz and Heloisa Murgel Starling

I have never learned about the history of Brazil and I found this introduction enjoyable. 4/5

The Silk Roads by Peter Frankopan

I loved the retelling of human history focused on Persia (and later Iran). 4/5

Louis D. Brandeis by Melvin I. Urofsky

I have never read about a supreme court justice before. This was a well-written biography and introduction to the history of law and law education. 4/5

Personal History by Katharine Graham

Once again I love reading about newspapers and media and the business and history. This was told by the publisher of The Washington Post. 5/5

The Snowball by Alice Schroeder

I've had this book about Warren Buffett on my shelf for nearly 10 years and finally went through it this year. A delightful and easy read despite the bulk. I only am unhappy that it focused more on family drama than on business decisions. Par for the course with biographies unfortunately. 4/5

The Cartiers by Francesca Cartier Brickell

This story spanned three or four major wars and a couple of continents. I didn't think I'd be interested in the history of luxury businesses but it has a lot in common with certain modern industries in tech too. You put premiums on relationships and building good faith and so on. 4/5

Secret Formula by Frederick Allen

A history of Coca-Cola over the last 100 years or so. Lessons on how they dealt with competition (Pepsi and Keurig Dr. Pepper) and product revitalization (New Coke, Diet Coke, etc.). Quite an interesting read. 5/5

Fiction

The Buffalo Hunter Hunter by Stephen Graham Jones

I got into horror fiction last year (not so much slashers but more just one of the better written categories of genre fiction). This book was one of my two favorite novels of the year.

It's a fictional retelling of American history where a Native American becomes a vampire and takes revenge on American colonizers in the American West. 5/5

The Handmaid's Tale by Margaret Atwood

This was my other favorite novel of the year. I am embarrassed not to have read it before. It's a dystopian story about the USA if all women were required to give birth to deal with a fertility crisis. 5/5

The Invisible Life of Addie Larue by V.E. Schwab

A French woman gets to live forever but everyone she meets forgets her after leaving her presence. An easy and enjoyable read. 4/5

Vampires of El Norte by Isabel Cañas

I love a good vampire story, and I love fictional retellings using fantastical horror elements to emphasize atrocities. Vampires employed by the US military help the US in the 1840s take Texas from Mexico. 4/5

The Very Secret Society of Irregular Witches by Sangu Mandanna

This was a cute cozy mystery about British witches forced to hide from society, learning how to accept themselves and develop trust in their community. 4/5

Pride and Prejudice by Jane Austen

I am likewise embarrassed I have not read this before, nor anything else by Austen. I'm told I haven't rated it highly enough. I will undoubtedly reread it. I loved the wit. It required closer reading than I expected. 4/5

James by Percival Everett

This was a retelling of Adventures of Huckleberry Finn as told by the slave, Jim. If you saw the movie American Fiction a few years ago, it's the same author (of the original book). Everett has very interesting ideas and I look forward to reading more by him. 4/5

My 2025 year in books. 18 to recommend among the 50 I read. pic.twitter.com/mIcbPk7e5x
— Phil Eaton (@eatonphil) December 24, 2025

Micro Review: Money for Nothing by Thomas Levenson

print "Me" — Tue, 16 Dec 2025 15:51:19 GMT

I listened to this last summer, but it took me a long time to write something so here it is: the book talks about the first great financial crisis, the South Sea bubble. The actual full title is “Money for Nothing: The Scientists, Fraudsters, and Corrupt Politicians Who Reinvented Money, Panicked a Nation, and Made … Continue reading "Micro Review: Money for Nothing by Thomas Levenson"

The post <span class='p-name'>Micro Review: Money for Nothing by Thomas Levenson</span> appeared first on print "Me".

Assign an AHK Global Hotkey to Send Text or Copy It to Your Clipboard

From Development to Production on Nick Janetakis — Tue, 16 Dec 2025 02:00:00 GMT

This could be very handy if you find yourself wanting to send a specific phrase to an application from time to time.

What does ChatGPT think about Viktor Orban's 15 years as Hungary's leader?

Bytepawn - Marton Trencseni — Mon, 15 Dec 2025 01:00:00 GMT

I asked ChatGPT to summarize the top 10 positives and negatives of Orban Viktor's 15 years as Hungary's leader.

PaxosLease: Extending and Releasing Leases

Bytepawn - Marton Trencseni — Mon, 15 Dec 2025 01:00:00 GMT

This article extends the basic PaxosLease algorithm with practical lease renewal and explicit release semantics, showing a full working Python/Flask implementation and explaining why the protocol remains safe without disks or synchronized clocks.

Dependency cooldowns, redux

ENOSUCHBLOG — Sat, 13 Dec 2025 02:00:00 GMT

Three weeks ago I wrote about how we should all be using dependency cooldowns.

Implementing PaxosLease in Python with HTTP Flask

Bytepawn - Marton Trencseni — Sat, 13 Dec 2025 01:00:00 GMT

This article introduces PaxosLease, a simplified Paxos variant that uses expiring leases instead of persistent values to enable fast, reliable master election in distributed systems.

I'm moving to Berlin

Nemo's Home — Fri, 12 Dec 2025 02:00:00 GMT

Haven’t posted here in a while, but this is worth an update: I’m moving to Berlin. I’ve loved living in Bangalore for the last decade (I moved here just after the first HillHacks in May 2015), but it is time for an adventure.

For the last couple of years, I’ve spent my time living in Indiranagar, building communities, helping underline.center, working on blr.today, organizing events, and really doing things that I care about. It has been a wonderful time, but we wanted to experience life elsewhere, and Berlin seems to fit the bill.

Bangalore’s infrastructure has been in the news, and it is a part of why I’m moving. An unwalkable footpath on CMH road meant we spent half a year dealing with a broken elbow this year. At Takshashila, I was taught that I’m not allowed to bring up civic problems without also coming up with solutions. But Bangalore is an unsolvable paradox: A city with 50+ unicorns and no walkable footpaths.

I’ve fought my personal share of battles against the state, but this is one where I don’t have any hope of making a difference. I appreciate the work that the Bangalore Civil Society is doing in attempting to hold the missing-government accountable - it just isn’t the kind of work that I want to do. Fighting for basic necessities (clean air, walkable footpaths, open public spaces, well-funded public transit) shouldn't be anyone's job in a city as large as Bangalore. I’m picking Exit (for now).

Why Berlin? Mainly because I have lots of friends there. Our immigration journey is still quite early, so maybe I can write about it when things have stabilized a bit. For now, if you have any Berlin recommendations or connections, please send them my way.

FreeBSD 15.0 post-upgrade reboot loop

The NeoSmart Files — Tue, 09 Dec 2025 21:34:30 GMT

This post is less of a deep dive into a bug I ran into upgrading an x86_64 machine from FreeBSD 14.3 to FreeBSD 15 and more of a PSA: I have a possible workaround for anyone that runs into the … Continue reading →

The post FreeBSD 15.0 post-upgrade reboot loop first appeared on The NeoSmart Files.

Why Is Video Editing So Bad on Linux Compared to Windows with Camtasia?

From Development to Production on Nick Janetakis — Tue, 09 Dec 2025 02:00:00 GMT

I recently tried Kdenlive, Shotcut, DaVinci Resolve, Blender and Lightworks on Linux. They're not even close to Camtasia.

Manuel was annoyed

Annoying Technology — Mon, 08 Dec 2025 20:49:11 GMT

All three of the files on top have been created/exported/transferred in the exact same way: Exported from the recently discarded (severely underrated, btw) Clips app, then sent (all in one transfer) via AirDrop. How on earth is it possible for them to have different cases on the file extension?!

The Bird-Poker Deck

Send More Paramedics — Mon, 08 Dec 2025 10:07:49 GMT

Whereby I talk about turning a common American Checkers set into an "arcade" for deep games of abstract strategy.

New Search Filtering in Web and API

Weblog on marginalia.nu — Mon, 08 Dec 2025 02:00:00 GMT

The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI.

This was originally going to be an incredibly chaotic update, both annuncing the new features and doing a technical walkthrough of the changes but that ambition turned out a bit too chaotic, so let’s split them up and focus on the feature announcement bit today.

New Search Filtering GUI

It’s now possible to define a custom filter in the GUI, on the marginalia-search.com version of the website!

Manuel was annoyed

Annoying Technology — Sun, 07 Dec 2025 21:31:00 GMT

Ogres Websites are like onions, they have layers! Thanks for the submission, Private Sender said: Is this what is meant by the Content Mines? We have to dig through layers of cruft to get to what we’re looking for?

How To Podcast

swyx's site RSS Feed — Sun, 07 Dec 2025 07:31:41 GMT

Through my 4 (!) podcasts I obviously have built up a lot of opinions on podcasting over the years. Here's some of them. The two outlier podcasts of our time are Dwarkesh and TBPN, and I will explain my mental model of them in a separate post - basically the McClusky Curve applied. the following advice is what I strongly believe for the "average" podcast that wants to do well but not necessarily being full time top tier influencer level. Also, this is primarily for professional/technical interview guest-led podcasts, but I also do have opinions on non-guest-led podcasts as well.

Multi-Paxos: Building a Simple Replicated Log in Python

Bytepawn - Marton Trencseni — Sun, 07 Dec 2025 01:00:00 GMT

This post builds a minimal Multi-Paxos system in Python with Flask HTTP endpoints, showing how repeated Paxos rounds form a fault-tolerant replicated log.

Paxos: Lamport's Crown Jewel for Replicated State Machines

Bytepawn - Marton Trencseni — Thu, 04 Dec 2025 01:00:00 GMT

A straightforward explanation of Paxos using a simple Python Flask HTTP implementation to show how distributed systems achieve consensus in the presence of faults.

So you want to move from backend code to robotics systems

jodavaho.io — Tue, 02 Dec 2025 02:00:00 GMT

I got a really good question sent to me, and I’d like to archive the response here…

I mostly use Go (previously Java) and am exploring Rust for unmanned systems, particularly embedded work and autonomy like navigation and perception. Given your domain expertise, I’d love to hear your perspective:

I’m bullish on Rust long term. If you’re coming from Go/Java, then you’ve already got the “backend” covered, and a lot of robotics happens in “backend”-like code. So, rust here is more of a curiosity / lateral move for you. Great to try, not required. You could start right now by developing implementations of robotics-like things and making a backend.

Very little “solving” happens on small machines, it happens on on-system servers running in docker, usually. The small computers do little things, usually. So, you can definitely get started with pathfinding, optimization, mapping, image processing in Rust right now, and build small automous systems that mostly live in something you’d call a “backend”. Then plug a webcam in, and hit your “backend” with imagery and you’re off to the races.

If you’re more interested in “robotics systems”, then there’s really no substitute for learning ROS. While ROS is a bit dated, and most serious shops built their own version, ROS2.0 has a bunch of normal practices in it, and every custom robotics framework is defacto compared to ROS2.0 nowadays. This implies C++ and CMake, it’s really hard to get around that. Though there are Rust wrappers and ROS2.0 Rust compliant messaging libraries.

If you’re more interested in “on the metal” code, then ROS2.0 is still relevant, but we’re more into the firmware level things. C is the king here (still). Firmware generally does very little more than manage custom hardware like motors, low level sensors, etc. It often doesn’t do the “thinking”.

So if you’re looking to get into robotics, C and C++ are still the industry breadwinner.

Rust is a great language for this domain, but the libraries and industry adoption is lagging a little. There’s just no serious reason to rewrite your entire stack in Rust at this time, when robotics products are not very mature yet. I definitely have seen production Rust code that flies in warzones, so it does exist, but it’s commonly an outlier.

One huge benefit of Rust that is often overlooked is the immediate compatability of types, build systems, and dependency management across the firmware, system, and backend. This is what will eventually cause Rust to catch up to C++, but it may never replace it and doesn’t need to.

Using sed to Print Specific Lines in a File

From Development to Production on Nick Janetakis — Tue, 02 Dec 2025 02:00:00 GMT

This could be handy when a stack trace in a large file points to a specific line, now you can see those lines quickly.

Langjam Gamejam: Build a programming language and then use it to make a game in 7 days

Austin Z. Henley's Blog — Fri, 28 Nov 2025 22:00:01 GMT

https://austinhenley.com/blog/langjamgamejam.html

The Quiet Earth

Marc Stuff — Thu, 27 Nov 2025 02:42:10 GMT

Above: A still image from “The Quiet Earth” (1985)

Lately it’s been dark here. A dark time of year, a dark political era…hell, I am afraid I could just go on and on about darkness!

Everybody’s got their ways of coping with this, whether that’s done by the automatic / instinctive side of the personality, or in a more conscious way.

On the conscious side, one thing that reliably helps me out in dark times is engaging with the personal hobbies and interests that seem to call out to me for attention.

Personal Interests are Kinda Lit

This topic by itself has become a huge interest of mine over the years. I like to study the way our personal interests can guide us through life in a mysteriously helpful way. They often do this through symbolic language.

Our personal interests tend to bring light into the dark corners of our lives, if given proper attention. And even if they don’t seem directly related to “resolving issues in global politics” or “dealing with that one coworker”, they often do relate to those things, somehow. I’m often surprised to find just how direct the connection actually is!

People usually talk about “engagement with personal interests” as something that disconnects us from the broader world, for good or bad…but my experiences also seem to point out that this is quite the opposite in a lot of ways.

I believe there is a strong, symbiotic relationship between those two realms: Out there, and in here.

Getting Gripped (by an Interest)

Related, one of the techniques that’s seized me lately works a lot like this:

Get excited about something. Get sucked into it! (Wow, this movie, or wow, this book! Or hobby, or whatever it is…)
Identify that new world and its energy, as a specific type of view on things, or a specific approach to things. More on this below.
Live through that world—think about your life using the tools and viewpoints from that world.

Some of this process comes from my training in Jungian-oriented topics, and some from various corners of personality theory.

But, this past weekend, one of those excitements snuck up and gripped me something good. And I wanted to share how that worked out, mostly to review, but also in case it can help somebody else.

Roll Tape

This time, it was The Quiet Earth, an ’80s movie that grabbed me once again, and sucked me in. It’s definitely one of my favorite films.

Independent, plucky, unique, and deep, it’s also everything that doesn’t describe me on the average lazy Sunday.

In the movie, the main character wakes up to find that he’s the only person left alive. On the entire planet. This is the result of a scientific experiment in which he took part: Project Flashlight.

It soon becomes clear that this person bears some responsibility for ending up completely isolated, maybe even for killing almost everyone on the planet.

There is a tremendous moral burden on this character, but he’s also just barely realizing that he’s part of a broader, selfish, shortsighted, killing system…and now…

…what to do about it?

This Kind of Topic Helps Me Out Sometimes

Why this movie? Why this approach?

Personally, I didn’t pick this interest. But if I had to guess why my subconscious mind found it interesting: It often really helps me to look at the world from this kind of viewpoint: “Oh no, what’s happened? Things are so weird and disconcerting lately. And what can I do about it?”

Solving novel problems is one way to look at this, and that is one of the skills I have worked hard on in my professional practice in recent years. So, that connection makes sense to me.

Here are some more specific examples why this helps:

First, I notice that I get more energy if I decide that stuff is really weird lately. lol. It’s more fun to look at the world this way, for me. It opens up the imagination and anyway, my mind has always been naturally attuned to single out “something weird that’s going on.”

Second: “Maybe I had a part in it, but let’s find out.” I’m a character-motivated person. I like to think about my role in things. There’s some personal-discovery and maybe also a personal redemption arc here, as is usually true for all of us. One of us, is all of us.

Within each of us is every philosophical perspective, doing some sort of battle against another perspective from time to time. Is it better to be Simple with this project, or a bit more fancy? Is it better to be a pragmatic person, or more sensitive to broader issues? Looking back, or looking forward? Focused on the now, or lost in thought?

When one of those perspectives wins, another perfectly good perspective tends to lose. And that can be really, really unfortunate, because they’re also a full palette of helpful tools and techniques.

Third, I feel kind of alone in dealing with the problems at hand…but that’s also a strength. I mean, there’s always some clear need to link together, if possible, to solve huge problems in society. But there’s also a separate strength in looking at problems that only seem to affect us.

(Eventually, maybe we can even publish the problems in a way that helps us link up with others, so whether this “I’m alone in dealing with this” is a good thing is also an open, process-related question and a matter to sort out over time.)

And finally, I get the idea that I can probably make some sense of the problem at hand, but I need to watch closely and make observations. To activate some sensitivity, this powerful tool we can use to analyze and tackle problems.

How it worked out

Translating a personal interest, or a fantasy world, into personal perspectives and tools seems to be a really helpful practice so far.

And I have to say I love the advantages of the scientific angle:

Patience is rewarded. This is not an action movie. Think through the problem well.
Unique perspectives are welcome. Again, we need so many different tools in life. You never know what can help.
We embrace the fact that dark things can happen. Embracing darkness means that we don’t try to only look on the bright side of everything, because sometimes this is just the same as blinding ourselves. With closed-off perspectives, we close off access to many important problem-solving tools. And darkness is usually just “depth in disguise.”

In the event, I took up my notebook and drew out a pleasant, organized, scientific grid layout.

Then at the top of the paper, I wrote: PROJECT FLASHLIGHT LOG (a reference to the movie), merging the movie theme with my own inner world.

I worked through general descriptions: Something’s off. What will I try now? What do I need to do?

This soon included a little to-do list, on the side. “Drink some water.” Never leave out the mundane stuff!

But also: “Take crucial measurements.”

For me, this is a thing I do—I measure my personal energy, how I’m feeling. It’s an activity I do while I’m journaling.

What Does It Mean?

In the end…I moved ahead in that particular movie-mindset, as the scientist of who-knows-what-is-going-on.

I decided to do some very specific things next, as experiments, and I did those things, or scheduled them.

And on a Sunday, no less! Hah.

I don’t want to overstate this, but it really just helps to move ahead. For most of us, that’s most of what we need on any given day! The feeling of forging ahead in life doesn’t really have to be that fancy.

And the tools we use to move ahead may also seem fancy, but it’s important to realize: Even if they just help us keep going, that’s usually plenty of help, and who’s to say if that’s fancy, if it just works!

Still, everything’s dark, and I hate it, ugh

These are, in many ways, dark times. For all of us.

Tracking that, and managing its effect on our lives, can be very hard. Attention to global issues can easily cascade into dark fallout in our personal lives. This is generally true, the less-conscious we are of a need to find our own way of solving important problems in our lives, even the dark ones.

Circumstances seem dire, and the tools available to us even seem dire sometimes. Even scrolling on a social media site these days feels like flirting with dangerous mental conditions, inviting some new level of anxiety, or depression, or obsession with some disquieting thing in a disconcerting way.

So, in this case, recognizing and seizing a motivating personal interest, or personal fantasy, if I can call it that, is a very clear, straightforward win.

Every global problem is also a personal problem, and movement on either end of that chain is helpful.

(And maybe for another time: What the hell has gone wrong with social media, anyway?)

That’s it for now everybody! May you also find some weird thing in your life that’s really interesting and ideally, helpful. :-) —Marc

Video: The Midnight – The Equaliser (Not Alone)

Festive offers for books on Python, Linux, Regular Expressions, Vim and more!

learnbyexample — Wed, 26 Nov 2025 02:00:00 GMT

Hello!

Here are some awesome deals for programming books and courses during the 2025 festive season.

My ebooks🔗

Festive offers for my ebooks till 30-November-2025:

All 13 Books Bundle — $16 (normal price $36), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
Understanding Python re(gex)? — FREE (normal price $10)

Other deals🔗

Python related deals
The Pragmatic Bookshelf — 50% off
Deals on Django and Git books/software
Huge list of awesome deals — tools, productivity, books, courses, etc
blackfridaydeals.dev — Hottest Black Friday Deals for Developers

Happy learning :)

Is psql's scripting language Turing complete? Or: fibonacci in psql

Notes on software development — Wed, 26 Nov 2025 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Avoid Interview Prep Courses

Agentultra — Mon, 24 Nov 2025 02:00:00 GMT

Musings and ramblings on programming and things

We Who Endure

Digital Apocrypha — Sun, 23 Nov 2025 17:06:14 GMT

A short piece provoked by a recent day spent chopping wood. Link

We should all be using dependency cooldowns

ENOSUCHBLOG — Fri, 21 Nov 2025 02:00:00 GMT

TL;DR: Dependency cooldowns are a free, easy, and incredibly effective way to mitigate the large majority of open source supply chain attacks. More individual projects should apply cooldowns (via tools like Dependabot and Renovate) to their dependencies, and packaging ecosystems should invest in first-class support for cooldowns directly in their package managers.

The Role Bridging Problem

smcleod.net — Wed, 19 Nov 2025 17:10:00 GMT

An observation on functional correctness without domain quality.

Digdoku

Kevin Cox's Blog — Wed, 19 Nov 2025 03:45:00 GMT

I randomly had the idea that it would be interesting to “dig into” all possible Sudoku puzzles. So I created Digdoku. The landing page is an “empty” puzzle, each link leads to a puzzle with one additional square filled. Values that are not possible are excluded. So you can never get an invalid puzzle by clicking links (but you can by editing the URL).

The concept is simple. Basically for every unset cell it attempts to solve the puzzle with every number from 1-9. If any solution is found that value is possible, otherwise it is omitted.

This “eager” removal of impossible values means that it functions as a solver as well. By the time you click all of the provided squares (or often sooner) there will be only one possible grid, and it will be revealed.

However, implementing this simple solution is fairly slow. So some optimizations were necessary. These are primarily around skipping options that are easily determined to be impossible or were already found to be possible. The solver is still mostly a brute-force depth-first solver with some very basic rules put on top (such as eliminating duplicate values in a box/column/row group and if a value can only be in one place as a group it must be that).

The slowest case I am aware of takes about 100ms on my desktop. It could almost certainly be much faster, I suspect another order of magnitude would be fairly easy to achieve. But in the first version some cases took over six minutes, so I’m calling it good enough for now.

One downside is that the URL only keeps minimal state, so a lot of options need to be re-eliminated every step of the way. Maybe compiling to WASM and running an incremental solve client-side would be better. Might be something interesting to try. It would also avoid the risk that my single-core VPS gets overloaded if this gets popular. I do have caching enabled, but I suspect after a few clicks hits will become quite rare.

There is also something about a simple HTML response that I am very fond of. Theoretically you could pre-render every possible grid into a static site. But according to Wikipedia there are 6.6e27 valid Sudoku grids, so I’ll hold off on the pre-render for now.

Manuel was annoyed

Annoying Technology — Tue, 18 Nov 2025 21:07:48 GMT

Our corporate IT policy enforced the Tahoe upgrade on my work Macbook – now I’m getting these warnings when I connect it to my screen. I do so via a single USB-C connection for both display and power, and I’ve got half a dozen other devices (a webcam, my keyboard, the tiny amp for my mic, a Litra Glow, a Stream Deck…) connected through the integrated USB hub. Everything works as before, so why bother me with this?

2025 Nov Summary

jodavaho.io — Mon, 17 Nov 2025 02:00:00 GMT

Life is really coasting along. There are always bumps, but generally great. I spend too much time at home, working and such. Of course I’m not exercising enough. But generally, being here lets me see my kiddos the maximal amount, and also to spend more time on work.

Flight Science has been a blast. We’re now fully automated!. What a journey it’s been.

My kiddos are four. I can’t really justify posting pictures of them online, but watching them swing in the backyard, fall leaves everywhere, sun setting over the water. Pure bliss. Of course one was “swinging” in the traditional sense (on a swingset), and the other was “swinging” as in trying to hit them with a stick, but they both agreed they were swinging and had a good time with the pun.

It won’t always be this good. But it is pretty good. Just need to get out and go jogging / lifting, so if anyone reads this and wants to do accountability buddies or something, you are gonna save my life!

Work/Life milestone: automated messaging

jodavaho.io — Mon, 17 Nov 2025 02:00:00 GMT

Big news! As of last week, we’re sending fully autonomous flight optimizations directly to aircraft for our partner airlines. Huge personal milestone for me, as I worked exclusively on the optimization and estimation pipelines that do this!

Why am I sending messages to cockpit screens?

Flight Science tracks flights for partner airlines, and helps them improve operations. We plug into various and sundry commercial and customer data streams; collate, estimate, and optimize things in real time; generate alerts and recommendations custom built for their objectives; and send them to the cockpit and/or dispatchers to help improve the experience, safety, and efficiency of the flight.

My background here…

I joined in 2024, and when it was just me and Andrew, we banged out the initial planning system over a couple of months. It’s iterated a dozen times at least, getting faster, more precise, and more reliable over time. I’m fond of saying that 8 my 10 favorite systems I’ve worked on were here at Flight Science.

The system has generated and sent messages for almost 1 year, but it involved me sitting at my machine and approving or denying each message. No longer! It’s stable and the recommendations are good enough that we don’t need to do that anymore.

What do we send?

Notable examples are alerts: “Turbulence ahead”, Replan: “Do this to avoid turbulence”, Efficiency: “FL320 would save xxx kg of fuel from improved following wind”, and Logistical: “Flight YYYY is due to land in 13 minutes”

How does it work?

We have precise models for key parts, optimization engines for estimating some things, and excellent heuristics for others. You can imagine we have to track the weight of all passengers, luggage, fuel, etc, the temperature of every cubic meter of air they fly through, the fractional kilogram of fuel they use for every stage of flight, airline/pilots preferred flight profile, like climb rates and speeds, and so on… And, we have to track in real time any changes to their plans - so we can look ahead for them.

Then all this information has to be piped into a real-time planner and quickly provide a batch of alternative actions, which we can score against their preferences and decide to send an alert or plan or whatever. And for each alternative we have to re-calculate projected values for all of that stuff above.

Then all this goes to the cockpit through a global satellite/vhf/hf network, and appears on one of those awesome looking little green screens, which they can ignore if they like.

Like this screen (not our message):

It’s all a suggestion for situational awareness. We do measure acceptance rate by tracking the activity of the airplane to see if they adopt the plan, it’s much higher than I expected, but more on that later.

I’m super excited. None of this would be possible if our team wasn’t already super plugged into the airline industry. We have excellent early stage partners to develop and test all this.

Amazing time.

Philipp was annoyed

Annoying Technology — Sat, 15 Nov 2025 20:33:38 GMT

If only there was a way for a cloud service to do background tasks without relying on the user’s browser to stay open.

I converted a rotary phone into a meeting handset

Stavros' Stuff — Tue, 11 Nov 2025 03:48:23 GMT

The meeting stakes are high when you can get hung up on

As you may remember, or completely not know, I have a bit of a fascination with old rotary phones. Occasionally, when people learn about this fascination, they donate their old rotary phones to me, so I have ended up with a small collection.

The other thing I have a fascination with is meetings. Well, I say “fascination”, but it’s more of a burning hatred, really. One day, a few months ago, I was in one such meeting, as I have been every day since, and I jokingly pretended to get irate about something.

One of my coworkers laughed and said “I bet if this were a phone call, you’d slam the phone down right now”, and a dread spread over me. Why didn’t I have a phone handset I could slam down? Had I really become a corporate husk of my former, carefree self, puppeteered by

The Impossible Triangle of LLM Infra

swyx's site RSS Feed — Thu, 06 Nov 2025 21:47:03 GMT

another talk I am giving at Mastra's TypeScript AI conf today https://docs.google.com/presentation/d/1NnQ3H5Bki3vWRRJdVXoCFJ5dsNKH9QrC-eEQ2Z8olck/edit?usp=sharing

Replacing my old desktop with a high-end Linux PC

Ben E. C. Boyter — Mon, 03 Nov 2025 02:00:00 GMT

I am hoping to get another 5 years out of the machine as it is now.

My previous post about my desktop ended with the above. It has been about 5 years and while I was happy with my old desktop I had started looking into agentic workflows using LLM’s. After burning $50 in an afternoon just debugging some flows against OpenAI I decided I needed a better solution.

A little research and I decided to order and build a new desktop that would solve my immediate problems, as well as last another 10 years. My requirements were fairly simple, as much power as possible in as quiet a form as possible.

An individual can change an organization

Notes on software development — Mon, 03 Nov 2025 02:00:00 GMT

One of the biggest lessons I learned early in my career was from Drew DeVault at Linode, 10 years ago. He was one of the youngest developers in the company (only I was younger, at 20, at the time) but he cared really strongly about thinking through architecture and code decisions when the culture at the time was, and I love those guys, a little haphazard.

Drew had no special position. We all had the same title, "Developer". But he argued so persuasively and so doggedly even when the entire organization seemed against him and somehow he eventually transformed the entire engineering organization.

That's supposed to be impossible! It was entirely new to me. That you don't need to wait behind people with more experience to make the right decision. That you can be part of making the right decision if you can find the logic and the will to do it.

It isn't that simple of course. Politics is politics. But there are plenty of companies with people who will make a good faith effort to do what makes sense but might, without someone's unasked-for effort, do not what makes sense but what is popular because what's popular just kinda seems easiest. And I always like working for these companies, and for the most part have been able to identify them during the interview process.

I learned from Drew to put limited value in seniority. I learned that it's ok to debate. I learned to be prepared and to try to present the facts. I learned to be persistent when I wanted change. I learned that with these skills, it's possible for an individual to redirect the path of an organization.

It took a while longer (and me driving one or two people on my team to quit, to my great regret) to learn when to do these things and when to let things go. Still, this lesson from Drew on what's possible always stands out in my memory. Thank you, Drew.

Drew DeVault taught me a really important lesson early in my career, that it's possible for a person to change the direction of an organization. pic.twitter.com/EGBRdbIjVO
— Phil Eaton (@eatonphil) November 3, 2025

Pelican on a Bike - Raytracer Edition

Beetle Space — Sat, 25 Oct 2025 10:00:00 GMT

Simon Willison has a light-hearted benchmark for evaluating LLMs. He asks them to draw a Pelican riding on a bicycle in SVG format.

I decided to do a twist on this. How good are LLMs at drawing a Pelican on a bicycle using the POV-Ray ray tracer?

My prompt …

how to draw a tetrapod

Tony Finch's blog — Sat, 25 Oct 2025 00:40:51 GMT

Concrete tetrapods are used to dissipate wave energy in coastal defences.

There’s a bit of a craze for making tetrapod-shaped things: recently I’ve seen people making a plush tetrapod and a tetrapod lamp. So I thought it might be fun to model one.

I found a nice way to describe tetrapods that relies on very few arbitrary aesthetic choices.

Click here to play with an animated tetrapod which I made using three.js. (You can see its source code too.)

Click to skip to the recipe

background and reference material

There is a patent for tetrapods written by the inventors of the tetrapod, Pierre Danel and Paul Anglès d’Auriac. It says,

The half angle a at the apex of the cone corresponding to one leg is between 9° and 14° while the leg length base width ratio b/c for each leg is between 0.85 and 1.15 and is preferably substantially 1/1.

One manufacturer has a diagram suggesting a taper of 10°.

A lot of pictures of tetrapods (one, two, three) have chunky chamfers at the ends of the legs. In some cases it looked to me like the angle of the chamfer matches the angle of the edges of the tetrahedron that the tetrapod is constructed from, which is the same as the angle of the faces of the cube that enclose the tetrahedron.

a surprising experiment

So I thought I would try constructing a tetrapod from the corners of a cube inwards.

My eyeball guesstimate sketch produced a result that’s amazingly neat. It’s the main reason I’m writing this blog post!

Using build123d and CQ-editor, I started with a cube centred on the origin enclosing a tetrahedron, with a common vertex at (100,100,100). I drew a pair of circles near the common vertex, just touching the faces of the cube and edges of the tetrahedron. These circles outlined where I thought the edges of the chamfer might be.

Using a taper angle of 10°, I found another circle centred on the origin that would be the base of the leg. I drew truncated cones to make the leg and chamfer, and replicated them to the three other axes of the tetrahedron.

With a bit of fettling of the sizes of the first two circles, I had a very plausible tetrapod!

To validate this sketch, I found the diameter of the circle around the leg near the centre of the cube where it just touches the x,y,z axes, and the distance from that circle to the end of the leg.

To my surprise, the width and length of the leg both came out very close to 100!

It’s good that they are about the same, because that’s what the inventors of the tetrapod specified. That means my eyeball guesstimate got the proportions about right. What’s amazing is that they match half the side length of the cube that I started with!

recipe for a tetrapod

You can read the source code for a tetrapod drawn using three.js and play with the resulting animation.

preliminaries

Pick a size.

The size is the length and diameter of each tetrapod leg, half the height of the tetrapod as it is constructed balancing on two legs, and a few percent less than half the height of the tetrapod when it’s standing on three legs.
The tetrapod is constructed inside a cube centred at the origin, with vertices at (± size, ± size, ± size). Inside the cube is a tetrahedron, and the tetrapod’s legs point towards its vertices.

In the picture above, the cube is shaded green, and the tetrahedron is outlined with straight red lines.
We will construct the positive leg along its main axis, the vector from the origin to the common vertex (+size, +size, +size).

The three other legs will be cloned from the positive leg.
The leg is constructed from truncated cones.

A cone is described by its taper, r / a = tan(θ), where a is the distance from the apex of the cone, and r is the radius of the circular section of the cone at that distance.

Every cone’s axis is the main axis, but their apex varies.
A cone is truncated by a circle with radius r and distance d from the origin along the main axis.

(Note that the apex distance a and origin distance d differ when the apex of the cone is not at the origin.)
We need two scaffolding cones.

The axis cone has its apex at the orign and its sides just touch the x,y,z axes.
```
axis_taper = sqrt(2)
```
derivation
z = (0,0,1)
xyz = (1,1,1)
z · xyz = |z| |xyz| cos(θ)
1 = 1 * sqrt(3) * cos(θ)
a / h = cos(θ) = sqrt(1/3)
r² = h² − a² = |z|² − a² = 1 − 1/3
tan(θ) = r / a = sqrt(2/3) / sqrt(1/3) = sqrt(2)

The face cone has its apex at the common vertex. Its sides just touch the faces of the cube along the same lines as the edges of the tetrahedron.
```
apex_d = size * sqrt(3)
face_taper = sqrt(1/2)
```
derivation
xy = (1,1,0)
xyz = (1,1,1)
xy · xyz = |xy| |xyz| cos(θ)
2 = sqrt(2) * sqrt(3) * cos(θ)
a / h = cos(θ) = sqrt(2/3)
r² = h² − a² = |xy|² − a² = 2 − 2*2/3
tan(θ) = r / a = sqrt(2/3) / sqrt(4/3) = sqrt(1/2)

circles

The leg is constructed from four circles: waist, thigh, ankle, sole. In the picture above, you can see the thigh, ankle, and sole circles outlined in red. The waist is hidden inside.

As required by the inventors of the tetrapod, our chosen size is equal to the diameter of the thigh, and to the distance between the thigh and the sole.

The thigh circles just touch each other at the x,y,z axes.

The foot of the tetrapod, between the ankle and the sole, is the chamfered part. It is a section of the face cone.

The ankle circle is chosen so that the taper of the main part of the leg is about 10°. It turns out that a nice round number, r = size / 3, makes the taper 10.41°.

To fill in the core of the tetrapod, between the thigh circles of the four legs, we will extrapolate the leg taper to find the waist circle whose centre is the origin.

Hence,

Locate the thigh circle from its radius in the axis cone
```
thigh_r = size / 2
thigh_d = 0 + thigh_r / axis_taper
```

Locate the ankle circle from its radius in the face cone

ankle_r = size / 3
ankle_d = apex_d - ankle_r / face_taper

The thigh circle and leg length determine the position of the sole circle, from which we get its radius
```
sole_d = thigh_d + size
sole_r = (apex_d - sole_d) * face_taper
```

The thigh and ankle give us the taper of the leg

leg_taper = (thigh_r - ankle_r)
          / (ankle_d - thigh_d)

Extrapolate back to the origin to find the radius of the waist
```
waist_d = 0
waist_r = thigh_r + thigh_d * leg_taper
```

legs

Make the positive leg from two truncated cones:

one between the waist and ankle,

one between the ankle and sole.
To make another leg, take a copy of the positive leg and rotate it 180° about the x or y or z axis.

Do this for all three axes to get all of the legs.

The legs overlap where their waists spread out around the origin. Conveniently, the legs meet at planes, so we can clip at these planes to make a model where they don’t overlap.

The positive leg meets the other legs at the three planes through the origin with normals:

    (1,1,0)
    (1,0,1)
    (0,1,1)

vital statistics

When standing on three legs with one leg vertical, this tetrapod is

2.09 * size tall
2.52 * size max wide
2.25 * size min wide

The radii of the key circles of the leg are

1/2 * size thigh
1/3 * size ankle
0.27 * size sole

The radii from the vertical axis to the ends of the base legs are

1.08 * size to point touching the ground
1.37 * size to furthest point

The taper of the leg is 10.4°.

The underside of the tetrapod is surprisingly close to flat.

0.17 * size between the ground and the cusp between the legs
9.0° angle between skin of legs and ground

The middle of each face of the tetrahedron is close to the core of the tetrapod. That’s partly because of the small 9° angle between the legs and the ground, which is parallel to a face of the tetrahedron. It’s also because the thigh circles protrude a tiny amount through each face, because they are slightly larger than the circles that just touch the x,y,z axes and the faces of the tetrahedron. These inner circles have a radius of sqrt(6/25) * size.

The cusp where three legs meet is slightly inside the face, by 0.015 * size.
The thigh circles are larger than the inner circles by 0.010 * size.

epilogue

It almost feels like the shape of the tetrapod was discovered rather than designed.

A tetrahedron nested in a cube is a fun framework for a shape. Using half the cube’s side length to determine the length and width of the legs seems like a natural choice. Maybe the choice was based on how close sqrt(6/25) and 1/2 are?

A third of the side length for the taper is a plausible option that turns out quite elegant. Maybe the chamfer at the foot is designed? I like the way it both hints at the cube from which it was constructed, and presumably has good engineering properties such as making the corner less prone to damage and easier to remove from its mould.

One thing I have not yet managed to work out (except by punting the problem to a CAD kernel) is exactly what ellipse is formed where the legs intersect. Investigations continue…

Transaction pooling for Postgres with pgcat

Notes on software development — Fri, 24 Oct 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

That Time I Trashed The Company Mainframe, And The Lesson I Learned

The Codist — Wed, 22 Oct 2025 18:00:44 GMT

I started my career in the fall of 1981, and in the first six months, I wrote a Jovial language source code formatter in Fortran. I had no prior experience in programming jobs or college programming classes; all I knew was what I had taught myself at home.

That formatter

Running Linux on my gaming PC again

Ben Overmyer's Site — Wed, 22 Oct 2025 03:00:00 GMT

With everything that's going on in the world right now, I am increasingly sensitive to the security and privacy implications of the tech I use. While most of my devices are in the Apple ecosystem, there has been one big hole in my device privacy footprint: my gaming PC.

I have been restricted to the Windows ecosystem on my gaming PC because of the games I regularly play. A few months ago, I reluctantly "upgraded" from Windows 10 to Windows 11. This made me anxious at the time, but then it faded into the background as other things took up my mental bandwidth.

Recently, though, Linux came back into my awareness. This past month, KDE celebrated its 29th birthday. This news was all over Hacker News, Mastodon, and some other online social circles I run in. And with Windows 10 formally being sunset, I decided it was probably time to finally switch. Again.

See, I've tried running a Linux desktop before. Generally it works fine. However, two areas have made me stumble hard in the past: design software and games. I've relied on Affinity Suite for the former, which has no Linux version and no plans to have a Linux version. For the latter, I tend to play games that somehow always have game-breaking bugs on Linux, even with Proton.

This time around, I've tried a mixture of the latest developments in Linux and a willingness to change what software I use and what kinds of games I play in order to permanently break from Windows.

So, here's how it's going.

I chose Debian 13 with Plasma and Wayland. Debian is familiar because I've used Ubuntu a lot in the past, but has none of the bloat of the latter distribution. Wayland solves a lot of problems with some games I play, though it does create new problems with some other software. I'm still working through some of those, like Visual Studio Code having bad flickering unless I use a specific set of startup flags.

Incidentally, here're the options I use to fix the flickering for VSCode on Wayland:

code --use-gl=egl --ignore-gpu-blocklist --enable-gpu-rasterization --ozone-platform=wayland

Choosing new software for old problems has been the bigger jump. Instead of Affinity Photo, I'm trying out GIMP. GIMP is way better than it used to be. There's still a big learning curve, so it's an ongoing challenge. For vector work, I'm using Inkscape, which does what I need it to do. For office software, I've already been using LibreOffice on Windows for a long time, so that wasn't a hurdle.

I discovered that Fastmail has a desktop client for Linux. It's pretty new, apparently, but so far it works really well for me.

For games, well, that's been interesting. A lot of games that didn't work well on X11 now work on Wayland. Some don't. The Finals, which is my primary shooter these days, currently has a shader bug that makes it unplayable. It was just introduced, so I don't know when it will get fixed. Monster Train 2, though, works flawlessly. I'm starting to explore new games that are built for Linux specifically, though, which is where things are getting more interesting. I might write a future post just about that.

Spotify works just fine on Linux, but I'm starting to make use of and grow my MP3 library again. I just discovered KDE's JuK for playing music yesterday, and it's surprisingly awesome. That started me down the path of exploring other KDE tools, and I'm discovering a lot to love.

I think this is just the beginning of a paradigm shift in how I use my gaming PC. My Mac Mini and Macbook Pro feel a lot less necessary now, too. Time will tell how this shakes out.

A couple of handy KNX gadgets

John Graham-Cumming's blog — Mon, 20 Oct 2025 18:33:15 GMT

KNX is a European standard used in home automation. It's primarily based around a twisted pair bus transporting DC power (30V) and data at 9600 bits/second. Devices attached to this KNX bus draw power for their operation and receive and send messages called telegrams. The devices are typically things like switches, relays, dimmers, HVAC interfaces, small displays, and blind/shutter controllers.

If you're working with a KNX bus you connect a computer to it via bus interface. These are commonly either a USB interface or an Ethernet/WiFi connection. Either way the computer is able to send and receive telegrams and monitor the bus for debugging. This interface is also used for programming the various KNX devices on the bus (for example, associating pressing a button on a particular switch with a particular light circuit going on or off).

Sometimes it's useful to program devices "off the bus" (i.e. away from the actual installation). For this purpose I put together the simplest of minimal KNX buses: a power supply to inject the necessary 30V, a USB bus interface, and a dangling KNX connector (the red and black thing in this photo) that can be plugged into the device to be programmed.

Normally the power supply (on the left) would be wired in permanently, but here I am just using a standard power cord for desktop programming. The USB bus interface is in the middle. There's nothing special about these devices, there are hundreds of KNX manufacturers that interoperate; I just happen to be using an ABB power supply and ABB bus interface.

This works great: plug the device to be programmed into the connector on the right and hook the PC running the KNX programming software ETS into the bus interface via USB. And apart from the two ABB boxes, all that's needed is a power cord, three WAGO 243-211 connectors, and some twisted pair. I used random wires that were lying around, in the real world of KNX installations you'd use proper shielded twisted pair cables!

The other gadget I built is this custom PCB (possibly the world's simplest PCB):

It just connects four KNX terminals together in parallel. The pins at the top are the standard WAGO 243-131 pins compatible with the WAGO 243-211 connector. When soldered together it looks like this:

In conjunction with a bus interface it lets you tap into an existing KNX bus if there's no bus interface present. Or if it's just more convenient to connect where you are working. You simply unplug some device from the bus, plug the bus into this PCB and then plug the device into the dangling connector.

The bus will work normally and you'll have access for debugging and programming. (Note: you can do this because the KNX bus can be a tree and so it's acceptable to plug in a spur of a couple of devices to the main bus line).

The PCB design is based on Matthias Kleine's KNX Distributor; my version can be found here. (If you build this you'll also need Matthias Kleine's footprint library).

The Bird-Poker Deck

Send More Paramedics — Mon, 20 Oct 2025 11:47:49 GMT

Whereby I talk about a truncated pack of playing cards called the Bird-Poker deck...

The only Permanent Underclass are the ones who believe it is permanent

swyx's site RSS Feed — Mon, 20 Oct 2025 04:02:22 GMT

The design space of AI coding tools

Austin Z. Henley's Blog — Sat, 18 Oct 2025 23:00:01 GMT

https://austinhenley.com/blog/aidesignspace.html

Insights a 25 Year Old Movie Can Give Us on LLMs

Beetle Space — Wed, 15 Oct 2025 10:00:00 GMT

I often hear complaints that LLM coders are not like junior colleagues. The difference being that junior colleagues learn and grow and don’t make the same mistakes over and over. Time spent attending to a colleague bears long-term fruit.

LLMs that don’t learn from their experiences will be …

Fully Implementing PSR-16 Simple Cache is Less Than Simple

Donat Studios — Mon, 13 Oct 2025 12:42:59 GMT

Foreword: Please note this falls firmly into "nitpick" territory rather than shining light onto any sort of major issue. I just thought it was an interesting gotcha and wanted to share.

TL;DR: To fully implement the PSR-16 specification as defined, an implementation MUST be contravariant of the defined interface in its accepted parameters when implementing v2 or later.

I work on an application with a proprietary caching layer. Recently I wanted to build a PSR-16 "Simple Cache" adapter for it, and started going through the spec and provided interface in detail. I wanted my implementation to be as correct as possible.

While working my way through getMultiple/setMultiple/deleteMultiple I noted something that seemed like an oversight.

/**
 * Obtains multiple cache items by their unique keys.
 *
 * @param iterable<string> $keys    A list of keys that can be obtained in a single operation.
 * @param mixed            $default Default value to return for keys that do not exist.
 *
 * @return iterable<string, mixed> A list of key => value pairs. Cache keys that do not exist or are stale will have $default as value.
 *
 * @throws \Psr\SimpleCache\InvalidArgumentException
 *   MUST be thrown if $keys is neither an array nor a Traversable,
 *   or if any of the $keys are not a legal value.
 */
public function getMultiple(iterable $keys, mixed $default = null): iterable;

Take a moment to read this carefully

@throws \Psr\SimpleCache\InvalidArgumentException
  MUST be thrown if $keys is neither an array nor a Traversable
  or if any of the $keys are not a legal value.

Then, inspect the definition

public function getMultiple(iterable $keys, mixed $default = null): iterable;

The takeaway here should be:

iterable $keys

To implement getMultiple, we MUST throw a \Psr\SimpleCache\InvalidArgumentException when $keys is invalid, yet $keys is defined as iterable which makes it impossible to handle if an implementation matches this signature.

Passing anything non-iterable will throw a TypeError before the code even runs. This would make it impossible to throw an InvalidArgumentException.

Parameter types were added to the interface with the release of v2. My suspicion was that the @throws had simply not been updated, because if the parameter type is explicitly defined as iterable, the function cannot throw a custom exception.

I originally suspected this would make it impossible to fulfill the spec. However, I had not considered contravariance. The implementing methods accepted parameter types may be wider than the interface it is fulfilling. In this instance, $keys does not need to be defined as iterable in the implementation.

This means a technically complete implementation must omit iterable and look something like

public function getMultiple($keys, mixed $default = null): iterable {
    if(!is_array($keys) && !$var instanceof Traversable) {
        // Must implement \Psr\SimpleCache\InvalidArgumentException;
        throw new ExampleException;
    }
}

Note \Psr\SimpleCache\InvalidArgumentException is an interface. An implementation must define a concrete exception class that implements it before it can be thrown

So while it's not impossible to implement, it certainly is strange to implement. One can only correctly implement the spec by manually checking $keys type like this.

Doing a little investigation, it seems like most major implementations still target v1 || v2 || v3 and as v1 did not have typed parameters, neither do the implementations iterable parameters. This means most major implementations are implemented correctly.

However, there are at least a couple of implementations that get this wrong.

I'm sure there are others, but I found these three in my quick search. It is a very easy mistake to make when not implementing v1.

This is not Java. There are no checked exceptions forcing anyone's hand. In practice, this is a minor quirk of the spec. If a TypeError gets thrown instead of an \Psr\SimpleCache\InvalidArgumentException, almost certainly nothing meaningful breaks. I doubt anything depends on this behavior. It’s just an inconsistency between what the spec says and what implementations are doing. It is nothing to lose sleep over, but it might be worth fixing for consistency's sake.

In summary, to fully implement the getMultiple/setMultiple/deleteMultiple methods of the PSR-16 spec, one cannot simply copy their definitions directly from the CacheInterface. The implementation must check the validity of their iterable parameters and not have them defined as an iterable.

My startup diary: Techstars

Austin Z. Henley's Blog — Mon, 13 Oct 2025 02:00:01 GMT

https://austinhenley.com/blog/techstarsdiary.html

L.A. Noire review

ブラック — Sun, 12 Oct 2025 03:00:00 GMT

Cinematic brilliance on the Sunset Boulevard.

I absolutely adore the writing and the story in this game. It’s so masterfully done, both gritty and hilarious, and really pulls you into the story and the setting. This also applies to the characters, who all have excellent characterisation and dialogue, including your man Cole Phelps himself. Anyone who is a fan of detective novels, noir, or just great dialogue should definitely pick up L.A. Noire. I think the biggest negative thing I can say about any of it is that you can only experience it for the first time once.

The gameplay’s good too. Given the fact that this is an open-world game from Rockstar Games, one might assume that it’s basically a police version of Grand Theft Auto, but it really isn’t. You go around finding clues, interviewing witnesses and suspects, doing police legwork. Sure, some mechanics like tailing cars or engaging in firefights is similar to many other games, but it’s mostly a very unique, and detective-like gameplay experience. And really, even if it was just a lazier Grand Theft Auto clone, the story would still be enough to carry it.

I think the biggest criticism I’d have for the gameplay is that some of the interrogations require some pretty out-of-the-box thinking and leaps of logic. If someone completes this game with all of the right answers without looking them up, I’d have some concerns about their thought processes. Thankfully most of the interrogation questions are sane, and even the ones where Cole Phelps goes off on a limb do at least have some perverted logic behind it.

The facial capture technology in L.A. Noire is also quite fascinating. It really gives the characters a lot of emotion and range, a lot more than you’d expect out of a game from 2011. Quite impressively I was able to recognise some of the actors from their faces alone, so that should say something. Unfortunately the texture quality in the game isn’t the highest, so it does water down the magic of the technology a bit. I also don’t think Team Bondi really had to go so hard when it came to creating the facial capture technology, since a lot of the facial clues that you need to read aren’t that subtle. Thankfully too, since a lot of us are not trained detectives. However, hard they went and we got this very unique gaming experience that borders on a TV show as a result.

Apart from the super-unique facial scanning technology, graphics in the game are quite average. It’s not ugly, but shows its age. Unfortunately one of the bigger issues I have with the game is on the graphics side: the game is capped at a hard 30 frames per second. It also makes the game feel so sluggish that I took a long break in middle of my playthrough because it was so demotivating in an otherwise interesting game. Thankfully now you can play the game with the excellent V-Patch and enjoy a fairly problem-free 60 FPS, or even >100 FPS if you don’t mind having to turn it down for at least one of the cases. Personally, I stuck to 60 frames per second, as it seems to have no issues apart from more twitchy driving controls, and it fixed the game feeling so sluggish to play.

But even if the graphics aren’t that spectacular, you can really tell that a lot of effort was spent on the game. While I can’t speak for what Los Angeles was in 1947 (or even what it’s like even now), it does really feel like driving through a meticulous period piece. Team Bondi also really took the extra mile when it comes to the cars too. Despite featuring a lot of driving in an open world, L.A. Noire isn’t really a proper sandbox or driving game, and yet they saw it necessary to fit almost a hundred real-life cars from the period. I doubt that the illusion would’ve been broken if they had half the amount, especially given how often you’re using your police-issued vehicle, but they went hard here too.

The music and the sounds of the game are top notch too, and like the setting and cars, builds your immersion for a 1940s Los Angeles. I just had to go and listen the soundtrack after I ended my 48-hour completionist playthrough. And since the game is full of acting talent giving their motion and facial performances, the voice acting is also pretty much as good as it gets. Without going into spoilers, there is a particular scene where Aaron Staton gives off an emotional performance as Cole Phelps that I still think about.

L.A. Noire is such a special game and unfortunately we might not see another game quite like it ever again. The development of the game was tumultuous, Team Bondi folded after the development ended, and I have no idea if the motion capture technology they spent all of that time and money creating even exists anymore. And despite all of that, this wonderful, one-of-a-kind game got made. I would highly recommend everyone pick it up and experience it for themselves.

Marketing Effectiveness and Experimentation Measurement

Bytepawn - Marton Trencseni — Sun, 12 Oct 2025 01:00:00 GMT

A deep dive into how to structure a marketing measurement and experimentation framework — from universal control groups and campaign holdouts to attribution windows, outlier handling, and statistical rigor. Based on lessons from building a real-world policy for large-scale marketing analytics.

New Art Tool: Kashi's Pixel Playground

Marc Stuff — Sat, 11 Oct 2025 06:46:27 GMT

I’m happy to share a new art-creation project, Kashi’s Pixel Playground.

Here’s the direct link to have a play:

https://www.friendlyskies.net/other-projects/kpp/

Notes:

This project was inspired by some colored pencils I like to use, which have a bunch of different colors inside each pencil.
Desktops / laptops work best for now. Mobile support is on the to-do list.
This project was created using SpiderBasic, as a fun retro-coding-style project.
Detailed writeup w/ code: https://forums.spiderbasic.com/viewtopic.php?t=2850

Modern Monetary Theory: What’s Missing from the Narrative

Bytepawn - Marton Trencseni — Sat, 11 Oct 2025 01:00:00 GMT

I explore several points that are rarely explained in Modern Monetary Theory narratives: from taxation in non-tax economies to central bank independence, inflation targets, and the many forms of money that exist beyond government spending. These are areas where the story of MMT could be told more completely.

Status II

Digital Apocrypha — Fri, 10 Oct 2025 15:34:37 GMT

The MMT Case for Governments Funding Open Source Software

Bytepawn - Marton Trencseni — Tue, 07 Oct 2025 01:00:00 GMT

I argue that widely used digital infrastructure like operating systems, compilers and browsers deserve public investment, just like roads, hospitals, and schools.

Status I

Digital Apocrypha — Mon, 06 Oct 2025 22:33:00 GMT

Verizon's I'm A Teapot Error And Other Technology Fails

The Codist — Mon, 06 Oct 2025 22:20:32 GMT

This is not my week for using technology. None of these problems was my fault, other than my existing in the universe and the attraction that brought.

Today, I wanted to review my Verizon mobile account, and I noticed that the website had changed since my last visit. I clicked

Language Support for Marginalia Search

Weblog on marginalia.nu — Mon, 06 Oct 2025 03:00:00 GMT

One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish.

These changes are now live for testing, but with an extremely small corpus of documents.

As the search engine has been up to this point built with English in mind, some anglo-centric assumptions made it into its code. A lot of the research on search engines generally seems to embed similar assumptions.

Control Ultimate Edition review

ブラック — Mon, 06 Oct 2025 03:00:00 GMT

Rather be out of control.

The story of Control is weird. You go into an empty government building to look for your little brother, meet a strange Finnish man, and get appointed as the director of the bureau within minutes. And unfortunately, it never gets any better. One of the characters you meet at the beginning of the game succinctly describes the story: "This is fucking unbelievable."

I feel like the story is trying way too hard to be weird and mysterious, to a point where I just completely lost interest. There's just a limit to weird cutscenes with strange double talk happening that I can process before I just zone off, and after that, there’s no amount of extreme close-ups of Jesse’s eyes that can make me start to care about her. I'd love a nice mystery story, but this isn't a mystery. It's just bizarre for the sake of being bizarre with zero intrigue except for "when does this start making sense?" But it really never does and has very little narrative closure.

Really, the only good thing going for the story is that there isn’t much of it. It took me eight and a half hours to beat the game from start to finish, and I wasn’t even playing particularly efficiently. Hell, considering I stopped playing the game for two years out of sheer disinterest, I even had time to forget how to play. But despite its short duration, it still feels padded out. The chapter Finnish Tango doesn’t seem to really have any plot significance, and just highlights how Remedy is a Finnish company and gives Martti Suosalo some work.

The combat starts off super disappointing when you're given the pistol. It's a total peashooter and feels super unsatisfying to use. Enemies will take at least three shots to the head with it before they even consider dying. I actually found it easier to rapid-fire enemies to the body than bother wasting your time aiming for the head. Your player character is also squishy, so being quick on the trigger was less likely to result in your untimely death. I'm surprised that a character in the story managed to commit suicide using this gun considering how little damage a shot to the head from it will do.

Thankfully the combat does actually open up considerably like 30 minutes afterwards when you're given your first magic power. I'm surprised that the developers decided to not give it to the players immediately considering how bad the firearm is. Or at the very least, do very minimal combat with the firearm before you're given the ability to magically hurl objects very fast at enemies. That being said, even with the magical powers, the combat is still not really amazing. It just stops being awful and starts being playable.

There’s a new collectible to grab like every 10 metres, but like the story and the characters, I’m struggling to find a reason to care about any of them. There’s also some form of crafting and side missions, and I’m not particularly interested in a game’s side hustles if the main parts are uninteresting. Apparently there’s also more magic powers than just throwing and levitation, but since those are not on the critical path, they’re surely not that important either.

I would otherwise consider the graphics to be a rare good thing about Control, but unfortunately the graphics are extremely buggy so I can’t even say that. Some enemies have the magic power of spawning pitch-black rectangles all over the screen, which makes the game very hard to play. I’m playing on an RTX 4090, so at least the issue isn’t a lack of horsepower. Thankfully I wasn’t wasteful enough to get the RTX 5090 when it came out, since apparently the game is even buggier on the 50-series cards than it is on my 40-series.

Not that the game needed to even really shine that much graphically given how bland the game is stylistically. Granted, it’s set inside a government building so blandness is on brand, but it doesn’t make it any more appealing to the eyes. You will probably not lose any enjoyment even if you can’t see the office fluorescent tube not shine off the marble floor.

At least the music’s good. I’d definitely listen to Take Control again. But that’s about the only thing Control that I want to take with me. At least I don’t have to feel miffed about not having access to Alan Wake II on Steam.

Organizational Lock-in

PREPEND — Thu, 02 Oct 2025 03:00:00 GMT

I worked in an organization with lots of processes. I was in an office with no windows, a bit of a hole or maybe even oubliette. You couldn’t tell time of day, weather, etc but it was better than a cubicle.

My door opened to a 10 foot hallway with and about 5 feet down there was a large window. It was a weird L-shape and no one else’s office or cube or anything was in the hallway to nowhere. So it was kind of nice that no one walked by and it was off the beaten path.

The window in the hallway was closed with blinds. But if I opened them it was possible to get some ambient light effects so while I couldn’t see out it from inside my office, I could tell if it was day or night. So it was better.

A few weeks went by and the blinds were closed one day. That’s weird, no one comes by here. I asked some nearby coworkers if they knew who closed and why and they didn’t know. No one knew. The entire floor is probably the size of a football field and has 100 offices around the edge with 200 cubes and conference rooms and stuff in the middle. My team is probably 10 people and there’s lots of other groups.

No one I knew knew and didn’t really care. I maybe spent 2-3 minutes because I was surprised that anyone walked by here, much less cared. I opened the blinds and when about my business.

About a week later, the blinds were closed again. Weird, but again I checked and no one I knew cared. I opened them again.

3 days later, they were closed. Repeat a few times and now I was opening them every day and someone else was closing them. Weird and funny.

This went on for a few days and the blinds would close in the middle of the day. I spent about 3-6 hours out of my office doing different stuff so I come and go. So I would now open them whenever I noticed them back down during the day.

Finally, I’m in my office and I hear the metallic zip sound of blinds closing. I stick my head of of my office and say hello to someone from a different floor and group, I’ll call her Betty.

I ask Betty what’s up and she tells me that she’s closing the blinds. I ask why and she says because they are supposed to be closed. I explain why I like them open and that there’s no one else who sees this window and she again says the blinds must be down.

Interesting. I love arbitrary rules as a big board game fan. I ask about why, and she says that no building things can be changed without a business justification as required by OSHA, fire marshalls, and emergency coordinators.

I tell her that I think this wouldn’t apply to something as minor as a window blind, but she assured me it’s for everything.

I try a different tack and propose that the natural state is open, and that she is actually the one who needs a justification to close. But she’s too smart for that, she produces an inventory of the building that she makes each January that clearly shows the blinds closed for the past few years.

She is not joking or being ironic. She’s serious.

So I ask how business justifications are evaluated and she says that they are submitted to her and she evaluates them. There’s no template or format and no one has ever submitted one in the 10 years she was in this role.

Wow, ok. I go ahead and tell her I’ll write one and ask how to submit them.

This is funny to me, so later that evening, I spend a few minutes and write up a justification email. It’s pretty bullshit and basically says “Window blinds open makes me happier, and thus more productive. Closed blinds reduce productivity and thus reduce organization impact. Yadda yadda yadda.”

It’s maybe two paragraphs. She replies that my justification is insufficient and that the blinds will stay closed.

This is now less funny. So I look her up in the company director. She is a random workgroup’s secretary and is a contractor who has been in the building forever. She has contracted for three different companies, but has always sat there. She schedules appointments and orders staples and stuff for about 20 of the hundreds of people in the building. She’s on a different floor than me. There’s five floors. Her group is entirely on a different floor.

This is bizarre. With this knowledge, I resubmit a revised justification. She denies it again. I respond copying her contracting monitor or whatever you call the person who organizes the contract, pointing out this bizarre process. The contracting person is confused and responds that this is not in any contract, they don’t give a shit, and question why this is done.

After a few days of email silence, the contracting person responds that the blinds will stay open. I never hear from Betty again.

I moved out of that office about 5 years ago. The blinds stayed open while I was there.

About a year or two ago, I was in that building and I walked by my old area to talk to some former co-workers and noticed the blinds were down.

I gave a quick recap and the current occupant didn’t care and hadn’t noticed. But they did say that Betty was still there.

I think people get really invested in a process they think is important, but is only important to them. A co-worker called this “building imaginary castles in the sky.”

It was really cool to them, but meaningless to any observer. Sometimes it’s useful in the long run for something else, but hard to tell immediately.

There’s confusion that gets compounded and you have real stuff wrapped around it. It seems like it may be bullshit. It makes me think how much of what I care about is just organizational lock-in and not really meaningful.

Note: I found this post when I was cleaning out my phone and I think I wrote it in 2018. So I expect this story took place in 2013 or so. Given the current state of government shutdown and free time, I thought that it’s theme of people doing crazy stuff for unknown reasons was still relevant.

Reference

This reminds me a bit of the Business Urban Legends post story about the chimpanzees who don’t know why they do things.

2025–10–02: Luckfox Pico Mini

megi's PinePhone Development Log — Thu, 02 Oct 2025 01:00:00 GMT

Protecting your (my) passport from liquid damage

John Graham-Cumming's blog — Wed, 01 Oct 2025 20:05:09 GMT

After an unfortunate incident involving my passport and some alcohol-based hand gel in my bag, I decided I needed some sort of waterproof passport protection. But searching around for "passport wallets" mostly doesn't result in anything waterproof and all the wallets are large, bulky or ugly. But there is a really simple solution.

A 4 mil ziploc bag of the appropriate size.

What's the appropriate size? Well, there's an ICAO standard for machine-readable passports which uses the ISO/IEC 7810 ID3 size for the "passport booklet" (what you and I would call a passport).

So, that's 125 x 88mm. Happily, a pretty common ziploc bag size is 10 x 15cm (or 100 x 150mm, if you prefer). That gives you an extra 12mm horizontally (so the passport will slip inside) and 25mm vertically (space for the passport and the area with the ziploc closure).

I mentioned "4 mil" above. Switching to non-metric measures (which seem common for the thickness of plastic) that's four 1/1000s of an inch (note it's not "millimetres", it's "mils"). You also sometimes see plastic measured in microns (which is 1/1000 of a millimetre).

4 mil bags have a sturdiness to them that thinner plastics do not have. They are sometimes described as "extra thick". In the UK, these folks (not an affiliate link) sell through Amazon and have a variety of bags of the right size and thickness.

And now my (new) passport is carefully protected.

Aside: it turns out I actually once owned a waterproof passport wallet. I was gifted a SÜLgear passport wallet and lost the waterproof liner. The kind folks at SÜLgear pointed out that the waterproof liner was simply a 4" x 6", 4 mil ziploc bag (or, for those of the metric persuasion, 102 x 152mm).

“Every Hard Drive I’ve Owned Has Been Larger Than All My Previous Ones Combined”

Beetle Space — Sun, 28 Sep 2025 10:00:00 GMT

Back in 2004, I saw this Slashdot comment:

Every hard drive I’ve ever bought has been larger than all my previous hard drives combined. And this is without even trying.

– blancolioni

Reflecting on this, I realized that this was also true of all the hard drives I had ever …

Upcoming projects, Fall 2025 edition

--Mike-- — Fri, 26 Sep 2025 09:12:04 GMT

Upcoming projects include revisiting old ideas

Bidirectional compilation - a compiler that can work backwards to enable radical refactoring, possibly into other languages. I'll start with Pascal, because it doesn't really do macros, which break reverse causality. Memex - I want to create a markup language for existing documents that can actually bring an 80 year old idea to life. The main feature is the ability to browse the web and have it all captured and ready for sharing and making copies on demand, something IP lawyers and censors would abhor.

LiveSQL - a database that offers subscriptions to changes, and temporal features. There was an old database back in the MS-DOS days that had the idea of generations of changes to resolve locking and other issues. Maybe that becomes part of this. The main feature is that you could have a query that gives you new versions of the database as others make changes.

Setting up my own electronics repair shop. My friend who I help has been doing this since the 1950s, I need to get my own workbench and things up and running again, it's been decades since I've had my own shop, now that I'm retired, it's time to set it up.

De-hoarding - too much stuff that I won't use, and my child doesn't need to worry about when we're gone. Part of the shop process should be finding someone else to inherit that stuff, and save my kid the grief of finding it a home.

Multidimensional ratings - Imagine a way to rate things on a forum (just to make it easier to start out), but instead of up/down vote, you can make up your own dimensions as well... humour, is_it_political, spam, etc. There are things you really can't talk about because the moderation systems we use globally have to worry about spam of many forms, and nuance gets crushed. I'd like to help address that.

Revised Alternative Self Driving Car Levels

Blogity blog blog. Journal — Thu, 25 Sep 2025 23:00:00 GMT

I’m not the SAE, I’m just some dude, but imo the current SAE levels for driving automation jump from “driver assistance” to “partial automation” to “conditional automation” without capturing the nuanced progression of capabilities that actually matter for safety and real-world deployment. I propose these more granular levels as a framework that better reflects the technical challenges and safety milestones in the journey toward full autonomy.

Level 0: No Automation

Human does everything

Level 1: Speed Control

Can maintain speed (cruise control) Human controls steering and everything else

Level 2: Speed + Steering

Can control speed AND steering simultaneously Requires constant human supervision

Level 3: Stay-in-Lane Highway

Can stay in a single lane on the freeway Handles stop-and-go traffic Can come to complete stop and resume without intervention Has basic object detection (car ahead stopped = stop) CANNOT change lanes or route around obstacles This is SAE Level 2 today, but it’s a meaningful milestone on the path to SAE Level 3/4.

Can change lanes, merge, take exits Can navigate around obstacles by changing lanes Handles all normal highway scenarios autonomously Can pull over safely if confused Will stop instead of running into something.

Level 5: Advanced Object Recognition

Can distinguish between critical and harmless obstacles (child vs plastic bag) Understands object physics (ball rolling = child may follow) Can navigate simple urban environments and intersections Can handle parking lots

Level 6: Predictive Behavior Modeling

Can identify and predict behavior of pedestrians, cyclists, children, animals Understands social driving cues (hand waves, eye contact) Can handle construction zones with human flaggers Can navigate complex urban environments in good conditions

Level 7: AGI-Level Driving

Functions in all weather conditions (heavy snow, fog, torrential rain) Can violate traffic laws when necessary for safety Makes complex ethical decisions about least-harm outcomes Handles unmapped roads and novel situations Matches or exceeds best human driver in all scenarios

Commentary

Comma.ai very capabily drives me down the freeway with much less cognitive burden. If you spend long amounts of time driving, you should get one. If your car does not support it, upgrade your car. (I’m not being paid to say that, just a happy customer.) It doesn’t integrate with GPS so you’ll miss the freeway exit if you don’t pay attention.

Meanwhile, Waymo is better than 4, but it’s unclear exactly how much further it is. From experience riding in them, the Waymo Driver is already at 5, and some parts of 6.

The industry has come a long way. The argument over LIDAR or not; comma.ai’s abilit to drive on the freeway suggests LIDAR is not as critical as it was back in 2009 when Waymo was the Goolgle self-driving car project.

Homelab Failure Domains

Pete Keen — Thu, 25 Sep 2025 03:00:00 GMT

Everything is in shambles.

One machine is dead and probably not coming back. Services are randomly scattered amongst the survivors. My Kubernetes project is kinda-sorta paused.

I'm unnerved and want to get it all sorted out, but I need to do some thinking out loud first.

My infrastructure currently consists of:

Omicron, a Kubernetes cluster that is just barely functioning. It is currently running this site, VMSave, and basically nothing else. Consists of hypnotoad, crushinator, and a control plane VM running on a Wyse 5070 running Proxmox.
Nimbus, a Kubernetes cluster that is not functioning at all. I tried building out a GitOps-driven cluster as my second attempt and everything was going swimmingly until nibbler, a historically unreliable piece of hardware that when it works has more cores and memory that the rest of my infra combined, fell over again in yet another inexplicable way.
Lrrr, a box with an Intel N150 and 32GB of memory running a VM on top of Proxmox that is hosting almost everything that was previously on the two Kubernetes clusters.

This jumbled state of affairs is basically due to a series of impulsive hardware purchases and "oh that's neat, let's do that" infrastructure changes.

Let's talk about failure domains.

I think of a failure domain as a set of risks and mitigation strategies as applied to a particular instance of a service.

The canonical example in the software-as-a-service world is "production", i.e. the instance of the service that the customers touch. The one that makes the money. The primary risk is the money going away if the service goes down.

A SaaS shop may have a staging environment, where changes get tested before they hit production. The main risk in staging is inconveniencing your coworkers, but the consequences of that to the company are much less impactful.

Each developer in then hopefully has one or more of their own environments in which to actually make the software. These are practially risk free to the company as a whole, only inconvenicing one developer if something goes awry.

Overcomplicated home infrastructure doesn't map neatly into the same failure domains as a SaaS business, of course, but they still exist.

When I think about the users of the services in my home I imagine a sort of abstract "household delight" score. Points accrue implicitly when things are running fine and people are able to use the things I'm trying to provide. Points get deducted when they notice things aren't working or when they see me stomping around grumbling about full hard drives and boot errors.

By that logic I have three different failure domains (actually four but we'll get to that):

Critical production. The absense of service would be immediately noticed and commented upon, often affecting the comfort of the occupants of the house. Examples: network, DNS, Home Assistant and friends, IoT coordinators.
Production: The absense of service would be noticed eventually but even an extended outage wouldn't cause hardship. Examples: Jellyfin, Sonarr and friends, paperless-ngx.
Lab: I'm the only one affected by things breaking in the lab. A playground for testing and fucking around.

The fourth failure domain that doesn't neatly map into the above is production services for external users. VMSave and this site are the big ones but there are a few smaller things too.

When I'm brutally honest with myself I have to recognize that the biggest common source of failure in every domain is me. Trying things, adding hardware, replacing software, messing around, testing in production.

Often my partner will remark "I don't understand how things just fail!" They usually don't. Failure is an immediate or delayed result of me changing something without considering the impact.

So. What to do.

Obviously first I need to deliniate the lab from everything else. Separate hardware for sure, maybe even hide it all behind another router and subnet.

For production, one plan would be to just put everything critical and production on the one docker VM and let it be. The machine isn't struggling overall but Jellyfin isn't super great because the N150 doesn't quite have the oomph necessary to transcode some of the stuff we have in real time.

Another plan would be to split them onto two machines running docker VMs. This would reduce the churn on critical production and reduce the chances of a change messing things up.

Yet another plan would be to spin up a separate Kubernetes cluster for each, moving right along the overcomplicated continuum.

The thing is, Kubernetes makes sense to me now that I've worked with it in anger a little. I really think for my application it makes sense, and the problems with Nimbus come down to nibbler being flaky and k8s trying to self-heal without enough resources available.

I don't know what to do about external production. My intent was to have it at home out of principle (or maybe out of spite) but it would probably be better to have it in an isolated cloud environment.

The one Docker VM is working ok, but it's mixing failure domains which makes me uncomfortable. For now, things are how they are and I can't let myself worry about it too much.

Links in the footer if you have comments or ideas. I'd love to hear them.

The Year in Agents - AI Engineer Paris

swyx's site RSS Feed — Tue, 23 Sep 2025 17:46:13 GMT

I gave a short address for the first AIE Paris conference organized by Koyeb!

Using FZF and bash to quickly move about

jodavaho.io — Tue, 23 Sep 2025 03:00:00 GMT

Oh look

… another advanced cli utility that is a rewrite + a tiny bit of functionality. And bonus, it’s in Rust!

https://news.ycombinator.com/item?id=45342943

This particular utility, z was also posted last year. It’s part of the “Warp” shell, and one could be forgiven for feeling like a dusty left-behind with all these fancy shell features. Thing is, bash is turing complete. You can do all this with a tiny bit of learning about how to use the shell you have, vs learning an entirely new shell. Plus, you’ll never be lost when you find yourself on another machine that isn’t hyper-customized with your brittle setup!

Basic example

Let’s say you have fzf installed. If you don’t, you’ll probably want it for many reasons. If you refuse, then see below, but this does make things much easier. Try this: cd $(find . -type d | fzf). That will change to a directory using fzf, and honestly it’s quite functional as is. You could make a bash function from that and use it.

Try it with preview! cd $(find . type d | fzf --preview="ls {} -l").

Scale up

Slightly more advanced would be tab-complete for a z-like cd command.

It would work like this:

Make an alias fcd
Make a tab complete that does that for the command fcd

This is kind of 101 bash - just DIY.

Here’s mine:

(2) is the hardest part - just write something that works with complete and fzf. Nowadays this is childs play for any AI to just spit out.

fz_comp()
{
  COMPREPLY=()
  local cur="${COMP_WORDS[COMP_CWORD]}"
  local prev="${COMP_WORDS[COMP_CWORD-1]}"
  local opts="none"
  if [ -z "$cur" ];then
    COMPREPLY=($(find $1 -type d | fzf --preview="ls {} -l"))
    return
  fi
  COMPREPLY=($(find $1 -type d | grep $cur | fzf --preview="ls {} -l"))
}

(1) is just a) set the new command b) make the completion call c) map that call to completion.

alias fcd=cd
_fcd(){ fz_comp $(pwd) }
complete -F _fcd fcd

there you go. Now you can fcd <tab> to get fuzzy search on directories to change to.

Extending

Note this: fz_comp $(pwd)… This sets the current directory ($(pwd)) to be the root of the search. You can pick any directory. I have several aliases like fcd to change to my work root workspace, or my journal, or anywhere else.

Just add many commands like this:

# Change to a subfolder of .logs
alias lcd=cd
_lcd(){ fz_comp ~/.logs }
complete -F _lcd lcd

# Change to a subfolder of .journal
alias jcd=cd
_jcd(){ fz_comp ~/.journal }
complete -F _jcd jcd

etc…

No fzf for idealogical reasons?

OK, so you want to fight about fzf? That’s fine, try it this way (github.com/jodavaho/smartcd) which is pure bash and easily installed. Of course, I wrote it by hand long ago when I was too dumb to use fzf, and it’s in serious need of cleanup, but there you go.

Is a movie prop the ultimate laptop bag?

John Graham-Cumming's blog — Mon, 22 Sep 2025 23:54:25 GMT

A while back I attended a Cloudflare board meeting and carried my laptop in a brown paper bag from a grocery store. This elicited a few comments about needing a real bag. But I've often carried my laptop in inconspicuous bags. For me, the ultimate laptop bag is one that looks nothing like a laptop bag; it should look like nothing special at all.

Unhappily, grocery bags are not very strong, but there is a solution.

Movie prop grocery bags like this:

Looks like a classic brown paper grocery bag, but it isn't. It's actually a silent prop made for a film set where the noise from a crinkling paper bag would overwhelm the dialogue.

Instead of being made from paper, it's made from fabric and is more robust then an actual paper bag. It's also nearly silent (well, much, much quieter than the real thing).

It's also the ideal size for my trusty and vintage MacBook Pro. Especially when it's in its sleeve.

So, is a movie prop the ultimate laptop bag? No, clearly Betteridge's Law of Headlines applies, but it does fit my criteria for discretion and not being ostentatious.

Dear GitHub: no YAML anchors, please

ENOSUCHBLOG — Mon, 22 Sep 2025 03:00:00 GMT

TL;DR: for a very long time, GitHub Actions lacked support for YAML anchors.

I will remember Charlie Kirk

Rondam Ramblings — Thu, 18 Sep 2025 01:00:10 GMT

I don't watch right-wing media personalities much because I find it too painful. For that matter, I don't watch left-wing media personalities much either for the same reason. But the pain in each case arises from very different sources. The right-wing pain comes from being weary of the lies and the distortions and the brazen hypocrisy. The left-wing pain comes from

Why Is Xcode So Antagonistic To Reduced Vision?

The Codist — Thu, 18 Sep 2025 00:31:34 GMT

My eyesight is generally OK, but I have trouble reading small text, especially medium gray on light gray backgrounds. I usually code in 16 or even 18 point fonts, currently Atkinson Hyperlegible Mono. Every new release of Xcode makes seeing various UI elements harder, as if the designers all have

We are starting a company to rethink email

Austin Z. Henley's Blog — Mon, 15 Sep 2025 21:00:01 GMT

https://austinhenley.com/blog/startingacompany.html

In response to a developer asking about systems

Notes on software development — Mon, 15 Sep 2025 03:00:00 GMT

Sometimes I get asked questions that would be more fun to answer in public. All letters are treated as anonymous unless permission is otherwise granted.

Hey [Redacted]! It's great to hear from you. I'm very glad you joined the coffee club and met some good folks. :)

You asked how to learn about systems. A great question! I think I need to start first with what I mean when I say systems.

My definition of systems is all of the underlying software we developers use but are taught not to think about because they are so solid: our compilers and interpreters, our databases, our operating system, our browser, and so on. We think of them as basically not having bugs, we just count on them to be correct and fast enough so we can build the applications that really matter to users.

But 1) some developers do actually have to work on these fundamental blocks (compilers, databases, operating systems, browsers, etc.) and 2) it's not thaaaat hard to get into this development professionally and 3) even if you don't get into it professionally, having a better understanding of these fundamental blocks will make you a better application developer. At least I think so.

To get into systems I think it starts by you just questioning how each layer you build on works. Try building that layer yourself. For example you've probably used a web framework like Rails or Next.js. But you can just go and write that layer yourself too (for education).

And you've probably used Postgres or SQLite or DynamoDB. But you can also just go and write that layer yourself (for education). It's this habit of thinking and digging into the next lower layer that will get you into systems. Basically, not being satisfied with the black box.

I do not think there are many good books on programming in general, and very very few must-read ones, but one that I recommend to everybody is Designing Data Intensive Applications. I think it's best if you read it with a group of people. (My book club will read it in December when the 2nd edition comes out, you should join.) But this book is specific to data obviously and not interested in the fundamentals of other systems things like compilers or operating systems or browsers or so on.

Also, I see getting into this as a long-term thing. Throughout my whole career (almost 11 years now) I definitely always tried to dig into compilers and interpreters, I wrote and blogged about toy implementations a lot. And then 5 years ago I started digging into databases and saw that there was more career potential there. But it still took 4 years until I got my first job as a developer working on a database (the job I currently have).

Things take time to learn and that's ok! You have a long career to look forward to. And if you end up not wanting to dig into this stuff that's totally fine too. I think very few developers actually do. And they still have fine careers.

Anyway, I hope this is at least mildly useful. I hope you join the Software Internals Discord and nycsystems.xyz as well and look forward to seeing you at future coffee clubs!

Cheers,
Phil

I wrote a letter in response to a developer asking about how to learn systems. pic.twitter.com/2ILNpzl662
— Phil Eaton (@eatonphil) September 15, 2025

a few notes on ratelimiting

Tony Finch's blog — Sun, 14 Sep 2025 06:30:44 GMT

Last year I wrote a pair of articles about ratelimiting:

Recently, Chris “cks” Siebenmann has been working on ratelimiting HTTP bots that are hammering his blog. His articles prompted me to write some clarifications, plus a few practical anecdotes about ratelimiting email.

mea culpa

The main reason I wrote the GCRA article was to explain GCRA better without the standard obfuscatory terminology, and to compare GCRA with a non-stupid version of the leaky bucket algorithm. It wasn’t written with my old exponential ratelimiting in mind, so I didn’t match up the vocabulary.

In the exponential ratelimiting article I tried to explain how the different terms correspond to the same ideas, but I botched it by trying to be too abstract. So let’s try again.

parameters

It’s simplest to configure these ratelimiters (leaky bucket, GCRA, exponential) with two parameters:

limit
period

The maximum permitted average rate is calculated from these parameters by dividing them:

rate = limit / period

The period is the time over which client behaviour is averaged, which is also how long it takes for the ratelimiter to forget past behaviour. In my GCRA article I called it the window. Linear ratelimiters (leaky bucket and GCRA) are 100% forgetful after one period; the exponential ratelimiter is 67% forgetful.

The limit does double duty: as well as setting the maximum average rate (measured in requests per period) it sets the maximum size (measured in requests) of a fast burst of requests following a sufficiently long quiet gap.

how bursty

You can increase or decrease the burst limit – while keeping the average rate limit the same – by increasing or decreasing both the limit and the period.

For example, I might set limit = 600 requests per period = 1 hour. If I want to allow the same average rate, but with a smaller burst size, I might set limit = 10 requests per period = 1 minute.

anecdote

When I was looking after email servers, I set ratelimits for departmental mail servers to catch outgoing spam in case of compromised mail accounts or web servers. I sized these limits to a small multiple of the normal traffic so that legitimate mail was not delayed but spam could be stopped fairly quickly.

A typical setting was 200/hour, which is enough for a medium-sized department. (As a rule of thumb, expect people to send about 10 messages per day.)

An hourly limit is effective at catching problems quickly during typical working hours, but it can let out a lot of spam over a weekend.

So I would also set a second backstop limit like 1000/day, based on average daily traffic instead of peak hourly traffic. It’s a lower average rate that doesn’t forget bad behaviour so quickly, both of which help with weekend spam.

variable cost

Requests are not always fixed-cost. For example, you might want to count the request size in bytes when ratelimiting bandwidth.

The exponential algorithm calculates the instantaneous rate as

    r_inst = cost / interval

where cost is the size of the request and interval is the time since the previous request relative to the period.

I’ve edited my GCRA algorithm to make the cost of requests more clear. In GCRA a request uses up some part of the client’s window, a nominal time measured in seconds. To convert a request’s size into time spent:

    spend = cost / rate

So the client’s earliest permitted time should be updated like:

    time += cost * period / limit

(In constrained implementations the period / limit factor can be precomputed.)

how lenient

When a client has used up its burst limit and is persistently making requests faster than its rate limit, the way the ratelimiter accepts or rejects requests is affected by the way it updates its memory of the client.

For exim’s ratelimit feature I provided two modes called “strict” and “leaky”. There is a third possibility: an intermediate mode which I will call “forgiving”.

The “leaky” mode is most lenient. An over-limit client will have occasional requests accepted at the maximum permitted rate. The rest of its requests will be rejected.

When a request is accepted, all of the client’s state is updated; when a request is rejected, the client’s state is left unchanged.

The lenient leaky mode works for both GCRA and exponential ratelimiting.
In “forgiving” mode, all of a client’s requests are rejected while it is over the ratelimit. As soon as it slows down below the ratelimit its requests will start being accepted.

When a request is accepted, all of the client’s state is updated; when a request is rejected, the client’s time is updated, but (in the exponential ratelimiter) not its measured rate.

The forgiving mode works for both GCRA and exponential ratelimiting.
In “strict” mode, all of a client’s requests are rejected while it is over the ratelimit, and requests continue to be rejected after a client has slowed down depending on how fast it previously was.

When a request is accepted or rejected, both of the client’s time and measured rate are updated.

The strict mode only works for exponential ratelimiting.

I only realised yesterday, from the discussion with cks, how a “forgiving” mode can be useful for the exponential ratelimiter, and how it corresponds to the less-lenient mode of linear leaky bucket and GCRA ratelimiters. (I didn’t describe the less-lenient mode in my GCRA article.)

anecdote

One of the hardest things about ratelimiting email was coming up with a policy that didn’t cause undue strife and unnecessary work.

When other mail servers (like the departmental server in the anecdote above) were sending mail through my relays, it made sense to use “leaky” ratelimiter mode with SMTP 450 temporary rejects. When there was a flood of mail, messages would be delayed and retried automatically.

When their queue size alerts went off, the department admin could take a look and respond as appropriate.

That policy worked fairly well.

However, when the sender was an end-user sending via my MUA message submission servers, they usually were not running software that could gracefully handle an SMTP 450 temporary rejection.

The most difficult cases were the various college and department alumni offices. Many of them would send out annual newsletters, using some dire combination of Microsoft Excel / Word / Outlook mailmerge, operated by someone with limited ability to repair a software failure.

In that situation, SMTP 450 errors broke their mailshots, causing enormous problems for the alumni office and their local IT support. (Not nice to realise I caused such trouble!)

The solution was to configure the ratelimiter in “strict” mode and “freeze” or quarantine over-limit bulk mail from MUAs. The “strict” mode ensured that everything after the initial burst of a spam run was frozen.

When the alert was triggered I inspected a sample of the frozen messages. If they were legitimate newsletters, I could thaw them for delivery and reset the user’s ratelimit. In almost all cases the user would not be disturbed.

If it turned out the user’s account was compromised and used to send spam, then I could get their IT support to help sort it out, and delete the frozen junk from the quarantine.

That policy worked OK: I was the only one who had to deal with my own false positives, and they were tolerably infrequent.

One year of zizmor

ENOSUCHBLOG — Sun, 14 Sep 2025 03:00:00 GMT

This is a dual purpose post: I’ve released zizmor v1.13.0, and zizmor has turned one year old1! It depends on how you count: zizmor’s first commit was roughly ↩

On Christmas Eve, Daniel Stenberg called my code bad

jodavaho.io — Sat, 13 Sep 2025 03:00:00 GMT

I found out today that Daniel Stenberg the guy behind curl, which is the library behind basically every web-enabled tool you use, was nominated developer of the year in Sweden. Now, honestly, this guy works hard, and curl represents the premier F/OSS experience. It’s free, ubiquitous, actively maintained, and essentially vital infrastructure.

So, here’s a story about how he called my code bad.

Christmas eve, 2021, I had a little time off and was enjoying some casual coding. I’ve always loved C over C++, which at the time was my “day job” language.

So there I was nestled in, waiting for dinner finish, likely over at my family’s house listening to the siblings, nephews, etc bicker and play.

I was working my way through the curl book. I even found and submitted a teeny tiny fix! (github link)

My usual way to learn from docs is to “paraphrase” the code examples, re-developing them by hand rather than copy-pasting them.

So, I had done that, but was constantly finding a segfault. In some OSS projects, questions are sometimes welcome on github. It being Christmas eve, I figured I would try to formulate a well-structured question and check in after a few days.

Of course, nearly immediately, two curl maintainers, including Daniel Stenberg himself, pointed out my dumb mistake. You can read for yourself, or see this reply:

I’m not proud - that was bad code. But, it happens to us all, and C++ poisons your respect for api types with implicit type coercion, which is why I try to work only in C or Rust, but that is a conversation for another day.

What 4chan is up to since Charlie Kirk was murdered

Caseysoftware — Fri, 12 Sep 2025 18:11:59 GMT

Disclosure: I’m not involved in 4chan in any way but I have deep fear appreciation for the weaponized autists. What…

The post What 4chan is up to since Charlie Kirk was murdered appeared first on Caseysoftware.

first-class merges and cover letters

Tony Finch's blog — Thu, 11 Sep 2025 04:26:47 GMT

Although it looks really good, I have not yet tried the Jujutsu (jj) version control system, mainly because it’s not yet clearly superior to Magit. But I have been following jj discussions with great interest.

One of the things that jj has not yet tackled is how to do better than git refs / branches / tags. As I underestand it, jj currently has something like Mercurial bookmarks, which are more like raw git ref plumbing than a high-level porcelain feature. In particular, jj lacks signed or annotated tags, and it doesn’t have branch names that always automatically refer to the tip.

This is clearly a temporary state of affairs because jj is still incomplete and under development and these gaps are going to be filled. But the discussions have led me to think about how git’s branches are unsatisfactory, and what could be done to improve them.

branch

One of the huge improvements in git compared to Subversion was git’s support for merges. Subversion proudly advertised its support for lightweight branches, but a branch is not very useful if you can’t merge it: an un-mergeable branch is not a tool you can use to help with work-in-progress development.

The point of this anecdote is to illustrate that rather than trying to make branches better, we should try to make merges better and branches will get better as a consequence.

Let’s consider a few common workflows and how git makes them all unsatisfactory in various ways. Skip to cover letters and previous branch below where I eventually get to the point.

merge

A basic merge workflow is,

create a feature branch
hack, hack, review, hack, approve
merge back to the trunk

The main problem is when it comes to the merge, there may be conflicts due to concurrent work on the trunk.

Git encourages you to resolve conflicts while creating the merge commit, which tends to bypass the normal review process. Git also gives you an ugly useless canned commit message for merges, that hides what you did to resolve the conflicts.

If the feature branch is a linear record of the work then it can be cluttered with commits to address comments from reviewers and to fix mistakes. Some people like an accurate record of the history, but others prefer the repository to contain clean logical changes that will make sense in years to come, keeping the clutter in the code review system.

rebase

A rebase-oriented workflow deals with the problems of the merge workflow but introduces new problems.

Primarily, rebasing is intended to produce a tidy logical commit history. And when a feature branch is rebased onto the trunk before it is merged, a simple fast-forward check makes it trivial to verify that the merge will be clean (whether it uses separate merge commit or directly fast-forwards the trunk).

However, it’s hard to compare the state of the feature branch before and after the rebase. The current and previous tips of the branch (amongst other clutter) are recorded in the reflog of the person who did the rebase, but they can’t share their reflog. A force-push erases the previous branch from the server.

Git forges sometimes make it possible to compare a branch before and after a rebase, but it’s usually very inconvenient, which makes it hard to see if review comments have been addressed. And a reviewer can’t fetch past versions of the branch from the server to review them locally.

You can mitigate these problems by adding commits in --autosquash format, and delay rebasing until just before merge. However that reintroduces the problem of merge conflicts: if the autosquash doesn’t apply cleanly the branch should have another round of review to make sure the conflicts were resolved OK.

squash

When the trunk consists of a sequence of merge commits, the --first-parent log is very uninformative.

A common way to make the history of the trunk more informative, and deal with the problems of cluttered feature branches and poor rebase support, is to squash the feature branch into a single commit on the trunk instead of mergeing.

This encourages merge requests to be roughly the size of one commit, which is arguably a good thing. However, it can be uncomfortably confining for larger features, or cause extra busy-work co-ordinating changes across multiple merge requests.

And squashed feature branches have the same merge conflict problem as rebase --autosquash.

fork

Feature branches can’t always be short-lived. In the past I have maintained local hacks that were used in production but were not (not yet?) suitable to submit upstream.

I have tried keeping a stack of these local patches on a git branch that gets rebased onto each upstream release. With this setup the problem of reviewing successive versions of a merge request becomes the bigger problem of keeping track of how the stack of patches evolved over longer periods of time.

cover letters

Cover letters are common in the email patch workflow that predates git, and they are supported by git format-patch. Github and other forges have a webby version of the cover letter: the message that starts off a pull request or merge request.

In git, cover letters are second-class citizens: they aren’t stored in the repository. But many of the problems I outlined above have neat solutions if cover letters become first-class citizens, with a Jujutsu twist.

A first-class cover letter starts off as a prototype for a merge request, and becomes the eventual merge commit.

Instead of unhelpful auto-generated merge commits, you get helpful and informative messages. No extra work is needed since we’re already writing cover letters.

Good merge commit messages make good --first-parent logs.
The cover letter subject line works as a branch name. No more need to invent filename-compatible branch names!

Jujutsu doesn’t make you name branches, giving them random names instead. It shows the subject line of the topmost commit as a reminder of what the branch is for. If there’s an explicit cover letter the subject line will be a better summary of the branch as a whole.

I often find the last commit on a branch is some post-feature cleanup, and that kind of commit has a subject line that is never a good summary of its feature branch.
As a prototype for the merge commit, the cover letter can contain the resolution of all the merge conflicts in a way that can be shared and reviewed.

In Jujutsu, where conflicts are first class, the cover letter commit can contain unresolved conflicts: you don’t have to clean them up when creating the merge, you can leave that job until later.

If you can share a prototype of your merge commit, then it becomes possible for your collaborators to review any merge conflicts and how you resolved them.

To distinguish a cover letter from a merge commit object, a cover letter object has a “target” header which is a special kind of parent header. A cover letter also has a normal parent commit header that refers to earlier commits in the feature branch. The target is what will become the first parent of the eventual merge commit.

previous branch

The other ingredient is to add a “previous branch” header, another special kind of parent commit header. The previous branch header refers to an older version of the cover letter and, transitively, an older version of the whole feature branch.

Typically the previous branch header will match the last shared version of the branch, i.e. the commit hash of the server’s copy of the feature branch.

The previous branch header isn’t changed during normal work on the feature branch. As the branch is revised and rebased, the commit hash of the cover letter will change fairly frequently. These changes are recorded in git’s reflog or jj’s oplog, but not in the “previous branch” chain.

You can use the previous branch chain to examine diffs between versions of the feature branch as a whole. If commits have Gerrit-style or jj-style change-IDs then it’s fairly easy to find and compare previous versions of an individual commit.

The previous branch header supports interdiff code review, or allows you to retain past iterations of a patch series.

workflow

Here are some sketchy notes on how these features might work in practice.

One way to use cover letters is jj-style, where it’s convenient to edit commits that aren’t at the tip of a branch, and easy to reshuffle commits so that a branch has a deliberate narrative.

When you create a new feature branch, it starts off as an empty cover letter with both target and parent pointing at the same commit.
Alternatively, you might start a branch ad hoc, and later cap it with a cover letter.
If this is a small change and rebase + fast-forward is allowed, you can edit the “cover letter” to contain the whole change.
Otherwise, you can hack on the branch any which way. Shuffle the commits that should be part of the merge request so that they occur before the cover letter, and edit the cover letter to summarize the preceding commits.
When you first push the branch, there’s (still) no need to give it a name: the server can see that this is (probably) going to be a new merge request because the top commit has a target branch and its change-ID doesn’t match an existing merge request.
Also when you push, your client automatically creates a new instance of your cover letter, adding a “previous branch” header to indicate that the old version was shared. The commits on the branch that were pushed are now immutable; rebases and edits affect the new version of the branch.
During review there will typically be multiple iterations of the branch to address feedback. The chain of previous branch headers allows reviewers to see how commits were changed to address feedback, interdiff style.
The branch can be merged when the target header matches the current trunk and there are no conflicts left to resolve.

When the time comes to merge the branch, there are several options:

For a merge workflow, the cover letter is used to make a new commit on the trunk, changing the target header into the first parent commit, and dropping the previous branch header.
Or, if you like to preserve more history, the previous branch chain can be retained.
Or you can drop the cover letter and fast foward the branch on to the trunk.
Or you can squash the branch on to the trunk, using the cover letter as the commit message.

questions

This is a fairly rough idea: I’m sure that some of the details won’t work in practice without a lot of careful work on compatibility and deployability.

Do the new commit headers (“target” and “previous branch”) need to be headers?
What are the compatibility issues with adding new headers that refer to other commits?
How would a server handle a push of an unnamed branch? How could someone else pull a copy of it?
How feasible is it to use cover letter subject lines instead of branch names?
The previous branch header is doing a similar job to a remote tracking branch. Is there an opportunity to simplify how we keep a local cache of the server state?

Despite all that, I think something along these lines could make branches / reviews / reworks / merges less awkward. How you merge should me a matter of your project’s preferred style, without interference from technical limitations that force you to trade off one annoyance against another.

There remains a non-technical limitation: I have assumed that contributors are comfortable enough with version control to use a history-editing workflow effectively. I’ve lost all perspective on how hard this is for a newbie to learn; I expect (or hope?) jj makes it much easier than git rebase.

The idiomatic ZFS on Linux quickstart cheat-sheet

The NeoSmart Files — Wed, 10 Sep 2025 23:10:18 GMT

I’m a FreeBSD guy that has had a long, serious, and very much monogamous relationship with ZFS. I experimented with Solaris 9 to learn about ZFS, adopted OpenSolaris (2008?) back in the “aughts” for my first ZFS server, transitioned my … Continue reading →

The post The idiomatic ZFS on Linux quickstart cheat-sheet first appeared on The NeoSmart Files.

A simple clustering and replication solution for Postgres

Notes on software development — Wed, 10 Sep 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Cognition: The Devin is in the Details

swyx's site RSS Feed — Mon, 08 Sep 2025 18:26:44 GMT

My thesis for the future of software dev agents. This is a hastily written blogpost done on not a lot of sleep, so pardon poor pacing and structure and typos and mistakes but just getting it out there.

Ex-X

John Graham-Cumming's blog — Sun, 07 Sep 2025 15:07:55 GMT

I did have an X.com (username: 6a6763) account and I got bored of it so I deactivated it today.

The CoPilot productivity paradox

Weblog on marginalia.nu — Sat, 06 Sep 2025 03:00:00 GMT

I’ve been using the CoPilot plugin for IntelliJ on and off for the last few years, and while initially pretty enthusiastic, I’ve come to first disable it and then delete it altogether along with JetBrains’ local AI-completions, and generally felt this has been an improvement in productivity and a reduction of frustration.

CoPilot is pretty good at taking things that are already pretty fast, such as monotonous code transformations like mapping an object to a SQL statement, and then making that even faster.

A brief discussion of the Practical Byzantine Fault Tolerance (PBFT) algorithm

Bytepawn - Marton Trencseni — Sat, 06 Sep 2025 01:00:00 GMT

This article explains how Practical Byzantine Fault Tolerance (PBFT) builds on Lamport’s theoretical consensus algorithms by introducing leaders, quorums, and cryptographic optimizations to make Byzantine consensus practical, contrasts it with Paxos and crash-fault tolerance, and explores its influence on modern blockchain protocols.

LLMs are not really a horse or a model-t

jodavaho.io — Fri, 05 Sep 2025 03:00:00 GMT

This horse/car analogy for LLMs is very weird.

Who’s the engineer here?

Are we sure this isn’t a CEO perspective where they are annoyed things can’t magically go faster and why do we have to feed these dumb engineers?

Have we considered that cars can’t break new ground and require roads to and from the destination?

LLMs have a kind of have a “mental GPS” functionality sometimes: You can always check in with an LLM in “interactive encyclopedia mode” to get a vague sense of where you are and what’s around you, but it can be off on the details quite a bit.

They are kind of like a self-driving car vs a real car: If I need to go somewhere that millions of other people have gone and do so daily, I can get there with zero thought now but I have to pay attention still.

The closest analogy is CNC vs hand milling, honestly.

The problem is that CNC has a ton of up-front design work with computer-aided drafting to make sure the damn thing is fully sketched out to the millimeter. We have no such tool for software. And no LLM is precise to the millimeter so to speak.

AI like we have now is something new. Far short of the promises, but monumentally different than other tools we’ve built. It’s neat. And everyone I know uses it for casual things, so it’s here to stay. It’s impact on engineering will be permanent but possibly not as thorough as promised.

Coloring matched portions with GNU grep, sed and awk

learnbyexample — Fri, 05 Sep 2025 03:00:00 GMT

You might already know how to use the --color option to highlight matched portions with GNU grep. In this post, you'll see how to use ANSI escape sequences to format matched portions with GNU sed and GNU awk.

GNU grep🔗

Consider this sample input file:

$ cat fruits.txt
banana mango cherry pineapple
grape fig apple dragonfruit papaya
watermelon cashew tomato orange
almond lime grapefruit walnut

The output for grep --color -wE '[ago]\w+' fruits.txt is shown below:

See this section from my ebook on GNU grep for more details about this option. ripgrep has a more featured support for color formatting, see this section for an example.

Formatting with ANSI escape sequences🔗

Here are some examples to show how you can format text in the terminal using ANSI escape sequences:

Your choice of formatting goes between \033[ and m. You can use 01 for bold, 03 for italics and 04 for underline. 31 is for the color red, 32 for green and 34 for blue. Multiple formats can be specified by separating the parameters with a semicolon. Using 0 turns off the format (otherwise, it will persist in the current terminal session until turned off).

If you find a file that has accidentally saved such escape sequences, you can use cat -v to identify them.

$ echo 'one (two) three' | grep --color=always '(two)' | cat -v
one ^[[01;31m^[[K(two)^[[m^[[K three

GNU sed🔗

With GNU sed, you'll need to use \o to specify an octal escape sequence. Here's an example:

Here's an example for processing lines bounded by distinct markers:

$ cat blocks.txt
mango
icecream
--start 1--
dragon 1234
unicorn 6789
**end 1**
have a nice day
--start 2--
a b c
apple banana cherry
**end 2**
par,far,mar,tar

GNU awk🔗

With GNU awk, you can embed the ANSI escape sequences in a string similar to the printf example seen earlier.

Here's a field processing example:

$ cat marks.txt 
Dept    Name    Marks
ECE     Raj     53
ECE     Joel    62
EEE     Moi     68
CSE     Surya   81
EEE     Tia     59
ECE     Om      92
CSE     Amy     67

$ cat filter.txt 
ECE 70
EEE 65
CSE 80

Linux CLI ebooks🔗

Check out my ebooks if you are interested in learning more about Linux CLI basics, coreutils, text processing tools like GNU grep, GNU sed, GNU awk, perl and more! You can get them all as a single bundle via leanpub or gumroad.

18XX: A System of Systems

Send More Paramedics — Thu, 04 Sep 2025 11:44:49 GMT

Whereby I talk about 18XX -- the fantastic family of economic tabletop games...

Analytics query goes 6x faster with EDB Postgres Distributed's new analytics engine

Notes on software development — Thu, 04 Sep 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Addendum B: Client Entitlement Setup

Little Green Viper — Wed, 03 Sep 2025 20:07:16 GMT

Once the server is set up, we need to get entitlements working on the client end. What’s the Deal With “Entitlements”? Entitlements are Apple’s name for settings that are “baked into” the app, and signal the hardware to “open certain doors” for the software. Apple requires entitlements to be present, when accessing various technology SDKs ...

Addendum A: Server Setup

Little Green Viper — Tue, 02 Sep 2025 19:32:32 GMT

In this posting, we’ll discuss the server setup for the demo app. The passkey is used in two places: Registration (creating an account), and logging in. After login, the server works in the same way that most of these types of things work. Server Configuration NOTE: The server source files are located in this directory. ...

Baby's first type checker

Austin Z. Henley's Blog — Sun, 31 Aug 2025 21:00:01 GMT

https://austinhenley.com/blog/babytypechecker.html

Lamport's Byzantine Consensus algorithm with Signatures

Bytepawn - Marton Trencseni — Sun, 31 Aug 2025 01:00:00 GMT

This article, following Lamport's work, extends the original Byzantine Generals algorithm from “oral-messages” to signed messages, showing how digital signatures drastically simplify the problem and reduce message complexity, with an implementation given in Python using Flask HTTP nodes.

Loss as the Price of Love

Digital Apocrypha — Fri, 29 Aug 2025 23:21:21 GMT

It is perhaps a truism that to love is to acknowledge the inevitability of loss. Either by time or circumstances, all that you love will be lost. People will change, depart, die, eventually become dust. Nothing lasts, and we will lose all that we love. But still we love anyway. This has been on my … Continue reading

Login Passkey Code Walkthrough

Little Green Viper — Fri, 29 Aug 2025 22:21:48 GMT

Logging in is simpler, but we still have the challenge/response process, as we did with creation. With logging in, the client doesn’t need the server to prepare to create a new account, so it just needs a challenge. What to Look At The login process calls a number of methods within the client and server. ...

The Process of Login

Little Green Viper — Fri, 29 Aug 2025 16:38:17 GMT

Logging into a server, using an existing passkey, is a lot simpler than creating a new account and passkey. Prerequisite You can only login, using a passkey, if one has already been established, through the creation process. It is assumed that the server has properly stored at least the Public Key, and the Credential ID ...

The Jujutsu Effect

Beetle Space — Fri, 29 Aug 2025 10:00:00 GMT

I began using jujutsu only recently. One month in, I was really liking it, but couldn’t find a compelling reason to recommend it. Overall, it wasn’t a transformative experience - just a nicer one.

Or so I thought.

Then I switched to a project that needed pre-commit to work …

Create Passkey Code Walkthrough

Little Green Viper — Thu, 28 Aug 2025 15:14:00 GMT

In our code walkthrough, we’ll look at both the client code (Swift), and the server code (PHP). The creation process is the most code-intensive part of using passkeys. What to Look At In this post, we’ll be looking at some particular methods, within each of the implementation classes. These are: The Client The Server Steps ...

strongly typed?

Tony Finch's blog — Thu, 28 Aug 2025 04:33:05 GMT

What does it mean when someone writes that a programming language is “strongly typed”?

I’ve known for many years that “strongly typed” is a poorly-defined term. Recently I was prompted on Lobsters to explain why it’s hard to understand what someone means when they use the phrase.

I came up with more than five meanings!

how strong?

The various meanings of “strongly typed” are not clearly yes-or-no. Some developers like to argue that these kinds of integrity checks must be completely perfect or else they are entirely worthless. Charitably (it took me a while to think of a polite way to phrase this), that betrays a lack of engineering maturity.

Software engineers, like any engineers, have to create working systems from imperfect materials. To do so, we must understand what guarantees we can rely on, where our mistakes can be caught early, where we need to establish processes to catch mistakes, how we can control the consequences of our mistakes, and how to remediate when somethng breaks because of a mistake that wasn’t caught.

strong how?

So, what are the ways that a programming language can be strongly or weakly typed? In what ways are real programming languages “mid”?

Statically typed as opposed to dynamically typed?

Many languages have a mixture of the two, such as run time polymorphism in OO languages (e.g. Java), or gradual type systems for dynamic languages (e.g. TypeScript).
Sound static type system?

It’s common for static type systems to be deliberately unsound, such as covariant subtyping in arrays or functions (Java, again). Gradual type systems migh have gaping holes for usability reasons (TypeScript, again).

And some type systems might be unsound due to bugs. (There are a few of these in Rust.)

Unsoundness isn’t a disaster, if a programmer won’t cause it without being aware of the risk.

For example: in Lean you can write “sorry” as a kind of “to do” annotation that deliberately breaks soundness; and Idris 2 has type-in-type so it accepts Girard’s paradox.
Type safe at run time?

Most languages have facilities for deliberately bypassing type safety, with an “unsafe” library module or “unsafe” language features, or things that are harder to spot. It can be more or less difficult to break type safety in ways that the programmer or language designer did not intend.

JavaScript and Lua are very safe, treating type safety failures as security vulnerabilities. Java and Rust have controlled unsafety. In C everything is unsafe.
Fewer weird implicit coercions?

There isn’t a total order here: for instance, C has implicit bool/int coercions, Rust does not; Rust has implicit deref, C does not.

There’s a huge range in how much coercions are a convenience or a source of bugs. For example, the PHP and JavaScript == operators are made entirely of WAT, but at least you can use === instead.
How fancy is the type system?

To what degree can you model properties of your program as types? Is it convenient to parse, not validate? Is the Curry-Howard correspondance something you can put into practice? Or is it only capable of describing the physical layout of data?

There are probably other meanings, e.g. I have seen “strongly typed” used to mean that runtime representations are abstract (you can’t see the underlying bytes); or in the past it sometimes meant a language with a heavy type annotation burden (as a mischaracterization of static type checking).

how to type

So, when you write (with your keyboard) the phrase “strongly typed”, delete it, and come up with a more precise description of what you really mean. The desiderata above are partly overlapping, sometimes partly orthogonal. Some of them you might care about, some of them not.

But please try to communicate where you draw the line and how fuzzy your line is.

Snark, Ironic Detachment, Authenticity

Weblog on marginalia.nu — Thu, 28 Aug 2025 03:00:00 GMT

How you engage with the world changes how you experience the world, and how the world experiences you.

A snarky and cynical approach, by its default assumption that things are shit, or if they are not yet shit will inevitably turn to; such an approach will give your world a malodorous brownish tint.

Granted, snark gives you plausible deniability, a motte-and-bailey that protects you from direct criticism, encountering backlash you can always backpedal and say it was just a joke that you accidentally took a bit too far.

searchcode.com is being rebooted (maybe?)

Ben E. C. Boyter — Thu, 28 Aug 2025 03:00:00 GMT

searchcode HAS been rebooted see https://boyter.org/posts/searchcode-has-been-rebooted/ for details.

By the time you are reading this, searchcode.com’s DNS should have been updated to point at a placeholder index.html file I created.

In short I am shutting down the current iteration of searchcode.com, unless something drastic happens in the next few days.

The reasons for this… are few, but worth exploring.

While I have often described searchcode as a fun forever project, big enough to be interesting, but small enough to be done by one person, I probably should have clarified that this was true so long as one or two of the following things held true,

The Process of Creation

Little Green Viper — Wed, 27 Aug 2025 22:47:13 GMT

In order to use a passkey, the client needs to create the key, and coordinate with the server. We should note that this is a VERY BASIC EXAMPLE. It’s quite possible to get a lot more involved, and a lot more secure. Multi-Step Process There are a few steps that we take, in creating an ...

Connect Four game with a twist

learnbyexample — Tue, 26 Aug 2025 03:00:00 GMT

From wikipedia: Connect Four:

Connect Four is a game in which the players choose a color and then take turns dropping colored tokens into a six-row, seven-column vertically suspended grid. The pieces fall straight down, occupying the lowest available space within the column. The objective of the game is to be the first to form a horizontal, vertical, or diagonal line of four of one's own tokens.

As a twist, this TUI implementation also offers two more variations of the game:

form a square, i.e. four cells forming 90 degree angles and equidistant from each other
form a line or square

Installation🔗

This app is available on PyPI as connectsquare. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install connectsquare

# launch the app
$ connectsquare

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias connectsquare='/path/to/textual_apps/bin/connectsquare'

As an alternative to manually managing such virtual environments, you can use uv or pipx instead.

As yet another alternative, you can install textual==0.85.2 (see Textual documentation for more details), clone this repository and run the connect_square.py file.

Screenshots🔗

Adjust your terminal's dimension for the game widgets to appear properly, for example 80x30 (characters x lines). Sample screenshots are shown below:

Guide🔗

Press the n key to start a new game. Existing game, if any, will be abandoned
You can choose between Connect Four, Connect Square (default) and Both types of game
You can choose between Easy (default), Medium and Hard difficulty modes:
- In the Easy mode, the AI will make a random move
- In the Medium mode, the AI will make a random move based on certain weight calculations
- In the Hard mode, the AI will make the best move based on the weight calculations (the algorithm is based only on the current board state and thus it is not impossible for the user to win)
The first move is based on the User first (default) and AI first choices
Only the bottom most empty cell of each column will be considered as a valid move
Press the t key to toggle between light and dark themes
Press the q key to quit the app

User moves are denoted by the ⭕️ character and AI moves are denoted by the ✖️ character.

The text panel under the game board displays the current status of the game. If the game ends with one of the players forming a valid line or square, the cells forming the winning move will be highlighted.

Square tic tac toe🔗

If you liked this game, you might also enjoy Square tic tac toe.

If you are interested in learning more about the AI algorithm for the Connect Square game, check out my explanation here for Square tic tac toe — while there are a few differences between the two, the foundation is the same.

A Closer Look At Passkeys

Little Green Viper — Tue, 26 Aug 2025 00:21:19 GMT

This series will focus on demonstrating a simple client (iOS) and server (PHP/MySQL) implementation of the most basic type of passkey. This app has already been written, so we’ll look at it, and walk through its operation, in order to learn about implementing passkeys in an iOS app. Our guide through this journey, will be ...

Stop Polluting Context - Let Users Disable Individual MCP Tools

smcleod.net — Mon, 25 Aug 2025 18:10:00 GMT

If you’re building MCP servers, you should be adding the ability to disable individual tools.

NixOS PostgreSQL Major Version Upgrade

Kevin Cox's Blog — Sun, 24 Aug 2025 17:55:00 GMT

PostgreSQL major version upgrades is something that NixOS currently doesn’t handle for you. This is my process, documented mostly so that I can remember it next time I want to upgrade.

Configuration

I use the standard NixOS module for Postgres. My main config looks like this:

{ lib, pkgs, ... }: {
	services.postgresql = {
		enable = true;
		package = pkgs.postgresql_17;
		initdbArgs = [
			"--encoding=UTF8"
			"--no-locale"
		];
		ensureUsers = [{
			name = "kevincox";
		} {
			name = "root";
		}];
	};

	systemd.services.postgresql = {
		postStart = lib.mkAfter ''
			$PSQL -tAc "ALTER ROLE kevincox with SUPERUSER"
			$PSQL -tAc "ALTER ROLE root with SUPERUSER"
		'';
		serviceConfig = {
			TimeoutStartSec = "30min";
		};
	};
}

Notable features:

I pin the version of Postgres, to ensure that I upgrade when I want to.
I give myself and root superuser.

Preparation

Update your NixOS config but don’t deploy it.
Build the config to make sure everything is ready to go.

Upgrading

Log in as the postgres user. (sudo -su postgres)
Define the old version old=16
Get the old and new versions:
```
cd /var/lib/postgresql/
pg_old=$(nix-build --no-out-link -A postgresql_${old:?} '<nixpkgs>')
pg_new=$(nix-build --no-out-link -A postgresql_$((old+1)) '<nixpkgs>')
```
Note
If you are using extensions you will need them in $pg_old and $pg_new. If you are using the NixOS module you can make a package with nix-build "<nixpkgs/nixos>" -A config.services.postgresql.finalPackage. If you are using colmena you can do colmena eval -E "{nodes, ...}: nodes.mynode.config.services.postgresql.finalPackage".
Initialize the new data directory. Be sure to use the arguments from your config!
```
$pg_new/bin/initdb --encoding=UTF8 --no-locale $((old+1))
```
Copy the configuration (this is really only needed if you have extensions that need to be loaded, the main value we care about is shared_preload_libraries. But copying the whole file is easiest and the NixOS module will overwrite it anyways when starting the new version.).
```
cp -v {${old:?},$((old+1))}/postgresql.conf
```

Run the check

$pg_new/bin/pg_upgrade \
	--old-bindir=$pg_old/bin \
	--new-bindir=$pg_new/bin \
	--old-datadir=/var/lib/postgresql/${old:?} \
	--new-datadir=/var/lib/postgresql/$((old+1)) \
	-j16 \
	--clone \
	--check

Stop the old Postgres. systemctl stop postgresql
Warning
This is the point where we copy the data. If the old Postgres starts running again it will write new data to the old directory that will not be migrated. This will result in data loss. If that happens remove the new state directory and start again.
Run the migration. This is the same command as the check without --check.
Start the new PostgreSQL server. (This is typically done by activating the new configuration such as sudo nixos-rebuild switch.)

At this point the new version is running. Note that rolling back will cause data loss as new changes are being made only to the new data directory.

You may wish to rename the old directory so that if you do accidentally roll back it is more likely that things fail rather than losing data. But if you have insecure apps that let the first user be admin that may be a bad idea. Maybe you could do something like put a file there or a directory owned by root? But I haven’t tested any of these things.

Cleanup

As suggested by pg_migrate run vacuumdb --all --analyze-in-stages.
At some point in the future once you are confident that the new DB is running fine (a few weeks doesn’t hurt) you can remove the old data.
```
cd /var/lib/postgresql/
./delete_old_cluster.sh
rm -v delete_old_cluster.sh
```

Applications of Byzantine Consensus

Bytepawn - Marton Trencseni — Sat, 23 Aug 2025 01:00:00 GMT

This article examines where Lamport's original oral-messages Byzantine Generals algorithm does and does not apply in practice, contrasting unsuitable cases like distributed databases and rocket launches with a suitable case in redundant flight control systems.

Why Passkeys?

Little Green Viper — Thu, 21 Aug 2025 17:45:31 GMT

Security has always been a difficult balance between usability and safety. Passkeys are a way to “bridge” these seemingly opposed concepts. They allow enhanced security between a server and a client, while also removing the need for the end-user to manage login details. The connection is more secure than that provided by the classic “User ...

The long season of langdev

Send More Paramedics — Tue, 19 Aug 2025 15:44:49 GMT

Whereby I talk about programming language development and why, and where it might go...

MCP DevTools

smcleod.net — Sun, 17 Aug 2025 18:10:00 GMT

A single, modular MCP server for AI coding agents.

Faster Index I/O with NVMe SSDs

Weblog on marginalia.nu — Sat, 16 Aug 2025 03:00:00 GMT

The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes.

The index is already fairly large, but can sometimes feel smaller than it is, and paradoxically, query performance is a big part of why. If each query has a budget of 100-250ms, a design that finds and ranks results faster in that time period will produce better search results. There are other limitations as well, query understanding is still somewhat limited, where only minor changes to a query can unearth dozens of new related results.

On waste

bayindirh — Fri, 15 Aug 2025 03:00:00 GMT

Lately, I have been pondering the waste we generate as we live and the ways to reduce it. As I scale down, I fill trash bags with things I can’t use, which bothers me. The items that can be donated and have a second life are strictly excluded from this definition, though. They’re donated to people who need them.

The items that are disposed of include cardboard boxes, plastic containers, and items that have leaked or reached the end of their useful life but cannot be consumed, donated, or revived. Putting things in a trash bag and throwing them out is the least preferred thing I do; however, in reality, some things need to go through this system. The wrapping film used to protect a product from the environment can’t be reused in practice, for example.

It’s imperative to understand that every item has a purpose and useful life, and is rarely useful outside these. While things can be repurposed, achieving this becomes increasingly difficult as time and technology progress. Our current system views products as disposable items, a problem El-Pais recently wrote about. As a result, the products we buy or use have shorter lifespans, and their repairability is exponentially reduced. The discussion of the whys and hows of this phenomenon is worthy of another blog post for another day.

Humans are creatures of comfort and habit. If something brings us carefree comfort, we tend to cling to it until we need to face the consequences of our actions. This has brought us good things in the past and allowed us to improve our lives progressively; however, we have now passed the point where we need to drastically change our habits. This momentum is evident in our response to climate change. Currently, trying to save our planet is a torturous battle.

It’s worth noting that there are other factors driving consumption, which are rooted in the society we live in. Humans are social creatures, as are creatures of habit. We have a tendency to buy the things our peers use, wear, or have. Some of us prefer to express ourselves with material items. This creates another vector of excessive consumption. Even others who don’t prefer to do this are manipulated by mass media to buy things by playing on the emotions of owning things.

Personally, I have begun to view waste as something I need to minimize, both realistically and consciously. This small change had a domino effect on how I perceive consuming and buying things from a bigger perspective. The most significant change I made was reducing disposable items in my life and using items that can be safely and realistically recycled (i.e., glass, metal, paper). This change lends itself effortlessly to buying for life, which advocates for buying better items once and using them for a lifetime.

Buying for a lifetime is not without its own difficulties, though. The first step is to find something that can endure daily life for a long time, especially when everything else is made to be disposable. In the past, the price of an item was a semi-reliable indicator of its longevity. Now, this line has blurred beyond recognition. Next, assuming that the item exists, its reachability becomes a problem. Many of the higher-quality items are made in small quantities and sold locally, making it harder to obtain them than before. The last point is the price of the item in question. When the first two hurdles are cleared, the cost of the item becomes the final obstacle. The price is higher not only because the item is better made, but the prospective buyer needs to bear more of the fixed costs related to the production of the said item. However, when all hurdles are cleared, what remains is an item that can be used for a long time and passed down to future generations. When used for this long, the item not only pays for itself, but carries and accumulates worth beyond money. If you pardon the term, it becomes enchanted.

While this is not an easy to path to tread, and sometimes there’s no path, I’m happy and content by trying to reduce what I buy and use them for longer, because having long-lasting items and using them every day without thinking about how to deal with them is akin to wearing a familiar sweater at home and being comfortable.

And, I like being comfortable, sipping a tea, and remembering good times. It beats trying to keep up with imaginary beings, damaging the environment, and lining somebody else’s pockets big time.

Until next time,

Be kind.

Fun with finite state transducers

ENOSUCHBLOG — Thu, 14 Aug 2025 03:00:00 GMT

I recently1 solved an interesting problem inside zizmor with a type of state machine/automaton I hadn’t used before: a finite state transducer (FST). A few months ago. ↩

Set up a single-node EDB Postgres Distributed cluster on Ubuntu

Notes on software development — Thu, 14 Aug 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Windows XP as the future of AI companies

jodavaho.io — Wed, 13 Aug 2025 03:00:00 GMT

The Endgame for LLMs Might Look Like Windows XP Hear me out.

1. The commercial space already pays for heavy, local tools

Mechanical engineers pay thousands of dollars per seat for their productivity software. Those profits go directly into improving the tools. Sometimes there are cloud add-ons, but the bulk of the functionality is local and offline. The feature sets are deep: automated FEA, rendering, parts databases. You run this on powerful machines with serious GPUs, fast CPUs, and large amounts of memory.

Graphics artists do the same with rendering engines. The money funds better pipelines, shaders, SDKs for in-house work. All run locally on heavy hardware.

Electrical engineers buy Matlab, Simulink, and other specialized packages. Industrial engineers rely on AutoCAD, LabVIEW, and so on.

Across disciplines, the pattern is the same:

Hardware intensive to run
Indispensable for professionals
Purchased by companies that see them as essential to success

In some cases, like Excel/Word, work software becomes the defacto home software as well.

However …

2. Running LLMs as a service is expensive

LLM companies like OpenAI, Anthropic, and others face two big problems:

The core tech is very costly to run at scale for end users.
Building and maintaining the surrounding infrastructure adds even more cost.

I cannot make these arguments as well as this article does

This limits how cheap and open ended a hosted LLM service can realistically be. TFA above implies $100,000/y cost to run an AI agent!

3. A better model: license LLMs as installable software

LLMs aren’t Model Ts replacing horses. They’re the internal combustion engine, a transformative core technology that powers many different products, but isn’t a product by itself. Even a chat interface is a product built on top of the engine. So, there’s really two markets here: LLMs and the LLM-enabled produts.

AI companies are trying to capture LLM market and all the product market by licensing LLM access. Fine, but they’re doing it by also building the hardware the product runs on. This is fine for a text editor and catastrophic for a massively intense, insanely popular LLM.

In the future, Anthropic, OpenAI, Google, and others could:

Sell custom-made LLMs directly for customer use
Build a range of software products that run on them
Separate revenue streams for core tech and end-user tools

Imagine deleting the massive inference backend and instead selling a license for “the world’s best coding LLM” that you install and run locally. Claude Code already works fine as a concept. Many tools already support OpenAI style APIs. Consumer hardware can run small or quantized models today, and inference-optimized chips are only getting better. NVIDIA can remain king of training, while others optimize for running models.

You could offer:

Consumer models: smaller, optimized for laptops/desktops
Commercial models: larger, more capable, for professional workstations
OpenSource models: For those folks who don’t want to pay.

The companies already buying engineering grade PCs could easily justify the cost, the app ecosystem doesn’t have to pay the AI tax, and AI companies can stop endless capex and focus on building the best LLMs (and maybe some apps).

4. The Excel analogy

If LLMs are sold as “commercial offerings to power your local tools,” then someone gets to write the LLM equivalent of Word/Excel. Popular in professional environments, but with spillover into home use because people want the same capabilities everywhere.

And given the broad appeal of LLMs, they could end up embedded in every work machine making them as expected as having Excel installed.

5. The likely future: PCs+Phones ship with AI core hw/sw

The endgame could be a coherent “operating system” built around licensed LLMs, used both at work and at home. All your tools like code editors, web apps, productivity software, would run on it. You’d pay for the hardware and the software license, just like today’s PCs and phones. Apps are built with the assumption that the OS includes an LLM module to interface with.

How do I get second order effects from these agents?

jodavaho.io — Tue, 12 Aug 2025 03:00:00 GMT

I use coding agents daily. So far, they are good for first-order effects. Things like “Fast typewriter” (doing something I would do but without interface delays) or “super copy paste” (altering an example and applying it with nuance in several places) or “super tab complete” (suggesting large blocks of text/code vs just single symbols) or even “meta bash” (asking for scripts that do oneoff analysis or processing of a few data sets you have lying around).

Most of these are code-generation tasks. Sure, we can write code as fast as possible, but that doesn’t help the thinking, decision making, and understanding go faster. According to Amdahls law, that’ll become the blocker, and probably already has. Theoretically, giving up on understanding is the natural next step - essentially let agents build whatever they want. Aka vibe coding.

So we’ve already hit the AI agent first-order improvements limit??

What I want is second order effects. I want tooling around my tooling to help my tooling build my tooling so that I can be more productive and directly involved. That’s the only way to ensure that what I want comes to pass.

I want smart-jump 10x, smart-context popups 10x, live execution / simulation, integrated replay of test cases with backtracking, and a million other things I can’t imagine yet. Instead I get a github agent that automates leaving comments about my code style. Thanks.

Anything else is just unproductive laziness and gambling your future job.

A more general implementation of Lamport's Byzantine Consensus algorithm in Python

Bytepawn - Marton Trencseni — Tue, 12 Aug 2025 01:00:00 GMT

This follow-up factors Lamport’s Byzantine Generals algorithm into a generalized class that's independent of I/O and payloads. Any node can initiate a round (be "king"), values can be arbitrary strings, and multiple rounds can run concurrently.

Assessing students in the era of AI

Austin Z. Henley's Blog — Sun, 10 Aug 2025 21:00:01 GMT

https://austinhenley.com/blog/aihomework.html

It was time for a dim bulb current limiter

John Graham-Cumming's blog — Sun, 10 Aug 2025 20:21:04 GMT

One of my Minitels (the one that I modified a few years ago to run new firmware) started having power supply problems: the LED was on but nothing else was working. I suspected that the main power/CRT board needed new capacitors and so I recapped it:

Alas, despite some of the older capacitors measuring poorly compared to their specs, that wasn't the problem and so I need to go deeper. The only problem is there are a ton of nasty voltages on this board. Notably the 230V AC input and then the tens of kilovolts generated for the CRT.

Also the board seems to have a short somewhere.

So, it's time to a dim bulb current limiter. Which I didn't have. So I made one. Here it is:

It's a remarkably simple device which plugs into a standard power outlet (230V AC) and puts a lightbulb in series with one of the connections. The two wires coming in then go to a standard socket. Thus you plug your device (e.g. my Minitel) into the socket and the lightbulb limits the current.

These things were super common in the past for anyone working with mains voltages.

Works great if you have an incandescent bulb at, say, 100W. A 100W bulb on 230V AC would have a current running through it of I = V/P -> I = 100/230 (approximately 430mA). Limiting the current that way means you're less likely to damage the device you're working on if there's a short. Naturally, using a different bulb would give you a different current. 60W would give you around 230mA. 150W around 650mA.

As the Minitel's input fuse is 630mA, my current limiter will keep things well under its blowing current. Also, if there's a complete short then the bulb will glow brightly.

Here's a look inside:

Power comes in at the top, and goes to a DPST switch. I'm using DPST because it's not easy to know which input wire is live and which is neutral because the European Schuko/Type F plug/socket is not polarized and I like knowing that when I switch something off, it's off! And the DPST has an integrated lamp that gives me a visual reminder that it's on.

Because local power sockets aren't polarised, the power strip I am using for my bench gear uses DPST switches for each socket. It's relatively hard to find one that does this (and says it does). Thanks, Brennenstuhl!

But I digress.

To make my dim bulb current limiter do the right thing, I need a 100W incandescent bulb. These have been on their way to oblivion in the EU since 2009 and were, I believe, eliminated for home use in 2016. It is still possible to buy them, labelled for "industrial use", from some locations.

So, that limits the current and protects the device I'm working on, but what about protecting me? For that, I pair it with an isolating transformer. If you're not familiar with this then here's the basics. In a house the neutral line is connected to ground (typically at the point at which power comes into your house). This means that the 230V coming out the live wire is relative to ground. If I touch it the power will flow through me to ground.

The isolating transformer eliminates the link between live, neutral and ground. It's still outputting 230V but that voltage isn't relative to ground and so I won't get a shock if I touch one of the wires coming from the transformer's socket. Of course, touching both would send 230V through me.

So, having built that I can go back to understanding why my Minitel isn't powering on.

Becoming a Student Again

Ben Overmyer's Site — Sat, 09 Aug 2025 03:00:00 GMT

Early this year, I started applying to universities in Canada for master's degree programs. In March, I was accepted to the Master of Software Engineering program at Memorial University in St. John's, Newfoundland.

From that point forward, we began the planning process for a major life shift. In early July, we gave notice at our jobs, and our long journey to move to Canada became mostly public knowledge. I say "mostly" because I have been very quiet about it online. Until it was a real, for-sure thing and we wouldn't have to pivot, I didn't want to make it known more widely.

Well, on August 8 we crossed the border in Maine and formally got my study permit and Sarah's work permit. We are now officially temporary residents of Canada. Unless something catastrophic happens, we'll be here at least until I finish my degree, which means 2027 at the earliest.

So. That means I will be a university student for the first time since 2007, when I got my bachelor's degree in history at South Dakota State University. This time around, I will be a graduate student.

I'm really excited. I get to reinvent my life and maybe, depending on how I like the academic life, start a new career. At the very least, we get to escape the growing darkness in the United States.

Rigging-suspended installation of a marine wind generator

Henri Bergius — Sat, 09 Aug 2025 03:00:00 GMT

We cruise on a small boat, a 31ft double-ender. As we’re off-grid the vast majority of the time, all electricity needs to be produced from renewable sources. Solar produces a lion’s share, but other sources are needed for overcast days. We don’t have space for a permanently mounted wind generator, so we converted a Superwind 350 to rigging-suspended.

Why wind?

As our 2024 cruise to the not-so-sunny Scotland demonstrated, there would still be place for wind power in the renewables mix of a long-distance sailboat. My energy production simulations from 2023 also showed a lot of promise for wind power.

Our boat has a canoe stern and dual aft stays, meaning that there is not much space in the back of the boat. We had a conversation with Superwind back in 2022, and they were of the opinion that there simply isn’t a good space for installing one. And so we installed a hydrogenerator and decided that we’d go sailing if we ran out of power.

But the interest in wind generators remained.

And then one day, rowing around the Sainte-Anne anchorage in rainy Martinique, I saw a potential solution: one of the small cruising boats had a wind generator suspended from ropes in their foretriangle. I chatted a bit with the owners, and they confirmed that the system worked nicely. Time for some research!

Rigging-suspended wind generator

Rigging-suspended wind generators used to be common. Commercial models included the Ampair 100, WindBugger, and the Hamilton-Ferris. As solar power has become cheaper, wind generators in general have fallen out of favor. At the same time cruising boats have grown in size, enabling permanent mounting of a big wind turbine.

These effects combined mean that there are currently no commercial manufacturers of rigging-suspended wind generators for boats.

Of the models once manufactured, the Ampair 100 sounded especially promising. It was a modular system that could be used either as a rigging-suspended wind turbine, or as a “tow generator” for making power while under sail.

This modularity is a big advantage of a rigging-suspended wind generator, especially for ease of stowing. They can also be a lot quieter than the pole-mounted ones, as any vibrations are dampened by the suspension ropes. And of course they don’t cause any windage or shading while stowed.

I tried finding an Ampair for sale online with no luck. The second-hand chandlery in Grenada – Treasure Trove – had two units, but couldn’t locate the wind blades.

However, the wind generator market has evolved quite a bit. There are several good wind generators intended for permanent mounting. Superwind and the D400 provide the best alternatives, but are very expensive. On the cheap end, there are numerous Chinese wind generators from companies like Pikasola and Vevor starting at around $250.

Maybe I could design a bracket to convert one of these for rigging-suspended installation?

Building the bracket

Sitting in the windy anchorage at Spanish Water this idea started sounding more and more interesting. After some paper brainstorming, I grabbed FreeCAD and made an initial design.

The design parameters were:

Can be built somewhat cheaply by a local metal fabricator
Can facilitate the most common fixed-mount wind generators
Poles to keep the rigging lines clear of the propellers
Wind generator is held in place and the whole assembly turns into the wind

My original idea was a neat stainless ring around the wind generator body. However, different wind generators are of different height, and so in interests of both manufacturing cost and adaptability, I went with two brackets connected by threaded rod.

It took a couple of months to actually get a quote from a local fabricator, but now we finally have the finished brackets for ourselves and a neighboring boat.

You can find the FreeCAD file on GitHub. There are also STEP files for the top bracket and the bottom bracket.

Installation

Our neighbor installed a 400W Pikasola wind generator on theirs. That mounted on the bracket without any other adapting except for some rubber mat to isolate the stainless parts from the aluminium wind turbine body.

We had bought an old Superwind 350 from another boat, and so for us a small connecting piece (140mm long aluminium pipe with 55mm inner diameter) was needed to make that fit.

The wind generator is hoisted using our spinnaker pole topping lift, with a short strop riding on the inner forestay. Stabilization is with a three rope bridle connected to pad eyes on the deck.

We have wires running from the bottom bracket to the deck level, where they connect via MC4 connectors to to cables running to inside the boat. This way we can easily disconnect the wind turbine as needed. We are adding a stop/run switch soon as well to aid deployment.

Deployment is already documented in our boat handbook.

Conclusion

We only got the Superwind deployed yesterday evening, and so we are gathering the early experiences. However, right now we’re on track to producing about 0.6kWh on the first day. This is measured with a dedicated Victron SmartShunt wired to the wind generator regulator and logging into our time series database.

That 0.6kWh per day is like a whole second solar arch!

Noise levels are not too bad at all. Inside the boat you can’t hear anything. In cockpit, you can hear a slight whirr from the generator, but it is a lot quieter than one of the popular pole-mounted wind turbines on neighboring boat, heard from few hundred meters away.

Durability and handling of heavier winds will remain to be seen. As will the practicality of stowing and deploying when changing anchorages. Though we already do similar things with the mast-hoisted solar panels and the nesting dinghy.

Especially when going with one of the cheap Chinese models, this rigging-suspended method can be the way to add wind power to a boat in an affordable way. We calculated the total price for the Pikasola installation to be around the same as what marine wind generator manufacturers ask for just a mounting pole!

The hardware design should be quite easy to manufacture anywhere where you can find a stainless steel welder. After all, we were able to get ours fabricated on a tropical island.

For us the new wind generator can be seen as completing the circle of our deployable renewable options:

When under sail, power is generated with the hydrogenerator
When anchored in light winds, power is generated with the mast-hoisted FLINsail solar array
When anchored in heavier winds, power is generated by the rigging-suspended Superwind

On top of these we have the fixed solar panels.

What even is distributed systems

Notes on software development — Sat, 09 Aug 2025 03:00:00 GMT

Distributed systems is simply the study of interactions between processes. Every two interacting processes form a distributed system, whether they are on the same host or not. Distributed systems create new challenges (compared to single-process systems) in terms of correctness (i.e. consistency), reliability, and performance (i.e. latency and throughput).

The best way to learn about the principles and fundamentals of distributed systems is to 1) read Designing Data Intensive Applications and 2) read through the papers and follow the notes in the MIT Distributed Systems course.

For Designing Data Intensive Applications (DDIA), I strongly encourage you to find buddies at work or online who will read it through with you. You can also always join the Software Internals Discord's #distsys channel to ask questions as you go. But it's still best if you have some partners to go through the book with, even if they are as new to it as you.

I also used to think that you might want to wait a few years into your career before reading DDIA but when you have friends to read it with I think you need not wait.

If you have only skimmed the book you should definitely go back and give it a thorough read. I have read it three times already and I will read it again as part of the Software Internals Book Club next year after the 2nd Edition is published.

Keep in mind that every chapter of DDIA provides references to papers you can keep reading should you end up memorizing DDIA itself.

When you've read parts of DDIA or the MIT Distributed Systems course and you want practice, the Fly.io x Jepsen Distributed Systems Challenge is one guided option. Other options might include simply implementing (getting progressively more complex down the list):

two-phase commit
three-phase commit
single-decree Paxos
highly available key-value store on top of a 3rd-party consensus library
chain replication (or CRAQ), using a 3rd-party consensus library
Raft
epaxos

And if you get bored there you can see Alex Miller's Data Replication Design Spectrum for more ideas and variants.

And if you want more people to follow, check out the Distributed Systems section of my favorite blogs page.

If these projects and papers sound arcane or intimidating, know that you will see the problems these projects/papers solve whether or not you know and understand these solutions. Developers often end up reinventing hacky versions of these which are more likely to have subtle bugs.

While instead you can recognize and use one of these well-known building blocks. Or at least have the background to better reason about correctness should you be in a situation where you must work with a novel distributed system or you end up designing a new one yourself.

And again, if you want folks to bounce ideas off of or ask questions to, I strongly encourage you to join the Software Internals Discord and ask there!

I wrote a short post on learning the fundamentals of distributed systems, with a few suggested resources to read and a few suggested projects to try. pic.twitter.com/b0EhDP8K0t
— Phil Eaton (@eatonphil) August 9, 2025

p-fast trie, but smaller

Tony Finch's blog — Wed, 06 Aug 2025 20:20:42 GMT

Previously, I wrote some sketchy ideas for what I call a p-fast trie, which is basically a wide fan-out variant of an x-fast trie. It allows you to find the longest matching prefix or nearest predecessor or successor of a query string in a set of names in O(log k) cache misses, where k is the key length.

My initial sketch was more complicated and greedy for space than necessary, so here’s a simplified revision.

(“p” now stands for prefix.)

layout

A p-fast trie stores a lexicographically ordered set of names.

A name is a sequence of characters from some small-ish character set. For example, DNS names can be represented as a set of about 50 letters, digits, punctuation and escape characters, usually one per byte of name. Names that are arbitrary bit strings can be split into chunks of 6 bits to make a set of 64 characters.

Every unique prefix of every name is added to a hash table.

An entry in the hash table contains:

A shared reference to the closest name lexicographically greater than or equal to the prefix.

Multiple hash table entries will refer to the same name. A reference to a name might instead be a reference to a leaf object containing the name.
The length of the prefix.

To save space, each prefix is not stored separately, but implied by the combination of the closest name and prefix length.
A bitmap with one bit per possible character, corresponding to the next character after this prefix.

For every other prefix that matches this prefix and is one character longer than this prefix, a bit is set in the bitmap corresponding to the last character of the longer prefix.

search

The basic algorithm is a longest-prefix match.

Look up the query string in the hash table. If there’s a match, great, done.

Otherwise proceed by binary chop on the length of the query string.

If the prefix isn’t in the hash table, reduce the prefix length and search again. (If the empty prefix isn’t in the hash table then there are no names to find.)

If the prefix is in the hash table, check the next character of the query string in the bitmap. If its bit is set, increase the prefix length and search again.

Otherwise, this prefix is the answer.

predecessor

Instead of putting leaf objects in a linked list, we can use a more complicated search algorithm to find names lexicographically closest to the query string. It’s tricky because a longest-prefix match can land in the wrong branch of the implicit trie. Here’s an outline of a predecessor search; successor requires more thought.

During the binary chop, when we find a prefix in the hash table, compare the complete query string against the complete name that the hash table entry refers to (the closest name greater than or equal to the common prefix).

If the name is greater than the query string we’re in the wrong branch of the trie, so reduce the length of the prefix and search again.

Otherwise search the set bits in the bitmap for one corresponding to the greatest character less than the query string’s next character; if there is one remember it and the prefix length. This will be the top of the sub-trie containing the predecessor, unless we find a longer match.

If the next character’s bit is set in the bitmap, continue searching with a longer prefix, else stop.

When the binary chop has finished, we need to walk down the predecessor sub-trie to find its greatest leaf. This must be done one character at a time – there’s no shortcut.

thoughts

In my previous note I wondered how the number of search steps in a p-fast trie compares to a qp-trie.

I have some old numbers measuring the average depth of binary, 4-bit, 5-bit, 6-bit and 4-bit, 5-bit, dns qp-trie variants. A DNS-trie varies between 7 and 15 deep on average, depending on the data set. The number of steps for a search matches the depth for exact-match lookups, and is up to twice the depth for predecessor searches.

A p-fast trie is at most 9 hash table probes for DNS names, and unlikely to be more than 7. I didn’t record the average length of names in my benchmark data sets, but I guess they would be 8–32 characters, meaning 3–5 probes. Which is far fewer than a qp-trie, though I suspect a hash table probe takes more time than chasing a qp-trie pointer. (But this kind of guesstimate is notoriously likely to be wrong!)

However, a predecessor search might need 30 probes to walk down the p-fast trie, which I think suggests a linked list of leaf objects is a better option.

My 2025 New Mac Setup

swyx's site RSS Feed — Wed, 06 Aug 2025 18:27:30 GMT

I set up a new Mac for work today. Here's everything I use on a Mac for fullstack web development.

Losslessly Changing Video Framerate

Kevin Cox's Blog — Wed, 06 Aug 2025 03:05:00 GMT

I’ve been taking a few timelapse videos recently and as an ammeter I’ve just been using the default phone camera apps. Google Camera defaults to “Auto” for the speedup, which keeps the resulting video between 15 and 30s no matter how long you record for (with minimum of 5x speed). It also offers explicit 5, 10, 30 and 120x speeds. The iPhone forgoes explicit options and only offers an auto mode with 20-40s output target and a 20x minimum speed.

Both of these apps have a limitation that they can only output 30 fps video. The seems like a missed opportunity since the faster than real time playback already tends to produce jumpy video. Getting 60 fps (or even 120 fps) makes a huge improvement to smoothness. The hardware can easily support these output framerates as the capture framerate will still be lower than regular speed videos.

My preferred approach is to record the video at “twice the duration” as I actually want, then to speed it up after the fact. Recording at a higher input framerate always gives you more flexibility as you can always speed it up later, and if you carefully pick your input framerate you can do this losslessly to avoid degrading the video quality.

For example, if I want to end up with a 1 minute 60 fps video covering 1 hour of real time. I need to record at $60\u{s} \times 60\u{Hz} \div 1\u{h}$ which is 1 fps. Google Camera doesn’t show you the framerate directly, but you can easily convert the input speeds to framerates. Since it always expects to output 30 fps the calculation is just $30\u{fps} \div \v{speed}$ , so to capture at 1 fps I need to select 30x speed.

To losslessly convert the video you can use ffmpeg bitstream filters.

ffmpeg
	-i input-video.mp4 # Input file.
	-c copy # Don't transcode, remain lossless.
	-r 60 # Output framerate, this is mostly informational for the whole-video metadata.
	-fps_mode vfr # Don't drop frames to match the expected framerate, just preserve all input frames.
	-bsf:v "setts=ts=PTS/2" # Update the presentation time to 1/2 the previous one (play twice as fast).
	output.mp4 # Output file.

This can easily be adjusted to other framerates by changing the -r and -bsf:v values.

Revisiting the Kingdoms of Stone and Fire

Ben Overmyer's Site — Tue, 05 Aug 2025 03:00:00 GMT

I haven't touched tabletop role-playing games in a few months now. In preparation for my upcoming move to Canada, I sold most of my collection. I didn't expect to even think about these games until well after the move.

Then, I found Typst.

Typst is a computer language for laying out books and documents. It's similar in concept to LaTeX, a common tool for academics. In fact, I have been using LaTeX to lay out all of my role-playing games for several years now. However, despite its ubiquity, LaTeX is pretty terrible to work with. It's slow and hard to extend. So, when I saw that a newcomer to the field of layout tools was getting rave reviews on Hacker News, I thought I would give it a try.

So, I chose to rework Vox Draconis: Kingdoms of Stone and Fire into Typst instead of LaTeX. The result was awesome, and I adore the toolset. It was much easier to use.

What I didn't expect as a result of this experiment, though, was that it rekindled my interest in working on Kingdoms of Stone and Fire.

With the final death of Silver Gryphon Games last year, I did not expect to ever pick up game design again. Now I'm seeing all kinds of possibilities. I think my mistake was attempting to make a commercially-viable game. All of the different disciplines of marketing, finances, and so forth pulled me away from the thing I actually liked to do - design.

So. The PDF of Kingdoms of Stone and Fire that is on this site now is what it looks like after the conversion to Typst. It's far from complete, though.

The game was originally meant to be an updated version of my original retroclone, Vox Draconis. I think my first mention of it on this blog was back in April of 2021. Since then, I barely talked about it. I suppose this is as good an opportunity as any to explain what the game is about and what I'm trying to do with it.

Vox Draconis: Kingdoms of Stone and Fire is a tabletop role-playing game set in a fantasy, pre-iron world filled with dinosaurs, dragons, and warring kingdoms. It's anachronistic and whimsical. While it takes some cues from classic swords-and-sorcery, it's less about fearing the dark and more about carving out a place in the world. It's closer to "noblebright" than it is to "grimdark."

The rules are meant to be simple but not entirely narrative-driven. Many rules-light games out there, like Powered by the Apocalypse, are intended to just give a framework for a story. Kingdoms of Stone and Fire is meant to power a story, yes, but it's also meant to be an interesting game rules-wise. People who enjoy exploring mechanics and game progression should find it fun, if I design it right.

I also want to design the game so that it fits into an hour or two of playtime instead of the four or five hours that some games take. This might be the bigger challenge, and I have a lot of thinking to do here. It has to be more than just combat rules that are interesting and worth spending time playing. As it exists now, it's not there.

Finally, I want the game rules to be relatively short. Too many role-playing games have manuals that are hundreds of pages long. Kingdoms should be short enough that it can be understood and started in half an hour, but have enough content that it can be explored for months or years, if desired. That's also going to be difficult to pull off. I suspect that the crux will be making the main rulebook short and concise, and providing the rest via supplemental works.

p-fast trie: lexically ordered hash map

Tony Finch's blog — Mon, 04 Aug 2025 23:52:21 GMT

Here’s a sketch of an idea that might or might not be a good idea. Dunno if it’s similar to something already described in the literature – if you know of something, please let me know via the links in the footer!

The gist is to throw away the tree and interior pointers from a qp-trie. Instead, the p-fast trie is stored using a hash map organized into stratified levels, where each level corresponds to a prefix of the key.

Exact-match lookups are normal O(1) hash map lookups. Predecessor / successor searches use binary chop on the length of the key. Where a qp-trie search is O(k), where k is the length of the key, a p-fast trie search is O(log k).

This smaller O(log k) bound is why I call it a “p-fast trie” by analogy with the x-fast trie, which has O(log log N) query time. (The “p” is for popcount.) I’m not sure if this asymptotic improvement is likely to be effective in practice; see my thoughts towards the end of this note.

layout

A p-fast trie consists of:

Leaf objects, each of which has a name.

Each leaf object refers to its successor forming a circular linked list. (The last leaf refers to the first.)

Multiple interior nodes refer to each leaf object.
A hash map containing every (strict) prefix of every name in the trie. Each prefix maps to a unique interior node.

Names are treated as bit strings split into chunks of (say) 6 bits, and prefixes are whole numbers of chunks.
An interior node contains a (1<<6) == 64 wide bitmap with a bit set for each chunk where prefix+chunk matches a key.

Following the bitmap is a popcount-compressed array of references to the leaf objects that are the closest predecessor of the corresponding prefix+chunk key.

Prefixes are strictly shorter than names so that we can avoid having to represent non-values after the end of a name, and so that it’s OK if one name is a prefix of another.

The size of chunks and bitmaps might change; 6 is a guess that I expect will work OK. For restricted alphabets you can use something like my DNS trie name preparation trick to squash 8-bit chunks into sub-64-wide bitmaps.

In Rust where cross-references are problematic, there might have to be a hash map that owns the leaf objects, so that the p-fast trie can refer to them by name. Or use a pool allocator and refer to leaf objects by numerical index.

search

To search, start by splitting the query string at its end into prefix + final chunk of bits. Look up the prefix in the hash map and check the chunk’s bit in the bitmap. If it’s set, you can return the corresponding leaf object because it’s either an exact match or the nearest predecessor.

If it isn’t found, and you want the predecessor or successor, continue with a binary chop on the length of the query string.

Look up the chopped prefix in the hash map. The next chunk is the chunk of bits in the query string immediately after the prefix.

If the prefix is present and the next chunk’s bit is set, remember the chunk’s leaf pointer, make the prefix longer, and try again.

If the prefix is present and the next chunk’s bit is not set and there’s a lesser bit that is set, return the leaf pointer for the lesser bit. Otherwise make the prefix shorter and try again.

If the prefix isn’t present, make the prefix shorter and try again.

When the binary chop bottoms out, return the longest-matching leaf you remembered.

The leaf’s key and successor bracket the query string.

modify

When inserting a name, all its prefixes must be added to the hash map from longest to shortest. At the point where it finds that the prefix already exists, the insertion routine needs to walk down the (implicit) tree of successor keys, updating pointers that refer to the new leaf’s predecessor so they refer to the new leaf instead.

Similarly, when deleting a name, remove every prefix from longest to shortest from the hash map where they only refer to this leaf. At the point where the prefix has sibling nodes, walk down the (implicit) tree of successor keys, updating pointers that refer to the deleted leaf so they refer to its predecessor instead.

I can’t “just” use a concurrent hash map and expect these algorithms to be thread-safe, because they require multiple changes to the hashmaps. I wonder if the search routine can detect when the hash map is modified underneath it and retry.

thoughts

It isn’t obvious how a p-fast trie might compare to a qp-trie in practice.

A p-fast trie will use a lot more memory than a qp-trie because it requires far more interior nodes. They need to exist so that the random-access binary chop knows whether to shorten or lengthen the prefix.

To avoid wasting space the hash map keys should refer to names in leaf objects, instead of making lots of copies. This is probably tricky to get right.

In a qp-trie the costly part of the lookup is less than O(k) because non-branching interior nodes are omitted. How does that compare to a p-fast trie’s O(log k)?

Exact matches in a p-fast trie are just a hash map lookup. If they are worth optimizing then a qp-trie could also be augmented with a hash map.

Many steps of a qp-trie search are checking short prefixes of the key near the root of the tree, which should be well cached. By contrast, a p-fast trie search will typically skip short prefixes and instead bounce around longer prefixes, which suggests its cache behaviour won’t be so friendly.

A qp-trie predecessor/successor search requires two traversals, one to find the common prefix of the key and another to find the prefix’s predecessor/successor. A p-fast trie requires only one.

IaCConf 2025: Terraform in Depth QA and Fireside Chat

tedious ramblings — Fri, 01 Aug 2025 17:00:00 GMT

At DevOpsDays Chicago this year I found myself answering a lot of questions in random hallway talks, which inspired me to have an entire session at IaCConf this year devoted to answering questions from the audience. These questions and more are answered in my talk, but if you have your own questions not touched on...

Stack traces for Postgres errors with backtrace_functions

Notes on software development — Thu, 31 Jul 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

IaCConf 2025: Automate All the Things Panel Discussion

tedious ramblings — Wed, 30 Jul 2025 17:00:00 GMT

I was lucky enough to participate in this panel at IaCConf on AI and ML in the world of Infrastructure as Code. In the spirit of using AI tools to make my life easier, here's an AI generated summary of the panel. This panel discussion explores the impact of AI and ML on infrastructure as...

Lamport's Byzantine Generals algorithm in Python

Bytepawn - Marton Trencseni — Wed, 30 Jul 2025 01:00:00 GMT

I discuss Lamport's Byzantine Generals problem and why it requires a total of N≥3M+1 nodes with M malicious nodes, and then implement the solution using Python with HTTP Flask servers.

Punch Line review

ブラック — Tue, 29 Jul 2025 03:00:00 GMT

A punch line without the set-up.

It’s frankly amazing how little effort PQube has put into this release while asking 49.99€ for it. The Steam version is shipped broken. As soon as you launch the game from the launcher, Steam will mark Punch Line as not running and thus will not update your gameplay time, won’t show you the Steam Overlay, and most importantly, will not allow you to use Steam Input. Seeing as I was trying to play this with my DualSense controller, not having my controller working at all was quite the issue.

Thankfully there is a workaround for this: open the game installation directory, remove the MAGESgamelauncher.exe executable, make a copy of the Game.exe executable and rename that copy to MAGESgamelauncher.exe. Congratulations, you can now use the Steam overlay, Steam Input and you will be correctly marked as playing Punch Line on your profile.

I’m guessing that this issue stems from the launcher launching the game executable as a completely separate process instead of a child process, so Steam never hooks the overlay/Steam Input into the Game.exe process and thus breaks everything. I’m also guessing that PQube has absolutely no quality assurance process since this should be the thing you notice immediately. Punch Line also used to have achievements on Steam – except those achievements never worked, so they “fixed” the issue by just removing them entirely. And they have the gall to ask 49.99€ for this release.

Once you’ve managed to actually get the game working, what you have is not that great. Ostensibly Punch Line is a visual novel with puzzle elements, but the puzzle elements in the game are so poor, annoying and unrewarding that the game would be better as a straight visual novel. There’s not really any way to figure out the puzzles, so you’ll just end up taking wild guesses and then doing a reset if you guessed wrong. There isn’t even any kind of fun interactions for non-progressing choices like there is for example in the Somnium segments in Uchikoshi’s AI – The Somnium Files games. The game is also around 12 hours long, and the annoying puzzle segments stop appearing around eight hours in – really for the game’s benefit.

Besides the obvious launcher issue, there are a bunch of smaller issues that pop up while playing through the game. While it never prevented me from finishing the game, the default controller layout seems to be broken (at least on a DualSense controller), with buttons that should do something not doing it. The background audio sometimes eclipses dialogue audio, even with the default volume settings. And perhaps the most annoying issue: unskippable anime-style opening and ending themes that you will have to watch about twenty times each. It’s possible that these smaller issues could be fixed by tweaking the settings, but I don’t want to spend all of my time fixing this game on PQube’s behalf.

Visually the game looks about okay – the character visuals are nice, but the environmental details leave room for improvement and the video segments, despite this having a tie-in anime, are kinda bad. I think they’ve had to retime the animations for the dialogue and it ends up looking more awkward than I would’ve imagined. At least during the final hour of the game, which is almost entirely video, the quality is at least decent. Thankfully there are no issues to point out on the music and voice acting side. Well, beyond the volume mixing issue noted earlier that is.

Really the only highlight of Punch Line is the story. It’s definitely not Uchikoshi’s finest work, but there’s a fairly interesting story with likeable characters. A lot more lighthearted than a lot Uchikoshi works too, considering how much of it revolves around panties. The writing also does a good job of resolving all of the mysteries that it sets up, so you won’t be left with nagging questions at the end.

I actually really liked the Punch Line anime when I watched it over a decade ago, and it made me want to play through the game. But since it took several years for the game to actually come out in the west, I kinda forgot about the whole thing and only got to it now that the anime’s already over ten years old. Perhaps that was for the best, since I’d already forgotten a lot of the details about it, and most of the story is the exact same as in the anime. Really what the game has going for it over the anime is that there’s a different (and better) ending, and the story is more expanded, as the game has about double the runtime.

But alas, it’s really hard to recommend going through the story through the game rather than by watching the anime. Punch Line is a good story hidden in a bad game. There’s just way too many issues with this game and they will never ever be resolved. If you absolutely insist on playing the game instead of watching the anime, buy this on a deep discount. PQube simply does not deserve the full price of admission for this mess.

IaCConf 2025: Taste Testing Tofu

tedious ramblings — Mon, 28 Jul 2025 20:33:30 GMT

As someone who’s spent years diving into the intricacies of Terraform and OpenTofu, I’ve seen the chaos that comes with “yolo deploying” (rushing changes without verification, only to face outages, compatibility snags, or unexpected bugs). My recent presentation at the first ever Infrastructure as Code Conference highlights a critical shift every team should embrace: treating testing as a non-negotiable cornerstone of infrastructure development.

Getting the KIM-1 to talk to my Mac

John Graham-Cumming's blog — Mon, 28 Jul 2025 17:28:23 GMT

I've written before about my 1976 KIM-1 and code I wrote for a similar one long ago. But I hadn't done much with the KIM-1 and strongly believe that old hardware need to be living machines not still lives. But using the KIM-1 directly (via its keypad and little hexadecimal display) is painful. I wanted to be able to use it with a terminal.

Luckily, the KIM-1 was designed to interface to a terminal: a Model 33 ASR Teletype. Teletypes use a type of interface which is unusual to most people (unless you work with industrial machinery where it's pretty common). That interface is a current loop.

A current loop uses current (often 20mA) rather than voltage (such as the voltage levels seen in common serial interfaces). Prior to the creation of RS-232, current loops were very common. Current loops are still used in industrial settings partly because they have good noise immunity, can go for long distances, and are simple and reliable.

In the KIM-1 manual there's a picture of the way its current loop works when connected to a Teletype:

When the KIM-1 wants to send data to the Teletype it uses pins S and U of one of its edge connectors. S is permanently connected to +5V via a 150 ohm resistor. When the KIM-1 receives data from the Teletype's keyboard it uses pins R and T. Just like S, R is permanently connected to +5V. The important thing to note is that the KIM-1 is responsible for the current: the Teletype is entirely "dumb". This will matter later in this blog post.

I happened to have lying around a kit to make an adapter between RS-232 and the Siemens SIMATIC S5 current loop.

A simple kit to build. Since USB to RS-232 adapters are easy to find I'd be able to communicate with the KIM-1 using it. But first check the circuit diagram of this adapter. There's a small problem on the receive side.

As is pretty common with current loop applications, optoisolators are used. OK1 handles when the device connected to the RS-232 transmits something. And you can see from the diagram, it expects to be getting 5V, 20mA in on its pin 5 and then out again on pin 4. So, that'll work perfectly for the "keyboard" connection to the KIM-1 (pins R and T).

But the receive side is a different story. It's handled by OK2 and it's expecting 5V, 20mA on pin 1, but pin 2 goes to GND. So, there's no "loop" back to the device. This won't work with the KIM-1 without a small modification.

I built the kit and and tested it against the KIM-1 like this:

Looks messy but it validated the little modification I needed to make and that I could talk to the KIM-1. Here it is talking at 300 baud using the RS-232 to USB adapter seen above.

So, what's the modification? Well, pin 2 of OK2 goes to ground and we don't want it to, we want it to go the KIM-1's pin U. There are two ways to do this. I could have cut the track on the back of the board that goes from pin 2 to ground, and soldered a wire on the back.

But this kit came with all the chips socketed so I took an even simpler route. Bend pin 2 away from the socket and solder directly to it.

And that worked fine. The kit also came with a nice little box and so the final adapter looks neat.

The grey wire goes to the KIM-1, the white wire is a USB-A cable that I'm using to power the adapter and the KIM-1 itself.

The KIM-1 will happily communicate as 1200 baud with 7 bit ASCII and 2 stop bits. I'm using minicom to communicate with it. The main thing is to set the delete key to send DEL instead of BS since the KIM-1 needs to receive a DEL on start up so that it can measure the Mac's transmission speed.

One of the Teletype functions available is to load a program from punched tape. I don't have an actual Teletype but I can emulate the punched tape format and that'll allow me to upload programs easily.

The format itself is not complicated.

What Is A Good Programmer?

The Codist — Sun, 27 Jul 2025 18:11:00 GMT

Am I a good programmer? The short answer is: I don’t know what that means.

I have been programming for 52 years now, having started in a public high school class in 1973, which is pretty rare because few high schools offered such an opportunity back then. I

Becoming a High Taste Tester

swyx's site RSS Feed — Fri, 25 Jul 2025 11:03:40 GMT

theres a specific reason why i'm writing this post i can't disclose yet. but am sharing my prep work in public

Performance lessons of implementing lbzcat in Rust

Linux Engineer's random thoughts — Thu, 24 Jul 2025 01:00:00 GMT

This was originally published as a thread on Mastodon.

For fun I implemented lbzip2's lbzcat (parallel bzip2 decompression) clone in rust, using the bzip2 crate.

Baseline: bzip2 vs bzip2-rs

First, let's look at the baseline. The Trifecta Tech Foundation recently claimed that bzip2-rs had faster decompression than libbzip2's original C …

Ode to the anti-Mac: Panasonic Toughbook FZ-40

John Graham-Cumming's blog — Mon, 21 Jul 2025 14:25:57 GMT

I own two computers that I use regularly. The first is my main machine, a rather old Apple MacBook Pro, the other is a Panasonic Toughbook FZ-40. They are polar opposites. Where the Mac is designed with tightly coupled software and hardware and is all smooth curves and delicate design, the Toughbook is designed to be used wearing gloves, outside and in rough situations that would damage the Mac.

Its built-in handle is a sign that this thing is heavy and robust.

And where the MacBook has a feeble port selection, my Toughbook has VGA, Serial, two Ethernet sockets, HDMI, a micro SD card, SIM card, USB-A (SS5 and SS10), head and microphone jack, Thunderbolt and USB-C, and built in GPS. There's also a smart card reader.

It feels like it's designed to do stuff where connecting to the real world is involved.

The camera has a physical cover and all the ports are protected by locking doors.

There are also some special Panasonic buttons (P1 through P4). P2 drops you into "Concealed Mode" which instantly cuts all the light coming from the machine.

P3 drops you into "Night Mode" for all those moments where your submarine is being hunted by the Soviets and you need to preserve your night vision to be able to see the instruments ready. Sadly there's no "one ping" option.

The machine runs Windows 11 (which I don't love) but a lot of industrial software is only available on Windows (along with things like the LEGO MINDSTORMS software which was weak on the Mac and full-featured on Windows).

I may be the only person in the world programming LEGO on a Toughbook with the option to drop into Night Mode during, uh, LEGO emergencies. But, hey, LEGO worked for Iron Man.

And then there's the red F11 key with a raised bump around it to prevent accidental activation.

I Don't Like Imports

Kevin Cox's Blog — Mon, 21 Jul 2025 01:15:00 GMT

I think this is an unusual opinion, so I thought I would share.

I prefer not to import external symbols into the local scope. For example, I prefer:

let body = reqwest::get("https://kevincox.ca/feed.atom").await?.text().await?;

over

use reqwest::get; // At the top of the file.

// Inside some function.
let body = get("https://kevincox.ca/feed.atom").await?.text().await?;

Unambiguous

There are various reasons for this, but all the most important ones boil down to being unambiguous. When I am reading code I see library::Foo and know that it comes from library. I see crate::Foo and know it comes from elsewhere in my code and if I see just Foo I know it is from the current file (and possibly private, so I could be a bit more careful with how I use the API). I don’t need to guess or lookup what Foo I am working with. (And when I do inevitably guess I don’t need to have that background process asking “did I guess wrong?”)

I find when I am working on codebases where the convention is to import things I am very frequently tracking down the definition of some type or function as I don’t know whether it is local or from which library. This is especially painful when reading or reviewing code in a web interface. If you are reading a patch the imports are likely not expanded (if they didn’t change) and even if you have the whole file expanded you need to check if the referenced value is the global import from the top of the file or some local shadow.

Copy + Paste

Another thing I love about fully qualified names is that I can just copy and paste code around or move it from file to file without needing to figure out which imports are needed in the new file (and inevitably importing the wrong Error at least a few times). For the most part code in other files and examples online can just be pasted into the file and work.

It’s not some dramatic productivity boon, but it certainly makes cleanups and refactors much smoother and a frustrating tedious step that is necessary when working with import-heavy code.

But The Names Are So Long!

In some cases they are. But more often than not this is just a poor choice by the library author (in my opinion). For example in Rust the standard hash-map is std::collections::HashMap. But even that is just an alias. If you want to work on an Entry you are now typing std::collections::hash_map::Entry. I see no reason it couldn’t have been std::HashMap and std::HashMapEntry or std::HashMap::Entry. (I don’t mind a small amount of nesting when it is very strongly associated, but still don’t really see much point.)

I much prefer C++‘s style where almost everything is just under std. For example std::unordered_map. An iterator is slightly nested at std::unordered_map::iterator but if you ask me that is still a pretty reasonable name. And I would much rather see that in code than wonder what iterator we are working with.

If it hurts stop hitting yourself, don’t find a way to numb the pain.

Conflicts

I see this come up as an argument against, and I am pretty dumbfounded. Some people see that you have std::io::Error and std::fmt::Error. So they can’t both be std::Error. But that is kind of my point! When I see code referencing Error I don’t know which of error type from the dozens libraries that the code might be using it is! That is what makes the {library}::{name} pattern so great, it is conflict free. If I use that throughout my code I never have to worry if Write is std::fmt::Write, std::io::Write or crate::Write because it is spelt out! I would much rather the types be actually named std::FmtWrite and have everyone “agree” on the name (because they have no choice) than half the time having it called Write and half the time having it called FmtWrite, WriteFmt or DebugWrite (because some other Write was in scope and the programmer needed to come up with their own name).

This isn’t an imagined problem. I regularly find myself reading rustdoc documentation and needing to hover over a type name so that my browser displays the fully qualified path in the link target! I feel that which Error or which Write is being used is important enough that it be immediately visible, in both code and generated documentation.

Avoiding imports means that the library authors are responsible for solving conflicts. And we are back to the fact that when I see a type or function it is unambiguous and I don’t have to waste brain cycles figuring out what Error I am looking at.

But Sometimes They Are Really Long

I’m not a stickler to this rule. There are some cases where I will use imports. Sometimes the names are just too long and repetitive. In that case I strongly prefer a local use with a short scope. This ensures that I don’t need to look far to resolve the ambiguity.

use sentry::integrations::tracing::EventFilter;
match *meta.level() {
	tracing::Level::ERROR => EventFilter::Event,
	tracing::Level::WARN => {
		if module.starts_with("hickory_proto::") {
			EventFilter::Ignore
		} else {
			EventFilter::Event
		}
	}
	tracing::Level::INFO => EventFilter::Ignore,
	tracing::Level::DEBUG => EventFilter::Ignore,
	tracing::Level::TRACE => EventFilter::Ignore,
}

(And more often than not when I run into this case it isn’t because the name is intrinsically high-entropy, but because someone got carried away with hierarchy. Stop hitting yourself.)

Rust-Specific Notes

I’ve been using Rust as an example but been trying to avoid being Rust-specific. But it is my primary language, so I do have some thoughts to share. If you don’t care about Rust you can stop reading now.

Avoiding imports in Rust is going against the grain. I would recommend that all crates provide a flat API. However, for many projects avoiding use isn’t the right choice. Just follow the common coding convention and be happy. To see a real-world example of both a flat API and avoiding use you can check out rl-core which is a small project where I am quite glad I chose to break convention. A lot of my “for-me” or “internal” code is written in a similar style.

Provide a Flat API

By default, Rust wants to ship your file structure as your exports (Conway’s law). So for example mylib::Foo needs to be defined in src/lib.rs and mylib::submodule::Bar needs to be defined in src/submodule.rs or src/submodule/mod.rs.

With a single re-export per module this can be solved!

mod config; pub use config::*;
mod tracker; pub use tracker::*;

One extra statement to free you from hierarchy. Re-exports are well-supported by Rust so rustdoc and the compiler will pick mylib::Config as the path to show users rather than mylib::config::Config. (I don’t know the exact heuristics but when working like this there will be a single public export, so there isn’t much cleverness needed.)

pub use also only sets the maximum export visibility. This means that pub, pub(crate) and private things in the module will have the expected visibility. So the definitions in the files look the same as they normally would and the export lines only need to be added when you add or remove modules.

The only real downside of this approach is the src/lib.rs will see everything as local and can reference Config rather than crate::Config. But this is a very minor issue and the solution is not to put anything into lib.rs. (Keeping only module definitions and re-exports in lib.rs is a very common pattern anyways.)

The nice thing about just re-exporting everything is that it also frees you from deciding where to put things based on what you want the exported name to be. You don’t need to put std::collections::hash_map::HashMap and std::collections::hash_map::Entry in the same file to have them appear at the same place, you can put them into separate files if you wish or the same file if you need access to non-public APIs. (The Rust std implementation does actually re-export these from various places, re-exports are very common and not something to be feared.)

Personally I just put every single type and most functions into separate files. I find that it keeps everything simple and organized and I prefer switching between different files than different parts of a file when going back and forth. So for example in rl-core I define rl_core::Config in src/config.rs and rl_core::Tracker in lib/tracker.rs.

Traits

Traits are where things start to get hairy. To call a trait method as a method the trait needs to be in scope. So if you want to call foo.a_method() where a_method is somelib::TheType::a_method you need to use somelib::TheType.

However you always have the option to call the method using a fully-qualified path. For example somelib::TheType::a_method(&foo) or <Foo as somelib::TheType>::static_func(). For the occasional method call I often do this because I do like seeing where .a_method() comes from, it isn’t always clear if it is a method directly on foo or which trait defined it. However nesting a few calls with this quickly becomes messy and sometimes I make a practical choice to just use the use. I prefer to do this just for a short scope to help keep the origin information close to the code, but for very common traits I will sometimes add a global import like use std::io::Write as _ (make the trait methods available but don’t actually pull the name into scope).

`std`

The elephant in the room is that in Rust std (and core) love love love hierarchy and lots of names are super long. I wish I could have a long talk with whoever decided it was really important that std::sync::atomic really needed to be nested under sync. I think we would have been fine to reserve the top level std::atomic for these operations even though they are synchronization related.

It is what it is. I’m not going to advocate for The Great std Naming Reform even if I would prefer it. We have a pattern, and we can deal with it. Most of the names aren’t so long that they need to be imported so most of the time I just spell them out. Occasionally I find myself repeating a long name frequently and just import it. I have yet to be stricken by a bolt of lightning from the heavens.

Pimping my Casio: Part Deux

John Graham-Cumming's blog — Sat, 19 Jul 2025 16:05:59 GMT

Close to three years ago I wrote about using Oddly Specific Objects' alternate "motherboard" to modify a classic Casio F-91W watch: Pimping my Casio with Oddly Specific Objects' alternate motherboard and firmware. That blog post goes into the detail of swapping out the guts of the Casio and building and uploading the firmware.

Happily, Oddly Specific Objects is back with a "Pro" version of their alternative Casio internals which now features an accelerometer and an alternative LCD. The original Sensor Watch used the existing Casio LCD display but the custom LCD allows for more complicated text to be displayed on screen. It's not a dot-matrix display so there are lots of limitations but it's still a fun upgrade.

Also, the Sensor Watch Pro, as it's called, requires no soldering (unlike the original version). Here's one I prepared earlier (that is a Casio F-91W with new internals):

You can see the new display in action there showing the day as FRI. As with the original Sensor Watch there's a browser-based emulator that uses emscripten to get the watch running on your computer. This is pretty important because flashing new firmware to the watch requires dismantling it.

Since there's no soldering the process of upgrading your Casio is simply take it apart and put back in the parts that came in the kit (along with a small piece of metal that's transferred from the original Casio watch to make the battery connection):

Since I bought the optional accelerometer add on it's necessary to put in place a little piece of Kapton tape (you can see if covering the five pads on the image above) and the insert the accelerometer board itself:

The five vertical traces at the top are the micro USB interface used for flashing the firmware.

I modified the firmware to remove Imperial units and 12 hour clock (which I never want) and to choose my own selection of screens (I added the tally, accelerometer and light sensor screens).

The complete sequence of commands needed to build the firmware was:

brew install --cask gcc-arm-embedded
git clone https://github.com/joeycastillo/second-movement
cd second-movement
git submodule update --init --recursive
make BOARD=sensorwatch_pro DISPLAY=custom

And then to upload to the watch you connect the micro USB, click the tiny switch on the back twice very rapidly and a volume called WATCHBOOT appears on your machine. You can copy the firmware over to the watch with make install.

To run the emulator you build with emscripten:

brew install emscripten
emmake make BOARD=sensorwatch_pro DISPLAY=custom
python3 -m http.server -d build-sim

And then visit localhost:8000/firmware.html in your browser.

Migrated gemlog content into this site

Ben Overmyer's Site — Thu, 17 Jul 2025 03:00:00 GMT

I'm in the process of sunsetting my gemlog. I liked the idea of the Gemini protocol, but it never really grew beyond a specific set of niche topics. Further, it's just easier to write in Markdown, which is what I use for writing this site.

As such, I've migrated all of my gemlog posts into this site.

Here's a list of them:

Agentic Coding Adoption Cost Cycle

smcleod.net — Tue, 15 Jul 2025 18:10:00 GMT

The two common themes I see with engineers adopting agentic coding tools

Megadimension Neptunia VII review

ブラック — Tue, 15 Jul 2025 03:00:00 GMT

Not another Victory.

When I started playing through Megadimension Neptunia VII, I thought that this was Neptunia finally reaching a level of effort and polish never before seen in the franchise. Character sprites were improved, there were levels that weren’t just recycled from the previous games, enemies were voiced and so on. Unfortunately, despite the seeming jump in quality, I started noticing a problem: the game wasn’t fun to play.

You know how in RPG games the game decides to take away your party members for plot reasons, leaving your party understaffed and making those characters underleveled when they rejoin your party? And how that is annoying as all hell? Well, Compile Heart decided that this annoying mechanic was worth making into an entire game. It took like 17 hours for my party to be at full fighting capacity, and they don’t even stick around after that. You really don’t want to get attached to any single party member because the chances are that they’re going to be missing for most of the story.

It also feels like they’ve completely forgotten about basic game design that previous titles in the Neptunia franchise understood. Things like having save points before a boss fight. Oh, you thought to challenge the boss after only 25 minutes of grinding? Time to lose your 25 minutes worth of progression. There’s also a point where you can get your character’s health too low for a boss fight, with no possibility to heal. Hopefully you didn’t overwrite your previous save, since you’ll have to reload at that point too. Compile Heart really has no considerations towards the value of your time in this game.

The overworld traversal is also a great example of just wasting your time. Movement between dungeons, cities and other points of interest involve moving on a gameboard-like map where every single space has a chance to randomly pull you into combat. Trying to get from one overworld plot event to another might require you to fight three random groups of enemies. And as you progress through the game, your party level far outstrips the enemies, requiring you to just waste time beating small fries for no good reason. Even if you don’t have to fight a single mob, the gameboard-like movement is just slow to watch. There is a fast-forward button but what the game needs is a skip button. And to further add insult to injury, you’re also expected to spend credits to build the paths to dungeons and other cities. Could I just pay a bit extra and build a path that isn’t infested with low-level mobs?

I think Compile Heart even realises how awful many of these things are since they allow disabling a lot of them once you reach New Game+. Personally, I’d design my game in such a way that you don’t want to disable a dozen "features" after a single playthrough.

VII might also have the worst achievements in franchise history. Card Master is an awful waste of time that will give you carpal tunnel, headache and a disdain for life itself. Treasure Hunter, while not reaching similar peak awfulness, is also a giant time sink without really any fun aspects to it. If you decide to challenge these, hopefully you have a second monitor to watch YouTube videos on as you painfully grind through them all. Completionists beware.

While there’s some good and interesting changes to the combat, like having a combo system that discourages just spamming the same attack again and again, combat feels worse than for example in the Rebirth titles. The timing for symbol attacks, trying to align your attacks to hit multiple enemies, having to chase after enemies dispersing all over the place. Everything just feels that much worse. Then there’s things like boss fights where the boss is immune to damage until you hit him enough times to break his damage immunity item – a process that takes way too long to be remotely fun. It’s not even a hard fight since the boss only does chip damage, meaning that at no point are you actually in danger.

As for the story, it’s just okay. Definitely not as good as Victory’s but it’s a serviceable Neptunia story, at least towards the end. I’m not really a fan of how the game is split into three different parts. It doesn’t really even add much to the story since it’s all continuous anyway. The conditions to reach the true ending are also annoying to fulfil and probably better saved for the New Game+ when the game stops doing its most annoying features.

The technical side of VII is nothing short of a train wreck. The sound settings are awful, you can’t have controller and keyboard input working at the same time, the game is capped at 60 FPS and 1920×1080 resolution and the battle load times are bad considering I have an SSD and the game isn’t that graphically intensive. I also had issues with the controller button mapping with my DualSense but maybe that was the fault of Steam Input. Hasn’t happened in other games though. And to put a cherry on top of everything, using my KVM switch (which unplugs the keyboard and mouse to use them with another PC) makes the game crash.

Thankfully there is a mod that allows you to run the game at such arbitrary resolutions as 2560×1440, and mercifully above 60 FPS. While my PC is quite powerful, I saw no negative effects from running this mod and would recommend it to anyone playing on hardware more advanced than a Steam Deck. And since the mod works so wonderfully, it really makes me wonder why Compile Heart couldn’t implement these features themselves. Although I think I know the answer: Compile Heart is a two-bit kusoge developer that wants PC gamer money for the least possible effort.

What a shame that the sequel to the best Neptunia game, Victory, and the only mainline Neptunia game we’ve gotten in the last decade, is such an awful game. And this is by Neptunia standards, which have historically not been that high anyway. I don't know why they decided that VII stands for "Victory II" with a quality gap like this. Usually with these things you can at least recommend the game to fans of the series, but I don’t think you can really even recommend it to them. The game just isn’t fun to play and the story’s not interesting enough to justify it. If you insist on having to experience the game for yourself, at least do yourself a favour and grab the game on a sale. Maybe that way you’ll feel less bad about playing it.

Lamport's logical clocks and the Ricart–Agrawala Distributed Mutual Exclusion algorithm in Python

Bytepawn - Marton Trencseni — Sun, 13 Jul 2025 01:00:00 GMT

This post shows how to implement the Ricart–Agrawala Distributed Mutual Exclusion algorithm with Lamport's logical clocks using HTTP Flask servers.

Exploring Lamport's Bakery algorithm in Python

Bytepawn - Marton Trencseni — Fri, 11 Jul 2025 01:00:00 GMT

This post shows how to implement Lamport’s Bakery algorithm with Python HTTP Flask servers. The Python code is short and readable, helping newcomers grasp the algorithm’s logic without the syntactic overhead of C++. The asynchronous HTTP flow also generates plenty of overlap between workers, so the need for mutual exclusion is obvious. Exposing the variables through REST endpoints enables observation of ticket picking and critical section entry in real time, potentially making the flow of the algorithm easier to understand.

Programming ebooks by Sundeep Agarwal

learnbyexample — Thu, 10 Jul 2025 03:00:00 GMT

This post lists my programming ebooks with details like PDF/EPUB purchase links, GitHub repos, web versions, testimonials, etc. All my ebooks are self-published. You can get these ebooks individually or as part of bundles. You can also read them online for free.

Bundles 📚

Poster created using Canva

All books bundle: leanpub or gumroad
- 13 programming ebooks on Regular Expressions, Linux CLI tools, Python, Vim and more
Linux CLI Text Processing bundle: leanpub or gumroad
- GNU grep, sed, awk, Perl and Ruby one-liners, GNU coreutils, CLI computing
Awesome regex: leanpub or gumroad
- Python, Ruby, JavaScript Regular expressions
- GNU grep, ripgrep, GNU sed, GNU awk CLI tools (BRE/ERE, PCRE, Rust regex crate, PCRE2)
- Vim regexp
Magical one-liners: leanpub or gumroad
- GNU grep, ripgrep, GNU sed, GNU awk, Ruby, Perl CLI tools
Learn by example Python bundle: leanpub or gumroad
- Python introduction, Regular expressions and Projects
Ruby Text processing: leanpub or gumroad
- Ruby regular expressions, Ruby One-Liners Guide

Testimonials 😍

I love your books on regex...As a student from the Digital VLSI space, it is indeed useful now and definitely in the future. It's really well written and really easy to understand the examples.

— feedback on reddit

It's very thorough, written with care, and presented in a way that makes sense. Even as an intermediate Python programmer, I found use in this book.

— feedback by Andrew Healey on Hacker News for "100 Page Python Intro"

Step up your cli fu with this fabulous intro & deep dive into awk. I learned a ton of tricks!

— feedback on twitter

Your Practice Python Projects book is really helping me to reinforce my knowledge and mastery of Python as I'm learning.

— feedback on twitter

In my opinion the book does a great job of quickly presenting examples of how commands can be used and then paired up to achieve new or interesting ways of manipulating data. Throughout the text there are little highlights offering tips on extra functionality or limitations of certain commands. For instance, when discussing the shuf command we're warned that shuf will not work with multiple files. However, we can merge multiple files together (using the cat command) and then pass them to shuf. These little gems of wisdom add a dimension to the book and will likely save the reader some time wondering why their scripts are not working as expected.

— book review by Jesse Smith on distrowatch.com for "Command line text processing with GNU Coreutils"

Literally was having a mini-breakdown about not understanding Regex in algorithm solutions the other day and now I'm feeling so much better, so thank YOU! I genuinely feel like I'm developing the skill for spotting when and where to use them after so much practice!

— feedback on twitter

This Ruby one-liners cookbook is incredible. Pretty mind boggling all the stuff you can do.

— feedback on twitter

Hi, great work releasing this! Trying to explain vim concisely is always an interesting challenge and I had a great time reading your attempt in this book. I always find it really interesting on how people try to group certain vim functions in a way that makes sense to people that don't use vim. I think you cover that idea pretty well in your 'Vim philosophy and features' section whilst not making it overly abstract and keeping it relatable.

— feedback on Hacker News by doix for "Vim Reference Guide"

I consider myself pretty experienced at shell-fu and capable of doing most things I set out to achieve in either bash scripts or fearless one-liners. However, my awk is rudimentary at best, I think mostly because it's such an unforgiving environment to experiment in. These books you've written are great for a bit of first principles insight and then quickly building up to functional usage. I will have no hesitation in referring colleagues to them!

— feedback on Hacker News

Thank you for choosing to write and share your knowledge. I read your books on CLI and sed - I think they are very comprehensive and very well explained. Keep up the great work

— feedback on twitter

This is fantastic! 👏 I use Perl one-liners for record and text processing a lot and this will be definitely something I will keep coming back to - I’ve already learned a trick from “Context Matching” (9) 🙂

— feedback on Linux@lemmy.ml

Nice book! I just started trying to get into linux today and you have some tips I haven’t found elsewhere and the text is an enjoyable read so far.

— feedback on reddit

I discovered your books recently and they’re awesome, thank you! As a 20 year *nix they made me realize how much more there are to these rock solid and ancient tools, once you spend the time to actually learn the intricacies of them.

— feedback on reddit

I love the whole learn by example premise. Those exercises at the end are so valuable, as it often times leads me to find multiple solutions which helps me conceptualize how commands work with each other much better!

— feedback on reddit

100 Page Python Intro

Short, introductory guide for the Python programming language, suited for those already familiar with programming basics.

Understanding Python re(gex)?

Learn Python Regular Expressions step-by-step from beginner to advanced levels with 300+ examples. Both re and regex modules are covered. Exercises are also included to test your understanding.

Practice Python Projects

Know Python basics but don't know what to do next? Take the next step in your programming journey with real world inspired Python projects.

Understanding JavaScript RegExp

Learn JavaScript Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises.

CLI text processing with GNU grep and ripgrep

Example based guide to mastering GNU grep and ripgrep. Exercises are also included to test your understanding.

CLI text processing with GNU sed

Example based guide to mastering GNU sed. Exercises are also included to test your understanding.

CLI text processing with GNU awk

Example based guide to mastering GNU awk one-liners. Exercises are also included to test your understanding.

Understanding Ruby Regexp

Learn Ruby Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises.

Sample chapters
Pay what you want for pdf/epub:
- gumroad
- leanpub
GitHub repo for code snippets and more
web version
Feedback: Twitter

Ruby One-Liners Guide

Example based guide for text processing with Ruby from the command line. Exercises are also included to test your understanding.

Perl One-Liners Guide

Example based guide for text processing with Perl from the command line. Exercises are also included to test your understanding.

CLI text processing with GNU Coreutils

You might be already aware of popular coreutils commands like head, tail, tr, sort and so on. This book will teach you more than twenty of such specialized text processing tools provided by the GNU coreutils package.

Vim Reference Guide

This is intended as a concise learning resource for beginner to intermediate level Vim users. It has more in common with cheatsheets than a typical text book. Topics like Regular Expressions and Macros have more detailed explanations and examples due to their complexity.

Sample chapters
Buy pdf/epub from:
- gumroad
- leanpub
GitHub repo
web version
Feedback: Twitter

Linux Command Line Computing

This ebook aims to teach Linux command line tools and Shell Scripting for beginner to intermediate level users. The main focus is towards managing your files and performing text processing tasks. Plenty of examples are provided to make it easier to understand a particular tool and its various features. Exercises at the end of chapters will help you practice what you've learned and solutions are provided for reference. I hope this ebook would make it much easier for you to discover CLI tools, features and learning resources than my own blundering experience.

Sample chapters
Buy pdf/epub from:
- gumroad
- leanpub
GitHub repo
web version
Feedback: Twitter

Practice Python Projects book announcement

learnbyexample — Thu, 10 Jul 2025 03:00:00 GMT

Hello!

My "Practice Python Projects" ebook presents five beginner-to-intermediate level projects inspired by real world use cases:

To test your understanding and to make it more interesting, you'll also be presented with exercises at the end of each project. Resources for further exploration are also mentioned throughout the book.

Ebook links🔗

You can buy the PDF/EPUB versions of the ebook using these links:

You can also get them as part of the Learn by example Python bundle using these links:

Videos🔗

Check out my programming tips covering Python, command line tools and Vim:

Testimonials🔗

Your Practice Python Projects book is really helping me to reinforce my knowledge and mastery of Python as I'm learning.

— feedback on twitter

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/practice_python_projects/preface.html.

GitHub repo🔗

Visit https://github.com/learnbyexample/practice_python_projects for programs, example files, markdown source and other details about the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Feedback🔗

I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, Gumroad rating, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/practice_python_projects/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Gemini Nano in Chrome 137: notes for AI Engineers

swyx's site RSS Feed — Tue, 08 Jul 2025 06:52:00 GMT

at long last, Gemini Nano is almost here for all Chrome users (i was originally misinformed that it was in Chrome 138 - but i checked my own facts and since Chrome 137+ it is starting to be shipped unflagged in limited situations). I was reminded by this HN post. I expect it to be shipped fully unflagged by end of year.

Dune: Awakening - First Impressions

Ben Overmyer's Site — Tue, 08 Jul 2025 03:00:00 GMT

Well, I've been playing Dune: Awakening for about a month now. I joined a guild right away and spent the first few hours after launch exploring and doing early quests ("journeys" in Dune parlance). I spent most of that first day in Discord voice chat with my guild.

I love the game. I've sunk many hours into it already. As expected, I spend most of my time building interesting things and then going out and collecting materials to build more interesting things. I haven't participated in PVP yet, though my guild (Dementium) has some very dedicated PVP players.

The crafting system is fun. The gating of crafting certain items behind acquiring particular resources is well done. I would say I wish they had more available for PVE players but it's not that big a deal to be unable to craft the highest-tier items without a guild's support. You don't really need the highest-tier items unless you're participating in the PVP endgame.

It's still early in the game's life, and that shows. Cheaters are rampant right now, especially in the PVP area, the Deep Desert. I have yet to be personally affected, but it's a little worrying. I have hope that the developers will fix the issue before too long.

Two days ago I built my first assault ornithopter. She's a big beast! I haven't taken her out for more than an initial spin, but I think that'll change when I need to go gather resources in a hard-to-reach spot.

That's it for now. We'll see how long Dune keeps my interest, especially with university starting in a couple months.

notes from Naval (2025)

swyx's site RSS Feed — Mon, 07 Jul 2025 00:41:21 GMT

someone I resonate a lot with is Naval Ravikant - his classic "How to Get Rich Without Getting Lucky" (henceforth HTGR) is formative to a lot of my thinking, including How to Market Yourself Without Being A Celebrity and "Play Long Term Games with Long Term People". HTGR also includes "How to Get Lucky", but I came across that before Naval, from pmarca, and wrote up my mnemonic/insight (that HTGL misses the role of strategy) in How to Create Luck.

Exploring Lamport's Bakery algorithm in C++20

Bytepawn - Marton Trencseni — Sun, 06 Jul 2025 01:00:00 GMT

I explore Lamport’s Bakery algorithm and show how a naive C++20 implementation breaks on weak memory models, how to fix it with std::atomic, and how to implement bounded tickets.

Karpathy: AI is changing Software Engineering

Bytepawn - Marton Trencseni — Thu, 03 Jul 2025 01:00:00 GMT

Andrej Karpathy says we’re marching from hand-written “Software 1.0” to NN-driven “2.0” and, finally, LLM-powered “3.0.” In my view, neural nets haven’t eaten deterministic code—kernels, browsers, games, and most business logic still run old-school. What has changed is the supply chain: LLM co-pilots (“Software 1.5”) already touch nearly every commit, nudging us towards language and architectures that LLMs are good at. The future belongs to teams that harness those co-pilots while keeping humans firmly on the trigger.

clamp / median / range

Tony Finch's blog — Wed, 02 Jul 2025 04:33:08 GMT

Here are a few tangentially-related ideas vaguely near the theme of comparison operators.

comparison style

Some languages such as BCPL, Icon, Python have chained comparison operators, like

    if min <= x <= max:
        ...

In languages without chained comparison, I like to write comparisons as if they were chained, like,

    if min <= x && x <= max {
        // ...
    }

A rule of thumb is to prefer less than (or equal) operators and avoid greater than. In a sequence of comparisons, order values from (expected) least to greatest.

clamp style

The clamp() function ensures a value is between some min and max,

    def clamp(min, x, max):
        if x < min: return min
        if max < x: return max
        return x

I like to order its arguments matching the expected order of the values, following my rule of thumb for comparisons – and the description of what clamp() does. (I used this flavour of clamp() in my article about GCRA.) But I seem to be unusual in this preference, based on a few examples I have seen recently.

clamp is median

Last month, Fabian Giesen pointed out a way to resolve this difference of opinion: A function that returns the median of three values is equivalent to a clamp() function that doesn’t care about the order of its arguments.

This version is written so that it returns NaN if any of its arguments is NaN. (When an argument is NaN, both of its comparisons will be false.)

    fn med3(a: f64, b: f64, c: f64) -> f64 {
        match (a <= b, b <= c, c <= a) {
            (false, false, false) => f64::NAN,
            (false, false, true)  => b, //  a > b > c
            (false, true,  false) => a, //  c > a > b
            (false, true,  true)  => c, // b <= c <= a
            (true,  false, false) => c, //  b > c > a
            (true,  false, true)  => a, // c <= a <= b
            (true,  true,  false) => b, // a <= b <= c
            (true,  true,  true)  => b, // a == b == c
        }
    }

When two of its arguments are constant, med3() should compile to the same code as a simple clamp(); but med3()’s misuse-resistance comes at a small cost when the arguments are not known at compile time.

clamp in range

If your language has proper range types, there is a nicer way to make clamp() resistant to misuse:

    fn clamp(x: f64, r: RangeInclusive<f64>) -> f64 {
        let (&min,&max) = (r.start(), r.end());
        if x < min { return min }
        if max < x { return max }
        return x;
    }

    let x = clamp(x, MIN..=MAX);

range style

For a long time I have been fond of the idea of a simple counting for loop that matches the syntax of chained comparisons, like

    for min <= x <= max:
        ...

By itself this is silly: too cute and too ad-hoc.

I’m also dissatisfied with the range or slice syntax in basically every programming language I’ve seen. I thought it might be nice if the cute comparison and iteration syntaxes were aspects of a more generally useful range syntax, but I couldn’t make it work.

Until recently when I realised I could make use of prefix or mixfix syntax, instead of confining myself to infix.

So now my fantasy pet range syntax looks like

    >= min < max    // half-open
    >= min <= max   // inclusive

And you might use it in a pattern match

    if x is >= min < max {
        // ...
    }

Or as an iterator

    for x in >= min < max {
        // ...
    }

Or to take a slice

    xs[>= min < max]

style clash?

It’s kind of ironic that these range examples don’t follow the left-to-right, lesser-to-greater rule of thumb that this post started off with. (x is not lexically between min and max!)

But that rule of thumb is really intended for languages such as C that don’t have ranges. Careful stylistic conventions can help to avoid mistakes in nontrivial conditional expressions.

It’s much better if language and library features reduce the need for nontrivial conditions and catch mistakes automatically.

This feed is no longer updated. Please subscribe to https://susam.net/feed.xml for the new feed.

Susam Pal - Discontinued Feed — Wed, 02 Jul 2025 03:00:00 GMT

This feed is no longer maintained. Please subscribe to the new feed at https://susam.net/feed.xml to receive updates about new blog posts.

2025 Mid-Year Review

Ben Overmyer's Site — Tue, 01 Jul 2025 03:00:00 GMT

Well, 2025 has certainly gone differently than I originally expected on January 1^st.

Much and more has been written about the state of the world in general and the United States of America in particular. I won't go into those topics here.

In March of this year, I was accepted into the Master of Applied Science in Software Engineering program at Memorial University in St. John's, Newfoundland, Canada. I confirmed the acceptance and began the process of applying for a study permit and related logistics. Today I will be giving notice at my current job at Apiture. By the end of this month, I will be temporarily retiring from industry to become a full-time student again. That will feel strange. Good, I imagine, but strange.

I really fell off the fitness wagon for several months. Most of the year, really. I've only just gotten back on in the last few days. Now, though, I'm finally going for a daily morning jog again. It feels good, even if I am clearly in much worse shape than I was last year. Well, have to start somewhere. I still have a gym membership until the end of the month and I intend to use it.

Recently, I've felt that my alcohol consumption has been increasingly deleterious to my health, so I'm easing way off. I did not make quarters 1 and 3 alcohol-free as planned, but I'll make up for it in the latter half of the year.

In terms of my creative hobby plans, I did continue guitar until the end of May. My guitar will go into storage until I finish my studies, but I do plan to pick it up again. Painting was starting to become a more regular thing for me up until we started needing to pack up. I will definitely be picking that back up too, and maybe sooner than guitar. We'll see.

My finances have been doing really well. Paying cash for a master's program will put a significant dent in my savings, though.

Finally... I discovered that I'm allergic to dogs, so I will not be getting a dog. We tried, though. For about a week we fostered a feisty little rescue named Clover. She was some kind of pit mix, and only five months old. Besides being allergic, I couldn't really handle her energy at the time. That made me very sad.

Want to meet people, try charging them for it?

Notes on software development — Sat, 28 Jun 2025 03:00:00 GMT

I have been blogging consistently since 2017. And one of my goals in speaking publicly was always to connect with like-minded people. I always left my email and hoped people would get in touch. Even while my blog and twitter became popular, passing 1M views and 20k followers, I basically never had people get in touch to chat or meet up.

So it felt kind of ridiculous when last November I started charging people $100 to chat. I mean, who am I? But people started showing up fairly immediately. Now granted the money did not go to me. It went to an education non-profit and I merely received the receipt.

And at this point I've met a number of interesting people, from VCs to business professors to undergraduate students to founders and everyone in between. People wanting to talk about trends in databases, about how to succeed as a programmer, about marketing for developers, and so on. Women and men thoughout North America, Europe, Africa, New Zealand, India, Nepal, and so on. And I've raised nearly $6000 for educational non-profits.

How is it that you go from giving away your time for free and getting no hits to charging and almost immediately getting results? For one, every person responded very positively to it being a fundraiser. It also helps me be entirely shameless about sharing on social media every single time someone donates; because it's such a positive thing.

But also I think that in "charging" for my time it helps people feel more comfortable about actually taking my time, especially when we have never met. It gives you a reasonable excuse to take time from an internet rando.

On the other hand, a lot of people come for advice and I think giving advice is pretty dangerous, especially since my background is not super conventional. I try to always frame things as just sharing my opinion and my perspective and that they should talk with many others and not take my suggestions without consideration.

And there's also the problem that by charging everyone for my time now, I'm no longer available to people who could maybe use it the most. I do mention on my page that I will still take calls from people who don't donate, as my schedule allows. But to be honest I feel less incentivized to spend time when people do not donate. So I guess this is an issue with the program.

But I mitigated even this slightly, and significantly jump-started the program, during my 30th birthday when I took calls with any person who donated at least $30.

Anyway, I picked this path because I have wanted to get involved with helping students figure out their lives and careers. But without a degree I am literally unqualified for many volunteering programs. And I always found the time commitments for non-profits painful.

So until starting this I figured it wouldn't be until I retire that I find some way to make a difference. But ultimately I kept meeting people who were starting their own non-profits now or donated significantly to help students. Peer pressure. I wanted to do my part now. And 30 minutes of my time in return for a donation receipt has been an easy trade.

While only raising a humble $6,000 to date, the Chat for Education program has been more successful than I imagined. I've met many amazing people through it. And it's something that should be easy to keep up indefinitely.

I hope to meet you through it too!

I wrote about trying to meet like-minded people and fundraising for educational non-profits. pic.twitter.com/UJ9U6DIHGU
— Phil Eaton (@eatonphil) June 28, 2025

A remote adventure in the Southern Highlands of Iceland

The Road Chose Me — Fri, 27 Jun 2025 20:45:20 GMT

The “F” roads in Iceland lead to incredibly stunning places and traverse some pretty remote and rugged terrain. After meeting a few locals and getting the inside word, I learn the really, really good...

Golang and Let's Encrypt: a free software story

Tony Finch's blog — Thu, 26 Jun 2025 04:50:23 GMT

Here’s a story from nearly 10 years ago.

the bug

I think it was my friend Richard Kettlewell who told me about a bug he encountered with Let’s Encrypt in its early days in autumn 2015: it was failing to validate mail domains correctly.

the context

At the time I had previously been responsible for Cambridge University’s email anti-spam system for about 10 years, and in 2014 I had been given responsibility for Cambridge University’s DNS. So I knew how Let’s Encrypt should validate mail domains.

Let’s Encrypt was about one year old. Unusually, the code that runs their operations, Boulder, is free software and open to external contributors.

Boulder is written in Golang, and I had not previously written any code in Golang. But its reputation is to be easy to get to grips with.

So, in principle, the bug was straightforward for me to fix. How difficult would it be as a Golang newbie? And what would Let’s Encrypt’s contribution process be like?

the hack

I cloned the Boulder repository and had a look around the code.

As is pretty typical, there are a couple of stages to fixing a bug in an unfamiliar codebase:

work out where the problem is
try to understand if the obvious fix could be better

In this case, I remember discovering a relatively substantial TODO item that intersected with the bug. I can’t remember the details, but I think there were wider issues with DNS lookups in Boulder. I decided it made sense to fix the immediate problem without getting involved in things that would require discussion with Let’s Encrypt staff.

I faffed around with the code and pushed something that looked like it might work.

A fun thing about this hack is that I never got a working Boulder test setup on my workstation (or even Golang, I think!) – I just relied on the Let’s Encrypt cloud test setup. The feedback time was very slow, but it was tolerable for a simple one-off change.

the fix

My pull request was small, +48-14.

After a couple of rounds of review and within a few days, it was merged and put into production!

A pleasing result.

the upshot

I thought Golang (at least as it was used in the Boulder codebase) was as easy to get to grips with as promised. I did not touch it again until several years later, because there was no need to, but it seemed fine.

I was very impressed by the Let’s Encrypt continuous integration and automated testing setup, and by their low-friction workflow for external contributors. One of my fastest drive-by patches to get into worldwide production.

My fix was always going to be temporary, and all trace of it was overwritten years ago. It’s good when “temporary” turns out to be true!

the point

I was reminded of this story in the pub this evening, and I thought it was worth writing down. It demonstrated to me that Let’s Encrypt really were doing all the good stuff they said they were doing.

So thank you to Let’s Encrypt for providing an exemplary service and for giving me a happy little anecdote.

The discreet charm of the infrastructureless

John Graham-Cumming's blog — Wed, 25 Jun 2025 11:30:21 GMT

Early in the morning in a European city, I awoke and wondered for a moment where I was. The room was totally black apart from a small glow from the face of my Casio Lineage watch lying on the bedside table. The luminous dial had absorbed sunlight during the day and was now giving off gentle light in the night; enough brightness to read the time.

Infrastructureless #1: sunlight + chemistry = glow in the dark.

The watch had also absorbed sunlight through its face during the day to recharge its battery. In all the time I've owned this watch I've never seen the battery drop below the what Casio calls HI (there are levels MID and LO but I'd only seen MID once when I took the watch from its original packaging)

Infrastructureless #2: sunlight + semiconductors = electrical power.

Clicking the top right button to illuminate the face further I could see the small LED display showing the local time in the European city (the main hands show my home time zone). And on the LED display a little black rectangle under RCVD indicated that somewhere in the night (it turned out at 02:04) the watch had picked up the signal from the DCF77 transmitter in Germany and set itself.

Infrastructureless #3: radio = correct time.

OK, DCF77 isn't totally without infrastructure, but one of the reasons I became a radio amateur is lack of infrastructure. Sure, you're able to read this because of the Internet but it's an incredibly complex system: layers and layers of technology and cooperating entities just so you can read this.

On the other hand, radio amateurs just send out electromagnetic waves and talk to each other, no intervening infrastructure (well, ok, maybe you can count the ionosphere as infrastructure allowing long distance reception). And that's what DCF77, and the other five transmitters the Casio can receive in the US, UK, China and Japan, are doing: sending waves into the night to be picked up and decoded by whoever is listening.

Also, I'm just fascinated by radio propagation: my ancient Sharper Image clock sometimes picks up the US WWVB in Portugal.

What is Simplicity?

Dan Palmer — Wed, 25 Jun 2025 02:00:00 GMT

It’s uncontroversial to say that the code we write should be simple. In a discipline that is all about grappling with complexity, keeping things simple is critical to software development efforts, team productivity, scalability, and maintainability. Despite well known satirical counter-examples such as Enterprise FizzBuzz, simplicity is a goal for most software engineers.

But what is simplicity? Despite this near universal agreement that it’s important, throughout my career I’ve seen numerous instances of friction in design and code review where both parties are convinced that their solution is the simpler option. These disagreements are notoriously hard to resolve because we lack the language to talk about our preferences when it comes to simplicity, and simplicity is not a single concept or direction, but a set of competing priorities and trade-offs.

Guide to Working Weekends

jodavaho.io — Mon, 23 Jun 2025 03:00:00 GMT

Coffee on a monday after a productive weekend, here’s what went well this time. Much better than usual!

Don’t work weekends

If at all possible, keep time to yourself on the weekend. Use it to read, relax, catch up on housework, do hobbies, spend time with wife or kids (or date otherwise). Etc. You need breaks, mentally, to remain sharp, and physically, to remain energetic.

So, it should be the default mode to not work on the weekends. But, sometimes that thing is broken, the deadline is looming, the competition is knocking, etc…

So how do you recharge (or at least not ruin your next week), while still being productive? Here’s what just worked for me.

Have a supportive partner

The best weekend warrior advice I can give is to have a supportive partner, which is also the best life I can give. Partner up with someone who knows and appreciates what you’re going though, preferably having been there before. My powerhouse wife has a 10x work ethic over me, and also (somewhat telling) has 10x the work boundaries that I do.

Tell people you need the time

Give time family, friends, or anyone else who is counting on you to adjust their plans. I had been telling my wife for two weeks I was going to be in a crunch and needed the time. We worked together to make that time.

Work on highest net happiness tasks

Maybe there’s a bit of work you wanted to do, but it wasn’t the highest priority for everyone. Do that now! It’s your time, and assuming you’re a decent employee, you gave it everything you had during the week. If you must give more, at least prioritize the tasks that will provide the most relief or enjoyment once they are done.

Take breaks

Code has to compile, emails might need a reply, Claude is busy having an existential crisis. Either way, go outside, to the gym, read a little, watch part of a movie, vacuum the floor, play with legos, listen to an audiobook. Hell, go have a margarita on the back porch. This time is yours, spend it how you want.

Unless your livelihood is threatened with imminent demise (and it almost never is, even if your emotions might say it is), you should not be over-exerting yourself during personal time.

Be alone

There’s something really awful about explaining to your kids that you cannot go play bubbles outside because you have to work. For me, at least, even seeing my wife go about her day pulls me away from work.

Worse, if you’re following this advice, and everyone is around to see you take a margarita break during a Claude task, you might be in for some difficult misunderstandings. “I thought you had to work!” is the seed for guilt (for you) and feelings of being taken advantage of (for them).

Send the kids, partner, etc to a spa day or trip to grandparents, maybe, but don’t add to the guilt by having them mope around the house while you burn yourself out.

Make it up to them

No doubt your spouse is pulling extra weight while you’re working. Or your friends are noticing you missed the event. I can’t emphasize enough:

Life is short, and consistently distancing yourself from those you love will have long term consequences for your happiness.

You’ll want to make it up to them. Do extra work around the house, cover your partner when it is their turn. Buy dinner for friends next time.

Have something to show

For employer, family, and friends, you want to have something good to show for the extra time. Tell your spouse you’re in a better place now, give your boss some extra results, etc. Check in Monday morning right away with something to show.

Now, there’s a risk of coming across as a performative worker. Those are the worst. Depending on your team culture, it might be best to keep the results just between you and the boss, or to somewhat downplay them. Weigh the tradeoffs, but if someone had to bend their schedule to allow you to work when you normally don’t, you better have something to show for it.

Debugging memory leaks in Postgres, jemalloc edition

Notes on software development — Sat, 21 Jun 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Writing a simple async message queue server in C++, Python and JavaScript — Summary

Bytepawn - Marton Trencseni — Sat, 21 Jun 2025 01:00:00 GMT

I summarize thoughts and lessons learned of implementing the same async message queue server in C++, Python and JavaScript over the past 2 years.

Writing a simple C++20 async message queue server - Part III

Bytepawn - Marton Trencseni — Fri, 20 Jun 2025 01:00:00 GMT

I write the final version of the async message queue server in modern C++20, wire compatible with the Python and Javascript versions.

100 blog posts, 6 years, 5 million views

Austin Z. Henley's Blog — Thu, 19 Jun 2025 22:00:01 GMT

https://austinhenley.com/blog/100.html

Agentic Coding Workflow & Cline Demo

smcleod.net — Thu, 19 Jun 2025 18:10:00 GMT

Recording of a demo and Q&A session on my Agentic Coding workflow and Cline use.

How SaaS Startups Can Scale Accounting Without Slowing Development

Caseysoftware — Wed, 18 Jun 2025 15:53:00 GMT

SaaS startups live and die by their ability to iterate. Every feature launch, bug fix, or product tweak hinges on…

The post How SaaS Startups Can Scale Accounting Without Slowing Development appeared first on Caseysoftware.

Zero Time Dilemma review

ブラック — Wed, 18 Jun 2025 03:00:00 GMT

The final stretch.

Everyone knows that Uchikoshi loves nothing more than non-linear stories, and Zero Time Dilemma might be him taking it to its ultimate conclusion. It feels like Uchikoshi wrote the final part of the Zero Escape series, fed most of it through a woodchipper and then made you scavenge the plot in a random order from the remains. The game lasts for about 20 hours and you’ll spend at least the first ten hours absolutely clueless as to what is happening – and when.

Even though the previous Zero Escape games are definitely non-linear stories too, at least you had a sense of belonging in the story. Most of the game feels like such a mess that I was worried if they could even to sort everything out in a manner that makes sense. Thankfully the story starts to mesh together in the end, at least to a certain point. But before that meshing starts to happen, you really need to cling onto your love for Zero Escape to get through the drudgery and confusion. I was sure that I’d recommend against this game during the first half.

The drudgery is not helped at all by the graphics. Technically speaking, the level of the 3D graphics has taken a significant leap since Virtue’s Last Reward. And yet, despite the objective advancements, the end result is arguably worse. Animation quality is laughably poor, especially the lip syncing, which amounts to random lip movements completely divorced from the spoken lines. It’s even worse than the sockpuppet-like mouth flapping of Virtue’s Last Reward, since at least there was some connection between the audio and the animation. The low-poly VLR models doing a Sesame Street impression didn’t look nearly as awkward as the more realistic models of Zero Time Dilemma. I think there was also an instance where a character was supposed to look shocked or mortified during a life-and-death scenario, and it ended up looking like he was visibly bored.

And even though the 3D quality is higher than Virtue’s Last Reward, it’s still not that stellar. However janky The Somnium Files games might look, they’re still doing circles around Zero Time Dilemma. I would’ve liked to see a lot more work towards the models and the textures. Presumably the developers either had limited skillsets for 3D work, or they simply didn’t have the budget to spend the time. But what didn’t need as much work as it did was the character designs: returning characters don’t look at all like their previous selves despite there only being some months since we’ve seen them in 999 or VLR. Akane and Junpei can really only be identified by their names and voices, as their looks are completely different, making it much harder to treat them as the old characters. Sigma and Phi aren’t quite as bad but the differences still catch the eye.

The redesign of the old characters isn’t helped at all that quite a few of the returning characters don’t really even act like themselves in the story. This is even noted in the game, especially with Junpei, but we never really get great explanations for it. Sigma also acts out in a pretty weird way considering the life he’s lived thus far as detailed in Virtue’s Last Reward. I was sorta expecting there to be a plot twist about one of the returning characters being an impostor, but no, these are the real deals.

And the graphics aren’t even the only problem! The resolution selection in the launcher didn’t seem to work so I just set the resolution by rescaling the window and ended with the uncommon resolution of 1729 × 979. Probably should’ve just edited the configuration file directly. They’ve also landed on a worse control scheme than Virtue’s Last Reward, so I ended up using the mouse for way more things than I wanted. In VLR, I used the keyboard a lot more for camera control and for inventory usage. And considering the sheer amount of text inputs in the game, was it that much hard to implement keyboard typing? You really can tell that this is a console port.

On the core gameplay front, gone is the visual novel style storytelling. The story is instead told as (poorly-animated) cutscenes from a third-person perspective. While I understand the reasoning for it, since it ties into the storytelling with the revolving character focus, I do still much prefer the first-person perspectives of the prior instalments. It made you feel like a participant in the games and not a mere observer, robbed of all agency.

The escape room segments aren’t too dissimilar from the previous entries. I’d say that the difficulty level is somewhere between the straight-and-direct 999 escape segments and the more obtuse VLR escapes. You probably still want to have a notebook at hand like with Virtue’s Last Reward, as there’s quite a few things to remember. What was surprising was that there’s actually not that many escape segments. I got the achievement for solving everything with like a quarter of the game left.

Once you get to a point where the story starts coming together, it does start to feel like proper good old Zero Escape. To a point where I reconsidered my earlier opinion and felt like it was a game worth playing, even if was clearly a much weaker presentation than the earlier titles. And as per usual in the franchise, you also get some plot twists to open up the mystery. And when I got to the big plot twist, my faith started faltering again. It just felt like it was reaching a bit too much, a bit of an asspull. At this point I was once again starting to reconsider whether or not I actually liked this game.

Quite honestly, I’m still not sure. While the big twist wasn’t completely without any setup, the foreshadowing was extremely subtle. In the previous Zero Escape titles, I felt like Uchikoshi had thoroughly bamboozled me when I got an unexpected twist, and here my first reaction was that he’d cheated me. Even after scouring the Internet for all of the foreshadowing in the game, it still feels more unfair than the foreshadowing in Virtue’s Last Reward. I guess it is at least thematically fitting, as the game is about the inherent unfairness of life. Maybe this game is just as unfair.

There’s also things about the story that feel like jumping the shark. The Zero Escape world has always been pretty out there, with elements of sci-fi, pseudoscience and the supernatural all thrown in there. Still, it feels like Zero Time Dilemma takes it maybe a bit too far, with new and even wilder things coming out on a regular basis. I can’t be too mad about it, but I also can’t overlook it. They just seem like extremely crazy concepts that are mostly there as convenient plot devices.

Zero Time Dilemma – the first half of the story is a mess, the visual presentation feels like a bad student project, technical side has significant regressions from the prior games, the storytelling gameplay is less engaging, the big twist is potentially maddening, and some developments take it a bit too far. This doesn’t sound like a winning combination to me. And yet, I feel like I can at least somewhat recommend playing through it, despite its many flaws. This assuming that you are already invested into the Zero Escape world. Anyone else should run away screaming.

While some of the developments range from confusing to maddening, there’s still an interesting plot once it starts to come together. Uchikoshi at his worst is still Uchikoshi. Zero Time Dilemma also manages to do the thing that I was most expecting from it: shed some light on what questions remained from Virtue’s Last Reward. After I finished playing through the game, I asked myself if I regretted spending 20 hours on it, and the answer was “no”. Thankfully, as the final entry in the Zero Escape trilogy, it also doesn’t leave us players hanging for a continuation. Probably for the best, since who knows how hard Uchikoshi would need to stretch to continue the story and to pull twists on us again.

Sigh, here we go again.

Rondam Ramblings — Wed, 18 Jun 2025 01:07:45 GMT

You would think that after the disasters in Afghanistan and Iraq that Republicans would have learned that starting a war in that part of the world is a Really Bad Idea (tm). But no. After utterly failing to bring about regime change in both its eastern and western neighbors, the Trump administration is winding up to try yet again again in Iran. Maybe the third time will be the

No, Science is Not Just Another Religion

Rondam Ramblings — Tue, 17 Jun 2025 19:35:31 GMT

I want to debunk once and for all this idea that "science is just another religion". It isn't, for one simple reason: all religions are based on some kind of metaphysical assumptions. Those assumptions are generally something like the authority of some source of revealed knowledge, typically a holy text. But it doesn't have to be that. It can be as simple as assuming that

What makes a good side project?

Austin Z. Henley's Blog — Tue, 17 Jun 2025 03:00:01 GMT

https://austinhenley.com/blog/goodproject.html

Finding Dead Websites

Weblog on marginalia.nu — Tue, 17 Jun 2025 03:00:00 GMT

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change.

This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking.

A new adventure

ENOSUCHBLOG — Tue, 17 Jun 2025 03:00:00 GMT

This is a personal announcement post: after 7 years at Trail of Bits, I’m leaving to do something new. Specifically, I’ll be joining Astral to help them build the next generation of Python developer tooling.

Running and storing 3+ million LLM AI requests without spending $100,000

Ben E. C. Boyter — Mon, 16 Jun 2025 03:00:00 GMT

With the rise of AI LLM technology, I naturally wanted to learn more about using it. Both from a what can it do but also where can I use it in various tasks. Since I am also investing more time in searchcode.com these days I wanted to find a way I could use these tools to improve it.

One of the easiest things I could think of was by creating summaries of each file. This could then be used in both the HTML title, description and on the page assisting users with what they are looking at. I find these fairly useful at times, and I quite often ask LLM’s what something does when working with unfamiliar languages or codebases.

If the Ten Commandments Reflected Reality

Rondam Ramblings — Sun, 15 Jun 2025 21:44:42 GMT

And the Lord spoke unto Moses, saying: I am the Lord your God, who brought you out of Egypt, out of the land of slavery.You shall have no other gods before me. Except Donald Trump. If he says something that goes against my word, you shall believe him and not me.You shall not make for yourself any image in the form of anything in heaven above or on the earth beneath or in the waters

The fastest way to detect a vowel in a string

Austin Z. Henley's Blog — Fri, 13 Jun 2025 22:00:01 GMT

https://austinhenley.com/blog/vowels.html

Bypassing GitHub Actions policies in the dumbest way possible

ENOSUCHBLOG — Wed, 11 Jun 2025 03:00:00 GMT

TL;DR: GitHub Actions provides a policy mechanism for limiting the kinds of actions and reusable workflows that can be used within a repository, organization, or entire enterprise. Unfortunately, this mechanism is trivial to bypass. GitHub has told me that they don’t consider this a security issue (I disagree), so I’m publishing this post as-is.

Low-background Steel: content without AI contamination

John Graham-Cumming's blog — Tue, 10 Jun 2025 21:00:21 GMT

Somehow I forgot to blog my site: https://lowbackgroundsteel.ai/. I created it back in March 2023 as a clearinghouse for online resources that hadn't been contaminated with AI-generated content.

Low-background Steel (and lead) is a type of metal uncontaminated by radioactive isotopes from nuclear testing. That steel and lead is usually recovered from ships that sunk before the Trinity Test in 1945. The site is about uncontaminated content that I'm terming "Low-background Steel". The idea is to point to sources of text, images and video that were created prior to the explosion of AI-generated content that occurred in 2022.

It currently contains pointers to a Wikipedia dump from prior to the release of ChatGPT, the Arctic Code Vault, Project Gutenberg, and more.

If you know of other sources of non-contaminated content plus submit them!

performance of random floats

Tony Finch's blog — Mon, 09 Jun 2025 19:50:27 GMT

A couple of years ago I wrote about random floating point numbers. In that article I was mainly concerned about how neat the code is, and I didn’t pay attention to its performance.

Recently, a comment from Oliver Hunt and a blog post from Alisa Sireneva prompted me to wonder if I made an unwarranted assumption. So I wrote a little benchmark, which you can find in pcg-dxsm.git.

(Note 2025-06-09: I’ve edited this post substantially after discovering some problems with the results.)

recap

Briefly, there are two basic ways to convert a random integer to a floating point number between 0.0 and 1.0:

Use bit fiddling to construct an integer whose format matches a float between 1.0 and 2.0; this is the same span as the result but with a simpler exponent. Bitcast the integer to a float and subtract 1.0 to get the result.
Shift the integer down to the same range as the mantissa, convert to float, then multiply by a scaling factor that reduces it to the desired range. This produces one more bit of randomness than the bithacking conversion.

(There are other less basic ways.)

code

The double precision code for the two kinds of conversion is below. (Single precision is very similar so I’ll leave it out.)

It’s mostly as I expect, but there are a couple of ARM instructions that surprised me.

bithack

The bithack function looks like:

    double bithack52(uint64_t u) {
        u = ((uint64_t)(1023) << 52) | (u >> 12);
        return(bitcast(double, u) - 1.0);
    }

It translates fairly directly to amd64 like this:

    bithack52:
        shr     rdi, 12
        movabs  rax, 0x3ff0000000000000
        or      rax, rdi
        movq    xmm0, rax
        addsd   xmm0, qword ptr [rip + .number]
        ret
    .number:
        .quad   0xbff0000000000000

On arm64 the shift-and-or becomes one bfxil instruction (which is a kind of bitfield move), and the constant -1.0 is encoded more briefly. Very neat!

    bithack52:
        mov     x8, #0x3ff0000000000000
        fmov    d0, #-1.00000000
        bfxil   x8, x0, #12, #52
        fmov    d1, x8
        fadd    d0, d1, d0
        ret

multiply

The shift-convert-multiply function looks like this:

    double multiply53(uint64_t u) {
        return ((double)(u >> 11) * 0x1.0p-53);
    }

It translates directly to amd64 like this:

    multiply53:
        shr       rdi, 11
        cvtsi2sd  xmm0, rdi
        mulsd     xmm0, qword ptr [rip + .number]
        ret
    .number:
        .quad     0x3ca0000000000000

GCC and earlier versions of Clang produce the following arm64 code, which is similar though it requires more faff to get the constant into the right register.

    multiply53:
        lsr     x8, x0, #11
        mov     x9, #0x3ca0000000000000
        ucvtf   d0, x8
        fmov    d1, x9
        fmul    d0, d0, d1
        ret

Recent versions of Clang produce this astonishingly brief two instruction translation: apparently you can convert fixed-point to floating point in one instruction, which gives us the power of two scale factor for free!

    multiply53:
        lsr     x8, x0, #11
        ucvtf   d0, x8, #53
        ret

benchmark

My benchmark has 2 x 2 x 2 tests:

bithacking vs multiplying
32 bit vs 64 bit
sequential integers vs random integers

I ran the benchmark on my Apple M1 Pro and my AMD Ryzen 7950X.

These functions are very small and work entirely in registers so it has been tricky to measure them properly.

To prevent the compiler from inlining and optimizing the benchmark loop to nothing, the functions are compiled in a separate translation unit from the test harness. This is not enough to get plausible measurements because the CPU overlaps successive iterations of the loop, so we also use fence instructions.

On arm64, a single ISB (instruction stream barrier) in the loop is enough to get reasonable measurements.

I have not found an equivalent of ISB on amd64, so I’m using MFENCE. It isn’t effective unless I pass the argument and return values via pointers (because it’s a memory fence) and place MFENCE instructions just before reading the argument and just after writing the result.

results

In the table below, the leftmost column is the number of random bits; “old” is arm64 with older clang, “arm” is newer clang, “amd” is gcc.

The first line is a baseline do-nothing function, showing the overheads of the benchmark loop, function call, load argument, store return, and fences.

The upper half measures sequential numbers, the bottom half is random numbers. The times are nanoseconds per operation.

         old    arm    amd

    00  21.44  21.41  21.42

    23  24.28  24.31  22.19
    24  25.24  24.31  22.94
    52  24.31  24.28  21.98
    53  25.32  24.35  22.25

    23  25.59  25.56  22.86
    24  26.55  25.55  23.03
    52  27.83  27.81  23.93
    53  28.57  27.84  25.01

The times vary a little from run to run but the difference in speed of the various loops is reasonably consistent.

The numbers on arm64 are reasonably plausible. The most notable thing is that the “old” multiply conversion is about 3 or 4 clock cycles slower, but with a newer compiler that can eliminate the multiply, it’s the same speed as the bithacking conversion.

On amd64 the multiply conversion is about 1 or 2 clock cycles slower than the bithacking conversion.

conclusion

The folklore says that bithacking floats is faster than normal integer to float conversion, and my results generally agree with that, apart from on arm64 with a good compiler. It would be interesting to compare other CPUs to get a better idea of when the folklore is right or wrong – or if any CPUs perform the other way round!

A lesson on Trust but Verify and Managing Upward

Ben E. C. Boyter — Mon, 09 Jun 2025 03:00:00 GMT

Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual and the most incompetent individual I ever worked with.

I thought it was time to lay out one of my own failures, of which resulted in me adopting trust by verify for a all future interactions… at least until I am more familiar with the individual.

This was back when I was working on C#, SQL Server and some ETL processing software the name of which I had forgotten. I had mentioned to my manager at the time that I should probably get more experience using the ETL software because simply because there was so much work there and I had never used it. When a new project came up to build a system for managing HR backend jobs, to deal with vanity and real titles and organizational chart management. Think, renaming titles, organizing organizational hierarchies and the like.

Python regular expression cheatsheet and examples

learnbyexample — Mon, 09 Jun 2025 03:00:00 GMT

This blog post gives an overview and examples of regular expression syntax as implemented by the re built-in module (Python 3.13+). Assume ASCII character set unless otherwise specified. This post is an excerpt from my Understanding Python re(gex)? book.

Visualization created using debuggex for the pattern r'\bpar(en|ro)?t\b'

From docs.python: re:

A regular expression (or RE) specifies a set of strings that matches it; the functions in this module let you check if a particular string matches a given regular expression

Elements that define a regular expression🔗

Anchors	Description
`\A`	restricts the match to the start of string
`\Z`	restricts the match to the end of string
`^`	restricts the match to the start of line
`$`	restricts the match to the end of line
`\n`	newline character is used as the line separator
`re.MULTILINE` or `re.M`	flag to treat input as multiline string
`\b`	restricts the match to the start/end of words
	word characters: alphabets, digits, underscore
`\B`	matches wherever `\b` doesn't match

^, $ and \ are metacharacters in the above table, as these characters have special meaning. Prefix a \ character to remove the special meaning and match such characters literally. For example, \^ will match a ^ character instead of acting as an anchor.

Feature	Description
`\|`	multiple RE combined as conditional OR
	each alternative can have independent anchors
`(pat)`	group patterns, also a capturing group
	`a(b\|c)d` is same as `abd\|acd`
`(?:pat)`	non-capturing group
`(?P<name>pat)`	named capture group
`.`	Match any character except the newline character `\n`
`[]`	Character class, matches one character among many

Greedy Quantifiers	Description
`*`	Match zero or more times
`+`	Match one or more times
`?`	Match zero or one times
`{m,n}`	Match `m` to `n` times (inclusive)
`{m,}`	Match at least `m` times
`{,n}`	Match up to `n` times (including `0` times)
`{n}`	Match exactly `n` times
`pat1.*pat2`	any number of characters between `pat1` and `pat2`
`pat1.pat2\|pat2.pat1`	match both `pat1` and `pat2` in any order

Greedy here means that the above quantifiers will match as much as possible that'll also honor the overall RE. Appending a ? to greedy quantifiers makes them non-greedy, i.e. match as minimally as possible. Appending a + to greedy quantifiers makes them possessive, which prevents backtracking. You can also use (?>pat) atomic grouping to safeguard from backtracking. Quantifiers can be applied to literal characters, groups, backreferences and character classes.

Character class	Description
`[aeiou]`	Match any vowel
`[^aeiou]`	`^` inverts selection, so this matches any consonant
`[a-f]`	`-` defines a range, so this matches any of abcdef characters
`\d`	Match a digit, same as `[0-9]`
`\D`	Match non-digits, same as `[^0-9]` or `[^\d]`
`\w`	Match word characters, same as `[a-zA-Z0-9_]`
`\W`	Match non-word characters, same as `[^a-zA-Z0-9_]` or `[^\w]`
`\s`	Match whitespace characters, same as `[\ \t\n\r\f\v]`
`\S`	Match non-whitespace characters, same as `[^\ \t\n\r\f\v]` or `[^\s]`

Lookarounds	Description
lookarounds	custom assertions, zero-width like anchors
`(?!pat)`	negative lookahead assertion
`(?<!pat)`	negative lookbehind assertion
`(?=pat)`	positive lookahead assertion
`(?<=pat)`	positive lookbehind assertion
`(?!pat1)(?=pat2)`	multiple assertions can be specified in any order
	as they mark a matching location without consuming characters
`((?!pat).)*`	Negate a grouping, similar to negated character class

Flags	Description
`re.IGNORECASE` or `re.I`	flag to ignore case
`re.DOTALL` or `re.S`	allow `.` metacharacter to match newline characters
`flags=re.S\|re.I`	multiple flags can be combined using `\|` operator
`re.MULTILINE` or `re.M`	allow `^` and `$` anchors to match line wise
`re.VERBOSE` or `re.X`	allows to use literal whitespaces for aligning purposes
	and to add comments after the `#` character
	escape spaces and `#` if needed as part of actual RE
`re.ASCII` or `re.A`	match only ASCII characters for `\b`, `\w`, `\d`, `\s`
	and their opposites, applicable only for Unicode patterns
`re.LOCALE` or `re.L`	use locale settings for byte patterns and 8-bit locales
`(?#comment)`	another way to add comments (not a flag)
`(?flags:pat)`	inline flags only for this `pat`, overrides `flags` argument
	flags is `i` for `re.I`, `s` for `re.S`, etc, except `L` for `re.L`
`(?-flags:pat)`	negate flags only for this `pat`
`(?flags-flags:pat)`	apply and negate particular flags only for this `pat`
`(?flags)`	apply flags for whole RE, can be used only at start of RE
	anchors if any, should be specified after `(?flags)`

Matched portion	Description
`re.Match` object	details like matched portions, location, etc
`m[0]` or `m.group(0)`	entire matched portion of `re.Match` object `m`
`m[n]` or `m.group(n)`	matched portion of the nth capture group
`m.groups()`	tuple of all the capture groups' matched portions
`m.span()`	start and end+1 index of the entire matched portion
	pass a number to get span of that particular capture group
	can also use `m.start()` and `m.end()`
`\N`	backreference, gives matched portion of the Nth capture group
	applies to both search and replacement sections
	possible values: `\1`, `\2` up to `\99` provided no more digits
`\g<N>`	backreference, gives matched portion of the Nth capture group
	possible values: `\g<0>`, `\g<1>`, etc (not limited to 99)
	`\g<0>` refers to the entire matched portion
`(?P<name>pat)`	named capture group
	refer as `'name'` in `re.Match` object
	refer as `(?P=name)` in search section
	refer as `\g<name>` in replacement section
`groupdict`	method applied on a `re.Match` object
	gives named capture group portions as a `dict`

\0 and \100 onwards are considered as octal values, hence cannot be used as backreferences.

re module functions🔗

Function	Description
`re.search`	Check if given pattern is present anywhere in input string
	Output is a `re.Match` object, usable in conditional expressions
	r-strings preferred to define RE
	Use byte pattern for byte input
	Python also maintains a small cache of recent RE
`re.fullmatch`	ensures pattern matches the entire input string
`re.compile`	Compile a pattern for reuse, outputs `re.Pattern` object
`re.sub`	search and replace
`re.sub(r'pat', f, s)`	function `f` with `re.Match` object as the argument
`re.escape`	automatically escape all metacharacters
`re.split`	split a string based on RE
	text matched by the groups will be part of the output
	portion matched by pattern outside group won't be in output
`re.findall`	returns all the matches as a list
	if 1 capture group is used, only its matches are returned
	1+, each element will be tuple of capture groups
	portion matched by pattern outside group won't be in output
`re.finditer`	iterator with `re.Match` object for each match
`re.subn`	gives tuple of modified string and number of substitutions

The function definitions are given below:

re.search(pattern, string, flags=0)
re.fullmatch(pattern, string, flags=0)
re.compile(pattern, flags=0)
re.sub(pattern, repl, string, count=0, flags=0)
re.escape(pattern)
re.split(pattern, string, maxsplit=0, flags=0)
re.findall(pattern, string, flags=0)
re.finditer(pattern, string, flags=0)
re.subn(pattern, repl, string, count=0, flags=0)

Regular expression examples🔗

As a good practice, always use raw strings to construct RE, unless other formats are required. This will avoid conflict between special meaning of the backslash character in RE and string literals.

I wrote an interactive TUI app to help you experiment with the examples presented below. See PyRegexPlayground repo for installation instructions and usage guide. See PyRegexExercises repo for a TUI app with 100+ Python regex exercises.

examples for re.search()

>>> sentence = 'This is a sample string'

# need to load the re module before use
>>> import re
# check if 'sentence' contains the pattern described by RE argument
>>> bool(re.search(r'is', sentence))
True

# ignore case while searching for a match
>>> bool(re.search(r'this', sentence, flags=re.I))
True

# example for a pattern not found in the input string
>>> bool(re.search(r'xyz', sentence))
False

# re.search output can be directly used in conditional expressions
>>> if re.search(r'ring', sentence):
...     print('mission success')
... 
mission success

# use raw byte strings for patterns if input is of byte data type
>>> bool(re.search(rb'is', b'This is a sample string'))
True

string and line anchors

# match the start of the input string
>>> bool(re.search(r'\Ahi', 'hi hello\ntop spot'))
True

# match the start of a line
>>> bool(re.search(r'^top', 'hi hello\ntop spot', flags=re.M))
True

# match the end of strings
>>> words = ['surrender', 'up', 'newer', 'do', 'era', 'eel', 'pest']
>>> [w for w in words if re.search(r'er\Z', w)]
['surrender', 'newer']

# check if there's a whole line 'par'
>>> bool(re.search(r'^par$', 'spare\npar\ndare', flags=re.M))
True

examples for re.findall()

# match 'par' with optional 's' at start and optional 'e' at end
>>> re.findall(r'\bs?pare?\b', 'par spar apparent spare part pare')
['par', 'spar', 'spare', 'pare']

# numbers >= 100 with optional leading zeros
# you'd need r'\b0*[1-9]\d{2,}\b' if possessive quantifiers isn't used
>>> re.findall(r'\b0*+\d{3,}\b', '0501 035 154 12 26 98234')
['0501', '154', '98234']

# if multiple capturing groups are used, each element of output
# will be a tuple of strings of all the capture groups
>>> re.findall(r'([^/]+)/([^/,]+),?', '2020/04,1986/Mar')
[('2020', '04'), ('1986', 'Mar')]

# normal capture group will hinder ability to get the whole match
# non-capturing group to the rescue
>>> re.findall(r'\b\w*(?:st|in)\b', 'cost akin more east run')
['cost', 'akin', 'east']

# useful for debugging purposes as well
>>> re.findall(r':.*?:', 'green:3.14:teal::brown:oh!:blue')
[':3.14:', '::', ':oh!:']

examples for re.split()

# split based on one or more digit characters
>>> re.split(r'\d+', 'Sample123string42with777numbers')
['Sample', 'string', 'with', 'numbers']

# split based on digit or whitespace characters
>>> re.split(r'[\d\s]+', '**1\f2\n3star\t7 77\r**')
['**', 'star', '**']

# to include the matching delimiter strings as well in the output
>>> re.split(r'(\d+)', 'Sample123string42with777numbers')
['Sample', '123', 'string', '42', 'with', '777', 'numbers']

# multiple capture groups example
# note that the portion matched by b+ isn't present in the output
>>> re.split(r'(a+)b+(c+)', '3.14aabccc42')
['3.14', 'aa', 'ccc', '42']

# use non-capturing group if capturing is not needed
>>> re.split(r'hand(?:y|ful)', '123handed42handy777handful500')
['123handed42', '777', '500']

backreferencing within the search pattern

>>> words = ['effort', 'flee', 'facade', 'oddball', 'rat', 'tool']

# whole words that have at least one consecutive repeated character
>>> [w for w in words if re.search(r'\b\w*(\w)\1\w*\b', w)]
['effort', 'flee', 'oddball', 'tool']

working with matched portions

# re.Match object
>>> re.search(r'so+n', 'too soon a song snatch')
<re.Match object; span=(4, 8), match='soon'>

# retrieving the entire matched portion, note the use of [0]
>>> motivation = 'Doing is often better than thinking of doing.'
>>> re.search(r'of.*ink', motivation)[0]
'often better than think'

# capture group example
>>> purchase = 'coffee:100g tea:250g sugar:75g chocolate:50g'
>>> m = re.search(r':(.*?)g.*?:(.*?)g.*?chocolate:(.*?)g', purchase)
# to get the matched portion of the second capture group
>>> m[2]
'250'

# to get a tuple of all the capture groups
>>> m.groups()
('100', '250', '50')

examples for re.finditer()

# numbers < 350
>>> m_iter = re.finditer(r'\d+', '45 349 651 593 4 204 350')
>>> [m[0] for m in m_iter if int(m[0]) < 350]
['45', '349', '4', '204']

# start and end+1 index of each matching portion
>>> m_iter = re.finditer(r'so+n', 'song too soon snatch')
>>> for m in m_iter:
...     print(m.span())
... 
(0, 3)
(9, 13)

examples for re.sub()

# add something to the start of every line
>>> ip_lines = "catapults\nconcatenate\ncat"
>>> print(re.sub(r'^', '* ', ip_lines, flags=re.M))
* catapults
* concatenate
* cat

# replace 'par' only at the start of a word
>>> re.sub(r'\bpar', 'X', 'par spar apparent spare part')
'X spar apparent spare Xt'

# same as: r'part|parrot|parent'
>>> re.sub(r'par(en|ro)?t', 'X', 'par part parrot parent')
'par X X X'

# remove first two columns where : is delimiter
>>> re.sub(r'\A([^:]+:){2}', '', 'apple:123:banana:cherry')
'banana:cherry'

backreferencing in the replacement section

# remove any number of consecutive duplicate words separated by space
# use \W+ instead of space to cover cases like 'a;a<-;a'
>>> re.sub(r'\b(\w+)( \1)+\b', r'\1', 'aa a a a 42 f_1 f_1 f_13.14')
'aa a 42 f_1 f_13.14'

# add something around the matched strings
>>> re.sub(r'\d+', r'(\g<0>0)', '52 apples and 31 mangoes')
'(520) apples and (310) mangoes'

# swap words that are separated by a comma
>>> re.sub(r'(\w+),(\w+)', r'\2,\1', 'good,bad 42,24')
'bad,good 24,42'

# example with both capturing and non-capturing groups
>>> re.sub(r'(\d+)(?:abc)+(\d+)', r'\2:\1', '1000abcabc42 12abcd21')
'42:1000 12abcd21'

using functions in the replacement section of re.sub()

>>> from math import factorial
>>> numbers = '1 2 3 4 5'
>>> def fact_num(n):
...     return str(factorial(int(n[0])))
... 
>>> re.sub(r'\d+', fact_num, numbers)
'1 2 6 24 120'

# using lambda
>>> re.sub(r'\d+', lambda m: str(factorial(int(m[0]))), numbers)
'1 2 6 24 120'

examples for lookarounds

# change 'cat' only if it is not followed by a digit character
# note that the end of string satisfies the given assertion
# 'catcat' has two matches as the assertion doesn't consume characters
>>> re.sub(r'cat(?!\d)', 'dog', 'hey cats! cat42 cat_5 catcat')
'hey dogs! cat42 dog_5 dogdog'

# change whole word only if it is not preceded by : or -
>>> re.sub(r'(?<![:-])\b\w+', 'X', ':cart <apple -rest ;tea')
':cart <X -rest ;X'

# extract digits only if it is preceded by - and followed by ; or :
>>> re.findall(r'(?<=-)\d+(?=[:;])', '42 apple-5, fig3; x-83, y-20: f12')
['20']

# words containing 'b' and 'e' and 't' in any order
>>> words = ['sequoia', 'questionable', 'exhibit', 'equation']
>>> [w for w in words if re.search(r'(?=.*b)(?=.*e).*t', w)]
['questionable', 'exhibit']

# match if 'do' is not there between 'at' and 'par'
>>> bool(re.search(r'at((?!do).)*par', 'fox,cat,dog,parrot'))
False
# match if 'go' is not there between 'at' and 'par'
>>> bool(re.search(r'at((?!go).)*par', 'fox,cat,dog,parrot'))
True

examples for re.compile()

Regular expressions can be compiled using the re.compile() function, which gives back a re.Pattern object. The top level re module functions are all available as methods for this object. Compiling a regular expression helps if the RE has to be used in multiple places or called upon multiple times inside a loop (speed benefit). By default, Python maintains a small list of recently used RE, so the speed benefit doesn't apply for trivial use cases.

>>> pet = re.compile(r'dog')
>>> type(pet)
<class 're.Pattern'>
>>> bool(pet.search('They bought a dog'))
True
>>> bool(pet.search('A cat crossed their path'))
False

>>> pat = re.compile(r'\([^)]*\)')
>>> pat.sub('', 'a+b(addition) - foo() + c%d(#modulo)')
'a+b - foo + c%d'
>>> pat.sub('', 'Hi there(greeting). Nice day(a(b)')
'Hi there. Nice day'

Understanding Python re(gex)? book🔗

Visit my GitHub repo Understanding Python re(gex)? for details about the book I wrote on Python regular expressions. The book uses plenty of examples to explain the concepts from the basics and introduces more advanced concepts step-by-step. The book also covers the third-party regex module. The cheatsheet and examples presented in this post are based on the contents of this book.

You can get all my ebooks as a single bundle via leanpub or gumroad.

Hating Trump More Won't Make Things Better

Rondam Ramblings — Mon, 09 Jun 2025 02:49:16 GMT

It has been nearly five months now since I published my open letter to Democratic candidates and organizations. Since then I have, unsurprisingly, received dozens of texts and emails asking me to "Donate $5 now!" For a while I responded to every one pointing them to my Open Letter and asking them to read it. I was expecting (hoping for?) one of three responses. 1) "You are

Vibe Coding vs Agentic Coding

smcleod.net — Thu, 05 Jun 2025 18:10:00 GMT

From Creative Exploration to Production Quality

Mobile blogging, the past and the future

Henri Bergius — Thu, 05 Jun 2025 03:00:00 GMT

This blog has been running more or less continuously since mid-nineties. The site has existed in multiple forms, and with different ways to publish. But what’s common is that at almost all points there was a mechanism to publish while on the move.

Psion, documents over FTP

In the early 2000s we were into adventure motorcycling. To be able to share our adventures, we implemented a way to publish blogs while on the go. The device that enabled this was the Psion Series 5, a handheld computer that was very much a device ahead of its time.

The Psion had a reasonably sized keyboard and a good native word processing app. And battery life good for weeks of usage. Writing while underway was easy. The Psion could use a mobile phone as a modem over an infrared connection, and with that we could upload the documents to a server over FTP.

Server-side, a cron job would grab the new documents, converting them to HTML and adding them to our CMS.

In the early days of GPRS, getting this to work while roaming was quite tricky. But the system served us well for years.

If we wanted to include photos to the stories, we’d have to find an Internet cafe.

To the Alps is a post from these times. Lots more in the motorcycling category

SMS and MMS

For an even more mobile setup, I implemented an SMS-based blogging system. We had an old phone connected to a computer back in the office, and I could write to my blog by simply sending a text. These would automatically end up as a new paragraph in the latest post. If I started the text with NEWPOST, an empty blog post would be created with the rest of that message’s text as the title.

In the Caucasus is a good example of a post from this era

As I got into neogeography, I could also send a NEWPOSITION message. This would update my position on the map, connecting weather metadata to the posts.

As camera phones became available, we wanted to do pictures too. For the Death Monkey rally where we rode minimotorcycles from Helsinki to Gibraltar, we implemented an MMS-based system. With that the entries could include both text and pictures. But for that you needed a gateway, which was really only realistic for an event with sponsors.

Mystery of the Missing Monkey is typical. Some more in Internet Archive

Photos over email

A much easier setup than MMS was to slightly come back to the old Psion setup, but instead of word documents, sending email with picture attachments. This was something that the new breed of (pre-iPhone) smartphones were capable of. And by now the roaming question was mostly sorted.

And so my blog included a new “moblog” section. This is where I could share my daily activities as poor-quality pictures. Sort of how people would use Instagram a few years later.

Internet Archive has some of my old moblogs but nowadays, I post similar stuff on Pixelfed

Pause

Then there was sort of a long pause in mobile blogging advancements. Modern smartphones, data roaming, and WiFi hotspots had become ubiquitous.

In the meanwhile the blog also got migrated to a Jekyll-based system hosted on AWS. That means the old Midgard-based integrations were off the table.

And I traveled off-the-grid rarely enough that it didn’t make sense to develop a system.

But now that we’re sailing offshore, that has changed. Time for new systems and new ideas. Or maybe just a rehash of the old ones?

Starlink, Internet from Outer Space

Most cruising boats - ours included - now run the Starlink satellite broadband system. This enables full Internet, even in the middle of an ocean, even video calls! With this, we can use normal blogging tools. The usual one for us is GitJournal, which makes it easy to write Jekyll-style Markdown posts and push them to GitHub.

However, Starlink is a complicated, energy-hungry, and fragile system on an offshore boat. The policies might change at any time preventing our way of using it, and also the dishy itself, or the way we power it may fail.

But despite what you’d think, even on a nerdy boat like ours, loss of Internet connectivity is not an emergency. And this is where the old-style mobile blogging mechanisms come handy.

Any of the 2025 Atlantic crossing posts is a good example of this setup in action

Inreach, texting with the cloud

Our backup system to Starlink is the Garmin Inreach. This is a tiny battery-powered device that connects to the Iridium satellite constellation. It allows tracking as well as basic text messaging.

When we head offshore we always enable tracking on the Inreach. This allows both our blog and our friends ashore to follow our progress.

I also made a simple integration where text updates sent to Garmin MapShare get fetched and published on our blog. Right now this is just plain text-based entries, but one could easily implement a command system similar to what I had over SMS back in the day.

One benefit of the Inreach is that we can also take it with us when we go on land adventures. And it’d even enable rudimentary communications if we found ourselves in a liferaft.

There are various InReach integration hacks that could be used for more sophisticated data transfer

Sailmail and email over HF radio

The other potential backup for Starlink failures would be to go seriously old-school. It is possible to get email access via a SSB radio and a Pactor (or Vara) modem.

Our boat is already equipped with an isolated aft stay that can be used as an antenna. And with the popularity of Starlink, many cruisers are offloading their old HF radios.

Licensing-wise this system could be used either as a marine HF radio (requiring a Long Range Certificate), or amateur radio. So that part is something I need to work on. Thankfully post-COVID, radio amateur license exams can be done online.

With this setup we could send and receive text-based email. The Airmail application used for this can even do some automatic templating for position reports. We’d then need a mailbox that can receive these mails, and some automation to fetch and publish.

Sailmail and No Foreign Land support structured data via email to update position. Their formats could be useful inspiration

Mapping latitude and longitude to country, state, or city

Austin Z. Henley's Blog — Wed, 04 Jun 2025 03:00:01 GMT

https://austinhenley.com/blog/coord2state.html

Customizing pandoc to generate beautiful pdf and epub from markdown

learnbyexample — Wed, 04 Jun 2025 03:00:00 GMT

Either you've already heard of pandoc or if you have searched online for markdown to pdf or similar, you are sure to come across pandoc. This tutorial will help you use pandoc to generate pdf and epub versions from a GitHub style markdown file.

The main motivation for this blog post is to highlight the customizations I used for self-publishing my ebooks. It wasn't easy to arrive at the setup I ended up with, so I hope this will be useful for those looking to use pandoc for such a purpose. This guide is specifically aimed at technical books that has code snippets.

Poster created using Canva

Installation🔗

If you use a debian based distro like Ubuntu, the below steps are enough for the demos in this tutorial. If you get an error or warning, search that issue online and you'll likely find what else has to be installed.

I first downloaded deb file from pandoc: releases and installed it. Followed by packages needed for pdf generation.

# latest pandoc version as of 19 May 2025
$ sudo gdebi ~/Downloads/pandoc-3.7.0.1-1-amd64.deb

# note that download size is hundreds of MB
$ sudo apt install texlive-xetex
$ sudo apt install librsvg2-bin
$ sudo apt install texlive-science

For more details and instructions for other operating systems, refer to pandoc: installation.

Minimal example🔗

Once pandoc is working on your system, try generating a sample pdf without any customization.

See learnbyexample.github.io repo for all the input and output files referred in this tutorial.

$ pandoc sample_1.md -f gfm -o sample_1.pdf

Here sample_1.md is input markdown file and -f is used to specify that the input format is GitHub style markdown. The -o option specifies the output file type based on extension. The default output is probably good enough. But I wished to customize hyperlinks, inline code style, add page breaks between chapters, etc. This blog post will discuss these customizations one by one.

pandoc has its own flavor of markdown with many useful extensions — see pandoc: pandocs-markdown for details. GitHub style markdown is recommended if you wish to use the same source (or with minor changes) in multiple places.

It is advised to use markdown headers in order without skipping — for example, H1 for chapter heading and H2 for chapter sub-section, etc is fine. H1 for chapter heading and H3 for sub-section is not. Using the former can give automatic index navigation on ebook readers.

On Evince reader, the index navigation for above sample looks like this:

Chapter breaks🔗

As observed from the previous demo, there are no chapter breaks by default. Searching for a solution online, I got this piece of tex code:

\usepackage{sectsty}
\sectionfont{\clearpage}

This can be added using the -H option. From pandoc manual:

-H FILE, --include-in-header=FILE

Include contents of FILE, verbatim, at the end of the header. This can be used, for example, to include special CSS or JavaScript in HTML documents. This option can be used repeatedly to include multiple files in the header. They will be included in the order specified. Implies --standalone.

The pandoc invocation now looks like:

$ pandoc sample_1.md -f gfm -H chapter_break.tex -o sample_1_chapter_break.pdf

You can add further customization to headings, for example:

\sectionfont{\underline\clearpage} to underline chapter names
\sectionfont{\LARGE\clearpage} to allow chapter names to get even bigger

Here are some more links to read about various customizations:

Changing settings via -V option🔗

-V KEY[=VAL], --variable=KEY[:VAL]

Set the template variable KEY to the value VAL when rendering the document in standalone mode. This is generally only useful when the --template option is used to specify a custom template, since pandoc automatically sets the variables used in the default templates. If no VAL is specified, the key will be given the value true.

The -V option allows to change variable values to customize settings like page size, font, link color, etc. As more settings are changed, better to use a simple script to call pandoc instead of typing the whole command on the terminal.

#!/bin/bash

pandoc "$1" \
    -f gfm \
    --include-in-header chapter_break.tex \
    -V linkcolor:blue \
    -V geometry:a4paper \
    -V geometry:margin=2cm \
    -V mainfont="DejaVu Serif" \
    -V monofont="DejaVu Sans Mono" \
    --pdf-engine=xelatex \
    -o "$2"

mainfont is for normal text
monofont is for code snippets
geometry is for page size and margins
linkcolor will set the color for internal links
- this will also colorize other types of links
- set urlcolor if you want to distinguish URLs and so on for other types
to increase the default font size, use -V fontsize=12pt
- See stackoverflow: change font size if you need even bigger size options

Using xelatex as the pdf-engine helps to use any font installed in your system. One reason I chose DejaVu was because it supported Greek and other Unicode characters that were causing error with other fonts. See tex.stackexchange: Using XeLaTeX instead of pdfLaTeX for some more details.

The pandoc invocation is now through a script:

$ chmod +x md2pdf.sh
$ ./md2pdf.sh sample_1.md sample_1_settings.pdf

Do compare the pdf generated side by side with previous output before proceeding.

On my system, DejaVu Serif did not have italic variation installed, so I had to use sudo apt install ttf-dejavu-extra to get it.

Syntax highlighting🔗

One option to customize syntax highlighting for code snippets is to save one of the pandoc themes and editing it. See stackoverflow: What are the available syntax highlighters? for available themes and more details (as a good practice on stackoverflow, go through all answers and comments — the linked/related sections on sidebar are useful as well).

$ pandoc --print-highlight-style=pygments > pygments.theme

Edit the above file to customize the theme. Use sites like colorhexa to help with color choices, hex values, etc. For this demo, the below settings are changed:

# by default, background is same as normal text
# change it to a shade of gray to easily distinguish code and text
"background-color": "#f8f8f8",

# change italic to false, messes up comments with slashes
# change comment text-color to yet another shade of gray
"Comment": {
    "text-color": "#9c9c9c",
    "background-color": null,
    "bold": false,
    "italic": false,
    "underline": false
},

Inline code

Similar to changing background color for code snippets, I found a solution online for inline code snippets as well.

\usepackage{fancyvrb,newverbs,xcolor}

\definecolor{Light}{HTML}{F4F4F4}

\let\oldtexttt\texttt
\renewcommand{\texttt}[1]{
  \colorbox{Light}{\oldtexttt{#1}}
}

Add --highlight-style pygments.theme and --include-in-header inline_code.tex to the script and generate the pdf again.

With pandoc sample_2.md -f gfm -o sample_2.pdf the output would be:

With ./md2pdf_syn.sh sample_2.md sample_2_syn.pdf the output is:

For my Understanding Python re(gex)? book, by chance I found that using ruby instead of python for REPL code snippets syntax highlighting was better. Snapshot from ./md2pdf_syn.sh sample_3.md sample_3.pdf result is shown below. For python directive, string output gets treated as a comment and color for boolean values isn't easy to distinguish from string values. The ruby directive treats string value as expected and boolean values are easier to spot.

Bullet styling🔗

This stackoverflow Q&A helped for bullet styling.

\usepackage{enumitem}
\usepackage{amsfonts}

% level one
\setlist[itemize,1]{label=$\bullet$}
% level two
\setlist[itemize,2]{label=$\circ$}
% level three
\setlist[itemize,3]{label=$\star$}

Comparing pandoc sample_4.md -f gfm -o sample_4.pdf vs ./md2pdf_syn_bullet.sh sample_4.md sample_4_bullet.pdf gives:

PDF properties🔗

This tex.stackexchange Q&A helped to change metadata. See also pspdfkit: What’s Hiding in Your PDF? and discussion on HN.

\usepackage{hyperref}

\hypersetup{
  pdftitle={My awesome book},
  pdfauthor={learnbyexample},
  pdfsubject={pandoc},
  pdfkeywords={pandoc,pdf,xelatex}
}

./md2pdf_syn_bullet_prop.sh sample_4.md sample_4_bullet_prop.pdf gives:

Adding table of contents🔗

There's a handy option --toc to automatically include table of contents at top of the generated pdf. You can control number of levels using --toc-depth option, the default is 3 levels. You can also change the default string Contents to something else using the -V toc-title option.

./md2pdf_syn_bullet_prop_toc.sh sample_1.md sample_1_toc.pdf gives:

Adding cover image🔗

To add something prior to table of contents, cover image for example, you can use a tex file and include it verbatim. Create a tex file (named as cover.tex here) with content as shown below:

\includegraphics{cover.png}
\thispagestyle{empty}

Then, modify the previous script md2pdf_syn_bullet_prop_toc.sh by adding --include-before-body cover.tex and tada — you get the cover image before table of contents. \thispagestyle{empty} helps to avoid page number on the cover page, see also tex.stackexchange: clear page.

The bash script invocation is now ./md2pdf_syn_bullet_prop_toc_cover.sh sample_5.md sample_5.pdf.

You'll need at least one image in input markdown file, otherwise settings won't apply to the cover image and you may end up with a weird output. sample_5.md used in the command above includes an image. And be careful to use escapes if the image path can contain tex metacharacters.

Stylish blockquote🔗

By default, blockquotes (lines starting with > in markdown) are just indented in the pdf output. To make them standout, tex.stackexchange: change the background color and border of blockquote helped.

Create quote.tex with the contents as shown below. You can change the colors to suit your own preferred style.

\usepackage{tcolorbox}
\newtcolorbox{myquote}{colback=red!5!white, colframe=red!75!black}
\renewenvironment{quote}{\begin{myquote}}{\end{myquote}}

The bash script invocation is now ./md2pdf_syn_bullet_prop_toc_cover_quote.sh sample_5.md sample_5_quote.pdf. The difference between default and styled blockquote is shown below.

Customizing epub🔗

For a long time, I thought epub didn't make sense for programming books. Turned out, I wasn't using the right ebook readers. FBReader was good for novels but not ebooks with code snippets. When I used atril, foliate or calibre ebook-viewer, the results were good.

I didn't know how to use css before trying to generate the epub version. Somehow, I managed to take the default epub.css provided by pandoc and customize it as close as possible to the pdf version. The modified epub.css is available from the learnbyexample.github.io repo. The bash script to generate the epub is shown below and invoked as ./md2epub.sh sample_5.md sample_5.epub. Note that pygments.theme is same as the pdf customization discussed before.

#!/bin/bash

pandoc  "$1" \
        -f gfm \
        --toc \
        --standalone \
        --top-level-division=chapter \
        --highlight-style pygments.theme \
        --css epub.css \
        --metadata=title:"My awesome book" \
        --metadata=author:"learnbyexample" \
        --metadata=lang:"en-US" \
        --metadata=cover-image:"cover.png" \
        -o "$2"

Resource links🔗

More options and workflows for generating ebooks:

pandoc-latex-template — a clean pandoc LaTeX template to convert your markdown files to PDF or LaTeX
Writing a book with pandoc, make, and vim
Quarto — open source scientific and technical publishing system built on Pandoc
quarkdown — a modern Markdown-based typetting system
typst — a new markup-based typesetting system that is powerful and easy to learn
Jupyter Book — open source project for building beautiful, publication-quality books and documents from computational material
- See also fastdoc — the output of fastdoc is an asciidoc file for each input notebook. You can then use asciidoctor to convert that to HTML, DocBook, epub, mobi, and so forth
Mau — template-based markup language, heavily inspired by AsciiDoc
Asciidoctor
- Asciidoc book template
- pdf generation workflow with Asciidoc
Sphinx
- Self-publishing a book with reStructuredText, Sphinx, Calibre, and vim
Bookdown
Emacs orgmode
Markdeep

Miscellaneous

moka pot notes

Tony Finch's blog — Sun, 01 Jun 2025 18:56:02 GMT

In hot weather I like to drink my coffee in an iced latte. To make it, I have a very large Bialetti Moka Express. Recently when I got it going again after a winter of disuse, it took me a couple of attempts to get the technique right, so here are some notes as a reminder to my future self next year.

It’s worth noting that I’m not fussy about my coffee: I usually drink pre-ground beans from the supermarket, with cream (in winter hot coffee) or milk and ice.

basic principle

When I was getting the hang of my moka pot, I learned from YouTube coffee geeks such as James Hoffmann that the main aim is for the water to be pushed through the coffee smoothly and gently. Better to err on the side of too little flow than too much.

I have not had much success trying to make fine temperature adjustments while the coffee is brewing, because the big moka pot has a lot of thermal inertia: it takes a long time for any change in gas level to have any effect on on the coffee flow.

routine

fill the kettle and turn it on
put the moka pot’s basket in a mug to keep it stable
fill it with coffee (mine needs about 4 Aeropress scoops)
tamp it down firmly [1]
when the kettle has boiled, fill the base of the pot to just below the pressure valve (which is also just below the filter screen in the basket)
insert the coffee basket, making sure there are no stray grounds around the edge where the seal will mate
screw on the upper chamber firmly
put it on a small gas ring turned up to the max [2]
leave the lid open and wait for the coffee to emerge
immediately turn the gas down to the minimum [3]
the coffee should now come out in a steady thin stream without spluttering or stalling
when the upper chamber is filled near the mouths of the central spout, it’ll start fizzing or spluttering [4]
turn off the gas and pour the coffee into a carafe

notes

If I don’t tamp the grounds, the pot tends to splutter. I guess tamping gives the puck better integrity to resist channelling, and to keep the water under even pressure. Might be an effect of the relatively coarse supermarket grind?
It takes a long time to get the pot back up to boiling point and I’m not sure that heating it up slower helps. The main risk, I think, is overshooting the ideal steady brewing state too much, but:
With my moka pot on my hob the lowest gas flow on the smallest rings is just enough to keep the coffee flowing without stalling. The flow when the coffee first emerges is relatively fast, and it slows to the steady state several seconds after I turn the heat down, so I think the overshoot isn’t too bad.
This routine turns almost all of the water into coffee, which Hoffmann suggests is a good result, and a sign that the pressure and temperature aren’t getting too high.

Old Desktop to Linux - Wayland is still not ready

Ben E. C. Boyter — Sun, 01 Jun 2025 03:00:00 GMT

Back in August 2021 during the covid madness I updated my rather aged desktop to have new CPU, GPU, PSU and fans. The goal being to make it quieter and get more life out of the system. I also kept it running Windows at the time, since I already have a stable Windows 10 install on it. However it is now 2025 and we are getting close to the end of life time for it in October. I never had any major issues with Windows up till 10. However Microsoft’s increasingly hostile approach to the hardware I paid for can no longer be overlooked. Similiar to how I cancelled netflix recently, I am not paying in order to watch ad’s. Coupled with the invasion of privacy of not allowing local accounts it was time to swich back to some flavour of Linux.

the algebra of dependent types

Tony Finch's blog — Thu, 29 May 2025 03:26:38 GMT

TIL (or this week-ish I learned) why big-sigma and big-pi turn up in the notation of dependent type theory.

I’ve long been aware of the zoo of more obscure Greek letters that turn up in papers about type system features of functional programming languages, μ, Λ, Π, Σ. Their meaning is usually clear from context but the reason for the choice of notation is usually not explained.

I recently stumbled on an explanation for Π (dependent functions) and Σ (dependent pairs) which turn out to be nicer than I expected, and closely related to every-day algebraic data types.

sizes of types

The easiest way to understand algebraic data types is by counting the inhabitants of a type. For example:

the unit type () has one inhabitant, (), and the number 1 is why it’s called the unit type;
the bool type hass two inhabitants, false and true.

I have even seen these types called 1 and 2 (cruelly, without explanation) in occasional papers.

product types

Or pairs or (more generally) tuples or records. Usually written,

    (A, B)

The pair contains an A and a B, so the number of possible values is the number of possible A values multiplied by the number of possible B values.

So it is spelled in type theory (and in Standard ML) like,

    A * B

sum types

Or disjoint union, or variant record. Declared in Haskell like,

        data Either a b
            = Left a
            | Right b

Or in Rust like,

        enum Either<A, B> {
            Left(A),
            Right(B),
        }

A value of the type is either an A or a B, so the number of possible values is the number of A values plus the number of B values.

So it is spelled in type theory like,

    A + B

dependent pairs

In a dependent pair, the type of the second element depends on the value of the first. The classic example is a slice, roughly,

        struct IntSlice {
            len: usize,
            elem: &[i64; len],
        }

(This might look a bit circular, but the idea is that an array [i64; N] must be told how big it is – its size is an explicit part of its type – but an IntSlice knows its own size. The traditional dependent “vector” type is a sized linked list, more like my array type than my slice type.)

The classic way to write a dependent pair in type theory is like,

Σ len: usize . Array(Int, len)

The big sigma binds a variable that has a type annotation, with a scope covering the expression after the dot – similar syntax to a typed lambda expression.

We can expand a simple example like this into a many-armed sum type: either an array of length zero, or an array of length 1, or an array of length 2, … but in a sigma type the discriminant is user-defined instead of hidden.

The number of possible values of the type comes from adding up all the alternatives, a summation just like the big sigma summation we were taught in school.

\underset{a &in A}{&Sum}

When the second element doesn’t depend on the first element, we can count the inhabitants like,

\underset{A}{&Sum}

And the sigma type simplifies to a product type.

telescopes

An aside from the main topic of these notes, I also recently encountered the name “telescope” for a multi-part dependent tuple or record.

The name “telescope” comes from de Bruijn’s AUTOMATH, one of the first computerized proof assistants. (I first encountered de Bruijn as the inventor of numbered lambda bindings.)

dependent functions

The return type of a dependent function can vary according to the argument it is passed.

For example, to construct an array we might write something like,

        fn repeat_zero(len: usize) -> [i64; len] {
            [0; len]
        }

The classic way to write the type of repeat_zero() is very similar to the IntSlice dependent pair, but with a big pi instead of a big sigma:

Π len: usize . Array(Int, len)

Mmm, pie.

To count the number of possible (pure, total) functions A ➞ B, we can think of each function as a big lookup table with A entries each containing a B. That is, a big tuple (B, B, … B), that is, B * B * … * B, that is, B^A. Functions are exponential types.

We can count a dependent function, where the number of possible Bs depends on which A we are passed,

\underset{a &in A}{&Product}

danger

I have avoided the terms “dependent sum” and “dependent product”, because they seem perfectly designed to cause confusion over whether I am talking about variants, records, or functions.

It kind of makes me want to avoid algebraic data type jargon, except that there isn’t a good alternative for “sum type”. Hmf.

Best Way to Run Unsigned Binaries in Terminal on macOS

Donat Studios — Thu, 29 May 2025 03:00:17 GMT

With the introduction of Apple Silicon, macOS became much stricter about running unsigned binaries. It even refuses to execute them in the Terminal. This can be frustrating as many precompiled tools, once easily downloadable from sources like GitHub, may no longer work as expected.

By default, if you try to run an unsigned binary in Terminal, you will see an error like:

Apple could not verify "[binary]" is free of malware that may harm your Mac or compromise your privacy

Instead of offering any help, macOS simply offers to move the binary to the trash. Very frustrating.

You find a lot of complicated solutions online, such as manually unquarantining and self-signing the individual binaries with:

$ xattr -dr com.apple.quarantine ./[binary]
$ codesign -s - --deep --force ./[binary]

Fortunately, there's a simple solution: add Terminal as a system Developer Tool in System Settings.

This will just allow it to run unsigned binaries. No fuss, no muss.

To do this:

Open System Settings
Search for "developer"
Click Allow applications to use developer tools in the sidebar.

If Terminal is not already listed, click the + and search for it:

Search for Terminal in the file dialog that appears and select it.

Once added, ensure the toggle next to Terminal is enabled.

Finally, restart Terminal and everything should now work:

There you have it. You may now run any unsigned binary from Terminal without issues.

Profiling Websites

Weblog on marginalia.nu — Thu, 29 May 2025 03:00:00 GMT

The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements.

The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code.

It turns out to be somewhat difficult to determine what a website that has non-trivial javascript will look like based its source code alone, as this would require us to among other things solve the halting problem.

Better bindings for command line history search

learnbyexample — Tue, 27 May 2025 03:00:00 GMT

Do you find it confusing to use Ctrl+r for searching a command from shell history? I have the following mappings in the ~/.inputrc file, so that I can use the ↑ and ↓ arrow keys instead. Note that this assumes you are using Emacs-style key bindings instead of Vi mode.

"\e[A": history-search-backward
"\e[B": history-search-forward

Normally, when you use the ↑ arrow key, you'll get the previous command from the history. You can repeatedly press the key to get more entries. With the above settings active, this behavior will change if you have some characters already typed before pressing the arrow key — you'll only get entries from the history that match these characters from the start of the command. If there are multiple matches, you can use the ↑ and ↓ keys repeatedly to move backwards and forwards through the list.

If you want the matching to happen anywhere in command (same behavior as Ctrl+r and Ctrl+s), use the following lines instead:

"\e[A": history-substring-search-backward
"\e[B": history-substring-search-forward

The ~/.inputrc file affects any shell using the readline library (for example, programming language REPLs). You can use the bind command and put them in, ~/.bashrc for example, to affect only that particular shell.

bind '"\e[A": history-search-backward'
bind '"\e[B": history-search-forward'

Cheerleading

Notes on software development — Mon, 26 May 2025 03:00:00 GMT

At work we're so absorbed in the difficulties we face that it becomes easy to forget what we appreciate and what we value in our coworkers. On social media we can be so focused on our own work that we forget to interact with others'.

I have heard so many times from founders that they switch between heads down building, disappearing from the community, and then pop up to talk about what they've built. Engineers often say the exact same thing. This kills me. Though I probably still do it to a degree too.

The easiest way for no one to give a shit about what you've done is for you to only interact at periodic intervals that are only convenient for you.

With networking in general the most important time to interact and build relationships is when you do not need anything. By the time you want someone's support or you want someone to care about what you've done it is way too late to start building a relationship.

One of the cheapest and most effective avenues to build genuine relationships is to be the biggest cheerleader you can be.

At work this means when someone writes a fantastic blog post you call it out in a public channel. It means when a coworker's work is featured somewhere you call this out in a public channel. At least, this is what I do at work.

I don't care that I have no leadership role or haven't met these people before. If I see something amazing that I'd like to see more of, I mention it in public and praise the person.

Or when someone shares work they've done in a public channel, you hit all the emojis and you reply "that's awesome".

On social media this means engaging seriously and (more) deeply with what people post. If the default behavior on social media is to passively observe, engaging more deeply can be liking a post. If you already show support through liking, engaging more deeply can be commenting or asking questions or (kindly) pointing out mistakes.

Engagement is a spectrum. Genuine engagement consistently over time builds genuine relationships.

I would not suggest that you do this without feeling it, or I'd rather only encourage you to respond to the degree that you feel. But I personally do feel so strongly when I cheer people on.

So much of life and work is drudgery such that when you see something positive, someone taking initiative, someone with talent or potential doing something with their skills, how can you not feel an overwhelming urge to cheer them on and hope to see more of it? Hope to see it develop?

What's more, I want to be around people who are trying new things and improving themselves. I want to be around people who celebrate. So I in turn try new things and work to improve myself and I celebrate the people around me.

This energy is infectious. And I genuinely think even a single person in a group celebrating publicly changes the group dynamic.

And I don't expect people to reciprocate in the same way or to the same degree that I show support. I'm particularly weird and confident and vocal. But if my cheerleading for some person seems like a complete sink then I'll not continue it and invest my time and energy elsewhere.

Not everybody needs my support and that's ok! It is better that it's spent where it's most needed. There were people who used to cheer me on who don't much anymore and that's totally fine, I still have positive feelings for them for the time they did cheer me on.

Cheerleading electrifies the work atmosphere. It is the proper use of social media. Cheerleading is the imperative of caring individuals as they become more experienced, gain confidence, command a wider audience, and want to continue their own growth through the development of genuine and supportive networks.

And an ambitious, high-achieving, celebratory culture is simply the most fun to be a part of.

I wrote a post on being a cheerleader. pic.twitter.com/mESNjSjQ0n
— Phil Eaton (@eatonphil) May 26, 2025

Elixir/Phoenix Liveview was a mistake

swyx's site RSS Feed — Fri, 23 May 2025 09:36:20 GMT

I made an expensive technical decision on Phoenix Liveview for the Smol Talk webapp about a year ago that I now regret, and am jotting down some notes to self for why.

A 2030 morning routine

Weblog on marginalia.nu — Fri, 23 May 2025 03:00:00 GMT

You wake at 05:30 in the morning, feeling somewhat groggy.

Instead of the alarm clock ringing like it normally does, a cheerful hologram appears: “Hi! I’m Kyle, your new alarm clock assistant!” You get dressed as Kyle explains all of the fantastic things he is capable of.

You head over to the coffee machine. “Hey there! I’m Evan! Are you ready for AI in your coffee? But first - tell me about yourself!”. You ignore Evan’s monologue and close your eyes as the synthetic coffee replacement is brewing. Real coffee costs more than your coffee maker nowadays, so it has to suffice.

Toronto Fish & Chips

Kevin Cox's Blog — Thu, 22 May 2025 22:30:00 GMT

Over the past year I’ve been on a small adventure to find some excellent Fish & Chips in Toronto, Canada. While I haven’t found the perfect chip shop I have gone to a handful of the most popular places and would like to share what I learned.

I’ve been to quite a few restaurants, but I’m only going to discuss the ones that stand out. I don’t think you will be disappointed by any location on this list.

Locations

Len Duckworth Fish & Chips

lenduckworthfishandchips.ca - Last Visited 2025

Thanks to Mark Prosser who sent in this recommendation after the initial list post went out. This one had managed to escape my searching but I eagerly gave it a try.

The fish was excellent, I tried the halibut as they claim it to be their specialty, and it did not disappoint.

The chips were also stellar, if I really nitpick they could have been slightly crispier, but they were great as they are with a crisp outside and fluffy inside.

We shared the Sumptuous Seafood platter and everything on it was top notch. The shrimp was super flavourful and while I rarely enjoy battered scallops these were really good. I would strongly recommend this platter which fairly easily serves two. We also got a side of calamari which was also amazing with some sort of curry mayo dip. To top it off we got the pumpkin pie which was also delicious.

Olde Yorke Fish & Chips

oldeyorkefishandchips.com - Last visited 2025

This is probably the location that I have seen recommended the most. And for good reason! This place serves probably the best battered fish I have ever had. We tried the Cod and Haddock, both were great. The cod was more flavourful, but the haddock was closer to a classic white flaky fish. I definitely want to go back and try the halibut.

The chips however were a bit mediocre. Not bad by and stretch, but when it says “chips” above your door I expect better. The outside was a touch tougher than I would like, and the inside wasn’t fluffy like it should be.

We also tried onion rings and a deep-fried Mars bar which were both fantastic. (I don’t know if they allow subbing in onion rings for the chips but if they do, I would recommend that.)

McNies Fish & Chips

mcnies.com - Last visited 2024

This is an old classic of mine as I practiced karate next door for many years. It seems to mostly fly under the radar online, but I think it deserves to be better known. The highlight here is definitely the chips. They are just perfect with crispy outside and fluffy inside.

The fish however isn’t as good as it could be. The batter in particular is a bit breadier than I would like as opposed to being super light and crispy.

Other things I have tried are the onion rings, which are possibly my favourite ever (I’m super picky about rings) and the scallops which I wouldn’t recommend (but I don’t think battered scallops are my cup of tea).

Sea Witch Fish & Chips

seawitchfc.com - Last visited 2024

This is also an online favourite. One of the easiest to get to as it is just a streetcar or short walk from the 1 subway line. They also have an awesome mural on the side of the building, so it is hard to miss.

While I enjoyed my meal here it did not stand out in any way. Solid fish and chips that I would definitely go to often if I lived nearby, but if you are going out of your way I would probably try one of the other options first.

Overall

I prefer Len Duckworth’s overall. The fish may not have quite been as good as Olde York but it was still excellent and everything else was top of the charts. It is also just a short walk from Main Street Station which makes it very easy to get to from basically anywhere in Toronto. Second place goes to Olde York, the fish is just that good, and I can get great chips elsewhere. McNie’s still has a place in my heart and I would probably go from-time to time if it wasn’t so far away from where I live. While Sea Witch is solid all around I think I would only go back if I was in the area, if I am going out of my way I would probably find something new first.

Others?

Did I miss your favourite? Let me know and I’ll update the list after some thorough research!

Debugging memory leaks in Postgres, heaptrack edition

Notes on software development — Thu, 22 May 2025 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Arities as pseudo-protocol

Send More Paramedics — Mon, 19 May 2025 14:44:49 GMT

Whereby I talk about an under-appreciated technique in Clojure where function arities serve as a pseudo-protocol...

Kubernetes at the (homelab) Edge

Pete Keen — Mon, 19 May 2025 03:00:00 GMT

As I've mentioned before, the RF environment in my house is difficult. The house layout is roughly:

single level house, half of which has a basement under it
single level office / mother-in-law-suite / ADU / whatever you want it
two car garage in between
shed in the back yard

The house and ADU are built out of cinder block and brick on the outside and plasterboard (not drywall) on the inside with foil-backed insulation sandwiched in between. The garage is built in between the two buildings, half with the same cinder block and brick construction and half with more modern stick construction.

These buildings were built in the 1950s when labor was cheap, longevity was valued, and AM radio stations were extremely powerful.

The shed is just an ordinary stick and OSB sheathing shed, but it's quite a distance from the house proper.

I consider the three buildings and the garage as separate "RF zones", for lack of a better word. Zigbee and Wi-Fi at 2.4GHz and Wi-Fi 5 at 5GHz do not propagate through the block and foil very well or at all. Z-Wave (900MHz) has a slightly better time but because the house is so spread out and Z-Wave has a fairly low hop limit for mesh packets (4 hops, vs at least 15 for Zigbee) repeaters don't work very well. Lutron Caseta (434MHz) has phenominal range and penetrates the foil with zero issues, but the device variety is severely limited. In particular, there are no Caseta smart locks.

Each zone has:

At least one Wi-Fi access point
A Z-Wave hub
(sometimes) a Zigbee hub
(sometimes) RS232 or RS485 to USB converters for equipment like our generator and furnace

Previous Solutions

The Z-Wave and Zigbee "hubs" are mostly just USB sticks stuck into a free port on a Dell Wyse 3040 thin client. I've had these little machines running for almost five years through a few different setups.

First, I had ser2net running over Tailscale and had the gateway software, i.e. zwave-js-ui and zigbee2mqtt, running on a server in my house. This was fine, until I had some significant clock skew when a node rebooted (i.e. the CMOS battery was dead and the hunk of junk thought it was 2016) and Tailscale refused to start because it thought the SSL certificate for the Tailscale control plane wasn't valid.

The second draft was to just run the gateway software directly on the 3040s. This actually works fine. The 3040s are capable machines, roughly comparable to a Raspberry Pi 4, so they could run a little javascript program just fine. It was somewhat less responsive than running the gateway on the server, though.

The third version of this is to use a hardware gateway. I'm currently using one for Zigbee because the location of the house zone's 3040 is not ideal for some important Zigbee devices and they lose connection a lot. I positioned the hardware gateway in a spot that has good Wi-Fi coverage but no Ethernet port and now those devices are rock solid.

But what if Kubernetes?

While rolling out Kubernetes I didn't really plan on converting the 3040s because running the gateway software on them was working fine. Just as an experiment I attempted to install Talos on one of my spares. Amazingly, it worked great after making one tweak to the install image. The 3040s are very particular about certain things and they, like many of the SBCs that Talos supports, don't like the swanky console dashboard. After turning that off the machine came right up as a Kubernetes node in the cluster.

At idle the Kubernetes workloads plus my cluster's standard DaemonSet pods use about 40% of the machine's 2GiB of memory and roughly 30% of CPU.

That leaves way more than enough to run ser2net.

Automatic ser2net Config

I initially thought that I would use Akri to spawn ser2net. Akri is a project that came out of Microsoft that acts as a generic device plugin for Kubernetes as well as managing what they call "brokers", which are just programs that attach to whatever device and provide it to the cluster.

That sounded perfect for my purposes so I set it up and let it bake for a few days. It did not go well.

The big problem is that Akri is just not very stable. Things were randomly falling over in such a way that the Akri-managed Z-Wave ser2net brokers would crash loop overnight. I made no debugging progress so I started on my fallback idea: automatically managing a ser2net config.

Realistically, my needs are simple:

I want to present one or more serial devices to the network.
I want this to be reliable.
I want this to be secure.
I don't want to micromanage it.

It turns out, discovering USB serial devices on linux is actually pretty trivial. You just have to follow a bunch of symlinks.

This shell script is based on logic found in the go-serial project:

set -e
        set -x
        
        # Find all of the TTYs with names we might be interested in
        ttys=$(ls /sys/class/tty | egrep '(ttyS|ttyHS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO|ttymxc)[0-9]{1,3}')
        
        for tty in $ttys; do
        # follow the symlink to find the real device
        realDevice=$(readlink -f /sys/class/tty/$tty/device)
        
        # what subsystem is it?
        subsystem=$(basename $(readlink -f $realDevice/subsystem))
        
        # locate the directory where the usb information is
        usbdir=""
        if [ "$subsystem" = "usb-serial" ]; then
        # usb-serial is two levels up from the tty
        usbdir=$(dirname $(dirname $realDevice))
        elif [ "$subsystem" = "usb" ]; then
        # regular usb is one level up from the tty
        usbdir=$(dirname $realDevice)
        else
        # we don't care about this device
        continue
        fi
        
        # read the productId and vendorId attributes from the USB device
        productId=$(cat $usbdir/idProduct)
        vendorId=$(cat $usbdir/idVendor)
        
        snippetFile="$vendorId:$productId.yaml"
        
        if [ -f "$snippetFile" ]; then
        sed "s/DEVNODE/\/dev\/$tty/" $snippetFile
        fi
        done

The last few lines of the loop body look for a YAML snippet available for the specific vendorId/productId pair. If there is one, replace the constant DEVNODE with the actual device path and write it to stdout.

Here's what one of the snippets looks like:

# zigbee
        
        connection: &zigbee
        accepter: tcp,6639
        connector: serialdev,DEVNODE,115200n81,local,dtr=off,rts=off
        options:
        kickolduser: true

This snippet tells ser2net to open a listening connection on TCP port 6639 and wire it to a serial device at path DEVNODE. Use 115200n81 parity and keep the dtr and rts bits off (this is specific to the Sonoff zigbee stick I'm using). Further, when a new connection opens immediately kick off the old one.

I drop the above script, the YAML snippets, and this simple entrypoint.sh script into a container image based on the standard ser2net container.

#!/bin/bash
        
        set -e
        set -x
        set -o pipefail
        
        echo "Generating ser2net.yaml"
        
        mkdir -p /etc/ser2net
        ./discover.sh > /etc/ser2net/ser2net.yaml
        
        echo "Running ser2net"
        
        cat /etc/ser2net/ser2net.yaml
        
        exec ser2net -d -l -c /etc/ser2net/ser2net.yaml

I then deploy it to my cluster as a DaemonSet targeting nodes labeled with keen.land/serials=true:

apiVersion: apps/v1
        kind: DaemonSet
        metadata:
        name: ser2net
        namespace: iot-system
        labels:
        app: ser2net
        spec:
        selector:
        matchLabels:
        app: ser2net
        template:
        metadata:
        labels:
        app: ser2net
        spec:
        nodeSelector:
        keen.land/serials: "true"
        volumes:
        - name: devices
        hostPath:
        path: /dev
        imagePullSecrets:
        - name: ghcr.io
        containers:
        - name: ser2net
        image: "ghcr.io/peterkeen/ser2net-auto:main"
        securityContext:
        privileged: true
        volumeMounts:
        - name: devices
        mountPath: /dev
        restartPolicy: Always
        hostNetwork: true

The interesting things here are that the pod is running in privileged mode on the host network. I thought about using container ports but then I would have to somehow know what DaemonSet pod name mapped to which host. With hostNetwork: true I don't have to think about that and can just use the host's name in my gateway configs. There's an opportunity here to cook up a custom deployment type with something like Metacontroller, which I have installed in the cluster but as of yet haven't done anything with.

Pros and Cons

So all of this work and what do I have? Basically what I had when I started:

ser2net on the devices with the serial ports
gateway software running on a server

The big pro of this setup is that I have one consistent management interface. I can set up and tear down ser2net with the exact same interface I use to set up and tear down everything else in the cluster.

There are a couple of cons, too. First, this probably uses a touch more power than the old solution because in addition to ser2net the Wyse 3040s are running all of the Kubernetes infrastructure. These things use so little power as is that I don't think it really matters, but it's worth pointing out.

Second, there's more to go wrong. Before this I had Alpine running basically nothing except ser2net. The system was static in practice, meaning that there was very little that could break.

Now there are several components running all the time that could break with a bad upgrade and could require me to take a crash cart out to each machine.

This is also putting more stress on the drives. All of these machines are booting off of significantly overprovisioned high endurance SD cards now so that shouldn't be an issue, but it's still something to keep in mind. The nice thing is that they're entirely stateless so swapping the card and reinstalling should be a quick operation.

Ultimately I think this is a good move and I plan to continue down the path of making every non-laptop device run on Kubernetes with very few exceptions.

Gemini Figured Out My Nephew’s Name

Beetle Space — Sat, 17 May 2025 10:00:00 GMT

I wrote an MCP server to give LLMs read-only access to my emails. Here is the chat log.

ME:

I need to know Donovan’s son’s name. It is buried in some email somewhere. Devise a strategy for finding it. Do NOT use any tools till we’ve discussed …

The two kinds of technology minimalists

jodavaho.io — Sat, 17 May 2025 03:00:00 GMT

If you were to take all technologists and instill in them a values of simplicity and re-use and longevity, and an obsession with repurposing old tech, I think you’d find they roughly sort into two camps:

One would take on an intellectual journey of discovery, sailing off into the pacific to learn to live off grid, writing colorful narrative-driven articles, full of artwork, mission statements, and subtle suggestions of donations. They would emphasize the human nature of discovery and seek to engage you as they went. Come to the enlightened tech path! Some neat new technology would be built along the way around these principles, mostly for their own use.

https://100r.co/site/projects.html

But the other group would write a self-hosting Forth targeting the top 10 most used CPU platforms of all time and post a text-only HTML site with download links. It would work everywhere. You’d know nothing about them and they’d probably run linux on a 17 year old mac.

https://duskos.org/

Personally, I’m squarely in the second camp, but I wish dearly I had the creativity and spine to be in the first camp.

“But your blog is styled!” Yeah, well that was all AI, see!?

CLI computation with GNU datamash

learnbyexample — Fri, 16 May 2025 03:00:00 GMT

I'm hoping this post will serve as a quick reference for some of the use cases and tickle your curiosity if you haven't come across this nifty CLI text processing tool yet. There are also links for further reading at the end.

Installation and Documentation🔗

See download page for source code and instructions to install the software on various platforms. This blog post is based on the 1.8 version.

See datamash manual for links to documentation in HTML, plain text, PDF, etc.

Sum🔗

# file with a single number per line
$ cat nums.txt
42
-2
10101
-3.14
-75
$ datamash sum 1 <nums.txt
10062.86

$ echo '3.14 42 1000 -51' | tr ' ' '\n' | datamash sum 1
994.14

# summing a particular column
# tab is the default field separator
$ cat table.txt
brown bread mat hair 42
blue cake mug shirt -7
yellow banana window shoes 3.14
$ datamash -t' ' sum 5 <table.txt
38.14

Other such operations include count, min, max, mean, median, sstdev (standard deviation), etc.

Transpose and Reverse🔗

$ cat scores.csv
Name,Maths,Physics,Chemistry
Ith,100,100,100
Cy,97,98,95
Lin,78,83,80
Er,60,70,90

# interchange rows and columns
$ datamash -t, transpose <scores.csv
Name,Ith,Cy,Lin,Er
Maths,100,97,78,60
Physics,100,98,83,70
Chemistry,100,95,80,90

# reverse columns
$ datamash -t, reverse <scores.csv
Chemistry,Physics,Maths,Name
100,100,100,Ith
95,98,97,Cy
80,83,78,Lin
90,70,60,Er

Group by🔗

You can use the -g option to group items based on one or more columns. You can specify an operation such as collapse, sum, mean, count and so on. See Grouping rows by categories avoiding repetition for an example with unique.

# here, the first column items are already next to each other
# so, sorting is not needed
$ cat toys.txt
car blue
car red
car yellow
truck brown
bus green
bus maroon
rocket white

# by default a comma is used as the separator between collapsed items
# use 'unique' instead of 'collapse' to avoid duplicates
$ datamash -t' ' -g1 collapse 2 <toys.txt
car blue,red,yellow
truck brown
bus green,maroon
rocket white

# 'count' gives the number of items for the collapsed row
# 'rand' selects a random item for such collapsed rows
# 'first' and 'last' are other choices available
$ datamash -t' ' -g1 count 2 rand 2 <toys.txt
car 3 red
truck 1 brown
bus 2 green
rocket 1 white

Here's an example with header lines as well as having to sort the input (-s). The -c option helps to customize the separator for the grouped items. The -H option is equivalent to using both --header-in and --header-out.

$ cat books.txt
Author,Title
Will Wight,Cradle
John Bierce,Mage Errant
Brandon Sanderson,Mistborn
Domagoj Kurmaic,Mother of Learning
Brandon Sanderson,The Stormlight Archive
Will Wight,The Last Horizon
Brandon Sanderson,Warbreaker

# not sure if there's an option to retain the original header line as is
# you can instead use: (sed -u 1q; datamash -st, -c: -g1 collapse 2) <books.txt
# use --header-in if you don't want the header line in the output
$ datamash -H -st, -c: -g1 collapse 2 <books.txt
GroupBy(Author),collapse(Title)
Brandon Sanderson,Mistborn:The Stormlight Archive:Warbreaker
Domagoj Kurmaic,Mother of Learning
John Bierce,Mage Errant
Will Wight,Cradle:The Last Horizon

Here's an example of summing values based on column 3 items:

$ cat duplicates.csv
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
dark red,sky,rose,555
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333

$ datamash -st, -g3 sum 4 <duplicates.csv
bread,153
flower,333
rose,999
water,333

Average marks:

$ cat result.csv
Amy,maths,90
Amy,physics,75
Joe,maths,79
John,chemistry,77
John,physics,91
Moe,maths,81
Ravi,physics,84
Ravi,chemistry,70
Yui,maths,92

$ datamash -t, -g1 mean 3 <result.csv
Amy,82.5
Joe,79
John,84
Moe,81
Ravi,77
Yui,92

2025 Advice to my old selves

swyx's site RSS Feed — Thu, 15 May 2025 19:32:56 GMT

I turned a birthday recently and it was so busy (DataCouncil + sg flight) that i never really got the chance to sit and reflect. a lot of things are going well, lots more could be better. I'm the only person responsible for preserving the good and fixing the bad. I'm now back on a plane again and on the right mood/medication to write with honesty, sternness and lowered barriers.

the penultimate conditional syntax

Tony Finch's blog — Wed, 14 May 2025 00:49:41 GMT

About half a year ago I encountered a paper bombastically titled “the ultimate conditional syntax”. It has the attractive goal of unifying pattern match with boolean if tests, and its solution is in some ways very nice. But it seems over-complicated to me, especially for something that’s a basic work-horse of programming.

I couldn’t immediately see how to cut it down to manageable proportions, but after reading syntactic musings on match expressions by Yoshua Wuyts a couple of weeks ago, I had an idea. I’ll outline it under the “penultimate conditionals” heading below, after reviewing the UCS and explaining my motivation.

what the UCS?

The ultimate conditional syntax does several things which are somewhat intertwined and support each other.

An “expression is pattern” operator allows you to do pattern matching inside boolean expressions.

Like “match” but unlike most other expressions, “is” binds variables whose scope is the rest of the boolean expression that might be evaluated when the “is” is true, and the consequent “then” clause.
You can “split” tests to avoid repeating parts that are the same in successive branches. For example,
```
    if num < 0 then -1 else
    if num > 0 then +1 else 0
```
can be written
```
    if num < 0 then -1
           > 0 then +1
               else  0
```
The example shows a split before an operator, where the left hand operand is the same and the rest of the expression varies. You can split after the operator when the operator is the same, which is common for “is” pattern match clauses.
Indentation-based syntax (an offside rule) reduces the amount of punctuation that splits would otherwise need. An explicit version of the example above is
```
    if { x { { < { 0 then −1 } };
             { > { 0 then +1 } }; else 0 } }
```
(This example is written in the paper on one line. I’ve split it for narrow screens, which exposes what I think is a mistake in the nesting.)
You can also intersperse let bindings between splits. I doubt the value of this feature, since “is” can also bind values, but interspersed let does have its uses. The paper has an example using let to avoid rightward drift:
```
    if  let tp1_n = normalize(tp1)
        tp1_n is Bot then Bot
        let tp2_n = normalize(tp2)
        tp2_n is Bot then Bot
        let m = merge(tp1_n, tp2_n)
        m is Some(tp) then tp
        m is None then glb(tp1_n, tp2_n)
```
It’s probably better to use early return to avoid rightward drift.

The desugaring uses let bindings when lowering the UCS to simpler constructions.

whence UCS

Pattern matching in the tradition of functional programming languages supports nested patterns that are compiled in a way that eliminates redundant tests. For example, this example checks that e1 is Some(_) once, not twice as written.

    if e1 is Some(Left(lv)) then e2
             Some(Right(rv)) then e3
             None then e4

Being cheeky, I’d say UCS introduces more causes of redundant checks, then goes to great effort to to eliminate redundant checks again.

Splits reduce redundant code at the source level; the bulk of the paper is about eliminating redundant checks in the lowering from source to core language.

I think the primary cause of this extra complexity is treating the is operator as a two-way test rather than a multi-way match. Splits are introduced as a more general (more complicated) way to build multi-way conditions out of two-way tests.

There’s a secondary cause: the tradition of expression-oriented functional languages doesn’t like early returns. A nice pattern in imperative code is to write a function as a series of preliminary calculations and guards with early returns that set things up for the main work of the function. Rust’s ? operator and let-else statement support this pattern directly. UCS addresses the same pattern by wedging calculate-check sequences into if statements, as in the normalize example above.

out of scope

I suspect UCS’s indentation-based syntax will make programmers more likely to make mistakes, and make compilers have more trouble producing nice error messages. (YAML has put me off syntax that doesn’t have enough redundancy to support good error recovery.)

So I wondered if there’s a way to have something like an “is pattern” operator in a Rust-like language, without an offside rule, and without the excess of punctuation in the UCS desugaring.

But I couldn’t work out how to make the scope of variable bindings in patterns cover all the code that might need to use them. The scope needs to extend into the consequent then clause, but also into any follow-up tests – and those tests can branch so the scope might need to reach into multiple then clauses.

The problem was the way I was still thinking of the then and else clauses as part of the outer if. That implied the expression has to be closed off before the then, which troublesomely closes off the scope of any is-bound variables. The solution – part of it, at least – is actually in the paper, where then and else are nested inside the conditional expression.

penultimate conditionals

There are two ingredients:

The then and else clauses become operators that cause early return from a conditional expression.

They can be lowered to a vaguely Rust syntax with the following desugaring rules. The 'if label denotes the closest-enclosing if; you can’t use then or else inside the expr of a then or else unless there’s another intervening if.
then expr ⟼ && break 'if expr
else expr ⟼ || break 'if expr
else expr ⟼ || _ && break 'if expr

There are two desugarings for else depending on whether it appears in an expression or a pattern.

If you prefer a less wordy syntax, you might spell then as => (like match in Rust) and else as || =>. (For symmetry we might allow && => for then as well.)
An is operator for multi-way pattern-matching that binds variables whose scope covers the consequent part of the expression.

The basic form is like the UCS,

scrutinee is pattern

which matches the scrutinee against the pattern returning a boolean result. For example,
```
    foo is None
```
Guarded patterns are like,

scrutinee is pattern && consequent

where the scope of the variables bound by the pattern covers the consequent. The consequent might be a simple boolean guard, for example,
```
    foo is Some(n) && n < 0
```
or inside an if expression it might end with a then clause,
```
    if foo is Some(n) && n < 0 => -1
    // ...
```
Simple multi-way patterns are like,

scrutinee is { pattern || pattern || … }

If there is a consequent then the patterns must all bind the same set of variables (if any) with the same types.

More typically, a multi-way match will have consequent clauses, like

scrutinee is { pattern && consequent || pattern && consequent || => otherwise }

When a consequent is false, we go on to try other alternatives of the match, like we would when the first operand of boolean || is false.

To help with layout, you can include a redundant || before the first alternative.

For example,
```
    if foo is {
        || Some(n) && n < 0 => -1
        || Some(n) && n > 0 => +1
        || Some(n) => 0
        || None => 0
    }
```
Alternatively,
```
    if foo is { Some(n) && ( n < 0 => -1 ||
                             n > 0 => +1 ||
                             => 0 )
             || None => 0 }
```
(They should compile the same way.)

The evaluation model is like familiar shortcutting && and || and the syntax is supposed to reinforce that intuition. The UCS paper spends a lot of time discussing backtracking and how to eliminate it, but penultimate conditionals evaluate straightforwardly from left to right.

The paper briefly mentions as patterns, like

    Some(Pair(x, y) as p)

which in Rust would be written

    Some(p @ Pair(x, y))

The is operator doesn’t need a separate syntax for this feature:

    Some(p is Pair(x, y))

For large examples, the penultimate conditional syntax is about as noisy as Rust’s match, but it scales down nicely to smaller matches. However, there are differences in how consequences and alternatives are punctuated which need a bit more discussion.

dangling syntax

The precedence and associativity of the is operator is tricky: it has two kinds of dangling-else problem.

The first kind occurs with a surrounding boolean expression. For example, when b = false, what is the value of this?

    b is true || false

It could bracket to the left, yielding false:

    (b is true) || false

Or to the right, yielding true:

    b is { true || false }

This could be disambiguated by using different spellings for boolean or and pattern alternatives. But that doesn’t help for the second kind which occurs with an inner match.

    foo is Some(_) && bar is Some(_) || None

Does that check foo is Some(_) with an always-true look at bar

    ( foo is Some(_) ) && bar is { Some(_) || None }

Or does it check bar is Some(_) and waste time with foo?

    foo is { Some(_) && ( bar is Some(_) ) || None }

I have chosen to resolve the ambiguity by requiring curly braces {} around groups of alternative patterns. This allows me to use the same spelling || for all kinds of alternation. (Compare Rust, which uses || for boolean expressions, | in a pattern, and , between the arms of a match.)

Curlies around multi-way matches can be nested, so the example in the previous section can also be written,

    if foo is {
        || Some(n) && n < 0 => -1
        || Some(n) && n > 0 => +1
        || { Some(0) || None } => 0
    }

The is operator binds tigher than && on its left, but looser than && on its right (so that a chain of && is gathered into a consequent) and tigher than || on its right so that outer || alternatives don’t need extra brackets.

examples

I’m going to finish these notes by going through the ultimate conditional syntax paper to translate most of its examples into the penultimate syntax, to give it some exercise.

Here we use is to name a value n, as a replacement for the |> abs pipe operator, and we use range patterns instead of split relational operators:

    if foo(args) is {
        || 0 => "null"
        || n && abs(n) is {
            || 101.. => "large"
            || ..10 => "small"
            || => "medium"
        )
    }

In both the previous example and the next one, we have some extra brackets where UCS relies purely on an offside rule.

    if x is {
        || Right(None) => defaultValue
        || Right(Some(cached)) => f(cached)
        || Left(input) && compute(input) is {
            || None => defaultValue
            || Some(result) => f(result)
        }
    }

This one is almost identical to UCS apart from the spellings of and, then, else.

    if name.startsWith("_")
    && name.tailOption is Some(namePostfix)
    && namePostfix.toIntOption is Some(index)
    && 0 <= index && index < arity
    && => Right([index, name])
    || => Left("invalid identifier: " + name)

Here are some nested multi-way matches with overlapping patterns and bound values:

    if e is {
        // ...
        || Lit(value)
            && Map.find_opt(value) is Some(result)
            => Some(result)
        // ...
        || { Lit(value) ||
             Add(Lit(0), value) ||
             Add(value, Lit(0)) }
            => {
                print_int(value);
                Some(value)
            }
        // ...
    }

The next few examples show UCS splits without the is operator. In my syntax I need to press a few more buttons but I think that’s OK.

    if x == 0 => "zero"
    || x == 1 => "unit"
    || => "?"

    if x == 0 => "null"
    || x > 0 => "positive"
    || => "negative"

    if predicate(0, 1) => "A"
    || predicate(2, 3) => "B"
    || => "C"

The first two can be written with is instead, but it’s not briefer:

    if x is {
        || 0 => "zero"
        || 1 => "unit"
        || => "?"
    }

    if x is {
        || 0 => "null"
        || 1.. => "positive"
        || => "negative"
    }

There’s little need for a split-anything feature when we have multi-way matches.

    if foo(u, v, w) is {
        || Some(x) && x is {
            || Left(_) => "left-defined"
            || Right(_) => "right-defined"
        }
        || None => "undefined"
    }

A more complete function:

    fn zip_with(f, xs, ys) {
        if [xs, ys] is {
            || [x :: xs, y :: ys]
                && zip_with(f, xs, ys) is Some(tail)
                => Some(f(x, y) :: tail)
            || [Nil, Nil] => Some(Nil)
            || => None
        }
    }

Another fragment of the expression evaluator:

    if e is {
        // ...
        || Var(name) && Map.find_opt(env, name) is {
            || Some(Right(value)) => Some(value)
            || Some(Left(thunk)) => Some(thunk())
        }
        || App(lhs, rhs) => // ...
        // ...
    }

This expression is used in the paper to show how a UCS split is desugared:

    if Pair(x, y) is {
        || Pair(Some(xv), Some(yv)) => xv + yv
        || Pair(Some(xv), None) => xv
        || Pair(None, Some(yv)) => yv
        || Pair(None, None) => 0
    }

The desugaring in the paper introduces a lot of redundant tests. I would desugar straightforwardly, then rely on later optimizations to eliminate other redundancies such as the construction and immediate destruction of the pair:

    if Pair(x, y) is Pair(xx, yy) && xx is {
        || Some(xv) && yy is {
            || Some(yv) => xv + yv
            || None => xv
        }
        || None && yy is {
            || Some(yv) => yv
            || None => 0
        }
    }

Skipping ahead to the “non-trivial example” in the paper’s fig. 11:

    if e is {
        || Var(x) && context.get(x) is {
            || Some(IntVal(v)) => Left(v)
            || Some(BoolVal(v)) => Right(v)
        }
        || Lit(IntVal(v)) => Left(v)
        || Lit(BoolVal(v)) => Right(v)
        // ...
    }

The next example in the paper compares C# relational patterns. Rust’s range patterns do a similar job, with the caveat that Rust’s ranges don’t have a syntax for exclusive lower bounds.

    fn classify(value) {
        if value is {
            || .. -4.0 => "too low"
            || 10.0 .. => "too high"
            || NaN => "unknown"
            || => "acceptable"
        }
    }

I tend to think relational patterns are the better syntax than ranges. With relational patterns I can rewrite an earlier example like,

    if foo is {
        || Some(< 0) => -1
        || Some(> 0) => +1
        || { Some(0) || None } => 0
    }

I think with the UCS I would have to name the Some(_) value to be able to compare it, which suggests that relational patterns can be better than UCS split relational operators.

Prefix-unary relational operators are also a nice way to write single-ended ranges in expressions. We could simply write both ends to get a complete range, like >= lo < hi or like

    if value is > -4.0 < 10.0 => "acceptable"
    || => "far out"

Near the start I quoted a normalize example that illustrates left-aligned UCS expression. The penultimate version drifts right like the Scala version:

    if normalize(tp1) is {
        || Bot => Bot
        || tp1_n && normalize(tp2) is {
            || Bot => Bot
            || tp2_n && merge(tp1_n, tp2_n) is {
                || Some(tp) => tp
                || None => glb(tp1_n, tp2_n)
            }
        }
    }

But a more Rusty style shows the benefits of early returns (especially the terse ? operator) and monadic combinators.

    let tp1 = normalize(tp1)?;
    let tp2 = normalize(tp2)?;
    merge(tp1, tp2)
        .unwrap_or_else(|| glb(tp1, tp2))

antepenultimate breath

When I started writing these notes, my penultimate conditional syntax was little more than a sketch of an idea. Having gone through the previous section’s exercise, I think it has turned out better than I thought it might.

The extra nesting from multi-way match braces doesn’t seem to be unbearably heavyweight. However, none of the examples have bulky then or else blocks which are where the extra nesting is more likely to be annoying. But then, as I said before it’s comparable to a Rust match:

    match scrutinee {
        pattern => {
            consequent
        }
    }

    if scrutinee is {
        || pattern => {
            consequent
        }
    }

The || lines down the left margin are noisy, but hard to get rid of in the context of a curly-brace language. I can’t reduce them to | like OCaml because what would I use for bitwise OR? I don’t want presence or absence of flow control to depend on types or context. I kind of like Prolog / Erlang , for && and ; for ||, but that’s well outside what’s legible to mainstream programmers. So, dunno.

Anyway, I think I’ve successfully found a syntax that does most of what UCS does, but much in a much simpler fashion.

PDF to Text, a challenging problem

Weblog on marginalia.nu — Tue, 13 May 2025 03:00:00 GMT

The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months.

Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format.

It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”. These glyphs may be rotated, overlap, and appear out of order, with very little semantic information attached to them.

Stress And Programming

The Codist — Fri, 09 May 2025 21:04:03 GMT

Having spent four decades as a programmer in various industries and situations, I know that modern software development processes are far more stressful than when I started.

It's not simply that developing software today is more complex than it was back in 1981. In that early decade, none

Switching to Kubernetes

Pete Keen — Fri, 09 May 2025 03:00:00 GMT

And you may ask yourself, "How do I work this?"
And you may ask yourself, "Where is that large home server?"

Once upon a time I had a Mac mini. It was hooked up to the tv (because we only had the one) and it ran Plex. It was fine.

Later, my new spouse and I moved across the country into a house. I decided that I should get a server because I was going to be a big time consultant and I figured I would need a staging environment. A Dell T30 picked up on super sale arrived soon after.

The server sat, ignored, while we suffered through the first few years of one baby, then two babies.

Later, we moved to our forever house and I found Home Assistant. I picked up a Raspberry Pi 4. All was good.

Except it kind of sucked? A 1GB Pi 4 is pretty limited in what it can practically run. Home Assistant ran mostly ok but anything else was beyond it's capabilities. To eBay!

Oooh, shiny hardware

Over the past four years I've accumulated a modest menagerie of hardware:

Hypnotoad, a HP 800 G3 mini
Crushinator, a HP 800 G3 SFF
Morbo, another HP 800 G3 SFF
Lrrr, a Dell Wyse 5070 thin client
Roberto, another Dell Wyse 5070
Nibbler, a Lenovo M80S Gen 3 SFF
Shed, another Dell Wyse 5070 (such a boring name)
A pack of roving Dell Wyse 3040 thin clients
The original Pi 4

The T30, sadly, imploded when I tried to install a video card and fried the motherboard. It's name was Kodos and it was a good box.

Software, take 1 through N

As I was acquiring hardware I was also acquiring software to run on it and developed a somewhat esoteric way of deploying that software. The first interesting version was a self-deploying Docker container. It would get passed the Docker socket and run compose, deciding on the fly what to deploy based on the hostname of the machine.

This was fine, but it proved too much for the 3040s which have fragile 8GB eMMC drives.

A later version moved the script to my laptop and used Ansible to push Docker compose files out to all the machines.

Fine. Fiddly, but fine.

Software, take N + 1

Xe Iaso is a person that I've been following online for years. Recently they went through a homelab transformation, where for Reasons they decided to switch away from NixOS. After trying various things, much to everyone's chagrin, they settled on Kubernetes running on Talos.

Talos seemed to be what I wanted: an immutable, hardened OS designed for one thing and one thing only: Kubernetes.

Much like Xe, I had resisted Kubernetes at home for a long time. Too complex. Too much overhead. Just too much.

Taking another look at that hardware list, though, I do actually have a somewhat Kubernetes-shaped problem. I want to treat my hardware as respected but mostly interchangable pets.

My deployment script was sophisticated but had no ability to just put something somewhere else automatically. It was entirely static, so when something needed to move I would have to restore a backup to the new target and manually redeploy at least part of the world in order to get the ingress set up properly.

Kubernetes takes care of that stuff for me. I don't have to think about where any random workload runs and I don't have to think about migrating it somewhere else if the node falls over. DNS, SSL certificates, backups, it all just happens in the background.

What's it look like?

After a couple of weeks of futzing around and day dreaming I settled on this software stack:

Kubernetes (obvo)
Talos Linux driven by Talhelper
Helm for off the shelf components, driven by Helmfile
Longhorn for fast replicated storage
1Password Operator for secrets management
Tailscale Operator for private ingress and a subnet router to poke at services and pods directly
ingress-nginx for internal and external access to services
MetalLB to give local services a stable virtual IP address
cert-manager for automatic LetsEncrypt certificates for services
external-dns to drive DNS updates for services
Keel.sh for automatic image updates
Reloader to reload deployments when linked secrets and configs update
EMQX as the MQTT server for some of my IoT devices (mostly zigbee)
CloudNative PG for PostgreSQL databases

The next thing to decide was how to divide up the hardware into control plane and worker nodes. Here's what I have so far:

Three (3) control plane nodes: hypnotoad, crushinator, and lrrr
Seven (7) local worker nodes: hypnotoad, crushinator, nibbler, shed, three Wyse 3040s hosting Z-Wave sticks
One (1) cloud worker node

You might notice that several nodes are doing double duty.

Splitting the control plane off to dedicated nodes makes sense when you have a fleet of hundreds of machines in a data center. I don't have that.

A small VM running on Lrrr is the only dedicated control plane node. The only reason for that is because Lrrr also hosts my Unifi and Omada network controllers and I haven't worked up the gumption to move those from Proxmox LXCs to k8s workloads.

Hypnotoad, Crushinator, and Nibbler are general compute. Nibbler has an Nvidia Tesla P4 GPU, which is not particularly impressive but fun to play with. Both Hypnotoad and Nibbler have iGPUs capable of running many simultaneous Jellyfin streams. Crushinator is a VM taking up most of the host which is also serving as a backup NAS for non-media data.

Shed lives in the shed and is connected to a bunch of USB devices, including two SDR radios, a Z-Wave stick, and an RS232-to-USB adapter for the generator.

Morbo is running TrueNAS and has no connection to Kubernetes except that some stuff running in k8s uses NFS shares. It's also the backup target for Longhorn and the script I use to backup Talos' etcd database.

Self-hosting in the Cloud

Talos has a neat feature built in that they call KubeSpan. This is a Wireguard mesh network between all of the nodes in the cluster that uses a hosted discovery service to exchange public keys.

Essentially, you can flip a single option in your Talos configs and have all of your nodes meshed, with a bonus option to send all internal cluster traffic over the Wireguard interface. The discovery service never sees private data, just hashes. It's really cool.

I'm using KubeSpan to put one of my nodes on a VPS to get a public IP without exposing my home ISP connection directly. After initial setup I was able to change the firewall to block all inbound ports other than 80, 443, and the KubeSpan UDP port.

To actually serve public traffic I installed a separate instance of ingress-nginx that only runs on the cloud node. This instance is set up to directly expose the cloud node's public IP which gets picked up by external-dns automatically.

I'm still trying to decide if this single node is enough or if I should get really clever and use a proxy running on Fly to get a public anycast IP.

Ok, but what's running?

Learning how Kubernetes works has been great and this process filled in quite a few gaps in my understanding, but it probably wouldn't have been worth the effort without hosting something useful.

Currently I'm hosting a couple of external production workloads:

this blog
VMSave
a handful of very small websites

I'm also running a bunch of homeprod services:

Home Assistant
Whisper and Piper, speech-to-text and text-to-speech tools and components of the Home Assistant voice pipeline
four (4) instances of Z-Wave JS UI, one per RF "zone" (this house has wacky RF behavior)
two (2) instances of Zigbee2MQTT, one in each RF zone that has Zigbee devices
Genmon keeps tabs on our whole home standby generator
A Minecraft server for me and my kids
Paperless-ngx stores and indexes important documents
Ultrafeeder puts the planes flying overhead on a map
iCloudPD-web for effortless iCloud photo backups
Jellyfin, an indexer and server for TV shows, movies and music
Sonarr, Radarr, Prowlarr and SABnzbd form the core of our media acquisition system
Jellyseerr makes requesting new media easy for the other people who live in the house
Calibre Web Automated is an amazing tool that serves all of my eBooks to my Kobo eReader
Ollama and Open Web UI for dinking around with local LLMs
Homer to keep track of all of the above, set as my browser homepage

Left To Do

There are a few things left on the todo list. Roberto is hooked up to a webcam that watches my 3D printer and I haven't touched it yet because it is connected via Wi-Fi which Talos doesn't support at all.

I also haven't touched the raspberry pi, mostly for the same reason. The pi is serving as a gateway between a Wi-Fi SD card that lives in my CPAP machine and the rest of the network so that I can scrape the data off without having to pull the card or futz with my laptop's Wi-Fi every day.

The Wi-Fi SD card, you see, only exists as an access point. It cannot be put into a mode that connects to another network. The pi has a USB Wi-Fi adapter connected to the card's network and the built-in Wi-Fi connected to the home network with nginx in between serving as a proxy. I don't think this is something that I really want or need to move into k8s.

I want to set up some sort of central authn/authz system for the homeprod services. The current fashion seems to be Pocket ID but I haven't been able to get it working reliably.

I'm thinking about setting up a small ActivityPub server to play around with.

A photo viewer like Immich might be cool to set up.

Overkill?

Of course this is overkill. This could probably all live on a single Wyse 5070 with a couple big harddrives attached.

I think it's been worth it to use Kubernetes in anger. I'm really enjoying the ability to deploy whatever I want to the cluster without having to think about where it runs, where it stores data, etc.

I've also learned a ton and fixed a bunch of preconceived notions and it's already helped increase reliability in a few things that affect household acceptance in big ways.

Everything you need to know about sed substitution

learnbyexample — Wed, 07 May 2025 03:00:00 GMT

The command name sed is derived from stream editor. The most commonly used editing command is substitution, for which various examples are shown in this blog post.

The examples presented here have been tested with GNU sed. Syntax and features might differ for other implementations.

Basic Substitution🔗

The substitute command syntax is s/REGEXP/REPLACEMENT/FLAGS where:

s stands for the substitute command
/ is an idiomatic delimiter character to separate various portions of the command
REGEXP is the regular expression that defines the search portion
REPLACEMENT refers to the replacement string
FLAGS are options to change the default behavior of the command

# for each input line, change only the first ',' to '-'
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'
1-2,3,4
a-b,c,d

# you can change all the matches by adding the 'g' flag
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/=-=/g'
1=-=2=-=3=-=4
a=-=b=-=c=-=d

Filter and Substitute🔗

You can use line numbers, regular expressions or a combination of them to select lines and then apply a command to these filtered lines. See the Selective editing chapter from my ebook to learn more about the various kinds of addressing.

# make changes only to the first line
$ printf '1,2,3,4\na,b,c,d\n' | sed '1 s/,/::/g'
1::2::3::4
a,b,c,d

# apply substitution only if the input line does NOT contain '2'
$ printf '1,2,3,4\na,b,c,d\n' | sed '/2/! s/,/-/g'
1,2,3,4
a-b-c-d

Regular Expressions🔗

Only a handful of examples are shown here. See this chapter from my ebook for a more detailed discussion. See my blog post for a cheatsheet.

Anchors:

# lines starting with 'par'
$ printf 'car par\nparty\nspare\n' | sed 's/^par/tas/'
car par
tasty
spare

# words starting with 'par'
$ printf 'car par\nparty\nspare\n' | sed 's/\bpar/1234/'
car 1234
1234ty
spare

Alternation and Grouping:

# same as: sed -E 's/part|parrot|parent/X/g'
# -E option enables ERE (default is BRE)
$ echo 'par part parrot parent' | sed -E 's/par(en|ro)?t/X/g'
par X X X

Character Class and Quantifiers:

# numbers >= 100 with optional leading zeros
$ echo '0501 035 154 12 26 98234' | sed -E 's/\b0*[1-9][0-9]{2,}\b/X/g'
X 035 X 12 26 X

# retain only punctuation characters
$ echo ',pie tie#ink-eat_42' | sed -E 's/[^[:punct:]]+//g'
,#-_

Backreferences:

# remove two or more duplicate words separated by spaces
# \b prevents false matches like 'the theatre', 'sand and stone' etc
$ echo 'aa a a a 42 f_1 f_1 f_13.14' | sed -E 's/\b(\w+)( \1)+\b/\1/g'
aa a 42 f_1 f_13.14

# whole words that have at least one consecutive repeated character
$ echo 'effort flee facade oddball rat tool' | sed -E 's/\w*(\w)\1\w*/X/g'
X X facade X rat X

# match lowercase followed by underscore followed by lowercase
# delete the underscore and convert the 2nd lowercase to uppercase
$ echo '_fig aug_price next_line' | sed -E 's/([a-z])_([a-z])/\1\u\2/g'
_fig augPrice nextLine

Replace Specific Occurrences🔗

You can use a number as a flag to replace only that particular occurrence of the search term. If you combine this with the g flag, all occurrences after that particular match will also be replaced.

$ s='apple:banana:cherry:fig:mango'

# replace only the second occurrence
$ echo "$s" | sed -E 's/[^:]+/"&"/2'
apple:"banana":cherry:fig:mango

# replace all matches except the first occurrence
$ echo "$s" | sed -E 's/:/---/2g'
apple:banana---cherry---fig---mango

With the help of capture groups and backreferences, you can replace a specific occurrence from the end of the input line.

$ s='car,art,pot,map,urn,ray,ear'

# replace the last occurrence
$ echo "$s" | sed -E 's/(.*),/\1[]/'
car,art,pot,map,urn,ray[]ear

# generic version, where {N} refers to last but Nth occurrence
$ echo "$s" | sed -E 's/(.*),((.*,){3})/\1[]\2/'
car,art,pot[]map,urn,ray,ear

Executing External Commands🔗

The e flag helps to insert the output of a shell command within sed.

# replace the entire line with the output of a shell command
$ printf 'apple\nreplace this line\n' | sed 's/^replace.*/date/e'
apple
Wednesday 07 May 2025 10:25:34 AM IST

# after substitution, the command that gets executed is 'seq 3'
$ echo 'xyz 3' | sed 's/xyz/seq/e'
1
2
3

Different Delimiters🔗

The / character is idiomatically used as the REGEXP delimiter. But any character other than \ and the newline character can be used instead.

# instead of this
$ echo '/home/learnbyexample/reports' | sed 's/\/home\/learnbyexample\//~\//'
~/reports

# use a different delimiter
$ echo '/home/learnbyexample/reports' | sed 's#/home/learnbyexample/#~/#'
~/reports

In-place Editing🔗

The -i option is helpful to write back the changes to the original files itself. If you don't provide an argument to this option, backup of the original file won't be created.

$ cat colors.txt
deep blue
light orange
blue delight

# no output on terminal as the -i option is used
$ sed -i.bkp 's/blue/green/' colors.txt
# output from sed is written back to 'colors.txt'
$ cat colors.txt
deep green
light orange
green delight

# original file is preserved in 'colors.txt.bkp'
$ cat colors.txt.bkp
deep blue
light orange
blue delight

* in the argument to the -i option will be replaced with the input filename. So, -i'bkp.*' for f1.txt will create bkp.f1.txt as the backup. And if you use old/*, the backups will be under the same name but under the directory old (provided that directory already exists).

Manipulating Newlines🔗

By default, sed reads the input line by line (with \n considered as the line ending). The newline character, if present, is removed and then added back when the pattern space is printed. Which implies that you cannot directly manipulate the newline character, unless you use features that results in more than one line in the pattern space.

# append the next line to the pattern space
# and then replace newline character with a colon character
$ seq 7 | sed 'N; s/\n/:/'
1:2
3:4
5:6
7

# if line contains 'at', the next line gets appended to the pattern space
# then the substitution is performed on the two lines in the buffer
$ printf 'gates\nnot\nused\n' | sed '/at/{N; s/s\nnot/d/}'
gated
used

Slurping Input🔗

If the input doesn't have NUL characters, then the -z option is handy to process the entire input as a single string. This is effective only for files small enough to fit the available machine memory. It would also depend on the regular expression, as some patterns have exponential relationship with respect to the data size.

# add ; to the previous line if the current line starts with c
$ printf 'cater\ndog\ncoat\ncutter\nmat\n' | sed -z 's/\nc/;&/g'
cater
dog;
coat;
cutter
mat

Fixed String Substitution🔗

Typically, you'd need to escape \, & and the delimiter for the string used in the replacement section. For the search section, the characters to be escaped will depend upon whether you are using BRE or ERE.

# replacement string
$ r='a/b&c\d'
$ r=$(printf '%s' "$r" | sed 's#[\&/]#\\&#g')

# ERE version for the search string
$ s='{[(\ta^b/d).*+?^$|]}'
$ s=$(printf '%s' "$s" | sed 's#[{[()^$*?+.\|/]#\\&#g')
$ printf '%s\n' 'f*{[(\ta^b/d).*+?^$|]} - 3' | sed -E 's/'"$s"'/'"$r"'/g'
f*a/b&c\d - 3

# BRE version for the search string
$ s='{[(\ta^b/d).*+?^$|]}'
$ s=$(printf '%s' "$s" | sed 's#[[^$*.\/]#\\&#g')
$ printf '%s\n' 'f*{[(\ta^b/d).*+?^$|]} - 3' | sed 's/'"$s"'/'"$r"'/g'
f*a/b&c\d - 3

See my blog post for multiline fixed string substitution examples.

Programming ebooks🔗

Check out my ebooks on Regular Expressions, Linux CLI tools, Python and Vim. You can get them all as a single bundle via leanpub or gumroad.

Building Better SDKs with Generative AI

Caseysoftware — Mon, 05 May 2025 16:38:00 GMT

Through many years of working with APIs, I’ve always been struck by the fundamental problems of SDKs. Most are not…

The post Building Better SDKs with Generative AI appeared first on Caseysoftware.

Where are the proactive AI coding tools?

Austin Z. Henley's Blog — Mon, 05 May 2025 04:00:01 GMT

https://austinhenley.com/blog/proactiveai.html

Viewing adjacent French towns on Wikipedia

Linux Engineer's random thoughts — Mon, 05 May 2025 01:00:00 GMT

It all started with this really simple nerdsnipe by Joachim (french):

How many clicks would it take to go from the Wikipedia pages of (southmost Metropolitan France town) Cerbères to (northmost France town) Bray-les-Dunes, using only Wikipedia's "adjacent communes links" ?

Quite a simple question. Many surprises.

Querying Wikidata

So, we …

Exponentials in 3 Instructions

Speculative Branches — Sun, 04 May 2025 03:00:00 GMT

This post expands on an algorithm shown in the book I wrote on floating-point math.

It is very common in computing to want to do $e^x$ very quickly and not care very much about how accurately you computed it. This is increasingly true in ML and AI algorithms, which can be very tolerant to noise from numerical error and often use low bit precision either way. It also shows up doing things like exponentially-weighted moving averages and other similar functions in signal processing. Thankfully, you can replace $e^x$ in these situations with the following function:

White noise into random walks

Bytepawn - Marton Trencseni — Sun, 04 May 2025 01:00:00 GMT

A visual and analytical walk from pure white noise to a true random walk.

Dead Bird Features

Kevin Cox's Blog — Sat, 03 May 2025 15:00:00 GMT

Anyone who has had an outdoor cat will have experienced a thoughtful gift left on your doorstep in the morning, usually a dead bird or mouse. These gifts are made with love, and take effort to acquire and return to your doorstep. However, us humans generally do not appreciate them.

I often find similar features in software. For example, the wiki that our company uses recently released a new feature: when you copy from a document it would format the copied text as Markdown. For example if I copied part of a code snippet inside a second-level list it would be formatted as:

1.
    1. ```bash
       -funsafe-math-optimizations
       ```

Whoever implemented this feature clearly worked really hard to ensure that parent context like the two lists above were included in this copying. However ultimately it isn’t what users want. When copying the entire list keeping the formatting could be useful, however when copying just one word the context is typically not desired. In this case the result is that I can’t paste it into my terminal.

Often these features are technically interesting. It is a fun coding project to take a snippet of the document and turn it into accurate Markdown. It is easy to forget to take a step back and think if the user actually wants this.

I find that these features often occur when overriding default behaviours. Web apps are a common culprit. Overriding copying is a common example, but also overriding spell checking, form filling, link following or undo. Many times there is a key feature that the developer wanted to provide, and they made impressive effort re-implementing browser behaviour. At best the feature is useful, but the reimplemented browser behaviour falls short of native. But very often the feature that was provided isn’t actually even helpful.

Perfect Random Floating-Point Numbers

Speculative Branches — Sat, 03 May 2025 03:00:00 GMT

When I recently looked at the state of the art in floating point random number generation, I was surprised to see a common procedure in many programming languages and libraries that is not really a floating-point algorithm:

Generate a random integer with bits chosen based on the precision of the format.
Convert to floating point.
Divide to produce an output between 0 and 1.

In code, this looks like:

1func (r *Rand) Float64() float64 {
2    int64 rand_int = r.Int63n(1<<53)
3    return float64(rand_int) / (1<<53)
4}

This function is supposed to produce floating-point numbers drawn from a uniform distribution in the interval $[0, 1)$. Zero is a possible output, but one is not, and the distribution is uniform. The number "53" in the algorithm above is chosen in a way that is floating-point aware: the double-precision floating-point numbers have 53 bits of precision, so this algorithm only creates bits equal to the precision of the number system. It seems to fit the bill.

Driving up an active Volcano in Iceland! Hekkla

The Road Chose Me — Thu, 01 May 2025 16:53:34 GMT

Driving to the top of an active volcano may not be the smartest thing we’ve ever done, but it sure is one of the most beautiful! On today’s adventure we’re in the southern highlands...

testing data structures per element

Tony Finch's blog — Thu, 01 May 2025 04:25:16 GMT

Recently, Alex Kladov wrote on the TigerBeetle blog about swarm testing data structures. It’s a neat post about randomized testing with Zig. I wrote a comment with an idea that was new to Alex @matklad, so I’m reposing a longer version here.

differential testing

A common approach to testing data structures is to write a second reference implementation that has the same API but simpler and/or more obviously correct, though it uses more memory or is slower or less concurrent or otherwise not up to production quality.

Then, run the production implementation and the reference implementation on the same sequence of operations, and verify that they produce the same results.

Any difference is either a bug in the production implementation (probably) or a bug in the reference implementation (unlucky) or a bug in the tests (unfortunate).

This is a straightforward differential testing pattern.

problems

There are a couple of difficulties with this kind of basic differential testing.

grow / shrink

The TigerBeetle article talks about adjusting the probabilities of different operations on the data structure to try to explore more edge cases. To motivate the idea, the article talks about adjusting the probabilities of adding or deleting items: If adding and deleting have equal probability, then the test finds it hard to grow the data structure to interesting sizes that might expose bugs.

Unfortunately, if the probability of add is greater than del, then the data structure tends to grow without bound. If the probability of del is greater than add, then it tries to shrink from nothing: worse than equal probabilities! They could preload the data structure to test how it behaves when it shrinks, but a fixed set of probabilities per run is not good at testing both growth and shrinkage on the same test run on the same data structure.

One way to improve this kind of test is to adjust the probability of add and del dynamically: make add more likely when the data structure is small, and del more likely when it is big. And maybe make add more likely in the first half of a test run and del more likely in the second half.

random elements

The TigerBeetle article glosses over the question of where the tests get fresh elements to add to the data structure. And its example is chosen so it doesn’t have to think about which elements get deleted.

In my experience writing data structures for non-garbage-collected languages, I had to be more deliberate about how to create and destroy elements. That led to a style of test that’s more element-centric, as Alex described it.

element-wise testing

Change the emphasis so that instead of testing that two implementations match, test that one implementation obeys the expected behaviour. No need to make a drop-in replacement reference implementation!

What I typically do is pre-allocate an array of elements, with slots that I can set to keep track of how each element relates to the data structure under test. The most important property is whether the element has been added or deleted, but there might be others related to ordering of elements, or values associated with keys, and so on.

test loop

Each time round the loop, choose at random an element from the array, and an action such as add / del / get / …

Then, if it makes sense, perform the operation on the data structure with the element.

For example, you might skip an add action if the element is already in the data structure, unless you can try to add it and expect an error.

data structure size

This strategy tends to grow the data structure until about 50% of the pre-allocated elements are inserted, then it makes a random walk around this 50% point. Random walks can diverge widely from their central point both in theory and in practice, so this kind of testing is reasonably effective at both growing and (to a lesser extent) shrinking the data structure.

invariants

I usually check some preconditions before an action, to verify that the data structure matches the expected properties of the chosen element. This can help to detect earlier that an action on one element has corrupted another element.

After performing the action and updating the element’s properties, I check the updated properties as a postcondition, to make sure the action had the expected effects.

performance

John Regehr’s great tutorial, how to fuzz an ADT implementation, recommends writing a checkRep() function that thoroughly verifies a data structure’s internal consistency. A checkRep() function is a solid gold testing tool, but it is O(n) at least and typically very slow.

If you call checkRep() frequently during testing, your tests slow down dramatically as your data structure gets larger.

I like my per-element invariants to be local and ideally O(1) or O(log n) at worst, so they don’t slow down the tests too much.

conclusion

Recently I’ve used this pattern to exhibit concurrency bugs in an API that’s hard to make thread-safe. Writing the tests has required some cunning to work out what invariants I can usefully maintain and test; what variety of actions I can use to stress those invariants; and what mix of elements + actions I need so that my tests know which properties of each element should be upheld and which can change.

I’m testing multiple implementations of the same API, trying to demonstrate which is safest. Differential testing can tell me that implementations diverge, but not which is correct, whereas testing properties and invariants more directly tells me whether an implementation does what I expect. (Or gives me a useless answer when my tests are weak.)

Which is to say that this kind of testing is a fun creative challenge. I find it a lot more rewarding than example-based testing.

The design of software engineering course projects

Austin Z. Henley's Blog — Tue, 29 Apr 2025 01:00:01 GMT

https://austinhenley.com/blog/groupprojects.html

My Plan, Document, Act, Review flow for Agentic Software Development

smcleod.net — Sun, 27 Apr 2025 18:10:00 GMT

An overview of my agentic coding development flow.

Random Walks and the Dickey-Fuller Test - Part II

Bytepawn - Marton Trencseni — Sat, 26 Apr 2025 01:00:00 GMT

The article examines how simple autoregressive stochastic processes behave with the Dickey-Fuller test.

LLM Sampling Parameters Guide

smcleod.net — Thu, 24 Apr 2025 18:10:00 GMT

A practical guide to LLM sampling parameters for Ollama, llama.cpp, and MLX

An Appearance on Apropos

Send More Paramedics — Wed, 23 Apr 2025 22:44:49 GMT

I was fortunate to appear on the April 22, 2025 episode of the Apropos podcast...

Debugging A Crawler Stall

Weblog on marginalia.nu — Tue, 22 Apr 2025 03:00:00 GMT

Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client. This seemed like a good idea at the time, but has led to a fair number of headaches.

Java’s HttpClient has one damning flaw, and that that it doesn’t support socket timeouts.

Its only supported timeout values are time to connect, and time until first byte of the response. This means the client can get stuck on a read call if a server stops responding, potentially for a very long time!

Screw your title

Notes on software development — Tue, 22 Apr 2025 03:00:00 GMT

I've been a developer, a manager, a cofounder, and now I'm a developer again. I ran away from each position until being a founder because I felt like I was limited by what I was allowed to do.

But I reached an enlightenment of sorts during my career progression: that everyone around me was dying for someone to pick things up, for employees to show engagement and agency.

We think of our titles as our limits. We're quick to say and believe, "that isn't my job". While in reality titles reflect the minimum expected of us, not the maximum that is open to us.

Moving your career forward

Trying to figure out what (new minimum) you must do to get promoted seems kind of backwards to me, reinforcing our sense of our own limits. Instead, at every stage in your career, focus on doing the intersection of:

what you see needs to be done (that isn't being done)
what you are capable of doing
what you have the desire/energy (or would find fulfillment) doing

And this is the path to promotion and a successful and interesting career.

Screw your title. screw your job description. I mean, keep your boss happy for sure. Keep your teammates happy by supporting them and building them up and communicating well. But don't wait to be officially made a lead or given a new title to do what otherwise fits into that intersection above.

And if after doing this for some time, demonstrating this level of agency, you are not promoted, it just means you're not at the right company or right organization within your company and you should look elsewhere.

What's more, this work you did (at a company that doesn't appreciate your agency, if that happens to be the case) merely makes the case stronger for your successful interview at the next company. There's no downside.

The cynical, and perhaps realistic, alternative to this is to do politics to get promoted. Or to not do politics but to do things that don't align with your long-term goals. I'm not personally interested in either path so I'm not covering them here. I'm interested in the intersection of things that move me in the direction I want, things that are useful to the company, and things that I am capable of doing (in addition to whatever minimum work I must actually do).

Examples

Here's a peek at what this looks like for me as an individual contributor, a programmer, at EnterpriseDB.

I started the EDB Engineering Newsletter because it seemed like we needed to do a better job telling the world the awesome things our engineering team is doing. (You know we're one of the biggest contributors to Postgres? Bruce Momjian, Robert Haas, Peter Eisentraut, etc. work here? The guy who implemented the WAL and MVCC in Postgres is my teammate?) Nobody asked me to do that.

I started publishing blog views for the entire company once a month internally. Nobody asked me to do that.

I wrote a number of internal docs and tutorials on the product because we were just obviously missing them. Nobody asked me to do that.

I started a fortnightly incident review meeting for my team because it seemed like we were missing chances to update docs and teach each other. Nobody asked me to do that.

I write a odd posts for the company blog on what I've learned. Nobody asked me to do that.

These are just a few of the random things that seemed like a good idea for me to do on top of my Actual Work as a developer, which I think I do a decent job of on its own.

In closing

Don't burn out. Don't do things you aren't asked for and don't find rewarding. Or that won't pave the way toward the career you want. I'm trying to be very careful not to advocate anything along those lines.

But also don't wait to be asked to do something. Do what is interesting and obvious and rewarding to you. Interesting opportunities seem to come most reliably when you make them for yourself.

Just one more calculator, bro, just one more, I swear it's the last one

John Graham-Cumming's blog — Mon, 21 Apr 2025 16:17:31 GMT

I'm not much of a collector of things. Long ago I decided that on the retro-computing front I would not collect any machines that weren't very significant to me. So, my "collection" consists of a KIM-1, Sharp MZ-80K, Research Machines 380Z and 480Z, BBC Micro Model B, an IBM ThinkPad 701C, and a Minitel (well, two).

But calculators are where I come closest to not being able to resist. Calculators are there for you. They don't change UI because of the whim of some product manager. They don't need updating because their underlying OS is on some new, crazy version. Nothing gets in the way of you using them: no popups, enticements to buy some extra thing, no notifications, nothing. In an insane world, they are the sanest choice.

I've written in the past about restoring my Rockwell 8R and Commodore P50 (and a fun discovery that Sir Roger Penrose had the same calculator and wrote the same program as me: mine was faster, his more elegant (which is probably some sort of metaphor)). And I have my Casio fx-3600p which I used extensively at school.

I also have an original HP 16C (the one for programmers).

Some time ago I discovered a company called SwissMicros that makes modern replicas of classic HP calculators. I bought two of them. I have their version of the HP-42S:

And their version of the HP-16C:

But recently they started making a credit card-sized version of the HP-16C and I was unable to resist ordering one. And so, after dealing with Portuguese Customs, I got this:

And now I have full size and card size side by side:

I'll try to stop collecting calculators there.

model available not open source

Blogity blog blog. Journal — Sun, 20 Apr 2025 05:04:19 GMT

model available not open source

It’s not open source if I can’t see why the LLM won’t tell me how to make cocaine.

Mark Zuckerberg wants to say Llama is open source. It’s not. Don’t let him.

Open source is a long-storied term that goes back to before the Internet. Don’t get me wrong—it’s nice of him to share the model file. I appreciate that, especially for my home LLM setup. But it doesn’t deserve the “open source” label.

Let me put it this way: Say I gave you a car. Nice gesture, right? Now imagine I also claimed I fucked your partner. That’s not part of the deal. Just because I gave you a car doesn’t mean I get to make wild, unrelated claims.

The GPL dates back to 1989, born from Richard Stallman and the Free Software Foundation. It reflects core values. Copyleft is the ethos: if you use free software, you must share your changes under the same terms. Later versions patched loopholes, but the spirit remained—freedom through openness.

So what does that mean today, three decades later, with the Internet built on open source code?

The thing about source code is that we can read it. Understand what it’s doing. When you’re trying to create a new username and it’s rejected, it’s not because the computer hates you. The computer’s just executing instructions. If the code says “no usernames shorter than 8 characters,” that’s fair. We can read that. But if we can’t see the code? Then who knows? Maybe someone hardcoded a list of disallowed names, and yours is on it. That’s not a joke.

Transparency matters.

With something like GIM-Paint, you can go into the source, see what it’s doing, change it if you don’t like it, and build your own version. That’s freedom. That’s open source.

LLMs are different.

I’ve got access to a supercomputer that could rebuild Llama—if I had the source. I don’t. It’s not just the weights. It’s the training data. The scripts. The intent. And most folks don’t even have the compute to try.

Why do I want the source? Yeah, I’m curious. I like understanding things. That’s my dopamine button. But more than that: I want to see its alignment.

I want to see why it won’t tell me how to make cocaine. Not because I plan to—but because I want to know what other things it refuses to say. It won’t give me a pipe bomb recipe, fine. But it will tell me my grandma’s favorite cocktail recipe?

In China, brilliant engineers built DeepSeek on top of Llama. It won’t talk about Tiananmen Square. What else won’t it talk about? COVID? LGBTQ issues? Uyghur genocide?

If I can’t inspect it, I can’t know.

That’s why “open source” matters. If you don’t give me the source, I can’t see how it works. I can’t trust what it’s been told to do. And I definitely can’t tell if it’s hiding something.

So call it what it is: model-available. But don’t steal the term “open source” just because it sounds better. We know the difference.

Awaiting Dune

Ben Overmyer's Site — Sun, 20 Apr 2025 03:00:00 GMT

I pre-ordered Dune: Awakening at the highest level.

When I first heard that Dune: Awakening was being made by the people who made Conan: Exiles, I was interested but not excited. I thoroughly enjoyed Exiles. My friends were the ones who routinely hunted for larger and more powerful bosses to slay, but my enjoyment focused on taking the resources they would bring back and using them to build interesting structures. In all survival-type games, I tend to do this. I'll venture out and fight monsters, but only in service to acquiring the resources and items I need to build cool things.

The announcement for Dune: Awakening came in August of 2022. I had seen and loved the new Dune movie in 2021, so the new game piqued my interest.

And then... then, I watched an interview with the makers describing what their inspirations were and what features were planned.

They said that one of their primary inspirations was Star Wars: Galaxies.

At this point, my interest changed to rabid fascination. My favorite computer game of all time, Star Wars: Galaxies was the epitome of an open, living world. I sank many hundreds of hours into it before it was shut down in 2011.

To hear, ten years later, that a new game in the style of Conan: Exiles, inspired by Star Wars: Galaxies, and set in the world of Arrakis was being worked on... well, that set my interest on fire.

The second movie being released last year added fuel to that fire.

So of course, as soon as preorders were available, I bought the game's most expensive edition. And I took off two days of work for the early release in May.

Later, I was dismayed to hear that the release was delayed three weeks, to June. I can't take time off of work that particular time period because it coincides with a major work project that I have to lead. However, I will play as much as possible those first days. I may even record myself playing, so I can remember what it was like in the future. We'll see. I don't stream anymore because there are no streaming services outside of American control.

I'm very much looking forward to this.

Transactions are a protocol

Notes on software development — Sun, 20 Apr 2025 03:00:00 GMT

Transactions are not an intrinsic part of a storage system. Any storage system can be made transactional: Redis, S3, the filesystem, etc. Delta Lake and Orleans demonstrated techniques to make S3 (or cloud storage in general) transactional. Epoxy demonstrated techniques to make Redis (and any other system) transactional. And of course there's always good old Two-Phase Commit.

If you don't want to read those papers, I wrote about a simplified implementation of Delta Lake and also wrote about a simplified MVCC implementation over a generic key-value storage layer.

It is both the beauty and the burden of transactions that they are not intrinsic to a storage system. Postgres and MySQL and SQLite have transactions. But you don't need to use them. It isn't possible to require you to use transactions. Many developers, myself a few years ago included, do not know why you should use them. (Hint: read Designing Data Intensive Applications.)

And you can take it even further by ignoring the transaction layer of an existing transactional database and implement your own transaction layer as Convex has done (the Epoxy paper above also does this). It isn't entirely clear that you have a lot to lose by implementing your own transaction layer since the indexes you'd want on the version field of a value would only be as expensive or slow as any other secondary index in a transactional database. Though why you'd do this isn't entirely clear (I will like to read about this from Convex some time).

It's useful to see transaction protocols as another tool in your system design tool chest when you care about consistency, atomicity, and isolation. Especially as you build systems that span data systems. Maybe, as Ben Hindman hinted at the last NYC Systems, even proprietary APIs will eventually provide something like two-phase commit so physical systems outside our control can become transactional too.

Transactions are a protocol

short new post pic.twitter.com/nTj5LZUpUr
— Phil Eaton (@eatonphil) April 20, 2025

Random Walks and the Dickey-Fuller Test - Part I

Bytepawn - Marton Trencseni — Sun, 20 Apr 2025 01:00:00 GMT

This post explores how to use the Dickey–Fuller test to check the random‑walk hypothesis for time series using a pure NumPy implementation, and using Monte Carlo simulations to compute critical‑value tables.

Getting Started with Agentic Systems - Developer Learning Paths

smcleod.net — Tue, 15 Apr 2025 18:10:00 GMT

A curated learning path for engineers looking to gain practical experience with AI and agentic systems.

The Via Verde Stick

John Graham-Cumming's blog — Thu, 10 Apr 2025 12:01:13 GMT

Portugal has an incredibly well executed and efficient system for collecting road tolls called Via Verde. A small transponder in the car either collects a toll at an automated toll booth or, on some motorways, every few kilometres without slowing down at all.

Via Verde can also be used to pay for parking in many locations.

When entering a car park where Via Verde is an option there are two ways it works: in some car parks it's possible just to wait a few moments for the gate to decide that you've opted for Via Verde and not a ticket, in others you have to press a button.

The parking above has three options: use a card (for residents or people with a subscription), blue button for a ticket, and green for Via Verde. To alleviate difficulty in pressing the green button I created the "Via Verde Stick" from a short length of wooden broom stick handle, some green spray paint, and two felt floor protectors.

The felt floor protectors are on both ends so that it doesn't matter which way the stick is oriented.

I do wonder what the driver behind me thinks when a long, bright green stick appears out the window but...

Fictional Non-fiction?

Beetle Space — Sun, 06 Apr 2025 11:00:00 GMT

When we classify writing as fiction, we have no expectation of it being true.

When we classify writing as non-fiction, we have an expectation of it being factual.

The word fiction is overloaded: We use it in two ways:

Content where it is well understood that the contents are not …

An Appeal to Documentation Owners

Beetle Space — Sun, 06 Apr 2025 10:00:00 GMT

If you have documentation for your library/SW/service, please, please ensure there is an option to show the whole documentation in a form I can print. That could be:

Showing it all in one HTML page (and styled so it prints well)
Export to a document format like PDF …

Visualizing a search in searchcode.com through the blinkenlights

Ben E. C. Boyter — Sun, 06 Apr 2025 03:00:00 GMT

At GopherCon2023 I presented a talk about how I built a custom index for searchcode. In there somewhere I mentioned that having a visualization of how the search works is something I would like to add in.

With some free time on a weekend rather than “wasting” time playing a game I decided to actually build it, and as is the current trend on the internet attempted to vide code it, at least for the UI portion anyway.

So called Liberation Day, the day the Dollar died.

--Mike-- — Fri, 04 Apr 2025 12:19:01 GMT

The US created a new world order from the ashes of WWII. This gave us the incredible privilege of having other nations treat our Dollar as their reserve currency.

Trump has thrown this away, and it can't be recovered. We're now going to actually have to pay, in hard currency for things we import from now on. This means we're all going to see our standards of living fall by about 70% in the next decade.

It's important to remember this was a purely self inflicted wound, we did it to ourselves. We fucked around, and now we (and especially our kids) are going to find out.

The Cost of Agentic Coding

smcleod.net — Wed, 02 Apr 2025 18:00:01 GMT

The cost of not leveraging agentic coding is likely far greater than you think.

What Is Software Quality?

The Codist — Mon, 31 Mar 2025 21:07:36 GMT

Everyone wants the software they work on to produce quality products, but what does that mean? In addition, how do you know when you have it?

This is the longest single blog post I have ever written.

I spent four decades writing software used by people (most of the server

Debugging Lotus 1-2-3 by fax

John Graham-Cumming's blog — Mon, 31 Mar 2025 18:30:58 GMT

There isn't a lot to this story beyond the fact that in around 1990 I helped debug someone's Lotus 1-2-3 set up via fax. But it's a good reminder of how important the Zeroth Law of Debugging is (see below).

Without some sort of online connection with these folks, and with transatlantic phone calls being very, very expensive (I was in the UK, they were in the US) fax was the obvious answer.

I was reminded of this when I came across the actual fax itself:

What I asked them to do was give me the output of /ppomr. Hitting / in 1-2-3 started the menu system and each letter took you down a submenu until you hit an actual menu item. /ppomr is /Print Printer Options Margins Right (read all about it in the Quick Reference) and so I was asking for the size of the right margin when printing.

Here's what that looks like in action:

And then I instructed the person on the other end to alter the left margin with /ppoml (/Print Printer Options Margins Left). I also told them if they wanted a change to the page headers I'd need to "send them a disc" (i.e. mail a floppy disc to the US!)

If you are going to debug anything you need to tighten the loop between trying something and observing the output of your change. This should be the Zeroth Law of Debugging: get the smallest/fastest test case so you can iterate.

Worst case send faxes across the Atlantic.

A markup language and hypertext browser in 600 lines of code

Kartik Agaram — Sat, 29 Mar 2025 21:03:28 GMT

Here's a document containing a line of text:

{type='text', data={'hello, world'}}

I'm building in Lua, so I'm reusing Lua syntax. Here's how it looks:

Such text boxes are the workhorse of this markup language. There are 3 other kinds of boxes: rows, cols and filler. Rows and cols can nest other boxes. But let's focus on text boxes for a bit.

Here's a text box containing two lines of text:

    {type='text', data={
      'hello',
      'world'}}

Since it's hypertext, you get a few attributes. You can set color, bg (background color) and border (also color). Each of these is a Lua array of 3 or 4 elements for red, green, blue and transparency.

    {type='text', data={'hello, world'},
      bg={1,0.6,0.4},
      border={0,0,0},
      color={0,0.2,0.8}}

Notice that the text is not centered vertically. The browser doesn't know details about the font like how far down the baseline is. You have to center manually using line_height.

      {type='text', data={'hello, world'},
        bg={1,0.6,0.4},
        border={0,0,0},
        color={0,0.2,0.8},
        line_height=25}

I imagine this sort of thing will get old fast. Probably best to use bg and border sparingly.

Since this is a hypertext browser, the main attribute of course is hyperlinks. To turn a text box into a link you can click on, use the target attribute.

      {type='text', data={'hello, world'},
        target='x'}

Links get some default colors if you don't override them. The target is a file path that will open when you click on it; there's no networking here. Press alt+left to go back.

What else. The one remaining attribute text boxes support is font. You can use any font as long as it's Vera Sans (or you're welcome to open up my program and put more fonts in). But you can adjust the font size and select bold and italic faces. However, before we can see them in action I should discuss inline styles.

A text box contains lines. Each line so far has been a string. But it can also be a string augmented with attributes. Here's a line with an inline 'tag':

      {type='text', data={
        {'hello, style1@{world}',
          attrs={
            style1={font={italic=true}},
          }},
      }}

(So many irritating silly curly brackets! But I hope you'll stick with me here. The goal is a simple markup language that is easy to implement while still providing some basic graphical niceties.)

Inline segments of text are surrounded in @{...} and prefixed with an alphanumeric name. (So they have to begin at the start of a word, after whitespace or punctuation.) The name gets connected up with attributes inside a block called attrs.

To stretch our legs, here's a text box with two lines, each containing inline markup for font and color.

      {type='text', data={
        {'hello, style1@{world}',
          attrs={
            style1={font={italic=true}},
          }},
        {'style1@{hello}, style2@{world}',
          attrs={
            style1={font={bold=true}, color={1,0,0}},
            style2={font={italic=true}, bg={1,0,1}},
          }},
      }}

Each line's attributes are independent. So far you can't change font size or add borders inline, because it complicates matters to change line height within a line, and also seldom looks nice.

Inline attrs should arguably be used sparingly. The pattern I've been using more is to give each text block a uniform font and mix and match combinations of text boxes. There are 2 ways to combine text boxes: rows and cols. Here's a vertical array of text boxes:

      {type='rows', data={
        {type='text',
          data={'hello, world'}},
        {type='text',
          data={'goodbye, cruel world'}},
      }}

And here's a horizontal array:

      {type='cols', data={
        {type='text',
          data={'hello, world'}},
        {type='text',
          data={'goodbye, cruel world'}},
      }}

It's hard to see, so let's make the border more obvious. You can add attributes to rows and cols just like to text.

      {type='cols', bg={0.8,0.8,1}, data={
        {type='text', bg={1,0.8,0.8},
          data={'hello, world'}},
        {type='text',
          data={'goodbye, cruel world'}},
      }}

All children of rows share a width, and all children of cols share a height. Children can also share other attributes when specified in a default attribute:

      {type='rows',
          default={color={0.8,0,0}},
        data={
          {type='text',
            data={'hello, world'}},
          {type='text',
            data={'goodbye, cruel world'}},
      }}

Widths and heights will grow and shrink depending on what you put in them, but you can also fix a width in rows and text boxes, and lines will wrap as needed.

      {type='rows', width=200, data={
        {type='text', data={
          'This is a long sentence.'}},
        {type='text', data={
          'This is a second long sentence.'}},
      }}

Notice that it'll try to wrap at word boundaries if it can. But it'll chop mid-word if a line would be too empty without the entire word.

For completeness, here's a filler box. All it does is add padding within rows and cols. Within rows, filler needs to specify a height, and within cols, a width. The other dimension will resize as needed.

      {type='rows',
          default={color={0.8,0,0}},
        data={
          {type='text',
            data={'hello, world'}},
          {type='filler', height=50},
          {type='text',
            data={'goodbye, cruel world'}},
      }}

Putting it all together, here's a table:

      {type='rows', border={0,0,0},
          default={border={1,1,1}},
        data={
          {type='cols',
              default={
                font={size=21, bold=true}},
            data={
              {type='text',
                data={'widgets'}},
              {type='text',
                data={'quantity'}}}},
          {type='cols', data={
            {type='text', data={'foo'}},
            {type='text', data={'34'}}}},
          {type='cols', data={
            {type='text', data={'bar'}},
            {type='text', data={'7'}}}},
      }}

No wait, that's not right:

      {type='cols', border={0,0,0},
          default={border={1,1,1}},
        data={
          {type='rows', data={
            {type='text', font={bold=true},
              data={'widgets'}},
            {type='text', data={'foo'}},
            {type='text', data={'bar'}}}},
          {type='rows', data={
            {type='text', font={bold=true},
              data={'quantity'}},
            {type='text', data={'34'}},
            {type='text', data={'7'}}}},
      }}

That's annoying! You have to specify columns before rows, or you're stuck manually sizing widths. But, 600 lines of code! Here it is. You'll need LÖVE, but the code should be easy to port to other graphics toolkits, the markup to JSON literals, etc. The program is a zip file containing the source code.

awk idioms explained

learnbyexample — Fri, 28 Mar 2025 02:00:00 GMT

Do you find awk one-liners cryptic? Stuff like !a[$0]++, 1, $1=$1, NR==FNR and -v RS=? You'll find examples and brief explanations for such idioms in this post.

The examples presented here have been tested with GNU awk. These are likely to work with most other implementations of awk as well.

awk command structure🔗

awk 'cond1{action1} cond2{action2} ... condN{actionN}'

When a conditional expression isn't provided, the action is always executed. When an action isn't provided, the $0 variable (which has the contents of the current record being processed) is printed if the conditional expression evaluates to true.

Regexp filtering🔗

# same as: grep 'at' and sed -n '/at/p'
$ printf 'gate\napple\nwhat\nkite\n' | awk '/at/'
gate
what

# same as: grep -v 'e' and sed -n '/e/!p'
$ printf 'gate\napple\nwhat\nkite\n' | awk '!/e/'
what

The generic syntax is string ~ /regexp/ to check if the given string matches the regexp and string !~ /regexp/ to invert the condition.

/regexp/ is a shortcut for $0 ~ /regexp/{print $0}
!/regexp/ is a shortcut for $0 !~ /regexp/{print $0}

Idiomatic use of 1🔗

Non-zero numeric values and non-empty strings are truthy (zero and empty strings are falsy). Idiomatically, 1 is used as a conditional expression to print the contents of $0.

$ echo 'ring amazing jar' | awk '{sub(/ing/, "ed", $2)} 1'
ring amazed jar

$ seq 2 | awk 'BEGIN{print "---"} 1; END{print "==="}'
---
1
2
===

Special variables🔗

$0 contains the current record being processed
$1 first field
$2 second field and so on
FS input field separator
OFS output field separator
NF number of fields
RS input record separator
ORS output record separator
NR number of records (i.e. line number) for the entire input
FNR number of records per file

Removing duplicates🔗

awk '!a[$0]++' is one of the most famous awk one-liners. It eliminates line based duplicates while retaining the input order.

$ cat purchases.txt
coffee
tea
washing powder
coffee
tea
coffee milkshake
soap
tea
washing soda

$ awk '{print +a[$0] "\t" $0; a[$0]++}' purchases.txt
0	coffee
0	tea
0	washing powder
1	coffee
1	tea
0	coffee milkshake
0	soap
2	tea
0	washing soda

# only the entries with zero in the first column will be retained
$ awk '!a[$0]++' purchases.txt
coffee
tea
washing powder
coffee milkshake
soap
washing soda

a[$0] creates an uninitialized element in array a with $0 as the key (if the key doesn't exist yet). Thus, !a[$0] will succeed only on the first occurrence of an item (since an uninitialized value is falsy) and the post-increment operator will ensure that further instances of an item will fail the conditional expression.

Rebuild $0🔗

Sometimes you just want to change the field separator, or perform some record-level text processing and then print it with a new field separator. In such cases, you'll have to explicitly fake a field operation — otherwise the field separation update won't happen for $0.

$ s='sample123string42with777numbers'

$ echo "$s" | awk -F'[0-9]+' -v OFS=, '{$1=$1} 1'
sample,string,with,numbers

$ echo "$s" | awk -F'[0-9]+' -v OFS=- '{gsub(/[aeiou]/, ""); $1=$1} 1'
smpl-strng-wth-nmbrs

Paragraph mode🔗

When RS is set to an empty string, one or more consecutive empty lines is used as the input record separator.

$ cat para.txt
hello world

hi there
how are you

just doing
believe it

banana
papaya
mango

much ado about nothing
he he he
adios amigo

# uninitialized variable 's' will be empty for the first match
# afterwards, 's' will provide the empty line separation
$ awk -v RS= '/do/{print s $0; s="\n"}' para.txt
just doing
believe it

much ado about nothing
he he he
adios amigo

Two file processing🔗

For two files as input, NR==FNR will be true only when the first file is being processed. The next statement will skip the rest of the code for the current record.

$ cat marks.txt
dept    name    marks
ece     raj     53
ece     joel    72
eee     moi     68
cse     surya   81
eee     tia     59
ece     om      92
cse     amy     67

$ cat dept_mark.txt
ece 70
eee 65
cse 80

# match dept and minimum marks specified in dept_mark.txt
$ awk 'NR==FNR{d[$1]=$2; next}
       $1 in d && $3 >= d[$1]' dept_mark.txt marks.txt
ece     joel    72
eee     moi     68
cse     surya   81
ece     om      92

Note that the NR==FNR logic will fail if the first file is empty, since NR wouldn't get a chance to increment. You can set a flag after the first file has been processed to avoid this issue — for example, awk '!f{a[$0]; next} !($0 in a)' file1 f=1 file2. See this unix.stackexchange thread for more workarounds.

Forcing string and numeric context🔗

Strings are automatically converted to a number when used in an arithmetic expression (for example, "42" + 5). You can use the unary + and - operators to force numeric context. If the string doesn't start with a valid number (ignoring any starting whitespaces), it will be treated as 0.

$ seq 3 | awk '{sum += $0} END{print sum}'
6
$ awk '{sum += $0} END{print sum}' /dev/null

$ awk '{sum += $0} END{print +sum}' /dev/null
0

Similarly, you can concatenate a string to a number to force string context.

$ awk 'BEGIN{n1="5.0"; n2=5; if(n1==n2) print "equal"}'
$ awk 'BEGIN{n1="5.0"; n2=5; if(n1==n2".0") print "equal"}'
equal

See gawk manual: How awk Converts Between Strings and Numbers for more details.

Programming ebooks🔗

Check out my ebooks on Regular Expressions, Linux CLI tools, Python and Vim. You can get them all as a single bundle via leanpub or gumroad.

Crawl Order and Disorder

Weblog on marginalia.nu — Thu, 27 Mar 2025 02:00:00 GMT

A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains.

This has been actualized recently, since the migration to slop crawl data has dropped memory requirements of the crawler by something like 80%, and as such I’ve been able to increase the number of crawling tasks, which has led to a bizarre case where 99.9% of the crawling is done in 4 days, and the remaining 0.1% takes a week.

Sydney Go meetup March 2025

Ben E. C. Boyter — Thu, 27 Mar 2025 02:00:00 GMT

I was a speaker at the first Go meetup in Sydney 2025… the first speaker in fact. As is my habit I am posting my notes for everyone who attended to view them.

I believe it was recorded and will post the link to that if I ever get the link.

You can find the link to the talk based on the blog post searchcode.com’s SQLite database is probably 6 terabytes bigger than yours.

Things that go wrong with disk IO

Notes on software development — Thu, 27 Mar 2025 02:00:00 GMT

There are a few interesting scenarios to keep in mind when writing applications (not just databases!) that read and write files, particularly in transactional contexts where you actually care about the integrity of the data and when you are editing data in place (versus copy-on-write for example).

We'll go into a few scenarios where the following can happen:

Data you write never actually makes it to disk
Data you write get sent to the wrong location on disk
Data you read is read from the wrong location on disk
Data gets corrupted on disk

And how real-world data systems think about these scenarios. (They don't always think of them at all!)

If I don't say otherwise I'm talking about behavior on Linux.

The post is largely a review of two papers: Parity Lost and Parity Regained and Characteristics, Impact, and Tolerance of Partial Disk Failures. These two papers also go into the frequency of some of the issues discussed here. These behaviors actually happen in real life!

Thank you to Alex Miller and George Xanthakis for reviewing a draft of this post.

Terminology

Some of these terms are reused in different contexts, and sometimes they are reused because they effectively mean the same thing in a certain configuration. But I'll try to be explicit to avoid confusion.

Sector

The smallest amount of data that can be read and written atomically by hardware. It used to be 512 bytes, but on modern disks it is often 4KiB. There doesn't seem to be any safe assumption you can make about sector size, despite file system defaults (see below). You must check your disks to know.

Block (filesystem/kernel view)

Typically set to the sector size since only this block size is atomic. The default in ext4 is 4KiB.

Page (kernel view)

A disk block that is in memory. Any reads/writes less than the size of a block will read the entire block into kernel memory even if less than that amount is sent back to userland.

Page (database/application view)

The smallest amount of data the system (database, application, etc.) chooses to act on, when it's read or written or held in memory. The page size is some multiple of the filesystem/kernel block size (including the multiple being 1). SQLite's default page size is 4KiB. MySQL's default page size is 16KiB. Postgres's default page size is 8KiB.

Things that go wrong

The data didn't reach disk

By default, file writes succeed when the data is copied into kernel memory (buffered IO). The man page for write(2) says:

A successful return from write() does not make any guarantee that data has been committed to disk. On some filesystems, including NFS, it does not even guarantee that space has successfully been reserved for the data. In this case, some errors might be delayed until a future write(), fsync(2), or even close(2). The only way to be sure is to call fsync(2) after you are done writing all your data.

If you don't call fsync on Linux the data isn't necessarily durably on disk, and if the system crashes or restarts before the disk writes the data to non-volatile storage, you may lose data.

With O_DIRECT, file writes succeed when the data is copied to at least the disk cache. Alternatively you could open the file with O_DIRECT|O_SYNC (or O_DIRECT|O_DSYNC) and forgo fsync calls.

fsync on macOS is a no-op.

If you're confused, read Userland Disk I/O.

Postgres, SQLite, MongoDB, MySQL fsync data before considering a transaction successful by default. RocksDB does not.

The data was fsynced but fsync failed

fsync isn't guaranteed to succeed. And when it fails you can't tell which write failed. It may not even be a failure of a write to a file that your process opened:

Ideally, the kernel would report errors only on file descriptions on which writes were done that subsequently failed to be written back. The generic pagecache infrastructure does not track the file descriptions that have dirtied each individual page however, so determining which file descriptors should get back an error is not possible.

Instead, the generic writeback error tracking infrastructure in the kernel settles for reporting errors to fsync on all file descriptions that were open at the time that the error occurred. In a situation with multiple writers, all of them will get back an error on a subsequent fsync, even if all of the writes done through that particular file descriptor succeeded (or even if there were no writes on that file descriptor at all).

Don't be 2018-era Postgres.

The only way to have known which exact write failed would be to open the file with O_DIRECT|O_SYNC (or O_DIRECT|O_DSYNC), though this is not the only way to handle fsync failures.

The data was corrupted

If you don't checksum your data on write and check the checksum on read (as well as periodic scrubbing a la ZFS) you will never be aware if and when the data gets corrupted and you will have to restore (who knows how far back in time) from backups if and when you notice.

ZFS, MongoDB (WiredTiger), MySQL (InnoDB), and RocksDB checksum data by default. Postgres and SQLite do not (though databases created from Postgres 18+ will).

You should probably turn on checksums on any system that supports it, regardless of the default.

The data was partially written

Only when the page size you write = block size of your filesystem = sector size of your disk is a write guaranteed to be atomic. If you need to write multiple sectors of data atomically there is the risk that some sectors are written and then the system crashes or restarts. This behavior is called torn writes or torn pages.

Postgres, SQLite, and MySQL (InnoDB) handle torn writes. Torn writes are by definition not relevant to immutable storage systems like RocksDB (and other LSM Tree or Copy-on-Write systems like MongoDB (WiredTiger)) unless writes (that update metadata) span sectors.

If your file system duplicates all writes like MySQL (InnoDB) does (like you can with data=journal in ext4) you may also not have to worry about torn writes. On the other hand, this amplifies writes 2x.

The data didn't reach disk, part 2

Sometimes fsync succeeds but the data isn't actually on disk because the disk is lying. This behavior is called lost writes or phantom writes. You can be resilient to phantom writes by always reading back what you wrote (expensive) or versioning what you wrote.

Databases and file systems generally do not seem to handle this situation.

The data was written to the wrong place, read from the wrong place

If you aren't including where data is supposed to be on disk as part of the checksum or page itself, you risk being unaware that you wrote data to the wrong place or that you read from the wrong place. This is called misdirected writes/reads.

Databases and file systems generally do not seem to handle this situation.

A Guest Post at Wormwoodania

Send More Paramedics — Wed, 26 Mar 2025 09:51:09 GMT

As an attempt to branch out with my writing, I've taken to working on various fiction and non-fiction pieces...

CLI text processing with GNU awk book announcement

learnbyexample — Wed, 26 Mar 2025 02:00:00 GMT

Hello!

I am pleased to announce a new version of my CLI text processing with GNU awk ebook.

Learn the GNU awk command step-by-step from beginner to advanced levels with hundreds of examples and exercises. This book will dive deep into field processing, show examples for filtering features, multiple file processing, how to construct solutions that depend on multiple records, how to compare records and fields between two or more files, how to identify duplicates while maintaining input order and so on. Regular expressions will also be discussed in detail.

Release offers🔗

To celebrate the new release, you can download the PDF/EPUB versions of CLI text processing with GNU awk for FREE till 06-April-2025. You can still pay if you wish ;)

Here are some more amazing offers:

All 13 books bundle is $18 (normal price $36) — Leanpub or Gumroad
Linux CLI Text Processing is $10 (normal price $20) — Leanpub or Gumroad

What's new?🔗

Command version updated to GNU awk 5.3.1
Added details for the --csv option and the \u escape sequence
Corrected typos, updated exercises, descriptions and external links
Updated Acknowledgements section

Videos🔗

Check out my programming tips covering Python, command line tools and Vim:

Interactive TUI app🔗

I also wrote an interactive TUI app based on some of the exercises from the ebook. Reference solutions are also provided.

Table of Contents🔗

Preface
Installation and Documentation
awk introduction
Regular Expressions
Field separators
Record separators
In-place file editing
Using shell variables
Control Structures
Built-in functions
Multiple file input
Processing multiple records
Two file processing
Dealing with duplicates
awk scripts
Gotchas and Tips
Further Reading

Web version🔗

You can read the book online here: https://learnbyexample.github.io/learn_gnuawk/

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_gnuawk for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_gnuawk/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Marginalia Search receives second nlnet grant

Weblog on marginalia.nu — Tue, 25 Mar 2025 02:00:00 GMT

I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant.

All the details are not yet finalized, but tentatively the grant will go toward addressing most of the items in the project roadmap for 2025.

I’ve already been working full time on the project since summer 2023, and this grant secures additional development time, and extends the runway to a comfortable degree.

Will post more details as they are finalized.

Nostalgia

bayindirh — Tue, 25 Mar 2025 02:00:00 GMT

The people I'm following on the internet put me inside a small but vibrant retrocomputing bubble. When this is combined with the synthwave composers I tend to listen lately, I'm being transported to my earlier years on this planet, repeatedly. The fact that I have grown with a Commodore 64, 486 systems, boomboxes and big Hi-Fi racks makes this a rather harsh, but enjoyable ride every time.

While small, I have my trove of old things. PalmOS devices, MiniDisc decks, an enjoyable vintage HiFi system among other things. As I use these older devices (MiniDisc systems sound nice, btw.), I wonder why we used to dislike some aspects of these things then, or love them now. There's a big question there, and while nostalgia is a big factor, I don't think it paints the complete picture.

See, nostalgia is the longing for a "better" past. Not technologically, but emotionally. Longing for a life less complicated, with less responsibilities, with saner weather, less crowded cities and other small joys. On the other hand, we find a value in this long discontinued, old-fashioned items. They are simpler, built better, can be repaired, have better qualities where it matters, and their little quirks. However, I think there's another small, but crucial reason we hold them close and dear.

We can overcome their shortcomings with better technology of their tomorrow. All the retro technology we love to use today had their shortcomings on their heyday. In old PC days, disks were small and slow, computers made us wait, they were noisy and big, and didn't allow us to connect anywhere or fast if they did. Now, all these are overcame with neat solutions. There are floppy emulators which accept SD cards, IDE disk drives are replaced with CF cards, MiniDiscs can be recorded from your browser, better audio can be ripped from old CDs and verified instantly for quality, old systems can be connected to networks (see Marchintosh or SDF Vintage Systems), and even to the internet with modern electronics... In short, we removed their limitations and integrated them to our more modern worlds with their own terms while preserving their best qualities. This is no less than a miracle.

I have seen someone resurrected their first PC faithful to its original specs, and then some. There are new AppleTalk hardware is being designed for older Apple systems, somebody wrote a TOTP application for PlamOS. Legacy Macs, Amigas and older Windows systems get modern browsers as their OS and computing power allows. Even new demos are being written for old systems like Commodore64 (See the demo “NINE”) to push their limits even further.

In other words, we have not done developing new things for the older hardware, and continue to extract more performance and utility from them. However, paradoxically, the new things we do with these old systems is enabled by the new technology we built on top of them over the years.

As a result, Today's retrocomputing and nostalgia becomes something bigger than the items and the emotions it carries in the first place. Resurrecting these old systems and things not only allow us to relive the parts of the past we cherish, but to carry these systems to future and give them a proper second life, because while these systems are old, they can be useful, or enable us to do things we can't do as effectively with the new tools.

I think we should continue doing this not only because it’s fun, but it’s beneficial to everyone and everything involved in the process. Because while we develop things, we make tradeoffs, and we can’t understand what we lost in the process, if we can’t look back and relive and cherish the things we had back then.

Until next time,

Be kind.

Phil Eaton on Technical Blogging

Notes on software development — Tue, 25 Mar 2025 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

The Democratisation Paradox: What History Teaches Us About AI

smcleod.net — Sat, 22 Mar 2025 17:00:01 GMT

The Democratisation Paradox: What History Teaches Us About AI.

Improved ways to operate a rude crawler

Weblog on marginalia.nu — Sat, 22 Mar 2025 02:00:00 GMT

This text is satirical in nature.

Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt. In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup is the one.

The effects of prompt caching on Agentic coding

smcleod.net — Thu, 20 Mar 2025 13:00:01 GMT

What is prompt caching and why is it so important for Agentic coding?

Eigiau walk

Digital Apocrypha — Mon, 17 Mar 2025 23:40:00 GMT

After a weekend at Blaen y Nant I had time for a Sunday afternoon walk in the Carneddau— a circuit of the valley north of Llyn Eigiau taking in the two reservoirs near Dulyn bothy. Having recently started a new job, I was thinking about this new stage in my life. And also, with the … Continue reading

The reward for long-term resilience in simple trading strategies - Part II

Bytepawn - Marton Trencseni — Sun, 16 Mar 2025 01:00:00 GMT

In this follow-up, I analyze 10 years of data on a select group of stocks to show that while higher take-profit and stop-loss thresholds boost win ratios and Sharpe ratios, longer holding times can lower annualized returns.

Vibe coding searchcode a new UI and saving myself 40+ hours of work

Ben E. C. Boyter — Wed, 12 Mar 2025 02:00:00 GMT

I have been working on a secret project within https://searchcode.com/ and ran into a blocker where the UI had become a road-block. I will be the first to admit that I do not like styling things with CSS very much. In fact working with HTML and CSS is probably my least favorite part of modern web development.

I have never discussed any use of “AI” on this blog. Not for lack of using it, as I had been using ChatGPT and other tools since day one. I had always previously found that the models were a neat party trick, but had a lot of issues dealing with code. Given the new “vibe coding” shift that been happening in the industry I thought I would give it a go, and use it to redo the searchcode UI.

Supa Pecha Kucha

swyx's site RSS Feed — Tue, 11 Mar 2025 12:28:52 GMT

slug: supapechakucha

The reward for long-term resilience in simple trading strategies - Part I

Bytepawn - Marton Trencseni — Sun, 09 Mar 2025 01:00:00 GMT

In this article, I explore how adopting longer-term, resilient trading strategies with higher take-profit and stop-loss thresholds can deliver more consistent, risk-adjusted performance.

OS installation woes

learnbyexample — Tue, 04 Mar 2025 02:00:00 GMT

I prefer stability and thus I let my Linux LTS distributions to last almost till the entire end of support time. Having learned from previous transition periods, I started maintaining notes on what softwares I install/purge and some of the customization stuff. This has vastly reduced the pain of a fresh OS installation, but the fact remains that there'll always be some persistent and annoying trouble.

The one I encountered this time around is really perplexing and I'm afraid of trying to figure out the root cause. For now, I'm happy with the workaround I ended up with.

I use redshift to set the color temperature of computer display. It is a simple temperature setting that doesn't depend on time or place, so my config is simple. However, it wasn't working when I moved from good old Ubuntu to Linux Mint (because Ubuntu is no longer user friendly). As per this discussion on Linux Mint forum, geoclue2 no longer works. That shouldn't matter for me since I don't need location services. Whatever, that same thread mentioned xsct as a simpler alternative that exactly fits my need (with a bonus of changing screen brightness!).

I installed it and a really really simple command from the terminal was all it needed to work. So, what was the annoying issue? I couldn't get it to autostart on login no matter what I did! At first, I thought perhaps I needed to use the full path of the command, but that obviously didn't solve my troubles. After fruitless search on the internet, I almost thought to ask a question on the forum. Before that though, I had the bright idea (really basic debugging rule) to first check if my autostart setup was working at all. I chose to autostart a terminal on login via the xfce4-terminal command and it did work!

So, why wasn't xsct working? No idea. But reading the man page of the terminal emulator showed that I can choose to execute a command with the -e option. So, that was my inelegant workaround! The desktop entry is shown below if you are curious. If you know why Exec=xsct 3500 0.9 doesn't work compared to Exec=xfce4-terminal -e 'xsct 3500 0.9', do let me know!

[Desktop Entry]
Encoding=UTF-8
Version=0.9.4
Type=Application
Name=displaytemperature
Comment=
Exec=xfce4-terminal -e 'xsct 3500 0.9'
OnlyShowIn=XFCE;
RunHook=0
StartupNotify=false
Terminal=false
Hidden=false

Update:

The above solution didn't work as I hoped. The screen setting would activate some of the time but most often it didn't. I resorted to creating a keyboard shortcut, which I'd use when the activation failed.

The terminal would always show up briefly on login though, which meant the issue wasn't due to the autostart failing altogether. One day I happened to notice that the screen setting would actually take effect before immediately relapsing to the normal temperature. And thus I arrived at the current solution that hasn't failed yet. I wrote a shell script that first used the sleep command to create a delay of one second before calling xsct. The autostart desktop entry now calls this script — again, using xfce4-terminal -e because for some reason calling the script directly still fails 🤷.

Marginalia Search: 4 Years

Weblog on marginalia.nu — Mon, 03 Mar 2025 02:00:00 GMT

This update is a few days late, the canonical birth date of the project is Feb 26.

It has been another year of Marginalia Search. The project is still ongoing, still my full time job, although the project is entering a somewhat more mature phase of development, most of the big pieces are in place and do a decent job at what they do.

The roadmap for the project is available on GitHub.

Minimal downtime Postgres major version upgrades with EDB Postgres Distributed

Notes on software development — Fri, 28 Feb 2025 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

I Had a Sad Feeling For a Moment …

Beetle Space — Mon, 17 Feb 2025 10:00:00 GMT

Then it passed.

searchcode.com’s SQLite database is probably 6 terabytes bigger than yours

Ben E. C. Boyter — Sun, 16 Feb 2025 02:00:00 GMT

searchcode.com’s SQLite database is probably one of the largest in the world, at least for a public facing website. It’s actual size is 6.4 TB. Which is probably 6 terabytes bigger than yours.

-rw-r--r-- 1 searchcode searchcode 6.4T Feb 17 04:30 searchcode.db

At least, I think its bigger. I have no evidence to the contrary, being by far the largest I have ever heard of. Poking around the internet did not find anyone talking publicly about anything bigger. The largest I could find using any search or LLM was 1 TB (without source), and a few people on HN and Reddit claiming to be working with SQLite database’s around the size of 10’s of gigabytes.

Old and New Experiences

Ben Overmyer's Site — Sat, 15 Feb 2025 02:00:00 GMT

The last three years have been quite the experience. I'm not really sure how to differentiate between new (and therefore top-of-mind) and old (and therefore more nostalgic/emotional) recollections.

As I write this I'm listening to the soundtrack for Age of Conan, as replayed on a Jellyfin server in my house.

It reminds me of Conan Exiles, which I've spent way more time in than Age of Conan, but which repurposed the soundtrack because it was cheaper than creating new music.

I have fond memories of Final Fantasy XIV. They are not as vivid, lately. But they're more vivid than my memories of Star Wars: Galaxies. SWG was the only game I've devoted creative enterprise to; my most-viewed YouTube video is a SWG video.

Neither game, though, has garnered my attention lately.

I've been spending more of my attention on a game called Reality Break. It's a Diablo-like action RPG where you play as a starship pilot. It has mechanics that feel rewarding.

This seems disingenuous. Current events make video games and other forms of entertainment seem really pointless.

From web developer to database developer in 10 years

Notes on software development — Sat, 15 Feb 2025 02:00:00 GMT

Last month I completed my first year at EnterpriseDB. I'm on the team that built and maintains pglogical and who, over the years, contributed a good chunk of the logical replication functionality that exists in community Postgres. Most of my work, our work, is in C and Rust with tests in Perl and Python. Our focus these days is a descendant of pglogical called Postgres Distributed which supports replicating DDL, tunable consistency across the cluster, etc.

This post is about how I got here.

Black boxes

I was a web developer from 2014-2021†. I wrote JavaScript and HTML and CSS and whatever server-side language: Python or Go or PHP. I was a hands-on engineering manager from 2017-2021. I was pretty clueless about databases and indeed database knowledge was not a serious part of any interview I did.

Throughout that time (2014-2021) I wanted to move my career forward as quickly as possible so I spent much of my free time doing educational projects and writing about them on this blog (or previous incarnations of it). I learned how to write primitive HTTP servers, how to write little parsers and interpreters and compilers. It was a virtuous cycle because the internet (Hacker News anyway) liked reading these posts and I wanted to learn how the black boxes worked.

But I shied away from data structures and algorithms (DSA) because they seemed complicated and useless to the work that I did. That is, until 2020 when an inbox page I built started loading more and more slowly as the inbox grew. My coworker pointed me at Use The Index, Luke and the DSA scales fell from my eyes. I wanted to understand this new black box so I built a little in-memory SQL database with support for indexes.

I'm a college dropout so even while I was interested in compilers and interpreters earlier in my career I never dreamed I could get a job working on them. Only geniuses and PhDs did that work and I was neither. The idea of working on a database felt the same. However, I could work on little database side projects like I had done before on other topics, so I did. Or a series of explorations of Raft implementations, others' and my own.

Startups

From 2021-2023 I tried to start a company and when that didn't pan out I joined TigerBeetle as a cofounder to work on marketing and community. It was during this time I started the Software Internals Discord and /r/databasedevelopment which have since kind of exploded in popularity among professionals and academics in database and distributed systems.

TigerBeetle was my first job at a database company, and while I contributed bits of code I was not a developer there. It was a way into the space. And indeed it was an incredible learning experience both on the cofounder side and on the database side. I wrote articles with King and Joran that helped teach and affirm for myself the basics of databases and consensus-based distributed systems.

Holding out

When I left TigerBeetle in 2023 I was still not sure if I could get a job as an actual database developer. My network had exploded since 2021 (when I started my own company that didn't pan out) so I had no trouble getting referrals at database companies.

But my background kept leading hiring managers to suggest putting me on cloud teams doing orchestration in Go around a database rather than working on the database itself.

I was unhappy with this type-casting so I held out while unemployed and continued to write posts and host virtual hackweeks messing with Postgres and MySQL. I started the first incarnation of the Software Internals Book Club during this time, reading Designing Data Intensive Applications with 5-10 other developers in Bryant Park. During this time I also started the NYC Systems Coffee Club.

Postgres

After about four months of searching I ended up with three good offers, all to do C and Rust development on Postgres (extensions) as an individual contributor. Working on extensions might sound like the definition of not-sexy, but Postgres APIs are so loosely abstracted it's really as if you're working on Postgres itself.

You can mess with almost anything in Postgres so you have to be very aware of what you're doing. And when you can't mess with something in Postgres because an API doesn't yet exist, companies have the tendency to just fork Postgres so they can. (This tendency isn't specific to Postgres, almost every open-source database company seems to have a long-running internal fork or two of the database.)

EnterpriseDB

Two of the three offers were from early-stage startups and after more than 3 years being part of the earliest stages of startups I was happy for a break. But the third offer was from one of the biggest contributors to Postgres, a 20-year old company called EnterpriseDB. (You can probably come up with different rankings of companies using different metrics so I'm only saying EnterpriseDB is one of the biggest contributors.)

It seemed like the best place to be to learn a lot and contribute something meaningful.

My coworkers are a mix of Postgres veterans (people who contributed the WAL to Postgres, who contributed MVCC to Postgres, who contributed logical decoding and logical replication, who contributed parallel queries; the list goes on and on) but also my developer-coworkers are people who started at EnterpriseDB on technical support, or who were previously Postgres administrators.

It's quite a mix. Relatively few geniuses or PhDs, despite what I used to think, but they certainly work hard and have hard-earned experience.

Anyway, I've now been working at EnterpriseDB for over a year so I wanted to share this retrospective. I also wanted to cover what it's like coming from engineering management and founding companies to going back to being an individual contributor. (Spoiler: incredibly enjoyable.) But it has been hard enough to make myself write this much so I'm calling it a day. :)

I wrote a post about the winding path I took from web developer to database developer over 10 years. pic.twitter.com/tf8bUDRzjV
— Phil Eaton (@eatonphil) February 15, 2025

† From 2011-2014 I also did contract web development but this was part-time while I was in school.

Using Xcode's AI Is Like Pair Programming With A Monkey

The Codist — Sat, 15 Feb 2025 00:03:57 GMT

I've never used any other AI "assistant," although I've talked with those who have, most of whom are not very positive. My experience using Xcode's AI is that it occasionally offers a line of code that works, but you mostly get junk

I have come to loathe acronyms

John Graham-Cumming's blog — Thu, 13 Feb 2025 19:59:46 GMT

Unless they are widely, widely known they make communication worse. And they have a tendency to make the writer seem pompous, or a member of some special clique, while making a reader who doesn't know the acronym feel foolish.

They are the opposite of clear communication, and are often accompanied by unclear, complex language.

The writer's goal (whether of a book, an email, or whatever the hell we call a tweet now) should be to be understood. Acronyms are an impediment to that and are often more difficult to read or say than actual words.

And don't fall into the awful trap of defining an acronym in a document and then using it later. That's a substitute for using plain language. Always use plain language.

Unless your goal is to be misunderstood.

Small web and self-hosting in a world in crisis

Ben Overmyer's Site — Thu, 13 Feb 2025 02:00:00 GMT

As I write this, the new President of the United States is trying to replace the democratic republic form of government with a fascist one. It appears that he's succeeding. Despite the focus on the United States right now though, this trend of far-right groups taking power is not limited to this country. Several countries in Europe, like Italy and Spain, are seeing a resurgence of conservative ideology. The regime change in the USA has been accompanied by an overt shift of Big Tech - that is, Meta, Microsoft, Google, and others - towards fascist policy. This makes the centralized version of the Web that most people are familiar with a hostile place.

There are refuges from this, though. The "small web" - personal websites run independently from the big platforms - is one component of that. The website you're reading this on is an example of the content you can find on the small web. It's built and run by me. Many such sites exist. The trick is in finding them. Following links from other small web sites is one way. Directories exist too. There are even search engines dedicated specifically to indexing the small web. I will put links to some of these at the end of this post.

At the time of writing, this website is hosted on Netlify, which is a platform for static websites. To better safeguard the site, I will move it in the near future to a server I control.

If you don't have the technical knowledge to build and host a website, then you can try using a site like Neocities. It hosts the website for you, much like the old Geocities.

Beyond social media, though, there is still much of the mainstream Internet that we use in our day-to-day life. There are messaging applications like Facebook Messenger or WhatsApp, video sites like YouTube, event websites like Meetup, and so on. To replace things like this, we are best served by using federated and decentralized alternatives. For those unfamiliar with the concept, "federation" refers here to multiple different instances of a particular service being run independently of each other but sharing content with each other. In that sense, it's kind of like email; no matter who runs your email service, you can send email to any other email service.

Messenger can be replaced with Matrix. YouTube can be replaced with PeerTube. Meetup can be replaced with Mobilizon. There is a more complete list at the end of this post.

I run a Matrix homeserver for family and friends. It's encrypted end-to-end, so governments and bad actors can't (easily) gain access to it or its data. I also use Signal, but as that is centrally controlled by an American company, I trust it less.

This site and Iron Arachne will be moving to servers I control in the near future, too.

I already run a server at home for videos and music, but that is inaccessible outside a select few devices. I use Tailscale to control access to that.

Resources

Awesome Self-Hosted: a list of self-hosted alternatives
Fedi Tips: more information about what the "Fediverse" and federated services are
Gemini: about the Gemini protocol, an alternative to the Web

Services

Jitsi Meet: video calls
Mastodon: social media
Matrix: chat
Mobilizon: event management
Neocities: platform for do-it-yourself websites
PeerTube: video sharing
Pixelfed: image sharing
Tailscale: zero-trust networking

Discovery Tools

Marginalia: a search engine that prioritizes non-commercial content
Wiby: a search engine specifically targeting the small web

Dieting and Differential Equations: Part 4, What's missing

jodavaho.io — Wed, 12 Feb 2025 02:00:00 GMT

Summary

In the first part of this series, I explored a primitive ‘quantitative diet’ idea. In the second part I showed how to use a simple model and some differential equations to estimate the effect of a diet on weight loss, and ‘derive’ a diet to acheive a weightloss goal. In the third part, I discussed how I track my diet using ChatGPT-4. In this part, I can share some pitfalls, including local fluctuations, model fitting, and so on.

Reducing uncertainty

While tracking weight and calories, I’ve noticed a problem.

When you decrease your calorie intake sharply (from a fast or beginning of a new diet), of even change the spread of fat vs carb vs protein significantly, I’ve notice an immediate change in weight.

When beginning a diet, this early loss vastly outstrips the predicted loss. These two factors were something I previously attributed to just poor caloric estimates, or, shall we say “food accumulation in my guts” (or rather, the loss of accumulated recent meals causing sharp weight changes).

Here’s my wow plot:

Predictions (See dieting-differential-equations-2)
   (Y)     ^
      Lbs: |
      >228 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >226 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >224 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >222 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >220 | +-----start weight⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >218 | ⡏⠙⠳⢶⢦⡤⢄⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >216 | ⡇⠀⠀⠀⠉⠙⢕⣒⠬⢍⡒⠒⠤⠤⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >214 | ⣇⢀⠀⠐⠀⠀⠀⠐⠑⠢⢌⡑⠢⠤⣀⡈⠉⠑⠒⠢⠤⢄⣀⡀⠀⠀<---sedentary⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >212 | ⡇⠀⠉⠀⠊⠐⠒⠀⠂⠀⠀⠈⠑⠢⣀⡈⠑⠒⢄⣀⠀⠀⠀⠈⠑⠒
      >210 | ⡇⠀⠀⠀⠀^wts ⠀ ⠀⠀⠀⠀⠈⠢⠤⡀⠀⠉⠑⠢⠤⣀⡀⠀⠀⠀
      >208 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀  ⠀⠀⠀⠈⠑⠢⢄⡀⠀⠀⠈⠑⠢⢄⣀<--light
      >206 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀        mod--->⠈⠑⠤⢄⠀
      >202 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀
      >200 | ⣇⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀
      >198 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀GOAL
      >196 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >194 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >192 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
      >190 | ⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
-----------|-|---------|---------|---------|--------
           | >0        >15       >30       >45       >60
           | Days

On the first couple days of the diet, I lost about 8 lbs, and then levelled off for the next week. Plotting the rest of the weights out for two months would clutter it, but let’s just say it’s mostly level.

So, there’s two curiosities. First, what’s with the early loss? Second, why did I level off?

Early loss

Any random on the internet will call this “water weight”, but what does that mean? (By the way, the opposite effect holds too: after cheat days I can gain 5 lbs easily.)

The answer allegedly is that the body stores energy in the form of glycogen, which is a carbohydrate. Glycogen is stored in the liver and muscles, and is used for quick energy. It’s also stored with water, which is why it’s so heavy. When you start a diet, you would think your body uses up the glycogen stores, and the water is released. That would explain some early loss, except I’m not in a total energy deficit. In fact, I only cut 10% of my calories.

From what I can tell, the liver stores 2000 calories or so. It should take me days to deplete that, unless I’m completely fasting. Even depleting 10% would be 200 calories, which is 50g of glycogen, maybe accompanied by 200g of water. That’s 0.5 lbs, not 8.

It turns out that as soon as I switched to my reduced calorie load, I also switched what I was eating - I wanted to go higher fat/protein so I would not feel so hungry all the time.

It turns out switching from mostly carbs to mostly protein can cause an immediate and severe dip in weight.

Here’s what I think is going on. First, look at how much water is retained to store each gram per macro:

grams of water
------------------
Carbs ........ 3-4
Protein ...... 2
Fat .......... 1

This just means that every gram of carbs causes you to retain 3-4 grams of water, etc. If the “weight” you have to carry is the gram of macro + its water, then we have 5 for carbs, 3 for protein, and 2 for fat.

"Carry" grams
------------------
Carbs ........ 5
Protein ...... 3
Fat .......... 2

However, not all calories are the same:

calories per gram
------------------
Carbs ........ 4
Protein ...... 4
Fat .......... 9

Therefore:

grams to carry per calorie
--------------------------
Carbs ............... 1.25
Protein ............. 0.75
Fat .................. 2/9

Take my word for it that 117 g each of protein, carbs, and fat is 2000 calories. Then that’s 117*5 + 117 * 3 + 117 * 2 = 1170 grams to carry. For a given calorie load, you carry C/2000 * 1170 ~=0.6 * C grams, or 0.0006 * C kg. That’s around 2.5 lbs just to hold your food.

Corollary: Every 80 carbs you eat is +1 lb of carry weight.

However, a high carb diet might have 500g of carbs, which is up to 2500g = 5.5 lbs. Vs a high fat diet, for which 2000 calories requires 222g of fat, which is 444g of carry, or just over 1 lb. So switching macros can cause a 4-5 lb drop in weight. And presumably this happens basically instantly, not over time. Body composition hardly changes, but the weight swings based on macro balance.

If you’re like me, and you go out swinging before the first day of the diet, you can go from 3000 calories carb-heavy pizza and beer, to below 1000 calories, mostly protein on day 1. (trust me I’ve done this a lot). It wouldn’t be unreasonable to see a 10 lb drop in weight in the first few days. And believe me you feel smaller, but it’s not fat loss.

Leveling off

It’s probably obvious now why I “level” off. The question isn’t “Why doesn’t the weight loss continue”, because there is no early functional weight loss. Just a macro balance change causing changes to water content in my body.

The more subtle problem is that my expected rate of weight loss is based on starting weight. And my calorie deficit and predicted weight all come from that.

So I’m basically completely misrepresenting my weight as too high, then expecting a steeper slope down from a given deficit. That is, my basal metabolic rate (calculated from weight) is much lower than I think because my weight is 5-10% higher from water due to pizza and beer binges before my diet.

All that to say, maybe third time is the charm.

Closing

To ensure that yes, this year I will actually do it, I mailed a large check to a friend who can cash it if I do not stand on a scale in front of him by the end of July and clock in at 200 lbs. I’m cautiously optimistic.

What did you expect? Of course the Trump administration will defy court orders!

Rondam Ramblings — Tue, 11 Feb 2025 00:30:25 GMT

I'm not sure whether to laugh or cry over this:Federal judges have issued President Donald Trump stinging legal rebukes in the early clashes over his blitz of executive orders, and two of his top aides have responded by suggesting that his administration defy the courts and move forward with its agenda.There’s no indication that Trump has adopted such a strategy, although a U.S. judge in Rhode

Archiving hardware projects

John Graham-Cumming's blog — Mon, 10 Feb 2025 19:09:09 GMT

From time to time I do some project involving old hardware that requires connecting it to a modern computer. For example,

Getting the KIM-1 to talk to my Mac

Resurrecting a Dataman S4 PROM programmer

Ripping old mini DV video tapes on a Mac

And I've come to the conclusion that archiving the related hardware is important. For example, I got this Dataman S4 talking to my Mac using a few different cables. I bought a set of cables and archived them in a bag for the next time I need to do this.

The Dataman S4 project required three things: a USB-C to USB-A adapter, a USB-A to RS-232 adapter and a 9 pin to 25 pin serial cable.

These things are fairly inexpensive making it viable to archive them. Now I only have to worry about the drivers being available!

Multiline fixed string search and replace with CLI tools

learnbyexample — Mon, 10 Feb 2025 02:00:00 GMT

This post shows how you can use the ripgrep, perl and sd commands to perform multiline fixed string search and replace operations from the command line. Solutions with GNU sed is also discussed, along with its limitations.

Fixed string matching🔗

The below sample input file will be used in the examples in this post.

$ cat ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

ripgrep🔗

ripgrep supports the -U option to allow multiline matching. The -F option turns off regexp matching, i.e. the search string is treated literally. In the bash shell (and likely most other shells), you can press enter key to insert literal newline character for quoted values. When you do so, the next line starts with the secondary prompt PS2, which is usually > and a space character. This isn't shown in the examples below to make it easier to copy-paste the commands.

$ rg -UF 'like . () * [] $ {}
| ^ + ? \ and' ip.txt
4:like . () * [] $ {}
5:| ^ + ? \ and ' and so on.

# use the -l option to get only the filename instead of matching lines
$ rg -lUF 'like . () * [] $ {}
| ^ + ? \ and' ip.txt
ip.txt

You'll have an issue if your search string itself contains single quote characters. Avoid using double quotes as a workaround, as that has its own set of special characters. You can work around by concatenating multiple strings next to each other, along with escaped single quote characters as needed.

# the -N option disables line number prefix
$ rg -NUF 'like . () * [] $ {}
| ^ + ? \ and '\'' and' ip.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

If your search string starts with the - character, you'll have to use -- before the search argument.

$ rg -NUF -- '-string multiline
search' ip.txt
-string multiline
search with cli tools.

perl🔗

You can use the -0777 option with perl to slurp the entire input as a single string. Another advantage with perl is that you can use files to pass the search and replace strings. Thus, you don't have to worry about any character that may clash with shell metacharacters. See my Perl One-Liners Guide if you are not familiar with using perl from the command line.

$ cat search_1.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

# display filename if the given search string matches
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_1.txt ip.txt
ip.txt

However, you'll have to make sure the file doesn't end with a newline if you are providing partial lines for searching, or take care of it within the perl script.

$ cat search_2.txt
-string multiline
search

# no output because there's a newline at the end of search_2.txt file
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt

# this will remove newline from the end of file before assigning to $s
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt
ip.txt

By default, ripgrep gives entire matching lines. To get rest of the line with perl, you'll have to explicitly add a pattern around the search string.

# $& variable has the entire matching portion
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $&' search_2.txt ip.txt
-string multiline
search

# use 'say $& while /.*\Q$s\E.*/g' if there are multiple matches
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /.*\Q$s\E.*/ && say $&' search_2.txt ip.txt
-string multiline
search with cli tools.

Fixed string substitution🔗

ripgrep🔗

ripgrep also supports replacing the matched string with something else using the -r option. By default, you'll see only matched lines in the output. Use the --passthru option to display all the input lines, even if they do not match the given search string. See my blog post for more details about the -r option and various ways you can use it for substitution requirements.

$ rg --passthru -NUF 'like . () * [] $ {}
| ^ + ? \ and' -r '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

Apart from having to workaround single quote, you'll have to use $$ instead of $ as it is used for backreferences in the replacement section.

$ echo 'sample input' | rg --passthru -F 'in' -r '$a'
sample put
$ echo 'sample input' | rg --passthru -F 'in' -r '$$a'
sample $aput

perl🔗

With perl, you can use files for both the search and replace strings. And, you can easily choose to replace the first or all occurrences, unlike ripgrep where it always replaces all the matches.

$ cat replace.txt
---------------------
$& = $1 + $2 / 3 \ 4
=====================

$ perl -0777 -ne '$#ARGV==1 ? $s=$_ : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_1.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

As seen before, you'll have to remove newline from the search string for partial line matching.

# use $r=s/\n\z//r to avoid trailing newline from replace.txt
$ perl -0777 -ne '$#ARGV==1 ? $s=s/\n\z//r : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_2.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
=====================
 with cli tools.

sd🔗

sd supports a fixed string option and Rust regexp based substitution. Unlike ripgrep, the -s option for fixed string will apply to both the search and replacement sections. sd does in-place editing for file inputs by default, you can use -p to preview results on the terminal. Multiline matching is automatically performed by default.

$ echo 'sample input' | sd -s 'in' '$a'
sample $aput

$ sd -ps 'like . () * [] $ {}
| ^ + ? \ and' '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

Saving file contents to a variable🔗

Trailing newlines and ASCII NUL characters will be lost if you wish to save contents of a file as bash variables using the var=$(< filename) command. See stackoverflow: pitfalls of reading file into shell variable for more details.

$ printf '\na\0b\n123\n\n\n\n\n\n\n\n' > t1
$ a=$(< t1)
bash: warning: command substitution: ignored null byte in input

# NUL character is lost after the assignment
# all the trailing newlines are lost as well
$ printf '%b' "$a" | cat -A
$
ab$
123

ripgrep🔗

If your search string doesn't have multiple trailing newlines or ASCII NUL characters, then you can save file contents to variables and then pass them to ripgrep. Single trailing newline will not normally cause an issue for searching operations as ripgrep will append a newline while displaying results anyway. If you want to make sure input file also contains the trailing newline, then you can manually concatenate a newline character to the search string.

$ s=$(< search_1.txt)
# use "$s"$'\n' if you want to match trailing newline as well
$ rg -NUF "$s" ip.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

# use -- if the search string starts with a - character
$ s=$(< search_2.txt)
$ rg -NUF -- "$s" ip.txt
-string multiline
search with cli tools.

For substitution operations, you'll have to preprocess the replacement file to replace $ with $$.

$ s=$(< search_1.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

# here, removal of trailing newline doesn't cause an issue,
# as it evens out between the search and replace strings
$ rg --passthru -NUF "$s" -r "$r" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

Here, partial line has to be matched. So, $() assignment works well for the search string. If the trailing newline of the replacement string isn't needed, then $() assignment again is good enough. Otherwise, you can modify the replacement string as -r "$r"$'\n'

$ s=$(< search_2.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

$ rg --passthru -NUF -r "$r" -- "$s" ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.

sd🔗

As mentioned before, the -s option for sd applies to both the search and replacement sections. So, the usage is lot simpler compared to ripgrep.

# -- is needed here because replace.txt starts with a - character
$ sd -ps -- "$(< search_1.txt)" "$(< replace.txt)" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

GNU sed🔗

To follow a similar approach with GNU sed, you'll have to preprocess the strings to escape metacharacters. Assuming input doesn't have ASCII NUL characters, you can use -z option to slurp the entire input as a single string.

Here's an example for multiline search.

# escape all BRE metacharacters
# replace literal newlines with \n
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n/\\n/g' search_1.txt)

# since newlines are replaced with \n,
# trailing newlines will be preserved here
$ echo "$s"
like \. () \* \[] \$ {}\n| \^ + ? \\ and ' and so on\.\n

# display filename if input matches the given multiline search string
# tr is used to change the NUL character after filename to newline
$ sed -nz '/'"$s"'/F' ip.txt | tr '\0' '\n'
ip.txt

And here's an example for multiline substitution.

# last newline is removed here to allow partial line matching
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n$//; s/\n/\\n/g' search_2.txt)

# escape all replacement section metacharacters
# and prefix \ character to literal newlines, except the last line
$ r=$(sed 's:[\\/&]:\\&:g; $!s/$/\\/' replace.txt)
$ echo "$r"
---------------------\
$\& = $1 + $2 \/ 3 \\ 4\
=====================

# if you need the trailing newline from replace.txt,
# use sed -z 's/'"$s"'/'"$r"'\n/g'
$ sed -z 's/'"$s"'/'"$r"'/g' ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.

Linux CLI ebooks🔗

Check out my ebooks if you are interested in learning more about Linux CLI basics, coreutils, text processing tools like GNU grep, GNU sed, GNU awk and perl.

Agentic Coding - Live Demo / Brownbag

smcleod.net — Thu, 06 Feb 2025 17:00:01 GMT

Video recording of a brownbag presentation / live demo I ran on Agentic Coding using Cline/Roo Code

Twenty years of the "GNU Make Standard Library"

John Graham-Cumming's blog — Wed, 05 Feb 2025 11:50:10 GMT

Twenty years ago, on February 3, 2005, I released v1.0.0 of a project I called "GNU Make Standard Library" (GMSL). That first release can still be found in its original location on SourceForge. I moved the project from SourceForge to GitHub and newer releases are there. The latest release is v1.2.2 on March 30, 2024.

The project has its own homepage at https://gmsl.jgc.org/. The page describes the GMSL as "a collection of functions implemented using native GNU Make functionality that provide list and string manipulation, integer arithmetic, associative arrays, stacks, and debugging facilities."

GMSL grew out of work that eventually became my book "The GNU Make Book". What can it do?

Well, things like this:

include gmsl

VAR := Here is a string with some words in it.
$(info $(VAR))

# Convert VAR to lowercase
LC_VAR := $(call lc,$(VAR))
$(info $(LC_VAR))

# Get the length of LC_VAR as a string
$(info $(call strlen,$(LC_VAR)))

# See if VAR and LC_VAR are the same string
$(info $(if $(call seq,$(VAR),$(LC_VAR)),Same,Different))

# Treat LC_VAR as a list and get the last element
$(info $(call last,$(LC_VAR)))

# Get the number of list elements in LC_VAR
$(info $(call length,$(LC_VAR)))

# Apply the strlen function to each element of LC_VAR
$(info $(call map,strlen,$(LC_VAR)))

# Deduplicate the list of lenghts of elements of LC_VAR
$(info $(call uniq,$(call map,strlen,$(LC_VAR))))

# Split LC_VAR on the letter e and replace with 3
$(info $(call merge,3,$(call split,e,$(LC_VAR))))

# Create an associative array mapping the lengths of words
# in LC_VAR to a count for each length
increment_aa = $(call set,LC_VAR_AA,$1,$(call inc,$(call get,LC_VAR_AA,$1)))
$(call map,increment_aa,$(call map,strlen,$(LC_VAR)))
dump_key = $(info $1: $(call get,LC_VAR_AA,$1))
$(call map,dump_key,$(call keys,LC_VAR_AA))

And a whole lot more. The entire GMSL is written using native GNU Make functionality and pull requests are welcome!

Update 2025-02-05: I decided GMSL was missing a fold function so I added it. So, latest release is today, v1.2.3.

Giving Junior Engineers Control Of A Six Trillion Dollar System Is Nuts

The Codist — Wed, 05 Feb 2025 01:05:17 GMT

For some purpose, the DOGE people are burrowing their way into all US Federal Systems. Their complete control over the Treasury Department is entirely insane.

Unless you intend to destroy everything, making arbitrary changes to complex computer systems will result in destruction, even if that was not your intention. No

Practicing graphical debugging using too many visualizations of the Hilbert curve

Kartik Agaram — Fri, 31 Jan 2025 19:06:44 GMT

Sorry, this article is too wide for my current website design so you'll need to go to it →

Edit for clarity

Notes on software development — Wed, 29 Jan 2025 02:00:00 GMT

I have the fortune to review a few important blog posts every year and the biggest value I add is to call out sentences or sections that make no sense. It is quite simple and you can do it too.

Without clarity only those at your company in marketing and sales (whose job it is to work with what they get) will give you the courtesy of a cursory read and a like on LinkedIn. This is all that most corporate writing achieves. It is the norm and it is understandable.

But if you want to reach an audience beyond those folks, you have to make sure you're not writing nonsense. And you, as reviewer and editor, have the chance to call out nonsense if you can get yourself to recognize it.

Immune to nonsense

But especially when editing blog posts at work, it is easy to gloss over things that make no sense because we are so constantly bombarded by things that make no sense. Maybe it's buzzwords or cliches, or simply lack of rapport. We become immune to nonsense.

And even worse, without care, as we become more experienced, we become more fearful to say "I have no idea what you are talking about". We're afraid to look incompetent by admitting our confusion. This fear is understandable, but is itself stupid. And I will trust you to deal with this on your own.

Read it out loud

So as you review a post, read it out loud to yourself. And if you find yourself saying "what on earth are you talking about", add that as a comment as gently as you feel you should. It is not offensive to say this (depending on how you say it). It is surely the case that the author did not know they were making no sense. It is worse to not mention your confusion and allow the author to look like an idiot or a bore.

Once you can call out what does not make sense to you, then read the post again and consider what would not make sense to someone without the context you have. Someone outside your company. Of course you need to make assumptions about the audience to a degree. It is likely your customers or prospects you have in mind. Not your friends or family.

With the audience you have in mind, would what you're reading make any sense? Has the author given sufficient background or introduced relevant concepts before bringing up something new?

Again this is a second step though. The first step is to make sure that the post makes sense to you. In almost every draft I read, at my company or not, there is something that does not make sense to me.

Do two paragraphs need to be reordered because the first one accidentally depended on information mentioned in the second? Are you making ambiguous use of pronouns? And so on.

In closing

Clarity on its own will put you in the 99th percentile of writing. Beyond that it definitely still matters if you are compelling and original and whatnot. But too often it seems we focus on being exciting rather than being clear. But it doesn't matter if you've got something exciting if it makes no sense to your reader.

This sounds like mundane guidance, but I have reviewed many posts that were reviewed by other people and no one else called out nonsense. I feel compelled to mention how important it is.

Wrote a new post on the most important, and perhaps least done, thing you can do while reviewing a blog post: edit for clarity. pic.twitter.com/ODblOUzB3g
— Phil Eaton (@eatonphil) January 29, 2025

Basic examples for the Linux date command

learnbyexample — Tue, 28 Jan 2025 02:00:00 GMT

I rarely ever use the date command, but when I need it I almost always struggle to get the right incantation. So, I'm just going to record such examples in this blog post (and some good to know features).

There'll also be learning resources linked at the end of the post.

Really basic examples

The date command by itself shows the current time. But that's rarely what I need, since I could just use the calendar widget at the bottom of my desktop screen. Perhaps useful to copy the string format and modify system time with the -s option?

# use the -u option for UTC (Coordinated Universal Time)
$ date
Wednesday 31 July 2024 03:53:01 PM IST

Instead, I need particular parts in a particular format. For example, to represent the time component in a dynamically constructed filename as part of a shell script.

# same as: date +%F or date -I or date --iso-8601
$ date +%Y-%m-%d
2024-07-31

$ date +%Y/%m/%d
2024/07/31
$ date +%y-%m-%d
24-07-31

# use 'b' and 'B' for month names
$ date +%a
Wed
$ date +%A
Wednesday

You can use %x to get the locale representation:

$ date +%x
31/07/24

For hours, minutes and seconds:

# same as: date +%T
$ date +%H:%M:%S
16:00:32

# same as: date +%Y-%m-%dT%H:%M:%S%:z
$ date -Iseconds
2024-07-31T16:09:27+05:30

Displaying and converting epoch seconds

# total seconds since the epoch (1970-01-01 00:00:00 UTC)
$ date +%s
1722422393

$ date -d @1722422393 +'%F %T'
2024-07-31 16:09:53

You can also provide an input file for conversion using the -f option:

$ cat epochs.txt
@0000000000
@1234567890
@2222222222

# recall that the -u option gives you UTC
$ date -u -f epochs.txt +'%F %T'
1970-01-01 00:00:00
2009-02-13 23:31:30
2040-06-02 03:57:02

Date arithmetic

$ date -I
2024-08-02
$ date -d '+1 month 4 days'
Friday 06 September 2024 01:24:44 PM IST

# same as: date -d '-20 days' +%F
# you can also use '20 days ago'
$ date -I -d '-20 days'
2024-07-13

For my learnbyexample weekly newsletter, I use a script to generate a template issue. I use the arithmetic feature as shown below:

# prev_date variable gets the value from the previous newsletter issue
$ prev_date='2024-02-23'
$ date -d "$prev_date"' +7 days' +%F
2024-03-01

Resource links

The CP/M emulator now works better!

Steve Kemp's Blog — Sun, 26 Jan 2025 01:00:00 GMT

I keep saying I'm "done" with my CP/M emulator, but then I keep overhauling it in significant ways. Today is no exception. In the past the emulator used breakpoints to detect when calls to the system BIOS, or BDOS, were made. That was possible because the BIOS and BDOS entry points are at predictable locations. For example a well-behaved program might make a system call with code like this:

    LD A,42
    LD C,4
    CALL 0x0005

So setting a breakpoint on 0x0005 would let you detect a system-call was being made, inspect the registers to see which system-call was being made and then carry out the appropriate action in your emulator before returning control back to the program. Unfortunately some binaries patch the RAM, changing the contents of the entry points, or changing internal jump-tables, etc. The end result is that sometimes code running at the fixed addresses is not your BIOS at all, but something else. By trapping/faulting/catching execution here you break things, badly.

So today's new release fixes that! No more breakpoints. Instead we deploy a "real BDOS" in RAM that will route system-calls to our host emulator via a clever trick. For BDOS functions the C-register will contain the system call to operate, our complete BDOS implementation is:

    OUT (C),C
    RET

The host program can catch writes to output ports, and will know that "OUT (3), 3" means "Invoke system call #3", for example. This means binary patches to entry-points, or any internal jump-tables won't confuse things and so long as control eventually reaches my BIOS or BDOS code areas things will work.

I also added a new console-input driver, since I have a notion of pluggable input and output devices, which just reads input from a file. Now I can prove that my code works. Pass the following file to the input-driver and we have automated testing:

A:
ERA HELLO.COM
ERA HELLO.HEX
ERA HELLO.PRN
hello
ASM HELLO
LOAD HELLO
DDT HELLO.com
t
t
t
t
t
t
t
t
t
C-c
EXIT

Here we:

Erase "HELLO.COM", "HELLO.HEX", "HELLO.PRN"
Invoke "hello[.com]" (which will fail, as we've just deleted it).
Then we assemble "HELLO.ASM" to "HELLO.HEX", then to "HELLO.COM"
Invoke DDT, the system debugger, and tell it to trace execution a bunch of times.
Finally we exit the debugger with "Ctrl-C"
And exit the emulator with "exit"

I can test the output and confirm there are no regressions. Neat.

Anyway new release, today. Happy.

How to download YouTube Videos quickly

swyx's site RSS Feed — Sat, 25 Jan 2025 20:33:59 GMT

I used to use yt5s all the time to rip and remix videos:

What I Miss And Don't From Working As A Programmer

The Codist — Sat, 25 Jan 2025 04:36:14 GMT

I retired almost four years ago after nearly 40 years as a programmer. While I still write code daily, I do so to support my generative art rather than get paid for it.

Most of my career was spent building new applications, and no matter what my title was, I

An explosion of transitive dependencies

Notes on software development — Sat, 25 Jan 2025 02:00:00 GMT

A small standard library means an explosion in transitive dependencies. A more comprehensive standard library helps you minimize dependencies. Don't misunderstand me: in a real-world project, it is practically impossible to have zero dependencies.

Armin Ronacher called for a vibe shift among programmers and I think that this actually exists already. Everyone I speak to on this topic has agreed that minimizing dependencies is ideal.

Rust and JavaScript, with their incredibly minimal standard libraries, work against this ideal. Go, Python, Java, and C# in contrast have a decent standard library, which helps minimize the explosion of transitive dependencies.

Examples

I think the standard library should reasonably include:

JSON, CSV, and Parquet support
HTTP/2 support (which includes TLS, compression, random number generation, etc.)
Support for asynchronous IO
A logging abstraction
A SQL client abstraction
Key abstract data types (BTrees, hashmaps, sets, and growable arrays)
Utilities for working with Unicode, time and timezones

But I don't think it needs to include:

Excel support
PostgreSQL or Oracle clients
Flatbuffers support
Niche data structures

Neither of these are intended to be complete lists, just examples.

Walled gardens

Minimal standard libraries force growing companies to build out their own internal collection of "standard libraries". As one example, Bloomberg did this with C++. And I've heard of companies doing this already with Rust. This allows larger companies to manage and minimize the explosion of transitive dependencies over time.

All growing companies likely do something like this eventually. But again, smaller standard libraries incentivize companies to build this internal standard library earlier on. And the community benefits relatively little from these internal standard libraries. The community would benefit more if large organizations contributed back to an actual standard library.

Smaller organizations do not have the capacity to build these internal standard libraries.

Maybe the situation will lead to libraries like Boost for JavaScript and Rust programmers. That could be fine.

Versioning

A comprehensive standard library does not prevent the language developers from releasing new versions of the standard library. It is trivial to do this with naming like Go has done with the v2 pattern. math/rand/v2 is an example.

Conclusion

I'm primarily thinking about maintainability, not security. You can read about the security risks of using a language with an ecosystem like Rust from someone who is an expert on the matter.

My concern about the standard library does not stop me from using Rust and JavaScript. They could choose to invest in the standard library at any time. We have already begun to see Bun and Deno to do exactly this. But it is clearly an area for improvement in Rust and JavaScript. And a mistake for other languages to avoid repeating.

While zero dependencies is practically impossible, everyone I've spoken to agrees that minimizing dependencies is ideal. Rust and JavaScript work against this ideal. But they could change at any time. And Bun and Deno are already examples of this.https://t.co/qkSh6oW1Yd pic.twitter.com/mY1MNErZG7
— Phil Eaton (@eatonphil) January 25, 2025

Understanding Python re(gex)? book announcement

learnbyexample — Wed, 22 Jan 2025 02:00:00 GMT

Hello!

I just published a new version of my Understanding Python re(gex)? ebook.

This book will help you learn Python Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises. The standard library re as well as the third-party regex module are covered in this book.

Release offers🔗

To celebrate the new release, you can download the PDF/EPUB versions of Understanding Python re(gex)? for FREE till 31-Jan-2025.

Here are some more amazing offers:

All 13 books bundle is $18 (normal price $36) — Leanpub or Gumroad
100 Page Python Intro is FREE (normal price $10) — Leanpub or Gumroad

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

What's new?🔗

Python version updated to 3.13
- deprecated features, SyntaxWarning, name change from re.error to re.PatternError
Corrected typos, updated descriptions, timing results and external links
Exercises are now numbered instead of using alphabets

Videos🔗

Check out my programming tips covering Python, command line tools and Vim:

re(gex)? playground🔗

To make it easier to experiment, I wrote on an interactive app. See PyRegexPlayground repo for installation instructions and usage guide. A sample screenshot is shown below:

re(gex)? exercises🔗

I wrote another TUI app to help you solve exercises from this book interactively. See PyRegexExercises repo for installation steps and app_guide.md for instructions on using this app. Here's a sample screenshot:

See my blog post Python regex cheatsheet for a quick reference.

Table of Contents🔗

Preface
Why is it needed?
re introduction
Anchors
Alternation and Grouping
Escaping metacharacters
Dot metacharacter and Quantifiers
Interlude: Tools for debugging and visualization
Working with matched portions
Character class
Groupings and backreferences
Interlude: Common tasks
Lookarounds
Flags
Unicode
regex module
Gotchas
Further Reading

Web version🔗

You can read the book online here: https://learnbyexample.github.io/py_regular_expressions/

GitHub repo🔗

Visit https://github.com/learnbyexample/py_regular_expressions for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, rating/review, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/py_regular_expressions/issues
E-mail: learn by example.net@gmail.com (without the spaces)
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Embedding Python in Rust (for tests)

Notes on software development — Wed, 22 Jan 2025 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Adventure to the heart of Iceland – The F821 Eyjafjarðarleið

The Road Chose Me — Mon, 20 Jan 2025 21:39:50 GMT

I’ve been eager to get up into the heart of Iceland since we arrived, knowing the inland roads won’t stay snow free forever. To get into the middle of the country we drive the...

“Fortunately, My Learning Turned Out to be Profitable”

Beetle Space — Mon, 20 Jan 2025 10:00:00 GMT

Isaac Berman had, by his second wife, Tamara, four children in his old age. The first was a girl, my mother, Anna Rachel, and then three sons. Isaac Berman died in 1901, when my mother was six years old.

Apparently my father did not want to mention things he considered …

An Open Letter to Democratic Candidates and Organizations

Rondam Ramblings — Mon, 20 Jan 2025 09:31:52 GMT

The ink is barely dry on the (drama-free) certification of the election results and I'm already getting calls from organizations and candidates seeking donations. (Apparently my name is on a lot of lists.) In order to save everyone a lot of time, I thought I'd write up my current thinking so I don't have to have the same conversation dozens of times.First and foremost: I really do

Home Assistant Voice Preview is an unusable mess.

swyx's site RSS Feed — Mon, 20 Jan 2025 02:37:56 GMT

I just got a Home Assistant Voice recently. I was so excited to try it out as a programmable Alexa.

Logical replication in Postgres: Basics

Notes on software development — Fri, 17 Jan 2025 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

I am (not) a Failure: Lessons Learned From Six (and a half) Failed Startup Attempts

Rondam Ramblings — Sun, 12 Jan 2025 21:49:48 GMT

A while back I wrote:I've had many, many failures in my life. (Hm, maybe I should write a blog post about that.)This is that post. I'm writing it not as a lament, but rather because I've ended up in a good place in life despite my extraordinary track record of failing at just about everything I've ever tried. If my younger self had heard these stories he might have had a less

The Unexpected Benefit of Consuming Articles Offline

Beetle Space — Sun, 12 Jan 2025 10:00:00 GMT

A while ago, I wrote about my shift in reading most articles on a paper as opposed to on a screen.

I also wrote about consuming articles via podcast SW.

In the first case, I switched to reading on paper. In the second, I consumed via audio in my car …

Simple Go Cache

Ben E. C. Boyter — Sun, 12 Jan 2025 02:00:00 GMT

Happy new year!

I am in the middle of converting searchcode.com over to SQLite, which is a far more interesting future post, and as a result had to rewrite some of the core service logic. The previously implemented code had grown over time so I decided to start fresh. Of course the catch of this is introducing regressions, which I did by stripping out some of the caching logic I had implemented.

Adding Fediverse Comments to My Static Site

Beetle Space — Fri, 10 Jan 2025 12:00:00 GMT

The thing one misses most when switching to static site generators is comments.

You can always use a 3rd party service like Disqus, but then it becomes a war against spam.

There are other creative options like creating a subreddit for your blog, and just moving the discussion there. It …

Just Present the First Draft!

Beetle Space — Fri, 10 Jan 2025 10:00:00 GMT

You write a working prototype.

You begin refactoring the code.

Your manager says “Don’t bother. It’s working. Why waste time rewriting when you could be making something new? Any competent programmer should be able to spend time and understand the code.”

You respond with “Sure. The next time …

Interactive exercises for GNU grep, sed and awk (TUI apps)

learnbyexample — Thu, 09 Jan 2025 02:00:00 GMT

Having an interactive program that automatically loads questions and checks the solution is immensely helpful to have while learning a topic. I've written TUI apps with plenty of beginner to intermediate level exercises for GNU grep, GNU sed and GNU awk.

Installation🔗

For the past few months, I've been using a Python framework called Textual to create interactive TUI apps.

You'll need Python for this. This app is available on PyPI as grepexercises, sedexercises and awkexercises. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install grepexercises sedexercises awkexercises

# launch the app, example shown for the grep command
$ grepexercises

To run the app without having to enter the virtual environment again, add aliases to .bashrc (or equivalent):

# you'll have to change the path
alias grepexercises='/path/to/textual_apps/bin/grepexercises'

# similarly, you can add aliases for the other apps as well

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install grepexercises sedexercises awkexercises
$ awkexercises

As yet another alternative, you can install textual==0.85.2 (see Textual documentation for more details), clone my TUI-apps repository and run the Python file from respective folders. For example, grep_exercises.py for the grep command.

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines).

You can use alternative CLI tools to solve these exercises as well. For example, perl instead of GNU awk or ripgrep instead of GNU grep and so on.

Brief Guide🔗

You can either click the buttons using mouse or press the key combinations listed below:

Press F1 to view the complete guide from within the app itself.
Press Ctrl+p and Ctrl+n to navigate the questions list.
Type the command in the box below the question.
Press Enter to execute the command.
- Output would be displayed below the command box.
- If the output matches the expected results, the command box will turn green and reference solutions will also be shown.
- Issues due to errors and timeout (about 2 seconds) will be displayed in red.
Press Ctrl+s to toggle the reference solution box.
Press Ctrl+t to toggle between light and dark themes.
Press Ctrl+q to quit the app.
Some basic readline-like shortcuts are supported, for example Ctrl+u, Ctrl+k, Ctrl+w, etc

Your progress is automatically saved when you close the app and restored when you launch it again later. Already answered questions will be skipped.

There is no safeguard against the command you are executing. They are treated as if you typed them from a shell session.

Ebooks🔗

The exercise questions in these apps have been adapted from my programming ebooks: https://learnbyexample.github.io/books/

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find these TUI apps useful. Happy learning :)

100+ Interactive Python Regex Exercises

learnbyexample — Thu, 09 Jan 2025 02:00:00 GMT

Having an interactive program that automatically loads questions and checks the solution is wonderful to have while learning a topic. This TUI app has beginner to advanced level exercises for Python regular expressions. There are more than 100 exercises covering both the builtin re and third-party regex module.

Installation🔗

This app is available on PyPI as regexexercises. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install regexexercises

# launch the app
$ regexexercises

If you are on Windows, using the Windows Terminal is recommended. See this issue for Virtual Environment commands and other details.

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias regexexercises='/path/to/textual_apps/bin/regexexercises'

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install regexexercises
$ regexexercises

As yet another alternative, you can install textual==0.85.2 (see Textual documentation for more details), clone my TUI-apps repository and run the pyregex_exercises.py file.

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines).

Video demo🔗

Brief Guide🔗

Type your solution in the input box below the question.
- Use ip variable to represent the sample input.
- Any single valid Python expression will be accepted.
- Some basic readline-like shortcuts are supported, for example Ctrl+u, Ctrl+k, Ctrl+w, etc
Press Enter to execute the code.
- Output would be displayed below the command box.
- If the output matches the expected results, the solution box will turn green and a reference solution will also be shown.
- Error messages due to exceptions will be displayed in red.
Press Ctrl+p and Ctrl+n to navigate the questions list.
Press Ctrl+r to toggle between str and repr — helps to spot characters like tabs, newlines, backspaces, etc.
Press Ctrl+b to toggle between expected and actual — helps to debug incorrect solutions.
Press Ctrl+s toggle reference solution
Press Ctrl+t to toggle between light and dark themes.
Press Ctrl+q to quit the app.
Press F1 to view a detailed guide within the app itself and press F2 to get back to the exercises.

Your progress will be automatically saved and restored. Already answered questions will be skipped.

There is no safeguard against the code you are executing. They are treated as if you executed them from a Python program.

See app_guide.md for more detailed instructions.

Ebook🔗

See my Understanding Python re(gex)? ebook to learn regular expressions with hundreds of examples and exercises.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find this TUI app useful. Happy learning :)

Python Regular Expressions Playground

learnbyexample — Thu, 09 Jan 2025 02:00:00 GMT

This TUI application is intended as an interactive playground for Python Regular Expressions. The app also includes a comprehensive cheatsheet and several interactive examples.

Installation🔗

This app is available on PyPI as regexplayground. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install regexplayground

# launch the app
$ regexplayground

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias regexplayground='/path/to/textual_apps/bin/regexplayground'

As an alternative, you can install textual==0.85.2 (see Textual documentation for more details), clone my TUI-apps repository and run the pyregex_playground.py file.

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines). Here's another screenshot:

Brief Guide🔗

You can type the search pattern in the Compile input box and press the Enter key (or Ctrl+r) to execute. For example, re.compile(r'\d') to match digit characters. Matching portions will be highlighted in red.

The compiled pattern is available via the pat variable and you can use ip to refer to the input string. You can transform or extract data by typing appropriate expression in the Action box. For example, pat.sub(r'(\g<0>)', ip) will add parenthesis around the matching portions.

You can skip the Compile box and directly use the Action box too. For example, [m.span() for m in re.finditer(r'\d+', ip)] to get the location of all the matching portions.

There is no safeguard against the commands you have typed. They are treated as if you executed them from a Python program.

Press F1 to view the detailed guide from within the app, F2 to get back to the Playground from other screens, F3 to view a cheatsheet and F4 for interactive examples.

For more detailed instructions, see app guide.

Ebook🔗

See my Understanding Python re(gex)? ebook to learn regular expressions with hundreds of examples and exercises.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find this TUI app useful. Happy learning :)

Interactive Linux CLI Text Processing Exercises

learnbyexample — Thu, 09 Jan 2025 02:00:00 GMT

Having an interactive program that automatically loads questions and checks the solution is wonderful to have while learning a topic. This TUI app has 60+ beginner to intermediate level exercises for Linux CLI text processing tools.

Installation🔗

Last month, I started learning a Python TUI framework called Textual. After working on a 4x4 board game, I made an interactive app to help you test your CLI text processing skills with 60+ beginner to intermediate level exercises.

You'll need Python for this. This app is available on PyPI as cliexercises. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install cliexercises

# launch the app
$ cliexercises

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias cliexercises='/path/to/textual_apps/bin/cliexercises'

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install cliexercises
$ cliexercises

As yet another alternative, you can install textual==0.85.2 (see Textual documentation for more details), clone my TUI-apps repository and run the cli_exercises.py file.

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines).

Video demo🔗

Brief Guide🔗

Press Ctrl+p and Ctrl+n to navigate the questions list.
Type the command in the box below the question.
Press Enter to execute the command.
- Output would be displayed below the command box.
- If the output matches the expected results, the command box will turn green and a reference solution will also be shown.
- Issues due to errors and timeout (about 2 seconds) will be displayed in red.
Press Ctrl+s to toggle the reference solution box.
Press Ctrl+t to toggle between light and dark themes.
Press Ctrl+q to quit the app.
Some basic readline-like shortcuts are supported, for example Ctrl+u, Ctrl+k, Ctrl+w, etc

Your progress is automatically saved and restored. Already answered questions will be skipped.

There is no safeguard against the command you are executing. They are treated as if you typed them from a shell session.

For more detailed instructions, visit https://github.com/learnbyexample/TUI-apps/tree/main/CLI-Exercises

Ebook🔗

The exercises in this app have been adapted from my Command Line ebooks.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find this TUI app useful. Happy learning :)

Designing a Logo

swyx's site RSS Feed — Wed, 08 Jan 2025 21:53:58 GMT

I recently kicked off a 99designs contest for a new logo: https://99designs.com/logo-design/contests/logo-brand-ai-engineering-podcast-help-define-industry-1307842/

My Books to Love & Hate from 2024

Caseysoftware — Wed, 08 Jan 2025 15:39:00 GMT

After college, I resolved to read one book a month. It can be fiction, non-fiction, technical, business-oriented, or whatever as the…

The post My Books to Love & Hate from 2024 appeared first on Caseysoftware.

The Westfijords of Iceland

The Road Chose Me — Mon, 06 Jan 2025 17:03:27 GMT

Iceland is a deceivingly large country, and although we don’t have a stated goal to drive every road on the island, the Westfijords are a place where that simply makes sense. Around every corner...

Actually Structured Journaling (Jan 2025)

swyx's site RSS Feed — Mon, 06 Jan 2025 12:12:26 GMT

I was very excited when I saw Cal Newport's episode on Structured Journaling come up, but I was disappointed by the recommendations:

Pseudocode for Intentionality

swyx's site RSS Feed — Mon, 06 Jan 2025 11:53:19 GMT

I've been interested in being more intentional with my work, time and life recently. Here is my working definition of how to be more intentional.

Iceland Aurora Borealis

The Road Chose Me — Fri, 03 Jan 2025 16:50:30 GMT

A lot of people ask if the Northern Lights look like they do in videos, or if that is an exaggeration. I can say that YES, they look almost exactly like in the video...

Date Me!

swyx's site RSS Feed — Thu, 02 Jan 2025 07:28:19 GMT

if you like me*, and you're interested but never thought to ask, just ask!

Appendix A: The DataProvider Struct

Little Green Viper — Wed, 01 Jan 2025 15:08:14 GMT

The real heart of the demonstration app, is something that we didn’t cover at all, in the tutorial. That’s the Data Model for the app, which is in the DataProvider.swift structural type. It’s a fairly basic type, that consumes some CSV (Comma-Separated-Values) data, and formats it into a configuration that can drive the chart. The ...

Looking Ahead to 2025

Ben Overmyer's Site — Wed, 01 Jan 2025 02:00:00 GMT

This is going to be a year of big changes.

Goals for 2025

My goals this year will be primarily setting new habits.

I started guitar lessons in December. I plan to continue practicing every day, excepting some holidays or travel.

Also on the creative side, I want to develop a painting habit. This will probably be less frequent than guitar, but I still want to paint at least once a week.

Continuing from last year's success with going alcohol-free, I will expand this to make Q1 and Q3 entirely alcohol-free. I expect this will have positive side effects.

For exercise, I'm getting back on the horse, as it were. I plan on setting a daily walking habit. I also hope to start jogging again, and complete at least one 5K.

Thoughts for 2025

Politically, the USA is likely to experience some upheaval this year. I hope nothing catastrophic comes of it. We'll see.

While it's not a goal, I'd like to pay more attention to finances this year and end the year with much more savings.

In 2025 I expect my life to change for the better as I devote myself to more physical and creative activities, and I'm very much looking forward to fostering a dog soon.

My 25-year adventure in AI and ML

Austin Z. Henley's Blog — Wed, 01 Jan 2025 01:00:01 GMT

https://austinhenley.com/blog/25yearsofai.html