Hacker News Personal Blogs 2024 | Top 100 Blogs

2024 Year in Review

Ben Overmyer's Site — Tue, 31 Dec 2024 02:00:00 GMT

2024 was a year of travel, challenges, endings, and beginnings. I went on five trips, four of which were for recreation and the fifth of which was almost a move to a different state. My fitness and health suffered, and Silver Gryphon Games shut down. However, I began taking music lessons, won an award at work, and regularly ran an RPG campaign for once.

Winning an Award at Apiture

At my day job, I won an award for my work on Apiture’s hosted sites service. I continued to be the sole infrastructure person for hosted sites, despite their quantity at the end of the year. My automation for them made it much easier for the designers to spin up new sites without requiring work from the platform engineering team.

Collapse of Exercise Habit

My exercise habit fell apart. I started the year strong, going to the gym regularly and focusing on a bodybuilding routine that was slowly starting to show progress. Then, I went to my annual physical, and the doctor was less than enthusiastic about my chosen exercise path. For some reason, I let that discourage me from exercising at all the rest of the year.

Towards the end of the year, though, I started going for longer walks occasionally. I have it as a daily to-do item, but I'm not consistent in doing it daily yet.

Alcohol-Free Months

At the beginning of the year I set out to have three full alcohol-free months in 2024. I achieved two full months and two half months. That is sort of a success, I guess, but I think I can do better. Next year will have a more ambitious goal. My interest in going alcohol-free has increased.

I've attempted to completely quit drinking a couple times before. This is not that, not yet. I can see this evolving into that though. Past attempts were not based on a holistic lifestyle change. Now, going without alcohol is an accessory to other changes I have made or am planning to make.

The Death of Silver Gryphon Games

While we tried to resurrect the company, neither Kevin nor I had the time or motivation to work on it. I formally shuttered the business in December. The second editions of both Aether and Ingenium will never be published, sadly. That’s too bad, because the cover art for both was awesome, and they fixed a lot of the problems with the first editions. However, a business needs sustained work in order to function, and that won’t happen for SGG.

Journaling About Trips

In 2024 I went on three major trips and I promised to journal about all of them. I wrote about the Cancun trip and the cruise, but I did not write about the eclipse trip. I thought I did, but I can’t find a journal entry for it. Two out of three, but not enough to count this goal achieved.

The journal entries that exist are nice and detailed, though. They'll be a good reminder in days to come.

Journaling Habit

While I did keep a regular work journal each week, I didn’t write a regular personal journal. Part of the problem is that I never set up the habit like I did with the work journal; there was no specific trigger. This is something I can fix in 2025.

I did start keeping a gaming journal just a few days ago. If I keep up with this, then I will have four different journals that I update: a personal journal, a work journal, a gaming journal, and a spiritual journal. Perhaps if I write regularly for all of them, that will start to bleed through into blogging, and I'll update this site more frequently.

Books Read

I didn’t read nearly as many books in 2024 as I did in 2023. Also, the variety was less, with 8 different authors in 2024 and 11 in 2023. I’m not sure why, but I imagine losing almost a full month of the year to trips where I didn’t read at all didn’t help.

My queue of books to read is growing. I have quite a few on my shelves. I may hold off on buying any more books until I've worked through that backlog a bit.

RPG Campaign

In November I started running a Dungeons & Dragons campaign in-person for a few random people. It is now the longest-running regular campaign I’ve ever run. It’s also the most difficult to run, as several of the players remember throwaway comments I make about setting flavor and accept that as canon. This means I have to take notes on everything I decide. On the other hand, though, these notes are making it easier to plan sessions and review what happened in past sessions.

The next session will be on January 6, after a several-week break for the holidays.

Beginning of Music Lessons

In December I began taking guitar lessons. This is the first time since I was a teenager that I’ve taken music lessons. I’ve been good about practicing almost every day, too; the only days I haven’t practiced were Christmas Eve and Christmas Day.

I've been practicing the usual beginner songs from a Hal Leonard book. As I get better, I want to try a few different styles, including Spanish guitar. I have a list of songs to attempt once I get good enough.

Previous Years

How I run a coffee club

Notes on software development — Tue, 31 Dec 2024 02:00:00 GMT

I started the NYC Systems Coffee Club in December of 2023. It's gone pretty well! I regularly get around 20 people each month. You bring a drink if you feel like it and you hang out with people for an hour or two.

There is no agenda, there is no speaker, there is no structure. The only "structure" is that when the circle of people talking to each other seems gets too big, I break the circle up into two smaller circles so we can get more conversations going.

People tend to talk in a little circle and then move around over time. It's basically no different than a happy hour except it is over a non-alcoholic drink and it's in the morning.

All I have to do as the organizer is periodically tell people about the Google Form to fill out. I got people to sign up to the list by posting about this on Twitter and LinkedIn. And then once a month I send an email bcc-ing everyone on the list and ask them to respond for an invite.

The first 20 people to respond get a calendar invite.

I mention all of this because people ask how they can start a coffee club in their city. They ask how it works. But it's very simple! One of the least-effortful ways to bring together people in your city.

If your city does not have indoor public spaces, you could use a food court, or a cafe, or a park during months where it is warm.

For example, the Cobble Hill Computer Coffee Club is one that meets outdoors at a park.

Good luck! :)

How I run a coffee club, a short guide for others who might be interested in running one. It's very simple!https://t.co/UgRWDQOA3v pic.twitter.com/5wYrLW7u6D
— Phil Eaton (@eatonphil) December 31, 2024

The CP/M emulator runs on Windows, maybe!

Steve Kemp's Blog — Tue, 31 Dec 2024 01:00:00 GMT

Today I made a new release of my CP/M emulator and I think that maybe now it will run on Microsoft Windows. Unfortunately I cannot test it!

A working CP/M implementation needs to provide facilities for reading input from the console, both reading a complete line of text and individual keystrokes. These input functions need to handle several different types of input:

Blocking, waiting for input to become available.
Non-blocking, returning any pending input if it is available otherwise nothing.
With echo, so the user can see what they typed.
Without echo, so the keys are returned by not displayed ot the user.

In the past we used a Unix-specific approach to handle the enabling and disabling of keyboard echoing (specifically we executed the stty binary to enable/disable echos), but this release adds a more portable solution, based around termbox-go which is the new default, and should allow our emulator to work on Microsoft Windows systems.

We always had the ability to select between a number of different output drivers, and as of this release we can now select between multiple input drivers too - with the new portable option being the default. This has been tested on MacOS X systems, as well as GNU/Linux, but sadly I don't have access to Windows to test that.

Fingers crossed it's all good now though, happy new year!

Culture talk: What I learned from building a distributed database that never took off

Bytepawn - Marton Trencseni — Tue, 31 Dec 2024 01:00:00 GMT

Recently, I delivered a culture talk on Scalien, my old database startup that never took off. I discussed the business, product and some technical challenges and lessons learned from developing ScalienDB, a Paxos-based distributed NoSQL database.

Adding Pinch-To-Zoom to the Chart

Little Green Viper — Mon, 30 Dec 2024 21:56:12 GMT

In this exercise, we’ll add a “pinch-to-zoom-in/out” feature to the chart. It will be very basic, but should work quite well. The Basics The zoom will be X-axis only. It doesn’t actually make sense to zoom in on the Y-axis, especially since we have the ability to select a bar, and get the exact values ...

Modern Portfolio Theory III: The Efficient Frontier in log volatility—return space

Bytepawn - Marton Trencseni — Mon, 30 Dec 2024 01:00:00 GMT

In the previous two articles, we explored the coverage of random portfolios in log volatility—return space, both with and without a risk-free asset. We now take the next step, and calculate the Efficient Frontier and Capital Market Line of Markowitz’s theory.

Optimising my Rust solutions for Advent of Code

Krishna's blog — Sun, 29 Dec 2024 02:00:00 GMT

I completed Advent of Code 2024 in Rust. Here are all the tricks I used to optimise my code.

O-1s, H-1Bs, and other ways US Immigration is Broken

Caseysoftware — Sat, 28 Dec 2024 18:37:27 GMT

With the big ongoing debate on immigration, it’s been amazing/horrifying watching people muddle everyone together into one bucket and then…

The post O-1s, H-1Bs, and other ways US Immigration is Broken appeared first on Caseysoftware.

The MMT Perspective on Crypto

Bytepawn - Marton Trencseni — Fri, 27 Dec 2024 01:00:00 GMT

From an MMT (chartalist) perspective, cryptocurrencies lack sovereign backing and fail to establish the enforced demand necessary to function as true currency. Because they rely solely on speculative sentiment rather than legal mandates (like taxation), their values are unsustainable in the long run and risk ultimately collapsing to zero.

The complete text of "All Watched Over by Machines of Loving Grace"

John Graham-Cumming's blog — Thu, 26 Dec 2024 17:34:02 GMT

Richard Brautigan's poem "All Watched Over by Machines of Loving Grace" is somewhat well known in tech. circles but I couldn't find a complete PDF of the original 1967 publication of it (and other poems) online.

The copyright notice in Brautigan's collection reads:

Permission is granted to reprint
any of these poems in magazines,
books and newspapers if they are
given away free.

I am interpreting "print" as including the availability of a free PDF and making a scan of the complete book available here.

A Reply to Don Geddis

Rondam Ramblings — Thu, 26 Dec 2024 10:12:48 GMT

Don Geddis left a comment on my last post. My reply grew far longer than would reasonably fit into a comment reply so I decided to post it as an article. Don wrote:I wonder if you've considered that perhaps you have more in common with the people who frustrate you, than your current self-image suggests.My reply:I've not just considered it, I will happily concede that I am not as

RSS Feeds and Real Time Crawling

Weblog on marginalia.nu — Thu, 26 Dec 2024 02:00:00 GMT

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed.

The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any given point in time. To help compensate for this, a new crawler and index partition has been added that focuses on recently updated content.

Picking up volleyball in NYC with Goodrec and New York Urban

Notes on software development — Thu, 26 Dec 2024 02:00:00 GMT

I was so intimidated to go at first, but it is in fact easy and fun to start playing beginner volleyball in New York. The people are so friendly and welcoming that it has been easy to keep playing consistently every week since I started for the first time this August. It's been a great workout and a great way to make friends!

The two platforms I've used to find volleyball games are Goodrec and New York Urban. While these platforms may also offer classes and leagues, I mostly use them to play "pickup" games. Pickup games are where you show up and join (or get assigned to) a team to play for an hour or two. Easy to go on your own or with friends.

I'm not an expert! My only hope with this post is that maybe it makes trying out volleyball in New York feel a little less intimidating for you!

Goodrec

With Goodrec you have to use their mobile app. Beginner tier is called "social" on Goodrec. So browse available games until you find one at the level you want to play. You enroll in (buy a place in) sessions individually.

Sessions are between 90-120 minutes long.

They ask you not to arrive more than 10 minutes early at the gym. When you arrive you tell the gym managers (usually in a desk up front somewhere) you're there for Goodrec and the tier (in case the gym has multiple level games going on at the same time). Then you wait until the Goodrec "host" arrives and they will organize everyone into teams.

Goodrec hosts are players who volunteer to organize the games. They'll explain the rules of the game (makes Goodrec very good for beginners) and otherwise help you out.

Always say thank you to your host!

New York Urban

With New York Urban, pickup sessions are called "open play".

There is no mobile app, you just use the website to purchase a spot in a session. The sessions are longer and cheaper than Goodrec. But there is no host; players self-organize.

The options are more limited too. You play at one of four high schools on either a Friday night or on Sunday. And session slots tend to sell out much more quickly than with Goodrec.

Big City Volleyball

You can also check out Big City Volleyball but I haven't used it yet.

Volo

I haven't ever done Volo but I think I've heard it described as "beer league". That even some of the beginner tier sessions with Goodrec and New York Urban are more competitive.

But also, Volo is built around leagues so you have to get the timing right. Goodrec's and New York Urban's pickup games make it easy to get started playing any time of year.

Making friends

It was super awkward to go at first! I went by myself. I didn't know what I was doing. I couldn't remember, and didn't know, many rules. I didn't have court shoes or knee pads.

But the Goodrec host system is particularly great for bringing beginners in and making them feel welcome. You have a great time even if you're terrible.

The first game I went to, I tried to hang out afterward to meet people. But people either came with their SO or with their friends or by themselves so they all just left immediately or hung out in their group.

So you can't just go once and expect to make friends immediately. But if you keep going at the same place and time regularly week over week, you'll see familiar faces. Maybe half the people I play with each week are regulars. If you're friendly you'll start making friends with these people and eventually start going out to bars with them after the games.

Improving

Even if you find yourself embarrassingly bad at first, just keep going! I'm 29, 6'1, 190lbs and from observation the past 5 months, age, height, and weight have a very indirect relation to playing ability.

Most of the people who play are self-taught, especially at the lower tiers I've played at. But some people played for the school team in high school or college. These people are fun to play with and you can learn a lot from them.

Most people who are self-taught seem to watch YouTube videos like Coach Donny, helpful for learning how to serve, set, block, etc. Or they take "clinics" (classes) with Goodrec or other platforms. (I have no idea about these, I've never done them before.)

At first I played 2 hours a week and I was completely exhausted after the session. Over time it got easier so I started playing 2-3 sessions a week (6-9-ish hours). With practice and consistency (after about 3-4 months), I started playing Intermediate tier with Goodrec and New York Urban. And I don't think I'll play Beginner/Social at all anymore.

I still primarily play for fun and for the workout and to meet people. But it's also fun to get better!

I played with one person much better than myself in an Intermediate session one time and he mentioned he will probably stop playing Intermediate and only play High Intermediate. He mentioned you get better when you keep pushing yourself to play with better and better players. Good advice!

I wrote a little post on picking up volleyball in new york.

It's fun, and a great workout, and you meet interesting people!https://t.co/jEWHbRWF6C pic.twitter.com/ipuIUB1ZnM
— Phil Eaton (@eatonphil) December 26, 2024

I ❤️ triathlons

Bytepawn - Marton Trencseni — Thu, 26 Dec 2024 01:00:00 GMT

In 2008, I completed my first half-Ironman — then, later that same year, I took on the full Ironman distance at the traditional ExtremeMan in Hungary, finishing in 13 hours and 38 minutes. By 2013, I had knocked my time down to 11 hours and 36 minutes. Since then, I’ve racked up a total of 56 triathlon finishes. 2024 was one of the best years so far, with a record of 10 triathlon races.

A Cryptographically Secret Santa

Speculative Branches — Wed, 25 Dec 2024 02:00:00 GMT

Twas about 4-6 weeks before Christmas, and all through the math department, not a creature was stirring, not even a plucky young undergrad. Cryptography professors Alice and Bob sat at the elliptically-curved conference table to plan the department's secret Santa. Mallory, the department secretary, had been given the task of organizing last year, and somehow managed to get three gifts while leaving several people disappointed. This year's math department thus resolved to do their secret Santa without a trusted party.

Pareto Hobbies 2025

jodavaho.io — Tue, 24 Dec 2024 02:00:00 GMT

Pareto Hobbies 2025

I decided to plot possible hobbies in 2025 in terms of career potential and personal enjoyment.

This is a silly thing to do, but it is also a uniquely software-engineering problem to have - for some reason we seem to be prone to lots of extra curricular learning, quantified self, and a constant need to optimize ourselves for our employability.

As a computer scientist and sometimes hardware hacker, I’m most interested in the following things:

Geospatial ML

I’m fascinated by taking spatial data and running predictors on it. This could be predicitng fine-grained weather patterns, predicting traffic, etc.

At flight science, this has lots of potential applications, so the usefulness is fairly high. I’ve also found that kriging is very useful for robotics though it has been 10 years since I’ve applied it.

Quantum Computing

The recent google announcement has reinvigorated my old interest in quantum computing. A friend from JPL who I highly respect has said QC is at the equivalent of the vaccuum tube era of computing.

Given we’re only now demonstrating benchmark problems, and that most the researchers in those areas are probably hardware and quantum PhDs …. well, I’m a bigshot PhD Computer Scientist, but doubt I could contribute to the field without years of retraining.

Relevance: Very low.

Radar Processing

Super-specific interest, but I want to use and SDR to build a homebrew radar and track boats at the in-laws lake cabin. Why? Who knows.

I’m super comfortable with tracking and estimation, but would need to learn how to extract a “measurement” from the timing and Doppler shift of the radar return.

Given I could see myself working in tracking and estimation in the future, this could be a useful skill.

AWS code as infra

Not very glamorous, but I hate the AWS console and would like to be able to do everything in code.

RL for planning

This is something that is near and dear to my heart. I have been working in planning systems for a while, and have used “RL” (POMDP, etc) in the past, but only as a tangential part of the team.

More generally, the use of end-to-end learned systems for control at least seems to be an increasingly dominant way of doing business. With RL+Planning, my goal is to eventually be able to tradeoff classical estimation and planning with end-to-end systems, and use the best parts of either for a better-than-either pipeline. At the jobs I’ve had I’ve seen a weird false dichotomy that produces decent results once committed, but there’s always this tension that “the other way” is better, producing weird politics.

Most recently, a classical planning pipeline was completely broken by an ML-based estimation of orientation of obstacles. On inspection they were using quite possibly the worst estimator you could use, and ignoring all kinds of good data, because (I think) they were counting on perfect “measurements” from the ML-based vision system. That kind of thing must happen all the time - get stuck in a local minimum of “tune just a little better” when a good filter on top of a noisy estimator could solve all your problems. There’s probably a million such examples in planning systems too.

For discrete, well structured problems with noisy data on them, such as we might encounter in flight science, I think there’s a lot of potential for RL to be extremely useful.

Deep Learning / Large Language Models

I’m not super interested in this, but it seems like a good thing to know.

Ratings

My interest and career potential vary. I’ve rated them on a scale of 1-100, with 100 being the highest.

This is a stupid way to determine hobbies, but I’m doing it anyway.

Item	Career relevance	Personal Interest
RL for planning	50	40
DL/LLM in general	20	35
Quantum Computing	0	90
AWS Code as infra	60	10
Radar / signal processing / tracking	20	80
Geospatial ML, kriging, weather data	80	40

Results

Here’s the plotted results:

How do we make a decision on this data? As a planning guy, I should be able to do this!

We’ll do a pareto analysis, using the reward function of ||(career_relevance , personal_interest)|| (i.e., magnitude of the vector).

So, let’s plot the pareto frontier for each:

In this case, the outermost circle is the “best”, and a solution “dominates” anything inside its circle. So, according to this, I should focus on Quantum Computing and Geospatial ML. If I had applied scalars to the career relevance and personal interest, those circles would have been ellipses, and the dominance ordering would change. Maybe later.

100 Page Python Intro book announcement

learnbyexample — Tue, 24 Dec 2024 02:00:00 GMT

Hello!

I am pleased to announce a new version of my 100 Page Python Intro ebook. This book is a short, introductory guide for the Python programming language. This book is well suited:

As a reference material for Python beginner workshops
If you have prior experience with another programming language
If you want a complement resource after reading a Python basics book, watching a video course, etc

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of the ebook for FREE till 02-Jan-2025. You can still pay if you wish ;)

Two of my bundles are on sale as well:

All 13 Books Bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
- Leanpub or Gumroad
Awesome Regex Bundle — $10 (normal price $20), Python, Ruby, JavaScript, BRE/ERE, PCRE and Vim regular expressions
- Leanpub or Gumroad

What's new?🔗

Python version updated to 3.13.0
Added more exercises and you can now practice some of them using this interactive TUI app
Descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
New cover image
Images centered for EPUB format

Videos🔗

Check out my programming tips covering Python, command line tools and Vim:

Testimonials🔗

It's very thorough, written with care, and presented in a way that makes sense. Even as an intermediate Python programmer, I found use in this book.

— feedback by Andrew Healey on an early draft of "100 Page Python Intro" mentioned in this Hacker News thread

Interactive TUI app🔗

I also wrote an interactive TUI app based on some of the exercises from the ebook. Reference solutions are also provided.

Table of Contents🔗

Preface
Introduction
Numeric data types
Strings and user input
Defining functions
Control structures
Importing and creating modules
Installing modules and Virtual environments
Exception handling
Debugging
Testing
Tuple and Sequence operations
List
Mutability
Dict
Set
Text processing
Comprehensions and Generator expressions
Dealing with files
Executing external commands
Command line arguments

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/100_page_python_intro/introduction.html.

GitHub repo🔗

Visit https://github.com/learnbyexample/100_page_python_intro for programs, example files, markdown source and other details about the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Feedback🔗

I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, Gumroad rating, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/100_page_python_intro/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Modern Portfolio Theory II: Random portfolio coverage in log volatility—return space with a risk free asset

Bytepawn - Marton Trencseni — Fri, 20 Dec 2024 01:00:00 GMT

In a previous article I simulated random portfolios using Monte Carlo methods for the 2023 daily closing prices of the 100 stocks constituting the Nasdaq-100. Here I add the risk-free asset to the portfolio and examine how it affects the coverage in log volatility—return space.

--Mike-- — Thu, 19 Dec 2024 12:35:44 GMT

I can't get over the feeling that I'm wasting whatever portion of my life I've got left.

In this video - https://www.youtube.com/watch?v=4jgTCayWlwc

"A rant on personal engineering projects" by BPS.shorts

The author makes the point that when you're working commercially on part of a project, the organization always has a way to keep the project going, even if you fail. If the cost goes over, they can usually change the scope or budget to fit it in. If the parts don't integrate well, they can buy a system already integrated, or even the company that has already done it.

However, on a personal project, if any part fails, the whole thing fails. The main hazard is simply not finishing the project, rather than doing any one stage optimally. I think there's a shit-ton of value in this analysis.

Good Quote from Pale Rose in the comments @ https://www.youtube.com/watch?v=4jgTCayWlwc&lc=Ugxjv3J38ajw82_bmtF4AaABAg

*Personal projects have a unique requirement that you must optimize for: 'finishing the damn project'. Everything else is secondary*

I want to be a high-?? individual, someone who can get shit done. High agency?

Yep... that's the term, from https://x.com/tferriss/status/1719035575305408551

During my podcast interview with @EricRWeinstein, Eric said “high-agency person” in passing, and I asked him to elaborate:
“When you’re told that something is impossible, is that the end of the conversation, or does that start a second dialogue in your mind, how to get around whoever it is that’s just told you that you can’t do something? So, how am I going to get past this bouncer who told me that I can’t come into this nightclub? How am I going to start a business when my credit is terrible and I have no experience?”"

Mike's plan for becoming high-agency

List the projects I have, maintain said list
Always be working to finish the things on the list, or pruning the list
Always be gathering resources, or using them, never just hording them

It's my Birthday, here... have a BitGrid

--Mike-- — Thu, 19 Dec 2024 12:31:02 GMT

If you've read about von Neuman, and wondered what it was like to bootstrap computing, and if there could be another way.... here's an interesting Birthday Gift from me to you... the BitGrid, and a writeup, on an HN thread

https://news.ycombinator.com/item?id=42115107

On The Road in Iceland!

The Road Chose Me — Wed, 18 Dec 2024 22:29:36 GMT

It’s been a long time since I posted a upate here on The Road Chose Me – I’ve been busy with YouTube videos, writing books, speaking at events, writing magazine articles, building a new...

How Many Hours Can You Code?

The Codist — Wed, 18 Dec 2024 22:03:18 GMT

How many hours a day can you write code, and at what point does the quality of your work go down? Even more important is how many weeks and months of that max effort you can still be effective.

In my life, there have only been three periods where I

swyx in 2024 End of Year wraps

swyx's site RSS Feed — Tue, 17 Dec 2024 20:03:35 GMT

i was involved in 3 end of year-ish recaps today:

badkeming: a site for kerning so bad it's keming

John Graham-Cumming's blog — Sun, 15 Dec 2024 18:16:23 GMT

I made another silly Tumblr (to go along with Movie Code and Low Background Steel). This times it's a Tumblr for all those font disasters and other typographical amusements. Welcome to the wonderful world of badkeming.com:

Submissions gladly accepted.

Individual Career Plan

Bytepawn - Marton Trencseni — Sun, 15 Dec 2024 01:00:00 GMT

This post provides a practical Individual Career Plan template you can use to guide self-assessment, prioritize your professional development, and set meaningful goals that align with your vision for the future. By outlining clear objectives and action steps, you create a framework for continuous improvement and more informed career decisions. I use this same template for the data organization I lead, filled it out myself, and shared it with others to inspire them to do the same.

Why I read history books

Bytepawn - Marton Trencseni — Mon, 09 Dec 2024 01:00:00 GMT

Hungary’s rushed entry into World War II and its botched attempt to break away from it show how stale leadership trapped in its own fantasy bubble can steer a nation toward disaster. By “political backtesting” these moments — comparing their underlying factors to the present — today’s citizens can weigh future outcomes, draw practical lessons, we can make more informed decisions about where and how to live.

"All your base are belong to us" introduction to my 2004 MIT Spam Conference talk

John Graham-Cumming's blog — Sun, 08 Dec 2024 19:28:43 GMT

As I've mentioned before my talk at the 2004 MIT Spam Conference was about using one machine learning system to beat another. In that case a spammer using machine learning to beat a machine learning spam filter.

The talk started with a short video playing off the All your base are belong to us meme. I very carefully edited the images to change the text to be about spam and incorporate Paul Graham (who was the instigator of the MIT Spam Conference and an early machine learning spam filter pioneer).

I thought I'd lost the audio of that introduction and randomly came across it today. So here is the restored introduction to the talk made on a Mac in 2004 (I still haven't found the audio of the actual talk).

Holding Space for Wicked (Part One)

swyx's site RSS Feed — Sun, 08 Dec 2024 04:57:10 GMT

Disable Slog Messages in Go Tests

Ben E. C. Boyter — Sun, 08 Dec 2024 02:00:00 GMT

As much as I like ZeroLog for Go logging Slog being added to the standard library fits in with my rule of avoiding 3rd party packages when they are not entirely required hence I have started moving anything I work on over to it.

One thing I find especially annoying however is log messages inside tests. As such here is a quick code snippet which turns off all slog outputs, but only for tests. Stick it into any _test.go file to disable logging outputs for that package.

Paper: The Unreasonable Effectiveness of Monte Carlo Simulations in A/B Testing

Bytepawn - Marton Trencseni — Thu, 05 Dec 2024 01:00:00 GMT

I run Monte Carlo simulations of content production over random Watts-Strogatz graphs to show various effects relevant to modeling and understanding Randomized Controlled Trials on social networks: the network effect, spillover effect, experiment dampening effect, intrinsic dampening effect, clustering effect, degree distribution effect and the experiment size effect. I will also define some simple metrics to measure their strength. When running experiments these potentially unexpected effects must be understood and controlled for in some manner, such as modeling the underlying graph structure to establish a baseline.

Bringing K/V Context Quantisation to Ollama

smcleod.net — Wed, 04 Dec 2024 22:00:02 GMT

K/V context cache quantisation has been added to Ollama. This enables significant reductions in VRAM usage, allowing users to realise the potential of expanded context sizes and run larger models at their existing context sizes.

20 Years of Blogging and I’m still a n00b

Caseysoftware — Wed, 04 Dec 2024 15:16:00 GMT

As of today, I’ve been blogging for 20 years. Holy crap, that’s a long time. I remember when I started,…

The post 20 Years of Blogging and I’m still a n00b appeared first on Caseysoftware.

A quieter place

bayindirh — Mon, 02 Dec 2024 02:00:00 GMT

While this blog was silent, my life was buzzing with activity. I have been trying to tend to multiple big projects at work and in my personal life, like a juggler trying not to drop anything.

Not everything went as planned. The blog got sidelined, personal projects lost momentum, and a couple of them are temporarily frozen. On the other hand, I managed to finish a big migration at work. As always, Murphy learned what we were doing and motivated us to finish faster by intervening at the most convenient time with maximum effect (at least from his perspective). A big pile of important things got handled in my personal and work life, keeping me busy or at the edge, or both.

Personally, I managed to get my digital garden to grow, write more Go, and love it in the process. I decluttered my life further in digital and physical realms. Decluttering things take a long time when you're trying to handle decades of clutter collected by a past me who didn't understand the consequences of what he's doing. On the other hand, we didn't have these organizational superpowers back in the day, and I didn't know better, to be honest.

This decluttering process is not limited to my collected cruft. I'm simplifying my digital life and the places I frequent. I'm not using X (formerly Twitter) and not adding new photos to my Instagram account anymore (since I don't want to provide AI training material to Meta, a subject worthy of its separate blog post). Instead, I moved to Mastodon and will use another service to share my photography. Recently, I subscribed to a high-quality local newspaper's digital edition.

These moves are a new iteration of the trend I started by switching to Kagi in 2023 (which again needs a separate blog post), which profoundly affected my daily life. Using a user-centric search engine without ads has an immediate and measurable effect on life quality since you don't mentally filter ads and other SEO spam with every search, which is mentally draining.

I felt the same after subscribing to the local newspaper. Higher editorial quality, almost no ads, and access to the paper's print version in PDF form made a profound difference in how I consumed the news and felt afterward (i.e., not mentally exhausted).

The movement from popular, mainstream sites to well-selected, smaller, yet higher-quality environments made me realize how advertisements, feed algorithms, and optimizations for eyes drain our willpower, energy, and capacity to focus and do meaningful work. It's the same with useless discussions with people one doesn't know and doesn't want to agree on anything.

From now on, I'll be much more selective about where I frequent, who I engage with, and how I spend my mental energy and focus. I'll build a quieter place because I want to be tired for more meaningful reasons when I go to bed.

Until next time,

Be kind.

Modern Portfolio Theory I: Random portfolio coverage in log volatility—return space

Bytepawn - Marton Trencseni — Sun, 01 Dec 2024 01:00:00 GMT

I examine the coverage of random portfolios in log volatility—return space using Monte Carlo methods with different randomization techniques for the 2023 daily closing prices of the 100 stocks constituting the Nasdaq-100.

Will AI Take My Job?

smcleod.net — Sat, 30 Nov 2024 00:00:02 GMT

It's probably not so much AI itself you have to fear.

Festive offers for books on Python, Linux, Regular Expressions, Vim and more!

learnbyexample — Fri, 29 Nov 2024 02:00:00 GMT

Hello!

Here are some awesome deals for programming books and courses during the 2024 festive season.

My ebooks🔗

Offers valid till 02-Dec-2024. You can get them on Leanpub:

All 13 Books bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
Linux CLI Text Processing bundle — $10 (normal price $20), grep, sed, awk, perl and ruby one-liners, GNU coreutils, CLI computing
Learn by example Python bundle — $8 (normal price $15), Python introduction, Regular Expressions and Projects
Understanding Python re(gex)? — FREE (normal price $10)

You can also avail these offers on Gumroad:

All 13 Books Bundle — $15 (normal price $32), learn Regular Expressions, Linux CLI tools, Python, Vim and more!
Linux CLI Text Processing bundle — $10 (normal price $20), grep, sed, awk, perl and ruby one-liners, GNU coreutils, CLI computing
Learn by example Python bundle — $8 (normal price $15), Python introduction, Regular Expressions and Projects
Understanding Python re(gex)? — FREE (normal price $10)

Indie creators🔗

7 Python books bundle — 50% off
Python books — 35% off with BF24 discount code
Ebooks on Django and Git — 50% off, plus purchasing power parity if applicable
- see also author's blog post for links to other Django-related deals
The Python Coding Place Membership — 40% off with black2024 discount code
Python Jumpstart — 50% launch discount
- see also author's blog post for links to other Python deals
Wizard Zines — 50% off with WIZARDPDF discount code
CSS Flex and Grid + Level up with Tailwind CSS — 60% off
Everyday Rails Testing with RSpec — 53% off

Other deals🔗

The Pragmatic Bookshelf — 40% off on all ebooks and audio books
Leanpub Black Friday Sale — offers for programming books, bundles and courses
Huge list of awesome deals — tools, productivity, books, courses, etc
InfoSec Hack Friday — InfoSec related software/tools
blackfridaydeals.dev — Hottest Black Friday Deals for Developers

Happy learning :)

AI: The Somnium Files - nirvanA Initiative review

ブラック — Thu, 28 Nov 2024 02:00:00 GMT

Back agAIn.

The Somnium Files are back except this time instead of running around as Date Kaname, you’re now given control of two different protagonists, who split the game’s plot into two halves: the past and the present. The past/present split gives the game a more linear presentation than the first game’s non-linear, wilder branching. Otherwise the gist is the same: run around like crazy and investigate a complex and bizarre murder mystery. Tone of the game is also the same: can take itself seriously but trends towards wackier, light-hearted comedy whenever it can.

Narratively nirvanA Initiative is good. It’s not as good as the original Somnium Files, but it’s good and enjoyable. If you liked the first story, you'll probably like this one too. Granted, some of the elements like the mastermind is weaker and the game is definitely more abstract, but I wouldn’t say that any part of the game ruins the journey. I also give it credit for not just retreading the track that the original game took, so it doesn’t feel like you’re just doing a replay. The characters in Nirvana Initiative, both returning and new, are still very strong and definitely made me want to keep playing through. While I don’t think that either of the new protagonists is as fun and full of a package as Date was, I still liked both of them.

Some bits of the story definitely feel like retcons, but I found the explanations given to make sense in-universe, so it didn’t really bother me. Not that any of them were so major that they’d be worth getting upset over, even if they left them completely unexplained. I also think that the additional backstory that they gave to some of the characters really rounded up some of the mysteries of the first game, so I’m a-okay with them.

There are definitely flaws in the storytelling though. Some bits were seemingly brought up to set up another scene without really ever being explained later on. Uchikoshi could’ve definitely done an additional pass through the story to make sure he didn’t introduce something only to drop it in the second scene. There were also some red herrings that seemed to not matter in any way, and felt like they only existed to fuck with you without any explanation why they were there in the first place. Could be that there was some information to explain away these red herrings, but with the game being almost 30 hours long and clues being spread quite thinly throughout that duration, it’s not easy to ascertain that there wasn’t a throwaway line somewhere that explained it. And even if there were, I’d still consider it a demerit that it wasn’t brought up in a bigger way when they were showcased as an indication of some kind of a conspiracy.

Then there’s the big twist. I wasn’t sure how to feel about the big twist at first, since it initially really felt like an asspull. However, given a bit of time and letting the game explain itself, it did start to mesh together to a point where I appreciated it more. There were also at least some hinting toward it, although I think there could’ve been more (unless I was just stupid and missed all of it), as the indication of the twist really just manifested as a sensation of something being really wrong. I imagine that basically no one managed to guess it correctly during the story. The big twist is also quite different from the big twist of the first game, so it doesn’t feel like just a repeat of the first game. So different that it’s probably why a lot of people seem to be turned away from the game entirely. You either like it or you hate it, whereas I think almost everyone liked the original game’s big twist.

The gameplay is almost the same as with the previous instalment, with most of the time being spent in the visual novel investigatory sections with Somniums in between. The biggest core change is that they’ve replaced the evidence-showing interrogation sections with virtual reality puzzle-solving sections, where you explore the scene of the crime in a 3D space and create a reconstruction of the events. Sorta like a mini-Somnium with a pop quiz at the end. I quite liked these VR sections, but I would’ve liked to have some of the evidence sections of the previous game too. Would’ve made it feel a bit more like you were doing proper detective work.

Quick-time events are still very much present, much to many people’s chagrin I imagine. They shouldn’t take anyone by surprise at this point, given the amount of Pavlovian conditioning the series has done by playing the same battle music every time you’re expected to hit random buttons in rhythm. I don’t exactly hate the QTE sections, but they’re hardly the best and brightest gameplay available. At best they're a change of pace from the point-and-clicking adventuring. At least the battle choreography for these sections is fun to watch.

While not drastically different, the Somnium sections seem like a step up from the first game. Not only do the actions feel less like taking stabs in the dark, but there’s also more actual puzzle-solving. Less of “I guessed it right” and more of “I solved the riddle.” Granted, there is still some amount of brute-forcing that you can do and I managed to send one Somnium by guessing the final answer, as my guess was just the correct size to fit the answer box. Still, the amount of variance in the Somnium gameplay mechanics really does make them feel more thought out than previously.

However, I’m not sure if there really was a need for so many Somnium segments, since a few of them didn’t really move the plot forwards all that much. Maybe they just wanted to break up the visual novel sections a bit more?

On the technical side, nirvanA Initiative is not much different from the first game. I could’ve done with better anti-aliasing and improved support for using a controller and keyboard simultaneously. The Somnium transition video is also compressed to hell and back. Surely there was no need to squeeze down the game’s install size this much? Thankfully this time they managed to make the ending movie play in the correct language, so there’s been at least some progress. A passing grade.

Localisation side is good but not perfect. I played through with the Japanese audio track (as one should) and some of the translated jokes really did not land as well in English as they did in the original script. I understand that some of the puns are definitely somewhere between hard and impossible to get right, but there were definitely places where there was a way to deliver the joke better. Some of it might’ve been due to trying to gear the game towards western audiences, with the expectation that the average player wouldn’t really get any references.

All in all, while not quite as brilliant as the first game, nirvanA Initiative is still a fun and interesting sequel, and one that I would recommend to anyone who liked the first game. If you didn't like it, you won't like this one either. And if you haven't played the original? Go play it instead. It’s a sequel, what’d you expect?

If they ever make a third game, aI’ll be sure to cobble up that one as well. A day one purchase, most likely.

1 million page views

Notes on software development — Thu, 28 Nov 2024 02:00:00 GMT

I was delighted to notice this morning that this site has recently passed 1M page views. And since Murat wrote about his 1M page view accomplishment at the time, I felt compelled to now too.

I started regularly blogging in 2018. For some reason I decided to write a blog post every month. And while I have definitely skipped a month or two here or there, on average I've written 2 posts per month.

Tooling

Since at least 2018 this site has been built with a static site generator. I might have used a 3rd-party generator at one point, but for as long as I can remember most of this site has been built with a little Python script I wrote.

I used to get so pissed when static site generators would pointlessly change their APIs and I'd have to make pointless changes. I have not had to make any significant changes to my build code in many years.

I hosted the site itself on GitHub Pages for many years. But I wanted more flexibility with subdomains (ultimately not something I liked) and the ability to view server-side logs (ultimately not something I ever do).

I think this site is hosted on an OVH machine now. But at this point it is inertia keeping me there. If you have no strong feelings otherwise, GitHub Pages is perfect.

I used to use Google Analytics but then they shut down the old version. The new version was incredibly confusing to use. I could not find some very basic information. So I moved to Fathom which has been great.

I used to track all subscribers in a Google Form and bcc them but this became untenable eventually after 1000 subscribers due to GMail rate limits. I currently use MailerLite for subscriptions and sending email about new posts. But this is an absolutely terrible service. They proxy all links behind a domain that adblockers hate and they also visually shorten the URL so you can't copy the text of the URL.

I just want a service that has a hosted form for collecting subscribers and a <textarea> that lets me dump raw HTML and send that as an email to my subscribers. No branding, no watermarks, no link proxying. This apparently doesn't exist. I am too lazy to figure out Amazon SES so I stick with MailerLite for now.

Evolution

In the beginning I talked about little interpreters in JavaScript, about programming languages, about Scheme. I was into functional programming. Over time I moved into little emulators and bytecode VMs. And for the last four years I became obsessed with databases and distributed systems.

I have almost always written about little projects to teach myself a concept. Writing a bytecode VM in Rust, emulating a subset of x86 in Go, implementing Raft in Go, implementing MVCC isolation levels in Go, and so on.

So many times when I tried to learn a concept I would find blog posts with only partial code. The post would link to a GitHub repo that, by the time I got to the post, had evolved significantly beyond what was described in the post. The repo code had by then become too complex for me to follow. So I was motivated to write minimal implementations and walk through the code in its entirety.

Even today there is not a single post on implementing TCP/IP from scratch that walks through entirely working code. (Please, someone write this.)

I have also had a blast writing survey posts such as how various databases execute expressions, analyzing non-V8 JavaScript implementations, how various programming language implementations parse code, and how various database systems build on top of key-value databases.

The last two posts have even each been cited in a research paper (here and here).

Editing

In terms of quality, my single greatest trick is to read the post out loud. Multiple times. Notice parts that are awkward or unclear and rewrite them.

My second greatest trick is to ask friends for review. Some posts like an intuition for distributed consensus and a write-ahead log is not a universal part of durability would simply not have been correct or credible without my fantastic reviewers. And I'm proud to have played that part a few times in turn.

We also have a fantastic #writing-and-drafts channel on the Software Internals Discord where folks (myself occasionally included) come for post review.

Context

I've lost count of the total number of times that these posts have been on the front page of Hacker News or that a tweet announcing a post has reached triple digits likes. I think I've had 9 posts on the front of HN this year. I do know that my single best year for HN was 12 months between 2022-2023 where 20 of my posts or projects were on the front page.

Every time a post does well there's a part of me that worries that I've peaked. But the way to deal with this has been to ignore that little voice and to just keep learning new things. I haven't stopped finding things confusing yet, and confusion is a phenomenal muse.

And also to, like, go out and meet friends for dinner, run meetups, run book clubs, chat with you fascinating internet strangers, play volleyball, and so on.

It's always been about cultivating healthy obsessions.

Benediction

In parting, I'll remind you:

It is definitely worth writing about, whatever "it" is
You're not writing enough
And some ideas for posts I want to hear about if you write about them

I wrote a little reflection on writing after noticing I passed 1M page views this morning.https://t.co/eIlMDVHNht pic.twitter.com/EKSiiDUz5G
— Phil Eaton (@eatonphil) November 28, 2024

Engineering AI Agents - OpenAI DevDay Singapore talk

swyx's site RSS Feed — Wed, 20 Nov 2024 04:06:58 GMT

this is the tracking doc for my talk on "Engineering AI Agents" for OpenAI DevDay Singapore. I'll add photos and notes and stuff when i'm done. note that the full slides contain a lot more info that i had to cut out of the 10 9 minute talk.

Bloom Filters and SQLite

Ben E. C. Boyter — Wed, 20 Nov 2024 02:00:00 GMT

I was talking to someone recently about how searchcode.com works with regards to its bloom filter. The question came up about why it stores the index in memory and not on disk considering how fast disks are these days.

Now the answer is firstly performance, but the second is that writing all the code to store the index on disk is a lot of work and I didn’t feel like working on it. However I was curious… since I have been using SQLite a lot recently, how about storing the bloom filter on disk using that? It comes with its own indexes for stepping though the filters and is likely out of the box to be better than anything I could come up with quickly.

Bill 212

Pxtl.ca — Wed, 20 Nov 2024 02:00:00 GMT

So in a fit of pettiness, the Government of Ontario is set to wrap up bike-lanes in a 50-foot-ball of red tape to prevent their construction. I asked on Bluesky what was the best way to complain - if there was a petition I should sign or something. Craig Burley recommended I send a fax of all things. It's got a good chance of being physical media in their space, and won't just be in a slushpile of emails they ignore, or forgettable names on a petition. So I wrote up the letters for the Premier and the Minister of Transportation at lunch on Tuesday and faxed them from my local UPS store. Also emailed them for good measure.

They're not my best writing but I was kind of in a rush.

The bodies of the letters are attached below.

Mirror: An LLM-powered programming-by-example programming language

Austin Z. Henley's Blog — Sun, 17 Nov 2024 20:00:01 GMT

https://austinhenley.com/blog/mirrorlang.html

My 2024 New Mac Setup

swyx's site RSS Feed — Sat, 16 Nov 2024 19:43:47 GMT

I set up a new Mac for work today. Here's everything I use on a Mac for fullstack web development.

Interactive Python Exercises and Quiz

learnbyexample — Thu, 14 Nov 2024 02:00:00 GMT

Having an interactive program that automatically loads questions and checks the solution is wonderful to have while learning a topic. This TUI app has beginner to intermediate level exercises and multiple-choice questions for Python learners.

Installation🔗

This app is available on PyPI as pythonexercises. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install pythonexercises

# launch the app
$ pythonexercises

If you are on Windows, using the Windows Terminal is recommended. See this issue for Virtual Environment commands and other details.

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias pythonexercises='/path/to/textual_apps/bin/pythonexercises'

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install pythonexercises
$ pythonexercises

As yet another alternative, you can install textual (see Textual documentation for more details), clone this repository and run the python_exercises.py file. You'll need to install textual[syntax] to enable syntax highlighting (see documentation for more details).

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines). Here's another screenshot:

Guide🔗

See app_guide.md for instructions.

Ebook🔗

The exercise and quiz questions in this app have been adapted from my 100 Page Python Intro ebook.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you find this TUI app useful. Happy learning :)

Getting comfortable with psql

CRAIG KERSTIENS — Mon, 11 Nov 2024 17:57:56 GMT

psql is a CLI editor that ships with Postgres. It’s incredibly powerful for working with Postgres, and doesn’t take too much of a learning curve to start to get comfortable so you can really feel like an expert working with your database.

Just a rundown of a few things to get you started:

Once connected in psql you can get an idea of all utility commands available with:

\?

A handy thing I use all the time is \d.

\d will describe the tables within database. You can also add a table/index/etc. onto it to describe that specific table such as:

\d accounts

There are a number of options you can set in your psqlrc (config) file to customize your CLI experience. But you can also toggle those when directly working in psql.

\timing will give you the time it took to run your query
\x auto will autoformat your text output
\pset pager 0 turns off your pager or 1 to turn it back on

Oh and for editing a query in your editor of choice. Make sure you set your $EDITOR enviroment variable to the editor of your choice, though the only right choice is vim:

\e

Just a few things to get you started working with psql.

Your browser does not support video

Exploring Postgres's arena allocator by writing an HTTP server from scratch

Notes on software development — Wed, 06 Nov 2024 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Notes on binary soup

Weblog on marginalia.nu — Tue, 05 Nov 2024 02:00:00 GMT

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before.

This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

Effective unemployment and social media

Notes on software development — Tue, 05 Nov 2024 02:00:00 GMT

Being unemployed can be incredibly depressing. So much rejection. Everything seems to be out of your control. Everything except for one thing: what you produce.

You might know that repeatedly posting on social media that you are looking for work is ineffective. That it looks (or at least feels) worse each time you say so. But there is at least one major caveat to this.

Every single time you create something and share it publicly is a chance to also reiterate that you are looking for work. And people actually appreciate and value this!

Whether you write a blog post or build some project, you are seen as working on yourself and contributing to the community. Positive things! And it is no problem at all to learn with each new post you write and each new project you publish that you are also looking for work.

Moreover, dynamics of the internet and social media basically require that you be regularly producing something new. Either regularly producing a new version of some existing project or regularly producing new projects (or blog posts) entirely.

What you did a week ago is old news on social media. What will you do next week?

This could itself feel depressing except for that it's probably actually a fairly healthy thing for yourself anyway! It is a motivation to keep your skills sharp as time goes on.

So while you're unemployed and able to muster the motivation, write about things that are interesting to you! Build projects that intrigue you. Leave a little note on every post and project that you are looking for work. And share every post and project on social media.

You'll expose yourself to opportunities and referrals. And even if no post or project "takes off" you will still be working on yourself and contributing back knowledge to the community.

I wrote a short post on some ideas for effective unemployment and social media.https://t.co/jmiJCOe2Nk pic.twitter.com/pK9AySNdHR
— Phil Eaton (@eatonphil) November 5, 2024

Moving in Big Cities: Lugg vs Clutter

swyx's site RSS Feed — Fri, 01 Nov 2024 11:12:31 GMT

> 2026 update: used Lugg to move a sectional sofabed couch and ended up regretting it - it cost $222.51 one way (44min labor pickup 55min labor dropoff, which i think was fraud, no way dropoff cost more than pickup) and total base fare 38 + 14.5 fee. for the two way move price i could have simply ordered a new couch and had a nice new couch.

Checking linearizability in Go

Notes on software development — Thu, 31 Oct 2024 02:00:00 GMT

You want to check for strict consistency (linearizability) for your project but you don't want to have to deal with the JVM. Porcupine, used by a number of real-world systems like etcd and TiDB, has you covered!

Importantly, neither Jepsen projects nor Porcupine can prove linearizability. They can only help you build confidence that you aren't obviously violating linearizability.

The Porcupine README is pretty good but doesn't give complete working code, so I'm going to walk through checking linearizability of a distributed register. And then we'll tweak things a bit by checking linearizability for a distributed key-value store.

But rather than implementing a distributed register and implementing a distributed key-value store, to keep this post concise, we're just going to imagine that they exist and we'll come up with some example histories we might see.

Code for this post can be found on GitHub.

Boilerplate

Create a new directory and go mod init lintest. Let's add the imports we need and a helper function for generating a visualization of a history, in main.go:

package main

import "os"
import "log"
import "github.com/anishathalye/porcupine"

func visualizeTempFile(model porcupine.Model, info porcupine.LinearizationInfo) {
    file, err := os.CreateTemp("", "*.html")
    if err != nil {
        panic("failed to create temp file")
    }
    err = porcupine.Visualize(model, info, file)
    if err != nil {
        panic("visualization failed")
    }
    log.Printf("wrote visualization to %s", file.Name())
}

A distributed register

A distributed register is like a distributed key-value store but there's only a single key.

We need to tell Porcupine what the inputs and outputs for this system are. And we'll later describe for it how an idealized version of this system should behave as it receives each input; what output the idealized version should produce.

Each time we send a command to the distributed register it will include an operation (to get or to set the register). And if it is a set command it will include a value.

type registerInput struct {
    operation string // "get" and "set"
    value int
}

The register is an integer register.

Now we will define a model for Porcupine which, again, is the idealized version of this system.

func main() {
    registerModel := porcupine.Model{
        Init: func() any {
            return 0
        },
        Step: func(stateAny, inputAny, outputAny any) (bool, any) {
            input := inputAny.(registerInput)
            output := outputAny.(int)
            state := stateAny.(int)
            if input.operation == "set" {
                return true, input.value
            } else if input.operation == "get" {
                readCorrectValue := output == state
                return readCorrectValue, state
            }

            panic("Unexpected operation")
        },
    }

The step function accepts anything because it has to be able to model any sort of system with its different inputs and outputs and current state. So we have to handle casting from the any type to what we know are the inputs and outputs and state. And finally we actually do the state change and return the new state as well as if the given output matches what we know it should be.

An invalid history

Now we've only defined the idealized version of this system. Let's pretend we have some real-world implementation of this. We might have two clients and they might issue concurrent get and set requests.

Every time we stimulate the system we will generate a new history that we can validate with Porcupine against our model to see if the history is linearizable.

Let's imagine these two clients concurrently set the register to some value. Both sets succeed. Then both clients read the register. And they get different values. Here's what that history would look like modeled for Porcupine.

    ops := []porcupine.Operation{
        // Client 3 sets the register to 100. The request starts at t0 and ends at t2.
        {3, registerInput{"set", 100}, 0, 100 /* end state at t2 is 100 */, 2},
        // Client 5 sets the register to 200. The request starts at t3 and ends at t4.
        {5, registerInput{"set", 200}, 3, 200/* end state at t3 is 200 */, 4},
        // Client 3 reads the register. The request starts at t5 and ends at t6.
        {3, registerInput{"get", 0 /* doesn't matter */ }, 5, 200, 6},
        // Client 5 reads the register. The request starts at t7 and ends at t8. Reads a stale value!
        {5, registerInput{"get", 0 /* doesn't matter */}, 7, 100, 8},
    }
    res, info := porcupine.CheckOperationsVerbose(registerModel, ops, 0)
    visualizeTempFile(registerModel, info)

    if res != porcupine.Ok {
        panic("expected operations to be linearizable")
    }
}

If we build and run this code:

$ go mod tidy
go: finding module for package github.com/anishathalye/porcupine
go: found github.com/anishathalye/porcupine in github.com/anishathalye/porcupine v0.1.6
$ go build
$ ./lintest
2024/10/31 19:54:08 wrote visualization to /var/folders/cb/v27m749d0sj89h9ydfq0f0940000gn/T/463308000.html
panic: expected operations to be linearizable

goroutine 1 [running]:
main.main()
        /Users/phil/tmp/lintest/main.go:59 +0x394

Porcupine caught the stale value. Open that HTML file to see the visualization.

A valid history

Let's say we fix the bug so now there's no stale read. The new history would look like this:

    ops := []porcupine.Operation{
        // Client 3 sets the register to 100. The request starts at t0 and ends at t2.
        {3, registerInput{"set", 100}, 0, 100 /* end state at t2 is 100 */, 2},
        // Client 5 sets the register to 200. The request starts at t3 and ends at t4.
        {5, registerInput{"set", 200}, 3, 200/* end state at t3 is 200 */, 4},
        // Client 3 reads the register. The request starts at t5 and ends at t6.
        {3, registerInput{"get", 0 /* doesn't matter */ }, 5, 200, 6},
        // Client 5 reads the register. The request starts at t7 and ends at t8.
        {5, registerInput{"get", 0 /* doesn't matter */}, 7, 200, 8},
    }

Rebuild, rerun lintest (it should exit successfully now), and open the visualization.

Great! Now let's make things a little more complicated by modeling a distributed key-value store rather than a distributed register.

Distributed key-value

The inputs of this system will be slightly more complex. They will take a key along with the operation and value.

type kvInput struct {
    operation string // "get" and "set"
    key string
    value int
}

And when we model the distributed key-value store with the state and output at each step being a map[string]int.

    kvModel := porcupine.Model{
        Init: func() any {
            return map[string]int{}
        },
        Step: func(stateAny, inputAny, outputAny any) (bool, any) {
            input := inputAny.(kvInput)
            output := outputAny.(map[string]int)
            state := stateAny.(map[string]int)
            if input.operation == "set" {
                newState := map[string]int{}
                for k, v := range state {
                    newState[k] = v
                }
                newState[input.key] = input.value
                return true, newState
            } else if input.operation == "get" {
                readCorrectValue := output[input.key] == state[input.key]
                return readCorrectValue, state
            }

            panic("Unexpected operation")
        },
    }

And now the history gets slightly more complex because we are now working with some specific key. But we'll otherwise use the same history as before.

    ops := []porcupine.Operation{
        // Client 3 set key `a` to 100. The request starts at t0 and ends at t2.
        {3, kvInput{"set", "a", 100}, 0, map[string]int{"a": 100}, 2},
        // Client 5 set key `a` to 200. The request starts at t3 and ends at t4.
        {5, kvInput{"set", "a", 200}, 3, map[string]int{"a": 200}, 4},
        // Client 3 read key `a`. The request starts at t5 and ends at t6.
        {3, kvInput{"get", "a", 0 /* doesn't matter */ }, 5, map[string]int{"a": 200}, 6},
        // Client 5 read key `a`. The request starts at t7 and ends at t8.
        {5, kvInput{"get", "a", 0 /* doesn't matter */}, 7, map[string]int{"a": 200}, 8},
    }

Build and run. Open the visualization.

And there we go!

What's next

These are just a few simple examples that are not hooked up to a real system. But it still seemed useful to show how you model one or two simple different systems and check a history with Porcupine.

Another aspect of Porcupine I did not cover is partitioning the state space. The docs say:

Implementing the partition functions can greatly improve performance. If you're implementing the partition function, the model Init and Step functions can be per-partition. For example, if your specification is for a key-value store and you partition by key, then the per-partition state representation can just be a single value rather than a map.

Perhaps that, and hooking this up to some "real" system, would be a good next step.

I wrote a short tutorial on using Porcupine to check for linearizability (without needing to deal with the JVM).https://t.co/kqeBz2jX76 pic.twitter.com/teXvlp2zcv
— Phil Eaton (@eatonphil) November 1, 2024

My Art And Color-After Tiling

The Codist — Tue, 29 Oct 2024 21:39:02 GMT

I make generative art with Swift and use tiling in many pieces. Truchet tiles are generally arranged randomly and contain everything appearing in the final image. What I do differently is to separate the layout of tiles from colorizing the image. I call this technique "Color-After Tiling."

For

The Unreasonable Effectiveness of Monte Carlo Simulations in A/B Testing

Bytepawn - Marton Trencseni — Sun, 20 Oct 2024 01:00:00 GMT

The article illustrates how Monte Carlo simulations serve as a powerful method for exploring statistical concepts in A/B testing, enhancing understanding, improving experimental design and analysis.

How I Defeated An MMO Game Hack Author

The Codist — Sat, 19 Oct 2024 20:25:59 GMT

In the late 2000's, I worked at a niche MMO game company. We had a small team, not a lot of money, but a loyal audience. It was a game of skill without any of the usual powerups and unreality, and the players enjoyed the challenge.

Then, one

The future of Postgres?

CRAIG KERSTIENS — Fri, 18 Oct 2024 19:50:56 GMT

I’m often asked what do I think the future for Postgres holds, and my answer has been mostly the same for probably 8 years now, maybe even longer. You see for Postgres itself stability and reliability is core. So where does the new stuff come from if it’s not in the stable core… extensions.

Extensions within Postgres are unlike most other databases allowing you to modify or well extend the standard Postgres behavior. You can build other storage backends, new types, etc. Postgres itself ships with a number of extensions within the “contrib”. The list of contrib extensions hasn’t changed any time recently, but even contrib is a small sampling of what is possible. Beyond core there is a whole world of extensions, I want dig into just a smalls sampling starting with a few in core…

pg_stat_statements is to me the most useful extension that exists. It records what queries were run, how long they took, and a number of other details about the queries. A key extension for managing performance of your database.

auto_explain another one in contrib that is helpful for performance. For queries that run over a certain period of time will automatically log the explain plan–helpful for performance debugging.

pg_prewarm useful to prewarming the cache ahead of a failover.

Let’s jump out of contrib a little bit now. Of course that isn’t everything that ships with Postgres there is more, explore for yourself.

Citus is one of the (to date) more advanced extensions ever created. Citus turns postgres into a sharded, distributed, horizontally scalable database. Citus is especially built to work well for B2B style multi-tenant apps, and now after being acquired years ago part of Microsoft.

Pg_search extends Postgres to support elastic-quality full text search directly within Postgres. I often say Postgres can do just about everything and be pretty capable. Things like time series and search it’s about 80% as good of some of the best in class options out there, but pg_search takes it further making it a full competitor to elastic, but Postgres.

Those are some bigger ones, but you’ve also got a lot of small focused ones as well.

Pg_cron is incredibly handy. Created originally by @marcoslot while at Citus, it’s a small extension that does what it sounds like run scheduled jobs within Postgres. It is now a standard across all the major cloud providers. We leverage it heavily at @crunchydata, but went a little further and built a UI on top so you don’t have to worry about crontab styling of the jobs.

Just yesterday at Crunchy Data we released pg_parquet. There have been some questions about what about other extensions that sort of do similar but not exact but also do a lot of other things. This allows you to seamless copy to and from Parquet files with Postgres. It follows the linux philosophy of small sharp tools. Marco described the why extremely well:

We wanted to create a light-weight implementation of Parquet that does not pull a multi-threaded library into every postgres process. When you get to more complex features, a lot of questions around trade-offs, user experience, and deployment model start appearing.

There is a whole lot of extensions out there. There are big robust ones that have been around forever (yeah I’m talking about you PostGIS), there are ones in contrib, there are ones that are literally created as jokes. Some try to do a lot others as I mentioned very narrow focus like pg_cron. Regardless of whether you believe in large complex extensions that do a lot or small ones that do one thing but do it really well, if you want to know where the future of Postgres is I can’t imagine a world where extensions do not play a heavy roll.

Generating Diagrams with with AI / LLMs

smcleod.net — Tue, 08 Oct 2024 04:00:10 GMT

Generating diagrams with AI / LLMs

Intuitions for Distributed Consensus by Phil Eaton

Notes on software development — Tue, 08 Oct 2024 03:00:00 GMT

This is an external talk of mine. Click here if you are not redirected.

I Finally Bought My Dream Airplane

MiscDotGeek — Mon, 07 Oct 2024 04:52:53 GMT

Aviation has been a love of mine since I was a very little person. Living in Nevada, seeing posters and ads for the Reno Air Races, specifically the Texans, transfixed me. I was hooked. As a teenager, I got an RC glider and dabbled in RC gliders and planes until my early 20’s. Later in …

The post I Finally Bought My Dream Airplane appeared first on MiscDotGeek.

Early Stage Company Offsites

swyx's site RSS Feed — Mon, 07 Oct 2024 02:38:50 GMT

> this post was mostly dictated off the top of my head with Wispr AI

A simple telnet based Taboo game in async Python

Bytepawn - Marton Trencseni — Sun, 06 Oct 2024 01:00:00 GMT

I built a simple Python-based Taboo game server using asyncio to provide a customizable multiplayer experience via Telnet.

Why I Use Swift To Make Generative Art

The Codist — Fri, 04 Oct 2024 18:47:41 GMT

Now that I am retired from programming for a living, I make generative art (not AI; see my post What Is Generative Art?) every day. I belong to a discord community of generative artists, yet I stick out because I am the only person using Swift as my chosen language.

AI Strategy in 2024 for the Rest of Us

Bytepawn - Marton Trencseni — Fri, 04 Oct 2024 01:00:00 GMT

My review of the AI landscape in late 2024, and what the road ahead looks like for those of us not building foundational models.

How Talking Over A Wall Changed My Direction As A Programmer

The Codist — Wed, 02 Oct 2024 21:21:51 GMT

I started my programming career in October 1981 at a large defense contractor (GD). At the time, my goal was to work for a couple of years and then continue my education with a Ph.D. in Chemistry (I had already been accepted).

The office I worked in was a

Ad hoc tools for gathering prompt context

Austin Z. Henley's Blog — Tue, 01 Oct 2024 06:00:01 GMT

https://austinhenley.com/blog/promptcontext.html

Phrase Matching in Marginalia Search

Weblog on marginalia.nu — Mon, 30 Sep 2024 03:00:00 GMT

Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query.

This is a write-up about implementing this change. This is going to be a relatively long post, as it represents about 4 months of work.

I’m also happy and grateful to announce that the nlnet people reached out after the run of the grant was over and asked me if I had more work in the pipe, and agreed to fund this change as well!

Build a serverless ACID database with this one neat trick (atomic PutIfAbsent)

Notes on software development — Sun, 29 Sep 2024 03:00:00 GMT

Delta Lake is an open protocol for serverless ACID databases. Due to its simplicity, scalability, and the number of open-source implementations, it's quickly becoming the DuckDB of serverless transactional databases for analytics workloads. Iceberg is a contender too, and is similar in many ways. But since Delta Lake is simpler (simple != better) that's where we'll focus in this post.

Delta Lake has one of the most accessible database papers I've read (link). It's kind of like the movfuscator of databases.

Thanks to its simplicity, in this post we'll implement a Delta Lake-inspired serverless ACID database in 500 lines of Go code with zero dependencies. It will support creating tables, inserting rows into a table, and scanning all rows in a table. All while allowing concurrent readers and writers and achieving snapshot isolation.

There are other critical parts of Delta Lake we'll ignore: updating rows, deleting rows, checkpointing the transaction metadata log, compaction, and probably much more I'm not aware of. We must start somewhere.

All code for this post is available on GitHub.

Delta Lake basics

Delta Lake writes immutable data files to blob storage. It stores the names of new data files for a transaction in a metadata file. It handles concurrency (i.e. achieves snapshot isolation) with an atomic PutIfAbsent operation on the metadata file for the transaction.

This method of concurrency control works because the metadata files follow a naming scheme that includes the transaction id in the file name. When a new transaction starts, it finds all existing metadata files and picks its own transaction id by adding 1 to the largest transaction id it sees.

When a transaction goes to commit, writing the metadata file will fail if another transaction has already picked the same transaction id.

If a transaction does no writes and creates no tables, the transaction does not attempt to write any metadata file. Snapshot isolation!

Let's dig into the implementation.

Boilerplate

Let's give ourselves some nice assertion methods, a debug method, and a uuid generator. In main.go:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "path"
    "slices"
    "strings"
)

func assert(b bool, msg string) {
    if !b {
        panic(msg)
    }
}

func assertEq[C comparable](a C, b C, prefix string) {
    if a != b {
        panic(fmt.Sprintf("%s '%v' != '%v'", prefix, a, b))
    }
}

var DEBUG = slices.Contains(os.Args, "--debug")

func debug(a ...any) {
    if !DEBUG {
        return
    }

    args := append([]any{"[DEBUG]"}, a...)
    fmt.Println(args...)
}

// https://datatracker.ietf.org/doc/html/rfc4122#section-4.4
func uuidv4() string {
    f, err := os.Open("/dev/random")
    assert(err == nil, fmt.Sprintf("could not open /dev/random: %s", err))
    defer f.Close()

    buf := make([]byte, 16)
    n, err := f.Read(buf)
    assert(err == nil, fmt.Sprintf("could not read 16 bytes from /dev/random: %s", err))
    assert(n == len(buf), "expected 16 bytes from /dev/random")

    // Set bit 6 to 0
    buf[8] &= ^(byte(1) << 6)
    // Set bit 7 to 1
    buf[8] |= 1 << 7

    // Set version
    buf[6] &= ^(byte(1) << 4)
    buf[6] &= ^(byte(1) << 5)
    buf[6] |= 1 << 6
    buf[6] &= ^(byte(1) << 7)

    return fmt.Sprintf("%x-%x-%x-%x-%x",
        buf[:4],
        buf[4:6],
        buf[6:8],
        buf[8:10],
        buf[10:16])
}

Is that uuid method correct? Hopefully. Efficient? No. But it's preferable to avoid dependencies in pedagogical projects.

Moving on.

Blob storage requirements

As mentioned above, the basic requirement is that we support atomically writing some bytes to a location if the location doesn't already exist.

On top of that we also need the ability to list locations by prefix, and the ability to read the bytes at some location.

We'll diverge from Delta Lake in how we name files on disk. For one, we'll keep all files in the same directory with a fixed prefix for metadata and another table name prefix for each data file. This simplifies the implementation of listPrefix a bit.

However, this also diverges from Delta Lake in that transactions will represent all tables. In Delta Lake that is not so. Delta Lake has a per-table transaction log. Only transactions that read and write the same table in Delta Lake achieve snapshot isolation.

So let's set up an interface to describe these requirements:

type objectStorage interface {
    // Must be atomic.
    putIfAbsent(name string, bytes []byte) error
    listPrefix(prefix string) ([]string, error)
    read(name string) ([]byte, error)
}

And this is literally all we need to get ACID transactions. That's crazy!

Atomic Put and cloud blob storage

We could implement the atomic putIfAbsent part of this interface in 2024 using conditional writes on S3. Or we could implement this interface with the If-None-Match header on Azure Cloud Storage. Or we could implement this interface with the x-goog-if-generation-match header on Google Cloud Storage.

Indeed a good exercise for the reader would be to implement this interface for other blob storage providers and see your serverless cloud database in action!

But the simplest method of all is to implement it on the filesystem, which is what we'll do next.

A filesystem blob store

If we had a server we could implement atomic putIfAbsent with a mutex. But we're serverless baby. Thankfully, POSIX supports atomic link which will fail if the new name is already a file.

So we'll just create a temporary file and write out all bytes. Finally, we link the temporary file to the permanent name we intended. For cleanliness (not correctness), if there is an error at any point, we'll remove the temporary file.

type fileObjectStorage struct {
    basedir string
}

func newFileObjectStorage(basedir string) *fileObjectStorage {
    return &fileObjectStorage{basedir}
}

func (fos *fileObjectStorage) putIfAbsent(name string, bytes []byte) error {
    tmpfilename := path.Join(fos.basedir, uuidv4())
    f, err := os.OpenFile(tmpfilename, os.O_WRONLY|os.O_CREATE, 0644)
    if err != nil {
        return err
    }

    written := 0
    bufSize := 1024 * 16
    for written < len(bytes) {
        toWrite := min(written+bufSize, len(bytes))
        n, err := f.Write(bytes[written:toWrite])
        if err != nil {
            removeErr := os.Remove(tmpfilename)
            assert(removeErr == nil, "could not remove")
            return err
        }

        written += n
    }

    err = f.Sync()
    if err != nil {
        removeErr := os.Remove(tmpfilename)
        assert(removeErr == nil, "could not remove")
        return err
    }

    err = f.Close()
    if err != nil {
        removeErr := os.Remove(tmpfilename)
        assert(removeErr == nil, "could not remove")
        return err
    }

    filename := path.Join(fos.basedir, name)
    err = os.Link(tmpfilename, filename)
    if err != nil {
        removeErr := os.Remove(tmpfilename)
        assert(removeErr == nil, "could not remove")
        return err
    }

    return nil
}

yencabulator on HN pointed out that an earlier version of this post had a buggy implementation of putIfAbsent (that attempted to manage atomicity solely via O_EXCL | O_CREAT) would leave around potentially bad metadata files if the os.Remove call ever failed.

The link approach works around that because the file is already fully and correctly written by the time we do the link.

listPrefix and read are minimal wrappers around filesystem APIs:

func (fos *fileObjectStorage) listPrefix(prefix string) ([]string, error) {
    dir := path.Join(fos.basedir)
    f, err := os.Open(dir)
    if err != nil {
        return nil, err
    }

    var files []string
    for err != io.EOF {
        var names []string
        names, err = f.Readdirnames(100)
        if err != nil && err != io.EOF {
            return nil, err
        }

        for _, n := range names {
            if prefix == "" || strings.HasPrefix(n, prefix) {
                files = append(files, n)
            }
        }
    }
    err = f.Close()
    return files, err
}

func (fos *fileObjectStorage) read(name string) ([]byte, error) {
    filename := path.Join(fos.basedir, name)
    return os.ReadFile(filename)
}

It is worth talking a bit about reading a directory though. Go doesn't provide a nice iterator API for us and I didn't want to implement this as callbacks with path/filepath.WalkDir.

We could use os.File.ReadDir but it allocates for all files in the directory. Sure, in a pedagogical project we don't worry about millions of files. But the ReadDir API, the error cases in particular, also isn't much simpler than Readdirnames.

What's more, even though we iterated through batches of directory entries, and did prefix filtering before accumulating, we still could have considered returning an iterator here ourselves. It seems possible and likely that the number of data files grows quite large in a production system. But I was lazy.

It would be nice if Go introduced an actual iterator API for reading a directory. :)

Delta Lake and stale reads

In any case the ACID properties of Delta Lake (and Iceberg) don't depend on being able to read up-to-date data.

This is because concurrent (or stale) transactions that write will fail on commit. And also because all files written (even metadata files) are immutable.

Since all data is immutable, we will always be able to read at least a consistent snapshot of data. But we will never be able to get SERIALIZABLE read-only transactions. This is just how Delta Lake and Iceberg work. And it is a similar or better consistency level to what any major SQL database gives you by default.

You'll see what I mean later on when we implement transaction commits.

Transaction boilerplate

Now that we've got a blob storage abstraction and a filesystem implementation of it, let's start sketching out what a client and what a transaction looks like.

In Delta Lake, a transaction consists of a list of actions. An action might be to define a table's schema, or to add a data file, or to remove a data file, etc. In this post we'll only implement the first two actions.

type DataobjectAction struct {
    Name  string
    Table string
}

type ChangeMetadataAction struct {
    Table   string
    Columns []string
}

// an enum, only one field will be non-nil
type Action struct {
    AddDataobject  *DataobjectAction
    ChangeMetadata *ChangeMetadataAction
    // TODO: Support object removal.
    // DeleteDataobject *DataobjectAction
}

These fields are all exported (i.e. capitalized, if you're not familiar with Go) because we will be writing them to disk when the transaction commits as the transaction's metadata.

In fact Actions and the transaction's id will be the only parts of the transaction we write to disk. Everything else will be in-memory state.

For our convenience we will track in memory a history of all previous actions, a mapping of table columns, and a mapping of unflushed data by table.

type transaction struct {
    Id int

    // Both are mapping table name to a list of actions on the table.
    previousActions map[string][]Action
    Actions         map[string][]Action

    // Mapping tables to column names.
    tables map[string][]string

    // Mapping table name to unflushed/in-memory rows. When rows
    // are flushed, the dataobject that contains them is added to
    // `tx.actions` above and `tx.unflushedDataPointer[table]` is
    // reset to `0`.
    unflushedData        map[string]*[DATAOBJECT_SIZE][]any
    unflushedDataPointer map[string]int
}

Only the current transaction will ever have transaction.previousActions filled out. transaction.tables will be populated when the transaction starts by reading through transaction.previousActions for ChangeMetadataActions, and we will also add onto it when we create a table in the current transaction.

We will append to transaction.Actions every time we write a new data file and every time we create a new table.

We will add rows to transaction.unflushedData for a table until transaction.unflushedDataPointer for that table reaches DATAOBJECT_SIZE upon which time we will write that data to disk and add a DataobjectAction entry to transaction.Actions.

Client boilerplate

A client will consist of an objectStorage implementation and a possibly empty *transaction. Empty meaning there is no current transaction.

type client struct {
    os objectStorage
    // Current transaction, if any. Only one transaction per
    // client at a time. All reads and writes must be within a
    // transaction.
    tx *transaction
}

func newClient(os objectStorage) client {
    return client{os, nil}
}

var (
    errExistingTx  = fmt.Errorf("Existing Transaction")
    errNoTx        = fmt.Errorf("No Transaction")
    errTableExists = fmt.Errorf("Table Exists")
    errNoTable     = fmt.Errorf("No Such Table")
)

Client or database?

In a previous version of my code I named this client struct database. But that's misleading. There is no central database. There is just the client and the blob storage.

Clients work with transactions directly and only when attempting to commit does the blob storage abstraction let the client know if the transaction succeeded or not.

Starting a transaction

When we start a transaction, we will first read all existing transactions from disk and accumulate the actions from each prior transaction.

We will interpret ChangeMetadataActions and materialize them into a current view of all tables.

And we will assign a transaction ID to this transaction to be 1 greater than the largest existing transaction ID we see.

Again it doesn't matter if the listPrefix call we use returns an up-to-date list. Notably on blob storage there are few guarantees about LIST operations recency. The Delta Lake paper mentions this too.

Out-of-date transactions attempting to write will be caught when we go to commit the transaction. Out-of-date transactions attempting only to read will still read a consistent snapshot.

func (d *client) newTx() error {
    if d.tx != nil {
        return errExistingTx
    }

    logPrefix := "_log_"
    txLogFilenames, err := d.os.listPrefix(logPrefix)
    if err != nil {
        return err
    }

    tx := &transaction{}
    tx.previousActions = map[string][]Action{}
    tx.Actions = map[string][]Action{}
    tx.tables = map[string][]string{}
    tx.unflushedData = map[string]*[DATAOBJECT_SIZE][]any{}
    tx.unflushedDataPointer = map[string]int{}

    for _, txLogFilename := range txLogFilenames {
        bytes, err := d.os.read(txLogFilename)
        if err != nil {
            return err
        }

        var oldTx transaction
        err = json.Unmarshal(bytes, &oldTx)
        if err != nil {
            return err
        }
        // Transaction metadata files are sorted
        // lexicographically so that the most recent
        // transaction (i.e. the one with the largest
        // transaction id) will be last and tx.Id will end up
        // 1 greater than the most recent transaction ID we
        // see on disk.
        tx.Id = oldTx.Id + 1

        for table, actions := range oldTx.Actions {
            for _, action := range actions {
                if action.AddDataobject != nil {
                    tx.previousActions[table] = append(tx.previousActions[table], action)
                } else if action.ChangeMetadata != nil {
                    // Store the latest version of
                    // each table in memory for
                    // easy lookup.
                    mtd := action.ChangeMetadata
                    tx.tables[table] = mtd.Columns
                } else {
                    panic(fmt.Sprintf("unsupported action: %v", action))
                }
            }
        }
    }

    d.tx = tx
    return nil
}

And we're set.

Creating a table

When we create a table, we need to add a ChangeMetadataAction to the transactions Actions. And we also want to add the table info to the in-memory transaction.tables field.

We don't do any of this durably. The change here will be written to disk on commit (if the transaction succeeds).

func (d *client) createTable(table string, columns []string) error {
    if d.tx == nil {
        return errNoTx
    }

    if _, exists := d.tx.tables[table]; exists {
        return errTableExists
    }

    // Store it in the in-memory mapping.
    d.tx.tables[table] = columns

    // And also add it to the action history for future transactions.
    d.tx.Actions[table] = append(d.tx.Actions[table], Action{
        ChangeMetadata: &ChangeMetadataAction{
            Table:   table,
            Columns: columns,
        },
    })

    return nil
}

Easy peasy. Now for the fun part, writing data!

Writing a row

This is the next area where we'll diverge from Delta Lake. For the sake of zero dependencies we are going to store data in-memory as an array of array of any. And when we later write rows to disk we'll write them as JSON. A real Delta Lake implementation would store data in-memory in Apache Arrow format, and write to disk as Parquet.

In line with Delta Lake though we will buffer data in memory until we get 64K rows. When we get 64K rows for a particular table we will flush all those rows to disk. (When we go to commit a transaction we will flush any outstanding rows.)

func (d *client) writeRow(table string, row []any) error {
    if d.tx == nil {
        return errNoTx
    }

    if _, ok := d.tx.tables[table]; !ok {
        return errNoTable
    }

    // Try to find an unflushed/in-memory dataobject for this table
    pointer, ok := d.tx.unflushedDataPointer[table]
    if !ok {
        d.tx.unflushedDataPointer[table] = 0
        d.tx.unflushedData[table] = &[DATAOBJECT_SIZE][]any{}
    }

    if pointer == DATAOBJECT_SIZE {
        d.flushRows(table)
        pointer = 0
    }

    d.tx.unflushedData[table][pointer] = row
    d.tx.unflushedDataPointer[table]++
    return nil
}

Now let's implement flushing.

Flushing a data object

Recall that data objects in Delta Lake (and Iceberg) are immutable. Once we've got enough data to write a data object, we give it a unique name, write it to disk, and add a AddObjectAction to the transaction's list of Actions.

type dataobject struct {
    Table string
    Name  string
    Data  [DATAOBJECT_SIZE][]any
    Len   int
}

func (d *client) flushRows(table string) error {
    if d.tx == nil {
        return errNoTx
    }

    // First write out dataobject if there is anything to write out.
    pointer, exists := d.tx.unflushedDataPointer[table]
    if !exists || pointer == 0 {
        return nil
    }

    df := dataobject{
        Table: table,
        Name:  uuidv4(),
        Data:  *d.tx.unflushedData[table],
        Len:   pointer,
    }
    bytes, err := json.Marshal(df)
    if err != nil {
        return err
    }

    err = d.os.putIfAbsent(fmt.Sprintf("_table_%s_%s", table, df.Name), bytes)
    if err != nil {
        return err
    }

    // Then record the newly written data file.
    d.tx.Actions[table] = append(d.tx.Actions[table], Action{
        AddDataobject: &DataobjectAction{
            Table: table,
            Name:  df.Name,
        },
    })

    // Reset in-memory pointer.
    d.tx.unflushedDataPointer[table] = 0
    return nil
}

That's it for writing data! Let's now look at reading data.

Scanning a table

We're going to make scanning mildly more complicated than it needed to be in pedagogical code because we'll have client.scan() return an iterator rather than an array with all rows.

The scanIterator will first read from in-memory (unflushed) data. And then it will read through every data object for the table that is still a part of this transaction. We will know which data objects are still a part of this transaction by reading through all AddDataobject actions. A future version of this project would also eliminate data object files from the list by observing DeleteDataobject actions. But we don't do that in this post.

func (d *client) scan(table string) (*scanIterator, error) {
    if d.tx == nil {
        return nil, errNoTx
    }

    var dataobjects []string
    allActions := append(d.tx.previousActions[table], d.tx.Actions[table]...)
    for _, action := range allActions {
        if action.AddDataobject != nil {
            dataobjects = append(dataobjects, action.AddDataobject.Name)
        }
    }

    var unflushedRows [DATAOBJECT_SIZE][]any
    if data, ok := d.tx.unflushedData[table]; ok {
        unflushedRows = *data
    }

    return &scanIterator{
        unflushedRows:    unflushedRows,
        unflushedRowsLen: d.tx.unflushedDataPointer[table],
        d:                d,
        table:            table,
        dataobjects:      dataobjects,
    }, nil
}

The scanIterator needs to track where we are in in-memory rows, in data objects, and within a particular data object.

type scanIterator struct {
    d     *client
    table string

    // First we iterate through unflushed rows.
    unflushedRows       [DATAOBJECT_SIZE][]any
    unflushedRowsLen    int
    unflushedRowPointer int

    // Then we move through each dataobject.
    dataobjects        []string
    dataobjectsPointer int

    // And within each dataobject we iterate through rows.
    dataobject           *dataobject
    dataobjectRowPointer int
}

And the scanIterator will be driven by a next() method that goes through in-memory data first and then through what's on disk.

func (d *client) readDataobject(table, name string) (*dataobject, error) {
    bytes, err := d.os.read(fmt.Sprintf("_table_%s_%s", table, name))
    if err != nil {
        return nil, err
    }

    var do dataobject
    err = json.Unmarshal(bytes, &do)
    return &do, err
}

// returns (nil, nil) when done
func (si *scanIterator) next() ([]any, error) {
    // Iterate through in-memory rows first.
    if si.unflushedRowPointer < si.unflushedRowsLen {
        row := si.unflushedRows[si.unflushedRowPointer]
        si.unflushedRowPointer++
        return row, nil
    }

    // If we've gotten through all dataobjects on disk we're done.
    if si.dataobjectsPointer == len(si.dataobjects) {
        return nil, nil
    }

    if si.dataobject == nil {
        name := si.dataobjects[si.dataobjectsPointer]
        o, err := si.d.readDataobject(si.table, name)
        if err != nil {
            return nil, err
        }

        si.dataobject = o
    }

    if si.dataobjectRowPointer > si.dataobject.Len {
        si.dataobjectsPointer++
        si.dataobject = nil
        si.dataobjectRowPointer = 0
        return si.next()
    }

    row := si.dataobject.Data[si.dataobjectRowPointer]
    si.dataobjectRowPointer++
    return row, nil
}

That's it for scanning a table! The final piece of the puzzle is committing a transaction.

Committing a transaction

When we commit a transaction we must flush any remaining data. A read-only transaction (one which has no Actions) is immediately done. There is no concurrency check.

Otherwise we will serialize transaction state and attempt to atomically putIfAbsent.

The only way this will fail is if there is another concurrent writer.

func (d *client) commitTx() error {
    if d.tx == nil {
        return errNoTx
    }

    // Flush any outstanding data
    for table := range d.tx.tables {
        err := d.flushRows(table)
        if err != nil {
            d.tx = nil
            return err
        }
    }

    wrote := false
    for _, actions := range d.tx.Actions {
        if len(actions) > 0 {
            wrote = true
            break
        }
    }
    // Read-only transaction, no need to do a concurrency check.
    if !wrote {
        d.tx = nil
        return nil
    }

    filename := fmt.Sprintf("_log_%020d", d.tx.Id)
    // We won't store previous actions, they will be recovered on
    // new transactions. So unset them. Honestly not totally
    // clear why.
    d.tx.previousActions = nil
    bytes, err := json.Marshal(d.tx)
    if err != nil {
        d.tx = nil
        return err
    }

    err = d.os.putIfAbsent(filename, bytes)
    d.tx = nil
    return err
}

func main() {
    panic("unimplemented")
}

This is the crux of Delta Lake. It's simple. And honestly it's a bit shocking. Real Delta Lake does support automatic retries in some cases. But primarily you are limited to a single writer per table, even if the writers are writing non-conflicting rows. Iceberg is basically the same here, it's just how metadata is tracked that differs.

As mentioned in another note above, our implementation is actually stricter than Delta Lake since it manages all table transaction logs together. This means you can get snapshot isolation across all tables (which Delta Lake doesn't support) but it will mean significantly more contention and failed write transactions.

The Delta Lake and Iceberg folks apparently wanted to avoid FoundationDB (i.e. the Snowflake architecture, which is mentioned in the Delta Lake paper) so much that they'd give up row-level concurrency to be mostly serverless.

Is it worth it? Dunno. Delta Lake and Iceberg are getting massive adoption. Many very smart people have worked, and continue to work, on both. Moreover it is apparently what the market wants. Every database-like product is implementing, or is planning to implement, Delta Lake or Iceberg.

Trying it out

Let's add a test in main_test.go to see what happens with concurrent writers. Follow the comments and debug logs for details:

package main

import (
    "os"
    "testing"
)

func TestConcurrentTableWriters(t *testing.T) {
    dir, err := os.MkdirTemp("", "test-database")

    if err != nil {
        panic(err)
    }

    defer os.Remove(dir)

    fos := newFileObjectStorage(dir)
    c1Writer := newClient(fos)
    c2Writer := newClient(fos)

    // Have c2Writer start up a transaction.
    err = c2Writer.newTx()
    assertEq(err, nil, "could not start first c2 tx")
    debug("[c2] new tx")

    // But then have c1Writer start a transaction and commit it first.
    err = c1Writer.newTx()
    assertEq(err, nil, "could not start first c1 tx")
    debug("[c1] new tx")
    err = c1Writer.createTable("x", []string{"a", "b"})
    assertEq(err, nil, "could not create x")
    debug("[c1] Created table")
    err = c1Writer.writeRow("x", []any{"Joey", 1})
    assertEq(err, nil, "could not write first row")
    debug("[c1] Wrote row")
    err = c1Writer.writeRow("x", []any{"Yue", 2})
    assertEq(err, nil, "could not write second row")
    debug("[c1] Wrote row")
    err = c1Writer.commitTx()
    assertEq(err, nil, "could not commit tx")
    debug("[c1] Committed tx")

    // Now go back to c2 and write data.
    err = c2Writer.createTable("x", []string{"a", "b"})
    assertEq(err, nil, "could not create x")
    debug("[c2] Created table")
    err = c2Writer.writeRow("x", []any{"Holly", 1})
    assertEq(err, nil, "could not write first row")
    debug("[c2] Wrote row")

    err = c2Writer.commitTx()
    assert(err != nil, "concurrent commit must fail")
    debug("[c2] tx not committed")
}

Try it out:

$ go mod init otf
$ go mod tidy
$ go test -run TestConcurrentTableWriters -- --debug
[DEBUG] [c2] new tx
[DEBUG] [c1] new tx
[DEBUG] [c1] Created table
[DEBUG] [c1] Wrote row
[DEBUG] [c1] Wrote row
[DEBUG] [c1] Committed tx
[DEBUG] [c2] Created table
[DEBUG] [c2] Wrote row
[DEBUG] [c2] tx not committed
PASS
ok      otf 0.311s

That's pretty cool.

And what about a reader and concurrent writer? Observe that the reader always reads a snapshot. Follow the comments again for detail:

func TestConcurrentReaderWithWriterReadsSnapshot(t *testing.T) {
    dir, err := os.MkdirTemp("", "test-database")

    if err != nil {
        panic(err)
    }

    defer os.Remove(dir)

    fos := newFileObjectStorage(dir)
    c1Writer := newClient(fos)
    c2Reader := newClient(fos)

    // First create some data and commit the transaction.
    err = c1Writer.newTx()
    assertEq(err, nil, "could not start first c1 tx")
    debug("[c1Writer] Started tx")
    err = c1Writer.createTable("x", []string{"a", "b"})
    assertEq(err, nil, "could not create x")
    debug("[c1Writer] Created table")
    err = c1Writer.writeRow("x", []any{"Joey", 1})
    assertEq(err, nil, "could not write first row")
    debug("[c1Writer] Wrote row")
    err = c1Writer.writeRow("x", []any{"Yue", 2})
    assertEq(err, nil, "could not write second row")
    debug("[c1Writer] Wrote row")
    err = c1Writer.commitTx()
    assertEq(err, nil, "could not commit tx")
    debug("[c1Writer] Committed tx")

    // Now start a new transaction for more edits.
    err = c1Writer.newTx()
    assertEq(err, nil, "could not start second c1 tx")
    debug("[c1Writer] Starting new write tx")

    // Before we commit this second write-transaction, start a
    // read transaction.
    err = c2Reader.newTx()
    assertEq(err, nil, "could not start c2 tx")
    debug("[c2Reader] Started tx")

    // Write and commit rows in c1.
    err = c1Writer.writeRow("x", []any{"Ada", 3})
    assertEq(err, nil, "could not write third row")
    debug("[c1Writer] Wrote third row")

    // Scan x in read-only transaction
    it, err := c2Reader.scan("x")
    assertEq(err, nil, "could not scan x")
    debug("[c2Reader] Started scanning")
    seen := 0
    for {
        row, err := it.next()
        assertEq(err, nil, "could not iterate x scan")

        if row == nil {
            debug("[c2Reader] Done scanning")
            break
        }

        debug("[c2Reader] Got row in reader tx", row)
        if seen == 0 {
            assertEq(row[0], "Joey", "row mismatch in c1")
            assertEq(row[1], 1.0, "row mismatch in c1")
        } else {
            assertEq(row[0], "Yue", "row mismatch in c1")
            assertEq(row[1], 2.0, "row mismatch in c1")
        }

        seen++
    }
    assertEq(seen, 2, "expected two rows")

    // Scan x in c1 write transaction
    it, err = c1Writer.scan("x")
    assertEq(err, nil, "could not scan x in c1")
    debug("[c1Writer] Started scanning")
    seen = 0
    for {
        row, err := it.next()
        assertEq(err, nil, "could not iterate x scan in c1")

        if row == nil {
            debug("[c1Writer] Done scanning")
            break
        }

        debug("[c1Writer] Got row in tx", row)

        if seen == 0 {
            assertEq(row[0], "Ada", "row mismatch in c1")
            // Since this hasn't been serialized to JSON, it's still an int not a float.
            assertEq(row[1], 3, "row mismatch in c1")
        } else if seen == 1 {
            assertEq(row[0], "Joey", "row mismatch in c1")
            assertEq(row[1], 1.0, "row mismatch in c1")
        } else {
            assertEq(row[0], "Yue", "row mismatch in c1")
            assertEq(row[1], 2.0, "row mismatch in c1")
        }

        seen++
    }
    assertEq(seen, 3, "expected three rows")

    // Writer committing should succeed.
    err = c1Writer.commitTx()
    assertEq(err, nil, "could not commit second tx")
    debug("[c1Writer] Committed tx")

    // Reader committing should succeed.
    err = c2Reader.commitTx()
    assertEq(err, nil, "could not commit read-only tx")
    debug("[c2Reader] Committed tx")
}

Run it:

$ go test -run TestConcurrentReaderWithWriterReadsSnapshot -- --debug
[DEBUG] [c1Writer] Started tx
[DEBUG] [c1Writer] Created table
[DEBUG] [c1Writer] Wrote row
[DEBUG] [c1Writer] Wrote row
[DEBUG] [c1Writer] Committed tx
[DEBUG] [c1Writer] Starting new write tx
[DEBUG] [c2Reader] Started tx
[DEBUG] [c1Writer] Wrote third row
[DEBUG] [c2Reader] Started scanning
[DEBUG] [c2Reader] Got row in reader tx [Joey 1]
[DEBUG] [c2Reader] Got row in reader tx [Yue 2]
[DEBUG] [c2Reader] Done scanning
[DEBUG] [c1Writer] Started scanning
[DEBUG] [c1Writer] Got row in tx [Ada 3]
[DEBUG] [c1Writer] Got row in tx [Joey 1]
[DEBUG] [c1Writer] Got row in tx [Yue 2]
[DEBUG] [c1Writer] Done scanning
[DEBUG] [c1Writer] Committed tx
[DEBUG] [c2Reader] Committed tx
PASS
ok      otf 0.252s

Sweet.

What's next?

As mentioned, we didn't touch a lot of things. Handling updates and deletes, transaction log checkpoints, data object compaction, etc.

Take a close look at the Delta Lake paper and the Delta Lake Spec and see what you can do!

Build a serverless ACID database with this one neat trick.

(New blog post)https://t.co/rHgfKSPY6q pic.twitter.com/1hmjsxIk6w
— Phil Eaton (@eatonphil) September 29, 2024

Techtalk: Why are A/B tests the gold standard of causal inference?

Bytepawn - Marton Trencseni — Sun, 29 Sep 2024 01:00:00 GMT

Recently, I delivered a techtalk on A/B testing to an audience of non-technical Product Managers and experienced Data Scientists.

200 articles

Bytepawn - Marton Trencseni — Sat, 28 Sep 2024 01:00:00 GMT

A review and introspect on the second 100 articles written on Bytepawn.

ChatGPT's critique of OpenAI CEO Sam Altman's vision of the AI future

Bytepawn - Marton Trencseni — Fri, 27 Sep 2024 01:00:00 GMT

A few days ago OpenAI CEO Sam Altman published an article titled The Intelligence Age. I used OpenAI's new o1-mini model to critique Sam Altman's writing of its own future.

Be someone who does things

Notes on software development — Mon, 23 Sep 2024 03:00:00 GMT

I wrote last month that what you want to do is one of the most useful motivations in life. I want to follow that up by saying that the only thing more important than wanting to do something is to actually do something.

The most valuable trait you can develop for yourself is to be consistent. It is absolutely something you can develop. And moreover it's kind of hard to believe that for anyone it is innate.

I meet so many people who say they want to do things. And I ask them what they're doing to get there and they get flustered. This is completely understandable.

I meet so many students who feel overwhelmed by what everyone else is doing. This is also understandable.

But it doesn't matter what anyone else is doing. It doesn't matter where anyone else is at. It matters where you are at. Compete with yourself before you compete with anyone else. What matters is that you get into a habit of consistently working on little goals.

If you pick something that is too complex, break it down. Keep on breaking problems or ideas down until you find a problem or idea you can solve.

Then keep on finding new problems to solve. Move on in complexity over time as you can and want to.

Don't worry about getting things perfect. Who can discredit you for doing your best? What shame is there when you're being earnest? The only thing that makes sense to feel bad about is not trying to do what you genuinely wanted to do.

And this doesn't have to be about projects or ideas outside of work. There may be things you want to do at work like improving documentation or writing better tests or adding new checks to code or blogging or interviewing customers or working with another team.

Like I said in Obsession, don't worry about what you do daily. That is too frequent to think about. Instead think about what you're doing once a month.

Make time once a month to publish a post or complete a small project. Whatever you want to do, I am confident you can find some small version of it that you could commit to doing once a month. Be consistent!

If a month is too often, pick a longer freqency. Find whatever cadence and whatever size of project that allows you to be consistent.

When you're consistent over the course of months I think you'll be astounded at what you accomplish in a year.

Shorter post tonight, may add to this later on.

Be someone who does things. And do these (little) things consistently.https://t.co/oVb6Sz8eEK pic.twitter.com/kNrZQ4pvTN
— Phil Eaton (@eatonphil) September 24, 2024

The sorry state of Java deserialization

Weblog on marginalia.nu — Sun, 22 Sep 2024 03:00:00 GMT

I’ve been on a bit of a frustration-driven quest to solve a problem I frequently encounter working on the search engine, that is, reading data from disk.

You’d think this would be a pretty basic thing, but doing this in a way that is half-way performant is surprisingly hard and requires avoiding basically all the high level tools at your disposal.

There’s a common sentiment that modern hardware is fast, so this may not matter, but we aren’t speaking a 30% performance hit, the the question is how many orders of magnitude you’re willing to forego.

How I manage my 1v1s

Bytepawn - Marton Trencseni — Sun, 22 Sep 2024 01:00:00 GMT

I detail my structured approach to managing one-on-one meetings within a 40-person data team, emphasizing people management principles like radical transparency, tailored meeting cadences, and strategies to navigate common managerial challenges to foster a supportive and productive work environment.

Arrogant programmers and Enshittification - A New Understanding

--Mike-- — Sat, 21 Sep 2024 20:44:39 GMT

A few weeks ago, I replied to a thread on Hacker News about a "leaked" talk by former Google CEO Eric Schmidt. The key phrase I tried to pick apart was

So imagine a non-arrogant programmer that actually does what you want

I had assumed he was referring to situations similar to my own experience as a software developer interacting with customers back in the 1980s. Programming was still a relatively new thing, and most people didn't understand how data was stored and processed. I considered it my job to probe what they really wanted to have, and figure out how I could meet their needs by impedance matching them to what was feasible. This smoothing over of their wants and needs with reality made everyone happy, in the end.

A few days ago, I watched Cory Doctorow's presentation at DefCon32 about the ongoing Enshittification of everything. He delivered, as usual for him, a passionate and fact stuffed critique of the way things are going, and how we got there. It's been quietly flowing through my mapper mind, bumping up against all of the other chunks of knowledge I've recently pondered.

This morning, I was reading a thread about Thea Lim's The Collapse of Self-Worth in the Digital Age.

She helped me see that though I have had my self-worth tied up in work, there was a deep cause of disastisfaction once I was out of the work force. I've always been good at explaining complex things in a way others can understand. (Impedance matching, again!) This bumped against some other things I was pondering, and then the penny dropped

I was wrong about what Eric Schmidt meant, and I see that now.

I had made a category error and assumed he was just trying to get good results, or meet some weird quirk of his design sense, like Steve Jobs is said to have done, and was getting frustrated. Oh boy, was I ever wrong.

Assuming that Schmidt was talking bout overriding ethical concerns is a way better fit for the amount of emotion this seems to hold for him.

Cory's right. We really do have to hold the line against this shit.

So there you have it... a brief journey through the thought process of a mapper, with a side of self-discovery

Apple M0110 Keyboard CPU replacement - part #1

--Mike-- — Tue, 10 Sep 2024 23:02:49 GMT

My friend Jim bought an Apple Macintosh at the Vintage Computer Festival Midwest on Saturday. It's got a bad keyboard. The keyboard has a bad Intel 8021 CPU.

I need to replace that chip. It's going to take wayyyy more time that I'm used to dumping into a project successfully. This is part of my process to push myself forward.

So far I've:

Searched for an 8048 series disassembler, found none
Downloaded the 8048 series manual from BitSavers
Decided to write an 8021 specific disassembler in pascal
Started a Lazarus project in my C:\res21 folder
Published the above to an MIT licensed GitHub repository

Let's see how this goes.

Go’s New Iterators Smell a Little Funny, but They’re Still Good

Donat Studios — Fri, 06 Sep 2024 02:28:12 GMT

Go's new Iterators were released on August 13th with Go 1.23. They're pretty useful.

They've had just a minute to marinate. In some parts of the community they've been an unpopular addition, however I don't particularly mind them. The old alternatives seem a little more "Go" to me, but the new iterators are fine.

One thing however I am mildly disappointed about is that there isn't just a generic iterable type similar to the generic comparable. It would be nice to have an imaginary iterable type that could accept maps, slices, arrays, and iterators. As I understand it, iterators combined with slices.All and maps.All are the blessed alternative to that. It works, I'm using it, but I don't love it.

All that's neither here nor there, however, and not what I'm here to talk about.

The part that smells funny to me is that range now accepts any already existing function that happens to fulfill either of the following interfaces.

func(func(any) bool)
func(func(any,any) bool)

It would be less weird if that wasn't also a little troublesome.

Example time!

The first interface, known as Seq takes a yield method as an argument that itself accepts any single parameter. The second interface, known as Seq2 similarly takes a yield method that accepts any two parameters as essentially a key/value tuple.

Note that yield is not a magic keyword as in many languages, but a passed-in callable. This is very akin to Go allowing you to name receivers rather than magically defining this.

At first blush to the uninitiated, this iterator looks fine. It seemingly runs fine as well.

package main

import "fmt"

func mightBeAnIterator(f func(string) bool) {
    f("foo")
    f("bar")
    f("baz")
}

func main() {
    for x := range mightBeAnIterator {
        fmt.Println(x)
    }
}

foo
bar
baz

What's the problem then?

The problem is iterators MUST handle early exit of the loop manually. With the current interface fulfilling "iterator", if we just add what seems a harmless break to our loop as follows, suddenly the code working moments ago panics.

    for x := range mightBeAnIterator {
        fmt.Println(x)
        break
    }

foo
panic: runtime error: range function continued iteration after function for loop body returned false

Somewhat surprisingly, as Go exits the loop, it allows the iterator to continue executing. In doing so, however, it signals the iterator through the result of the yielding method that it should return. This is to allow for teardown. For example, imagine an iterator that opens files or a socket; continuing execution allows it to close them before returning.

However, Go will panic if the signal to return is not honored and the yield function is then invoked again by the iterator.

To prevent our example iterator from panic-ing, we simply have to listen for the signal to exit, as shown below.

func mightBeAnIterator(f func(string) bool) {
    if !f("foo") {
        return
    }
    if !f("bar") {
        return
    }
    if !f("baz") {
        return
    }
}

And when run, now we receive simply

foo

I would imagine the reason for the panic is to prevent iterators from accidentally running wild.

To my personal taste, I would have strongly preferred honoring the exit signal to be optional, and following superfluous yield calls being quietly ignored and returning false.

I can certainly imagine cases where an iterator would want to finish regardless of early exits by its consumer. While still possible, it's awkward to implement.

We are certainly in a place where, should they decide to go back on the panic behavior, it would not break any existing code. That's not a terrible place to be.

All this is to say it smells a little funny that such a simple and likely somewhat common interface has this magic range logic available to it.

Expecting the implementers to implement it correctly and panic-ing if they don't smells kind of Worse is Better to me. While one could complicate Go's iterator interface to enforce stricter guarantees around safe iteration, doing so would add unnecessary complexity and detract from Go's philosophy of simplicity.

It's one of the rare occasions where I wish interfaces had to be intentionally met. It's strange to me that anything that smells like an iterator can be passed to range, even if it doesn't implement the intended handshake.

It smells funny, but that's probably OK. It almost certainly falls into the case of "things that will never be a real problem in the real world". I look forward to the oncoming brave new world of libraries expecting iterators.

scc vs stto a code counter head to head

Ben E. C. Boyter — Thu, 05 Sep 2024 03:00:00 GMT

A work colleague (Dazza!) messaged me this morning about a new code counter with some pretty serious performance claims. Not only was it claiming to be accurate it was claiming to have a 38x speed improvement over tools such as tokei!

Code counters are something I am interested in so I did some digging to find the repository https://github.com/mainak55512/stto Interesting it is written in Go.

My first though about the times was that all it was measuring is the startup time of a binary in whatever it’s written in to Go. No surprise Go is slower as it has GC and a runtime. However this is not the case here. That said, improving the startup time of scc for times below 20 ms is something I never worried about. There are two main reasons for that, use and human perception.

Breaking Bell's Inequality with Monte Carlo Simulations in Python

Bytepawn - Marton Trencseni — Sun, 01 Sep 2024 01:00:00 GMT

The article explains the Bell inequality using Monte Carlo simulations in Python, and shows how non-local action-at-a-distance can be used to break it with entangled qubits.

Two Rules of AI Business and Startups That Ignore Them

Svedic.org — Sat, 31 Aug 2024 16:23:15 GMT

These rules are not new, and they are not mine; I stole them from Andrew Ng and Benedict Evans, two men with a huge following. Still, a large majority of AI entrepreneurs and engineers don’t pay attention to them, maybe … Continue reading →

Evaluating your AI models in the wild

Austin Z. Henley's Blog — Sun, 25 Aug 2024 00:00:01 GMT

https://austinhenley.com/blog/aiinthewild.html

Obsession

Notes on software development — Sat, 24 Aug 2024 03:00:00 GMT

In your professional and personal life, I don't believe there is a stronger motivation than having something in mind and the desire to do it. Yet the natural way to deal with a desire to do something is to justify why it's not possible.

"I want to read more books but nobody reads books these days so how could I."

"I want to write for a magazine but I have no experience writing professionally."

"I want to build a company someday but how could someone of my background."

Our official mentors, our managers, through a combination of well-intentioned defeatism and well-intentioned lack of accomplishment themselves, among other things, are often unable to process big goals or guide you toward them.

I've been one of these managers myself. In fact I have, to my immense regret, tried too often to convince people to do what is practical rather than what they want to do. Or to do what I judged they were capable of doing rather than what they wanted to do.

In the best cases, my listener had the self-confidence to ignore me. They did what they wanted to do anyway. In the worst case, again to my deep regret, I've been a well-intentioned part of derailing someone's career for years.

So I don't want to convince anyone of anything anymore. If I start trying to convince someone by accident, I try to catch myself. I try to avoid sentences like "I think you should …". Instead "Here is something that's worked for me: …" or "Here is what I've heard works well for other people: …".

Nobody wants to be convinced. But intelligent people will change their mind when exposed to new facts or different ideas. Being convinced is a battle of will. Changing one's mind is a purely personal decision.

There are certainly people with discipline who can grind on things they hate doing and eventually become experts at it. But more often I see people grind on things they hate only to become depressed and give up.

For most of us, our best hope is (healthy) obsession. And obsession, in the sense I'm talking about, does not come from something you are ambivalent about or hate. Obsession can only come when you're doing something you actually want to do.

For big goals or big changes, you need regular commitment weekly, monthly, yearly. Over the course of years. And only obsession makes that work not actually feel like work. Obsession is the only thing that makes discipline not feel like discipline.

That big goals take years to accomplish need not be scary. Obsession doesn't mean you can't pivot. There is quite a lot to gain by committing to something regularly over the course of years even if you decide to stop and commit from then on to something else. You will learn a good deal.

And healthy obsession to me is more specifically measurable on the order of weeks, not hours or days. Healthy obsession means you're still building healthy personal and professional relationships. You're still taking care of yourself, emotionally and physically.

I do not have high expectations for people in general. This seems healthy and reasonable. But as I meet more people and observe them over the years, I am only more convinced of the vast potential of individuals. Individuals are almost universally underestimated.

I think you can do almost anything you want to do. If you commit to do doing it.

I'll end this with a personal story.

Until 11th grade, I hated school. I hated the rigidity. Being forced to be somewhere for hours and to follow so many rules. I skipped so many days of school I'm embarrassed by it. I'd never do homework at home. I never studied for tests. I got Bs and Cs in the second-tier classes. I was in the orchestra for 6 years and never practiced at home. I was not cool enough to be a "bad kid" but I did not understand the system and had no discipline whatsoever.

I found out at the end of 10th grade that I could actually afford college if I got into a good enough school that paid full needs-based tuition. It sounded significantly better than the only other option that seemed obvious, joining the military as a recruit. I realized and decided that if I wanted to get into a good school I needed to not half-ass things.

Somehow, I decided to only do things I could become obsessed with. And I decided to be obsessed in the way that I wanted, not to do what everyone else did (which I basically could not do since I had no discipline). If we covered a topic in class, I'd read news about it or watch movies about it. I'd get myself excited about the topic in every way I could.

It basically worked out. I ended high school in the top 10% of the class (up from top 40% or something). I got into a good liberal arts college that paid the entirety of my tuition. But I remained a basically lazy and undisciplined person. I never stayed up late studying for a test. I dropped out after a year and a half for family reasons.

But I've now spent the last 10 years in my spare time working on compiler projects, interpreter projects, parser projects, database projects, distributed systems projects. I've spent the last 6 years consistently publishing at least one blog post per month.

I didn't want to work the way everyone else worked. I wanted to be obsessed about what I worked on.

Obsession has made all of this into something I now barely register as doing. It's allowed me to continue adding activities like organizing book clubs and meetups to the list of things I'm up to. Up until basically this year I could have in good faith said I am a very lazy and undisciplined person. But obsession turned me into someone with discipline.

Obsession became about more than just the tech. It meant trying to fully understand the product, the users, the market. It meant thinking more carefully about product documentation, user interfaces, company messaging. Obsession meant reflecting on how I treat my coworkers, and how my coworkers feel treated by others in general. Obsession meant wanting an equitable and encouraging work environment for everyone.

And, as I said, it's about healthy obsession. I didn't really understand the "healthy" part until a few years ago. But I'm now convinced that the "healthy" part is as important as the "obsession" part. To go to the gym regularly. To play pickup volleyball. To cook excellent food. To read fiction and poetry and play music. To serve the community. To be friendly and encouraging to all people. To meet new people and build better genuine friendships.

And in the context of work, "healthy obsession" means understanding you can't do everything, even while you care about everything. It means accepting that you make mistakes and that you do your best; that you try to do better and learn from mistakes the next time.

It's got to be sustainable. And we can develop a healthy obsession while we have quite a bit of fun too. :)

I wrote an essay on my mistakes trying to convince people to do something, on doing what you want to do, and on obsession.

Ended with a personal note on developing healthy discipline, and having fun. :)https://t.co/4WWdtU6AhL pic.twitter.com/lBw7zlqWeq
— Phil Eaton (@eatonphil) August 24, 2024

Objective-C is just, like, a leaky abstraction over C

metablog — Thu, 22 Aug 2024 10:54:10 GMT

Recently I came across this 10 year old post by Robert Atkins: Objective C is like Jimi Hendrix. It is about reconciling admiration for Objective-C by the "old-timers" with the newcomers' somewhat less enthusiastic response.

His very cogent insight was that Objective-C, like Jimi Hendrix, introduced new concepts that were somewhat revolutionary (""mind blowing") at the time, but are now taken for granted:

So if you'e new to Objective-C and, as I am, struggling to come to terms with the fact that it's one great big leaky abstraction on top of C, put yourself in the shoes of an 80s C programmer and remember you get to use these neat "modern" features in a systems programming language.

As one of the very early, pre-NeXT, adopters of Objective-C, I have a slightly different take: That implicit "despite" is actually very much a "because" for me.

The modern features in Objective-C such as a dynamic messaging, a runtime with introspection and intercession etc. were not new at the time, and they were not really "mind-blowing". LISP and Smalltalk had had them for a long time. But so far having these features had required large, complex runtime environments that were very distant from the rest of the machine, and usually quite isolated from the rest of the machine. Still are, to this day. You don't program your computer with LISP or Smalltalk. Your computer runs a separate LISP or Smalltalk computer that you can then program on its own terms and in its own world. (And attempts to bring machines closer to these languages mostly did not succeed).

Objective-C provided these features with the slimmest of a sliver of an extension to a portable PDP-11 macro assembler.

Now that was mind-blowing!

And it goes beyond that: there actually was, at some point, Objective-Assembler. Objective-C was never intended to be "a language" like Java or Rust. The Objective part of Objective-C is a glue layer, an integration mechanism that can be added to any language. There was Objective-FORTRAN, Objective-Pascal etc. Helge Hess once put it almost perfectly: Objective-C is COM with language support. Or SOM. Or whatever interop mechanism they come up with again (Swift "library evolution", I am looking at you, can we have Objective-Swift?).

So it wasn't just mind-blowing, it also was, is, and remains incredibly useful.

So useful, in fact, that this slimmest sliver of an extension gradually replaced the portable PDP-11 macro assembler it was sitting on top of for most day-to-day use. So much so, that in the Apple developer ecosystem, the much larger C part started to be regarded as a completely separate language that most devs never dared touch.

Of course the fact that this is doable shouldn't be surprising, after all the Objective part is modeled after Smalltalk, and Smalltalk is a complete programming language. But Smalltalk requires a fairly large VM to run, typically coded in C or in Squeak's case, C plus BCPL-encoded-in-Smalltalk that gets automatically translated to C. Objective-C and Objective-Assembler showed that you don't actually need all that, just a tiny messaging function on top of bare machine is not just sufficient, it's also faster, integrates better and is easier to implement. Less is indeed more.

All that doesn't mean that Objective-C isn't an ad-hoc car crash of two languages with overlapping functionality and syntax, and the safety of C's memory model and Smalltalk's type system. That this disaster area works better in practice than most other languages tells you all you need to know about the state of PL design.

A more principled, if irreverent, exploration of the same approach can be found in Ian Piumarta's COLA (COmbined Lambda Architecture). He basically discovered the same principles, and came up with something that's even cooler, though at this point less practical.

My favorite bit (it's a tight field) is how he manages to build a message-dispatcher that is itself invoked by message-send. How does he resolve the infinite recursion inherent in the self-referential definition? By pre-populating a single cache.

Ian is also a really great speaker:

Alas this video of his Stanford EE380 talk is only 240p, so the text is illegible, but fortunately there are slides that you can follow along.

In Objective-S, the procedural/OO part is really based on just that sliver of an extension. The C part is removed (yes, a real "Objective-C without the C"), replaced by a few type declarations for indicating primitive types, special stores for raw memory access when needed and a generalization of message-sending that subsumes calling C functions.

Vim Reference Guide book announcement

learnbyexample — Tue, 20 Aug 2024 03:00:00 GMT

Hello!

I am pleased to announce a new version of my Vim Reference Guide ebook. This is intended as a concise learning resource for beginner to intermediate level Vim users. It has more in common with cheatsheets than a typical text book. Topics like Regular Expressions and Macros have more detailed explanations and examples due to their complexity. I hope this guide would make it much easier for you to discover Vim features and learning resources than my own blundering experience.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of the ebook for FREE till 31-Aug-2024. You can still pay if you wish ;)

Two of my bundles are on sale as well:

All Books Bundle is $15 (normal price $32) — all my 13 programming ebooks
Linux CLI Text Processing is 50% OFF — grep, sed, awk, perl and ruby one-liners, coreutils, cli computing

What's new?🔗

Updated ebook for Vim version 9.1
Corrected typos
Some of the examples, descriptions and external links were updated
New cover image

Videos🔗

Visit this playlist for video demos on most of the topics from the ebook.

Testimonials🔗

Got several suggestions and feedback when my submission about this book reached the front page of Hacker News.

Great job on this! — rendall

Hi, great work releasing this! Trying to explain vim concisely is always an interesting challenge and I had a great time reading your attempt in this book. I always find it really interesting on how people try to group certain vim functions in a way that makes sense to people that don't use vim. I think you cover that idea pretty well in your 'Vim philosophy and features' section whilst not making it overly abstract and keeping it relatable. — doix

Neat stuff! One piece of feedback is that I would include "+p and "+yy in the copy and paste section. — mrpotato

I learnt regular expression by reading your books, thank you for the great work. — LamJH

A comment from another Hacker News thread:

I stumbled upon your vi post a few days ago, really like the style. Keep it up!

Vim prank🔗

Did you know that Vim has an easy mode? It can be rather hard to use for those already familiar with Vim modes. I wrote a blog post about this mode, which was interesting enough to reach the front page of Hacker News!

Table of Contents🔗

Preface
Introduction
Insert mode
Normal mode
Command-line mode
Visual mode
Regular Expressions
Macro
Customizing Vim
CLI options

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/vim_reference/.

GitHub repo🔗

Visit https://github.com/learnbyexample/vim_reference for markdown source and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/vim_reference/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

What's the big deal about Deterministic Simulation Testing?

Notes on software development — Tue, 20 Aug 2024 03:00:00 GMT

Bugs in distributed systems are hard to find, largely because systems interact in chaotic ways. And even once you've found a bug, it can be anywhere from simple to impossible to reproduce it. It's about as far away as you can get from the ideal test environment: property testing a pure function.

But what if we could write our code in a way that we can isolate the chaotic aspects of our distributed system during testing: run multiple systems communicating with each other on a single thread and control all randomness in each system? And property test this single-threaded version of the distributed system with controlled randomness, all the while injecting faults (fancy term for unhappy path behavior like errors and latency) we might see in the real-world?

Crazy as it sounds, people actually do this. It's called Deterministic Simulation Testing (DST). And it's become more and more popular with startups like FoundationDB, Antithesis, TigerBeetle, Polar Signals, and WarpStream; as well as folks like Tyler Neely and Pekka Enberg, talking about and making use of this technique.

It has become so popular to talk about DST in my corner of the world that I worry it risks coming off sounding too magical and maybe a little hyped. It's worth getting a better understanding of both the benefits and the limitations.

Thank you to Alex Miller and Will Wilson for reviewing a version of this post.

Randomness and time

A big source of non-determinism in business logic is the use of random numbers—in your code or your transitive dependencies or your language runtime or your operating system.

Crucially, DST does not imply you can't have randomness! DST merely assumes that you have a global seed for all randomness in your program and that the simulator controls the seed. The seed may change across runs of the simulator.

Once you observe a bad state as a result of running the simulation on a random seed, you allow the user to enter the same seed again. This allows the user to recreate the entire program run that led to that observed bad state. Allows the user to debug the program trivially.

Another big source of non-determinism is being dependent on time. As with randomness, DST does not mean you can't depend on time. DST means you must be able to control the clock during the simulation.

To "control" randomness or time basically means you support dependency injection, or the old-school alternative to dependency injection called passing the dependency as an explicit parameter. Rather than referring to a global clock or a global seed, you need to be able to receive a clock or a seed from someone.

For example we might separate the operation of an application into the language's main() entrypoint and an actual application start() entrypoint.

# app.pseudocode

def start(clock, seed):
  # lots of business logic that might depend on time or do random things

def main:
  clock = time.clock()
  seed = time.now()
  app.start(clock, seed)

The application entrypoint is where we must be able to swap out a real clock or real random seed for one controlled by our simulator:

# sim.pseudocode

import "app.pseudocode"

def main:
  sim_clock = make_sim_clock()
  sim_seed = os.env.DST_SEED or time.now()
  try:
    app.start(sim_clock, sim_seed)
  catch(e):
    print("Bad execution at seed: %s", sim_seed)
    throw e

Let's look at another example.

Converting an existing function

Let's say that we had a helper method that kept calling a function until it succeeded, with backoff.

# retry.pseudocode
class Backoff:
  def init:
    this.rnd = rnd.new(seed = time.now())
    this.tries = 0

  async def retry_backoff(f):
    while this.tries < 3:
      if f():
        return

      await time.sleep(this.rnd.gen())
      this.tries++

There is a single source of nondeterminism here and it's where we generate a seed. We could parameterize the seed, but since we want to call time.sleep() and since in DST we control the time, we can just parameterize time.

# retry.psuedocode
class Backoff:
  def init(this, time):
    this.time = time
    this.rnd = rnd.new(seed = this.time.now())
    this.tries = 0

  async def retry_backoff(this, f):
    while this.tries < 3:
      if f():
        return

      await this.time.sleep(this.rnd.gen())
      this.tries++

Now we can write a little simulator to test this:

# sim.psuedocode
import "retry.pseudocode"

sim_time = {
  now: 0
  sleep: (ms) => {
    await future.wait(ms)
  }
  tick: (ms) => now += ms
}

backoff = Backoff(sim_time)

while true:
  failures = 0
  f = () => {
    if rnd.rand() > 0.5:
      failures++
      return false

    return true
  }
  try:
    while sim_time.now < 60min:
      promise = backoff.retry_backoff(f)
      sim_time.tick(1ms)
      if promise.read():
        break

    assert_expect_failure_and_expected_time_elapse(sim_time, failures)
  catch(e):
    print("Found logical error with seed: %d", seed)
    throw e

This demonstrates a few critical aspects of DST. First, the simulator itself depends on randomness. But allows the user to provide a seed so they can replay a simulation that discovers a bug. The controlled randomness in the simulator is what lets us do property testing.

Second, the simulation workload must be written by the user. Even when you've got a platform like Antithesis that gives you an environment for DST, it's up to you to exercise the application.

Now let's get a little more complex.

A single thread and asynchronous IO

The determinism of multiple threads can only be controlled at the operating system or emulator or hypervisor layer. Realistically, that would require third-party systems like Antithesis or Hermit (which, don't get excited, is not actively developed and hasn't worked on any interesting program of mine) or rr.

These systems transparently transform multi-threaded code into single threaded code. But also note that Hermit and rr have only limited ability to do fault injection which, in addition to deterministic execution, is a goal of ours. And you can't run them on a mac. And can't run them on ARM.

But we can, and would like, to write a simulator without writing a new operating system or emulator or hypervisor, and without a third-party system. So we must limit ourselves to writing code that can be collapsed into a single thread. Significantly, since using blocking IO would mean an entire class of concurrency bugs could not be discovered while running the simulator in a single thread, we must limit ourselves to asynchronous IO.

Single threaded and asynchronous IO. These are already two big limitations.

Some languages like Go are entirely built around transparent multi-threading and blocking IO. Polar Signals solved this for DST by compiling their application to WASM where it would run on a single thread. But that wasn't enough. Even on a single thread, the Go runtime intentionally schedules goroutines randomly. So Polar Signals forked the Go runtime to control this randomness with an environment variable. That's kind of crazy. Resonate took another approach that also looks cumbersome. I'm not going to attempt to describe it. Go seems like a difficult choice of a language if you want to do DST.

Like Go, Rust has no builtin async IO. The most mature async IO library is tokio. The tokio folks attempted to provide a tokio-compatible simulator implementation with all sources of nondeterminism removed. From what I can tell, they did not at any point fully succeed. That repo has now been replaced with a "this is very experimental" tokio-rs project called turmoil that provides deterministic execution plus network fault injection. (But not disk fault injection. More on that later.) It isn't surprising that it is difficult to provide deterministic execution for an IO library that was not designed for it. tokio is a large project with many transitive dependencies. They must all be combed for non-determinism.

On the other hand, Pekka has already demonstrated for us how we might build a simpler Rust async IO library that is designed to be simulation tested. He modeled this on the TigerBeetle design King and I wrote about two years ago.

So let's sketch out a program that does buggy IO and let's look at how we can apply DST to it.

# readfile.pseudocode
def read_file(io, name, into_buffer):
  f = await io.open(name)
  read_buffer = [4096]u8{}
  while true:
    err, n_read = await f.read(&read_buffer)
    if err == io.EOF:
      into_buffer.copy_maybe_allocate(read_buffer[0:sizeof(read_buffer)])
      return

    if err:
      throw err

    into_buffer.copy_maybe_allocate(read_buffer[0:sizeof(read_buffer)])

In our simulator, we will provide a mocked out IO system and we will randomly inject various errors while asserting pre- and post-conditions.

# sim.psuedocode
import "readfile.pseudocode"

seed = if os.env.DST_SEED ? int(os.env.DST_SEED) : time.now()
rnd = rnd.new(seed)

while true:
  sim_disk_data = rnd.rand_bytes(10MB)
  sim_fd = {
    pos: 0
    EOF: Error("eof")
    read: (fd, buf) => {
      partial_read = rnd.rand_in_range_inclusive(0, sizeof(buf))
      memcpy(sim_disk_data, buf, fd.pos, partial_read)
      fd.pos += partial_read
      if fd.pos == sizeof(sim_disk_data):
        return io.EOF, partial_read
      return partial_read 
    }
  }
  sim_io = {
    open: (filename) => sim_fd
  }

  out_buf = Vector<u8>.new()
  try:
    read_file(sim_io, "somefile", out_buf)
    assert_bytes_equal(out_buf.data, sim_disk_data)
  catch (e):
    print("Found logical error with seed: %d", seed)
    throw e

And with this simulator we would have eventually caught our partial read bug! In our original program when we wrote:

      into_buffer.copy_maybe_allocate(read_buffer[0:sizeof(read_buffer)])

We should have written:

      into_buffer.copy_maybe_allocate(read_buffer[0:n_read])

Great! Let's get a little more complex.

A distributed system

I already mentioned in the beginning that the gist of deterministic simulation testing a distributed system is that you get all of the nodes in the system to run in the same process. This would be basically impossible if you wanted to test a system that involved your application plus Kafka plus Postgres plus Redis. But if your system is a self-contained distributed system, such as one that embeds a Raft library for high availability of your application, you can actually run multiple nodes into the same process!

For a system like this, our simulator might look like:

# sim.pseudocode
import "distsys-node.pseudocode"

seed = if os.env.DST_SEED ? int(os.env.DST_SEED) : time.now()
rnd = rnd.new(seed)

while true:
  sim_fd = {
    send(fd, buf) => {
      # Inject random failure.
      if rnd.rand() > .5:
         throw Error('bad write')

      # Inject random latency.
      if rnd.rand() > .5:
        await time.sleep(rnd.rand())

      n_written = assert_ok(os.fd.write(buf))
      return n_written
    },
    recv(fd, buf) => {
      # Inject random failure.
      if rnd.rand() > .5:
         throw Error('bad read')

      # Inject random latency.
      if rnd.rand() > .5:
        await time.sleep(rnd.rand())

      return os.fd.read(buf)
    }
  }
  sim_io = {
    open: (filename) => {
      # Inject random failure.
      if rnd.rand() > .5:
        throw Error('bad open')

      # Inject random latency.
      if rnd.rand() > .5:
        await time.sleep(rnd.rand())

      return sim_fd
    }
  }

  all_ports = [6000, 6001, 6002]
  nodes = [
    await distsys-node.start(sim_io, all_ports[0], all_ports),
    await distsys-node.start(sim_io, all_ports[1], all_ports),
    await distsys-node.start(sim_io, all_ports[2], all_ports),
  ]
  history = []
  try:
    key = rnd.rand_bytes(10)
    value = rnd.rand_bytes(10)
    nodes[rnd.rand_in_range_inclusive(0, len(nodes)].insert(key, value)
    history.add((key, value))
    assert_valid_history(nodes, history)

    # Crash a process every so often
    if rnd.rand() > 0.75:
      node = nodes[rnd.rand_in_range_inclusive(0, 3)]
      node.restart()
  catch (e):
    print("Found logical error with seed: %d", seed)
    throw e

I'm completely hand waving here to demonstrate the broader point and not any specific testing strategy for a specific distributed system. The important points are that these three nodes run in the same process, on different ports.

We control disk IO. We control network IO. We control how time elapses. We run a deterministic simulated workload against the three node system while injecting disk, network, and process faults.

And we are constantly checking for an invalid state. When we get the invalid state, we can be sure the user can easily recreate this invalid state.

Other sources of non-determinism

Within some error margin, most CPU instructions and most CPU behavior are considered to be deterministic. There are, however, certain CPU instructions that are definitely not. Unfortunately that might include system calls. It might also include malloc. There is very little to trust.

If we ignore Antithesis, people doing DST seem not to worry about these smaller bits of nondeterminism. Yet it's generally agreed that DST is still worthwhile anyway. The intuition here is that every bit of non-determinism you can eliminate makes it that much easier to reproduce bugs when you find them.

Put another way: determinism, even among DST practitioners, remains a spectrum.

Considerations

As you may have noticed already from some of the pseudocode, DST is not a panacea.

Consideration 1: Edges

First, because you must swap out non-deterministic parts of your code, you are not actually testing the entirety of your code. You are certainly encouraged to keep the deterministic kernel large. But there will always be the non-deterministic edges.

Without a system like Antithesis which gives you an entire deterministic machine, you can't test your whole program.

But even with Antithesis you cannot test the integration between your system and external systems. You must mock out the external systems.

It's also worth noting that there are many areas where you could inject simulation. You could do it at a high-level RPC and storage layer. This would be simpler and easier to understand. But then you'd be omitting testing and error-handling of lower-level errors.

Consideration 2: Your workload(s)

DST is dependent on your creativity and thoroughness of your workload as much as any other type of test or benchmark.

Just as you wouldn't depend on one single benchmark to qualify your application, you may not want to depend on a single simulated workload.

Or as Will Wilson put it for me:

The biggest challenge of DST in my experience is that tuning all the random distributions, the parameters of your system, the workload, the fault injection, etc. so that it produces interesting behavior is very challenging and very labor intensive. As with fuzzing or PBT, it's terrifyingly easy to build a DST system that appears to be doing a ton of testing, but actually never explores very much of the state space of your system. At FoundationDB, the vast majority of the work we put into the simulator was an iterative process of hunting for what wasn't being covered by our tests and then figuring out how to make the tests better. This process often resembles science more than it does engineering.

Unfortunately, unlike with fuzzing, mere branch coverage in your code is usually a pretty poor signal for the kinds of systems you want to test with DST. At Antithesis we handle this with Sometimes assertions, at FDB we did something pretty similar, and I assume TigerBeetle and others have their own version of this. But of course the ultimate figure of merit is whether your DST system is finding 100% of your bugs. It's quite difficult to get to the point that it does. The truly ambitious part of Antithesis isn't the hypervisor, but the fact that we also aim to solve the much harder "is my DST working?" problem with minimal human guidance or supervision.

Consideration 3: Your knowledge of what you mocked

When you mock out the behavior of disk or network IO, the benefits of DST are tied to your understanding of the spectrum of behavior that may happen in the real world.

What are all possible error conditions? What are the extreme latency bounds of the original method? What about corruption or misdirected IO?

The flipside here is that only in deterministic simulation testing can you configure these crazy scenarios to happen at a configurable regularity. You can kick off a set of runs that have especially high IO latency or especially high corrupt reads/writes. Joran and I wrote a year ago about how the TigerBeetle simulator does exactly this.

Consideration 4: Non-reproducible seeds as code changes

Critically, the reproducibility of DST only helps so long as your code doesn't change. As soon as your code changes, the seed may no longer even get you to the state where the bug was exhibited. So the reproducibility of DST means more that it may help you convert the seed simulation run into an integration test that describes the precise scenario even as the code changes.

Consideration 5: Time and compute

Because of Consideration 4, you need to keep rerunning the simulator not just to keep finding new seeds and new histories but because the new seeds and new histories may change every time you make changes to code.

What about Jepsen?

Jepsen does limited process and network fault injection while testing for linearizability. It's a fantastic project.

However, it represents only a subset of what is possible with Deterministic Simulation Testing (if you actually put in the effort described above to get there).

But even more importantly, Jepsen has nothing to do with deterministic execution. If Jepsen finds a bug and your system can't do deterministic execution, you may or may not be able to reproduce that Jepsen bug.

Here's another Will Wilson quote for you on Jepsen and FoundationDB:

Anyway, we did [Deterministic Simulation Testing] for a while and found all of the bugs in the database. I know, I know, that’s an insane thing to say. It’s kind of true though. In the entire history of the company, I think we only ever had one or two bugs reported by a customer. Ever. Kyle Kingsbury aka “aphyr” didn’t even bother testing it with Jepsen, because he didn’t think he’d find anything.

Conclusion

The degree to which you can place faith in DST alone, and not time spent in production, has limits. However, it certainly does no harm to employ DST. And, barring the considerations described above, will likely make the kernel of your product significantly more stable. Furthermore, everyone who uses DST knows about these considerations. But I think it's worthwhile to list them out to help folks who do not know DST to build an intuition for what it's excellent at.

2024 Aug Quick Summary

jodavaho.io — Mon, 19 Aug 2024 03:00:00 GMT

I’m lucky to have found a new job that is essentially my dream position. I’m now a full-time Rust programmer, but still working on a full planning stack. It’s a very small startup, and I’m completely stoked.

This summer has been beautiful, and we’ve been lucky to have a lot of time with family and friends.

My only side project at the moment is a Cyberpunk-like rule system for a tabletop RPG, which is heavily inspired by the old R Talsorian Cyberpunk 2020 rules, and is designed so that the mechanics directly affect roleplay and player decisions, and any player decision should play out with the mechanics.

Quantum Entanglement, Locality, Determinism

Bytepawn - Marton Trencseni — Sun, 18 Aug 2024 01:00:00 GMT

This blog post explores foundational quantum mechanics concepts such as superposition, entanglement, and the challenges posed by locality and determinism, while highlighting the historical efforts of Einstein and Bohm to reconcile these phenomena with hidden variable theories.

Ranking the Hugo and Nebula Award Winners and Nominees

Beetle Space — Mon, 12 Aug 2024 10:00:00 GMT

I grew up reading science fiction. Although I enjoyed the genre a great deal, most of my reading was confined to Isaac Asimov and Arthur C. Clarke (with a dash of Ray Bradbury).

By the end of my college years, I had read most of the fiction from those two …

Using ESPHome Without the Home Assistant Addon

Pete Keen — Mon, 12 Aug 2024 03:00:00 GMT

The "blessed" flow for using ESPHome is the Home Assistant ESPHome Addon. This works well! It has a nice editor and it takes care of some housekeeping tasks for you. If you don't already have a comfortable development workflow it's a very nice way to start.

If you do already have a working style that doesn't involve a web UI and a browser editor you can still use ESPHome, you just have to handle those housekeeping tasks yourself.

Fundamentals

I think it can be helpful to step back and take a look at the fundamentals of a piece of software like ESPHome before diving headlong into the deep pool of non-standard workflows.

At it's base level ESPHome is a microcontroller firmware generator. That is, it reads your YAML config, generates a bunch of C++ files and config files, and then using a compiler and some helper programs it generates a binary program that your microcontroller (usually but not always an ESP32) can run.

ESPHome also has a few very useful helpers. First, it can do seamless over the air (OTA) updates once any ESPHome firmware has been installed on a device.

Second, it has a pretty powerful web-based UI and configuration editor.

Third, ESPHome ships with a "native" binary protocol it can use to talk to Home Assistant (previously) complete with Noise-based symmetric key encryption.

Lastly, it can be used as a Home Assistant addon, which as I said earlier takes care of a few things for you. The OTA update functionality requires a pre-shared key to validate that updates are coming from a known source. The addon takes care of generating that and the Noise secret and sharing these keys with Home Assistant so you don't have to care about them.

ESPHome on a Macbook?!

My ESPHome workflow doesn't involve the web UI or the addon at all. Instead, I install ESPHome on my Macbook with homebrew and manage the OTA and HA secret keys with 1Password and a small helper script. The script and all of my ESPHome configs live in this public GitHub repo.

This is the script as it exists today:

#!/bin/bash
        
        set -x
        set -eo pipefail
        
        trap "rm common/device_base.yaml" EXIT
        
        op inject --in-file common/device_base.yaml.in --out-file common/device_base.yaml
        
        command=$1
        shift
        
        if [ $# -eq 0 ]; then
        configs="*.yaml"
        else
        configs="$@"
        fi
        
        esphome $command $configs

All this is really doing is using 1Password's op inject tool to generate a file with my configured secrets, runs esphome, and makes sure to clean up the generated file with that trap line. The top of device_base.yaml.in looks like this:

substitutions:
        wifi_ssid: "op://keen.land secrets/ESPHome Secrets/ESPHOME_WIFI_SSID"
        wifi_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_WIFI_PASSWORD"
        fallback_ssid_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_FALLBACK_SSID_PASSWORD"
        home_assistant_encryption_key: "op://keen.land secrets/ESPHome Secrets/ESPHOME_HOME_ASSISTANT_ENCRYPTION_KEY"
        ota_password: "op://keen.land secrets/ESPHome Secrets/ESPHOME_OTA_PASSWORD"

All of those are just text entries in the ESPHome Secrets rich text item, but again they can be whatever you want. If you decide to make them password type entries I believe you'd need to add --reveal to the op inject command, but I'm not 100% certain on that.

This differs in kind of a fundamental way from the way the web UI / addon work, in so far as the addon will create and manage unique OTA and HA keys for each device. My setup instead uses two keys shared among all of my devices. I don't see this as a significant risk because I don't use esphome devices in higher security contexts (i.e. my door locks are not running esphome), but your threat model is likely different than mine so you should make your own decisions. Nothing is stopping you from using unique keys for every device with this setup, you just have more secrets to manage in 1Password.

Workflow

My workflow looks like:

$ <edit whatever.yaml>
        $ ./build.sh run whatever.yaml
        # a bunch of compiler output and then logging from the device itself
        $ git add whatever.yaml && git commit -m 'updates' && git push origin main

There aren't many hard edges with this setup. You can put whatever you want into common and you can organize your devices however you want.

One exception is that secrets.yaml has some confusing implicit behavior, so I just commit an empty one and use a different file for my secrets.

Updating ESPHome is not something I do on a regular basis, but when I do it's basically just brew upgrade esphome && ./build.sh.

The process to add a new ESPHome device to Home Assistant is also fairly streamlined. All you do is attach the ESP32 device to your computer with USB, create a new yaml file, and run ./build.sh run <your new file>.yaml. ESPHome will pick up that there's a serially attached device without firmware and handle flashing the new firmware to it.

Once ESPHome is running on the new device it should show up in the Home Assistant integrations page and something that can be added. Clicking the accept button will open a config flow where you can paste your Home Assistant key, and then it should work just like any other device.

Why I don't live in Hungary

Bytepawn - Marton Trencseni — Wed, 07 Aug 2024 01:00:00 GMT

In February 2016, I made a life-changing decision to leave Hungary and seek opportunities abroad, starting in London and later moving to Dubai. This essay explores the multitude of reasons behind my choice and why I have no immediate plans to return to Hungary.

“How to let go: one life ends while another begins”

The Story's Story — Tue, 06 Aug 2024 21:11:16 GMT

“How to let go: one life ends while another begins. I’m seven months pregnant with our daughter as Jake’s life comes to a close. How do I walk into an uncertain future without him?“

“No Salt”

The Story's Story — Tue, 06 Aug 2024 01:23:22 GMT

This is by my brother, Sam. I arrived to Arizona late Saturday, after learning that my brother has only a few days left before cancer ends him. Jake’s wife, Bess, confessed that she had neither the willpower or the energy to take care of the post-death rituals—in this case, cremation, followed by a celebration of […]

Starting hospice. The end

The Story's Story — Mon, 05 Aug 2024 03:47:49 GMT

I’m entering hospice. It’s time, and realistically past time. The squamous cell carcinoma tumors are growing, and the two doses of spot radiation I got on June 10 and 12 have utterly destroyed whatever quality of life I had. This weekend, a nurse came by and did some planning with Bess and me. Our extensive […]

How to solve it

Bytepawn - Marton Trencseni — Thu, 01 Aug 2024 01:00:00 GMT

I discuss Polya's How to Solve it approach to mathematical problem solving and the 2010 paper Teaching general problem-solving skills is not a substitute for teaching math.

How I program in 2024

Kartik Agaram — Wed, 31 Jul 2024 21:52:44 GMT

I talk a lot here about using computers freely, how to select programs to use, how to decide if a program is trustworthy infrastructure one can safely depend on in the long term. I also spend my time building such infrastructure, because there isn't a lot of it out there. As I do so, I'm always acutely aware that I'm just not very good at it. At best I can claim I try to compensate for limited means with good, transparent intentions.

I just spent a month of my free time, off and on, rewriting the core of a program I've been using and incrementally modifying for 2 years. I've been becalmed since. Partly this is the regular cadence of my subconscious reflecting on what just happened, what I learned from it, taking some time to decide where to go next. But I'm also growing aware this time of a broader arc in my life:

Back in 2015 I was suspicious of abstractions and big on tests and version control. Code seemed awash in bad abstractions, while tests and versions seemed like the key advances of the 2000s. I thought our troubles stemmed from bad incentives, using abstractions too much, and not using tests and versions enough. Mu1 was an attempt at designing a platform with tests and layers (more like versions, less like abstractions) as foundational constraints influencing everything else.
In 2017 I started reworking Mu1 into the current Mu. At the start I used all my new ideas for tests and layers. But over time I stopped using them. Mu today has tons of tests, but they are conventional tests, and I never got around to porting over my infrastructure for layers.
In 2022 I started working on Freewheeling Apps. I started out with no tests, got frustrated at some point and wrote thorough tests for a core piece, the text editor. But I struggled to find ways to test the rest, and also found I was getting by fine anyway.
Now it's 2024, and a month ago I deleted all my tests. I also started radically reworking my text editor, in a way that would have made me worried about merge conflicts with other Freewheeling Apps. In effect I stopped thinking about version control. Giving up tests and versions, I ended up with a much better program. The cognitive dissonance is now impossible to ignore.

After mulling it over for a few days, I think my current synthesis on programming durable things is:

Building durably for lots of people is too hard, just don't even try. Be ruled by what you know well, who you know well and Dunbar's number.
Most software out there is incurably infected by incentives to serve lots of people in the short term. Focus as far as possible on software without lots of logos on the website, stuff that is easy to build, has few dependencies, definitely doesn't auto-update. Once you filter by these restrictions, the amount of durable software humanity has created so far is tiny.
Small changes in context (people/places/features you want to support) often radically change how well a program fits its context. Our dominant milieu of short-termism doesn't prepare us for this fact.
Given this tiny body of past work and low coverage per program, any new program you decide to build is quite likely striking out into the unknown in some way or other. You often won't know quite what you're doing in some direction or other. (In my example above, I was trying to insert special "drawing lines" in a text editor. Questions that raised: can the cursor lie on a drawing? Can I try to draw in one line while the cursor is on another? Drawings are taller than text lines. Can a drawing be partially visible at top of screen? Can I draw on a partially visible drawing? My answers to these questions were sub-optimal for a long time, leading to hacks piled on hacks.)
Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. Use them to taste.
You'll inevitably end up over-using some of these tools, the ones you gravitate towards. The ideal quantity to use these tools is tiny, much more miniscule than any of us is trained to think by our dominant milieu of short-termism. The excess is tech debt. It keeps us from noticing that a program is unnecessarily complex, less durable than it could be, harder to change when the context shifts.
When your understanding of the context stabilizes, there's value in throwing away vast swathes of a program, and redoing it from scratch.
Before you set out to rewrite, you have to spend some time importing everything into your brain at once. Everything you want from the program, all the scenarios the program has to cater to. This is hard. The goal is to get to a point where you can build everything all at once.
Build everything all at once.

In my case, tests and versions actively hindered getting to the end of this evolution. Tests let me forget concerns. Version control kept me attached to the past. Both were counter-productive. It took a major reorientation to let go of them.

All the software I've written in my life — and all my Freewheeling Apps so far — are at level 6 in this trajectory. Only the output of the past month feels like it might have gotten to level 9. We'll see.

It seems likely that a program can grow so complex it becomes impossible to import into memory in level 8. That seems to describe most software so far, certainly most software written by more than a couple of people. Even my text editor, small as it is, was daunting enough I spent much of the month girding myself to face the terror.

Not all software necessarily needs to get to level 9. I think many of my Freewheeling Apps are simple enough and evolve slowly enough that they would stabilize to a bug-free state with just a handful of people using them, regardless of my initial design choices. Particularly now that I know how to streamline one complex piece at their core. Still, it's good to be aware of how things might be improved, if it becomes worthwhile.

One thing that feels definitely useful in getting to level 9 is data-oriented design. It's not a tool you can blindly apply but a way of thinking you have to grow into, to look past immediate data structure choices at the big picture of how your program accesses data. Just don't let tools like ECS blind you to the essential intellectual activity.

These levels are probably not quite right. I'm probably under-estimating tools I have less experience with.

I wonder what levels lie beyond these.

(I last wrote some thoughts on how I program back in 2019. It's nice to see signs of evolution.)

Less Coffee, Better Sleep

Weblog on marginalia.nu — Wed, 31 Jul 2024 03:00:00 GMT

As an experiment, I’ve reduced my coffee-intake to a single cup a day for about a week now. It’s made an enormous difference in sleep, mood and energy. I get tired at night, fall asleep quickly, and wake up refreshed.

As mentioned previously in the context of morning sunlight exposure—another thing that’s aided my sleeping habits, but is somewhat less practical to sustain as it requires fair weather—I’ve always been slow to get going in the morning, active at night, bad at getting to bed at sane hours. Tired when I should be awake, and awake when I should be tired.

Automatic Cash Management

Pete Keen — Wed, 31 Jul 2024 03:00:00 GMT

The Setup

One Fidelity Brokerage Account ("brokerage")
One Fidelity Cash Management Account ("CMA")
One Fidelity 2% Cash Back Credit Card
One Chase Freedom Unlimited Credit Card
One Amazon Prime Visa

How Money Flows

Payroll direct deposited into brokerage
All non-mortgage recurring expenses are paid with the Fidelity card
All Amazon expenses are paid with the Amazon card
All other expenses are paid with the Chase card
Bills autopay from brokerage (credit cards, insurance, billpay to household vendors)
Debit cards and live paper checks written against CMA

Why

We use a brokerage account because it lets us keep cash and investments in the same account. All of our cash, including working capital and reserves, sits in the brokerage's core position. Our core position is FZFXX, a Federal money market fund that pays ~5% interest as of July 2024.

We use the Fidelity credit card because it pays 2% cash back when it's set up to deposit rewards into a Fidelity account. It is also one of the only cards I've seen that can be set to automatically cash out deposits. Ours is set to deposit into the brokerage account. We only use this card for recurring expenses. Non-recurring expenses go on the Chase card because in our experience Chase's fraud detection and customer support is considerably better than Fidelity/Elan.

We use the Amazon Prime credit card for all Amazon expenses. This pays 5% back on Amazon purchases which makes it worth it for us. Your milage may vary.

We have the CMA so we don't expose the brokerage account number every time we write a paper check. This is probably overly paranoid and is the only significant complication in the entire system.

The CMA has "self-funded overdraft protection" turned on which automatically transfers from the brokerage account into the CMA to fund checks and debit card transactions.

The brokerage core position is not FDIC insured. I don't care about FDIC insurance. FZFXX is composed of ultra short term US Treasury bills. If Treasuries are suddenly not liquid enough to withdraw our money our society has much bigger problems.

The CMA's core position is FDIC insured, and the CMA is almost a full brokerage account, but we don't use it as the centerpiece account for two reasons. First, the CMA core position pays shit for interest. Second, the CMA cannot have margin turned on. We'll talk about why that's important in the next post.

Automatic Investing

Pete Keen — Wed, 31 Jul 2024 03:00:00 GMT

The Setup

Fidelity Brokerage Account ("brokerage") with margin enabled
Fidelity Roth IRAs
Fidelity Solo 401K
Fidelity Multi-Asset Index Fund (FFNOX)
iShares Core Aggressive Allocation ETF

How Money Flows

Every investable dollar is in FFNOX across all non-401k tax advantaged accounts
Every investable dollar in taxable accounts is invested in AOA
Automatically contribute to my employer's 401K plan every pay period into whatever fund

Why

I don't want anyone to have to think about where to pull money from at any time. I want me or my wife to be able to login to Fidelity and sell enough to cover cash needs with a very small number of clicks.

FFNOX and AOA are funds of funds. FFNOX consists of low cost Fidelity funds and AOA consists of low cost Blackrock iShares ETFs. They both invest in approximately 60% US total stock market, 25% international developed total stock market, and 15% US total bond market. This fits our family's desired asset allocation.

The brokerage account has margin enabled. Margin allows you to borrow up to 50% of the value of your investable assets (everything but cash and CDs) from your broker for any purpose whatsoever. It kicks in while you run out of cash and will automatically pay itself back when you deposit cash in the account.

We have margin turned on so that we don't have to worry about selling investments to raise cash while something awful is happening. We can login to Fidelity and sell some FFNOX when it's convenient rather than having to do it one some kind of schedule.

Contributing to my employer's 401K plan is an automatic tax break and also means I get my employer's matching contribution. I currently (for 2024) have it set up to max out my employer's match.

We don't use robo-investors. FFNOX's total expenses are capped at 0.08%, or $80 per $100,000 annually. Wealthfront charges 0.25% (more than 3x) on top of the fees for the actual ETFs (typically ~0.1%). Robo-investors are never going to offset their fees when compared to FFNOX or similar funds (Target Retirement funds at Vanguard or Freedom Index funds at Fidelity).

Automatic Finances

Pete Keen — Wed, 31 Jul 2024 03:00:00 GMT

My friend Amy Hoy recently tweeted about financial planning and personal finance. This particular tweet stuck out to me:

you'd probably think given my love of money that i'd be all over financial planning, but truly i only enjoy earning it and spending it. saving it feels like… freezing cake. i guess you could do it, but why, you have cake
— Amy Hoy ✨ (@amyhoy) February 19, 2019

In early 2018 I radically simplified my family's personal financial system and made everything as automatic as possible. Amy and Joel Hooks asked me to write up how it works, so this is the start of a series of short posts about how and why I set everything up.

A Little Bit of Why

I prefer to think of myself as a realist. Due to my health history my wife is likely going to be around longer than me. Her family has some very long lived female members as well. Her grandmother is 103 and her great aunt just passed in 2018 at the ripe old age of 95.

I want to make our finances as simple as possible so she doesn't have to worry about them when the inevitable happens.

In 2016 my wife and I welcomed our first child into the world and in late 2018 we welcomed our second. They are two more very important reasons why I want things to be simple. If something happens to both my wife and me, I want our intentions with regards to our finances as plain as possible.

This system got a trial run in late 2018. My wife was admitted to the hospital at 29 weeks pregnant for preeclampsia, a very dangerous condition that needs close monitoring. My daugther was born at 34 weeks and spent the next five weeks in NICU.

I didn't have to touch this system at all. Not once. I logged in a handful of times to check up on it, but everything just hummed along.

Delightful, production-grade replication for Postgres

Notes on software development — Tue, 30 Jul 2024 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

More isn’t always better: death and over-treatment as a downside of agenticness

The Story's Story — Mon, 29 Jul 2024 15:30:00 GMT

i. Aren’t we clever? Bess and I felt so very smart. In April, we congratulated ourselves for navigating the healthcare system to get two doses of spot radiation that, in conjunction with clinical trial drug PDL1V, likely shrunk, and perhaps temporarily eliminated, the squamous cell carcinoma tumors in my neck that have been strangling me […]

Ingest: Streamlining Content Preparation for LLMs

smcleod.net — Mon, 29 Jul 2024 04:00:10 GMT

A CLI tool for parsing directories into LLM-friendly markdown

Building a qBittorrent client with SwiftUI

ブラック — Sat, 27 Jul 2024 03:00:00 GMT

I recently got convinced to switch from rTorrent to qBittorrent for my Linux-downloading needs. Tried it out for a bit and decided that it was indeed the superior of the two, meaning I fully migrated all of my Linux ISOs to it. Unfortunately, I didn't really like any of the available clients for it, so I asked myself what any good software engineer would at that point: how hard can it be?

I first thought about writing one in Rust, since I've had some experience with it lately, first building an automatic file sorter and later an anime scrobbler, but after a cursory look at Cursive and Ratatui, I buried that idea. Cursive just seemed like a bad fit and while Ratatui seemed like it could be a good fit for building my application, the tutorials were in middle of a rewrite and I didn't want to spend too much time learning from examples that were already abandoned. Rust and TUIs are tricky enough as they are without a bunch of outdated documentation.

Since I didn't want to make yet another web client for qBittorrent, and I'd preferably have something with at least decent performance, I figured I could try writing a native desktop application. And since I'm on macOS and Swift is actually a really good language, Swift it was. I've had a bit of experience in writing Mac apps in Swift before, but I figured that while I was pursuing my half-baked idea, I might as well learn something new while I was at it.

Cue SwiftUI, Apple's hot new declarative user interface framework that's actually not that new since it was released almost five years ago. Not sure if it's even that hot since I never hear anyone actually using it to ship products. Well, it's the thought that counts anyways.

Getting started

The first thing I noticed about SwiftUI applications is that they take absolutely ages to build. People complaining about Rust having long compile times should try SwiftUI to get some perspective. Granted, most Rust programs don’t have graphical user interfaces, but it’s still quite a grating development experience. Change one line and it takes like 30–120 seconds to get a new debug build running on an M1 Max. I’ve done some stuff with Storyboard and XIB interfaces before and I do not remember any of them being nearly as bad.

I first thought that the long build times were just a me problem, but after bitching about it on Bluesky, I did get another developer chime in to confirm that yes, SwiftUI takes a long time to build.

The other thing that I quite quickly noticed was that SwiftUI does produce quite a lot of annoying and obscure roadblocks for you to decipher. For example, if you fuck up your types and are writing views that have any girth to them at all, Swift will just shame you about your massive views without telling what type error you've produced. That's a super fun one. Apparently Swift's type checker is just so slow that it decides to mercy kill the whole operation.

I've also had Xcode hard crash quite a few times while building out my user interface. I imagine most of them have been triggered by the preview window rendering things. Thankfully I've never lost any work from these crashes, only wasted a bit of time.

Redraw woes

One thing about SwiftUI's automagic logic of updating your views whenever the application state changes is that it has some very definite flaws in it. I wanted to show my data in a table view where the table rows are selectable, but because I also have OCD, I want to be able to deselect rows to get my application back to a serene unselected state. In order to fulfil my desires, I figured that I'd add a keypress handler to my table so that every time Escape is pressed, the table's selection binding is cleared and my selection would be gone. Basically, like this:

struct TableView: View {
    @EnvironmentObject var dataModel: DataModel

    @State private var tableSelection = Set<RowObject.ID>()

    var body: some View {
        Table(dataModel.tableCollection, selection: $tableSelection) { ... }
        .onKeyPress(.escape) {
            tableSelection.removeAll()
            return .handled
        }
    }
}

Good idea in theory. You update tableSelection that is the selection binding for the Table view and the table should redraw without any rows selected. Unfortunately, while it does clear the selection, the table won't show the selection disappearing until the data source for the table updates. Since I'm building a torrent client that fetches the current state every 𝓃 seconds, it might take up to 𝓃 seconds for the deselection to be visible to the user after they press it. Not great.

So, what if I just forced the data source to update? Would that work? Yeah, kinda.

struct TableView: View {
    @EnvironmentObject var dataModel: DataModel

    @State private var tableSelection = Set<RowObject.ID>()

    var body: some View {
        Table(dataModel.tableCollection, selection: $tableSelection) { ... }
        .onKeyPress(.escape) {
            tableSelection.removeAll()
            dataModel.objectWillChange.send()
            return .handled
        }
    }
}

By sending the objectWillChange event for my table's data source, I am able to force the Table to redraw, which will make the deselection action feel and look responsive. Worked absolutely perfectly in my testing. So where's the kinda part?

Unfortunately this (seemingly?) functional code is actually undefined behaviour and Xcode will give you a big warning saying it's verboten. So what can I do instead?

struct TableView: View {
    @EnvironmentObject var dataModel: DataModel

    @State private var tableSelection = Set<RowObject.ID>()

    var body: some View {
        Table(dataModel.tableCollection, selection: $tableSelection) { ... }
        .onKeyPress(.escape) {
            DispatchQueue.main.async {
                tableSelection.removeAll()
                dataModel.objectWillChange.send()
            }
            return .handled
        }
    }
}

Turns out that the solution is to wrap the code in DispatchQueue.main.async and you're golden. It works perfectly fine and gives zero errors. Not sure if this is actually much better than the previous solution, since we're just kicking off the code to be executed asynchronously in the main thread instead of doing it here and now. Shouldn't even be any difference in the thread since onKeyPress() handler should already be on the main thread, as calling DispatchQueue.main.sync instead will horribly crash your application. Would definitely be better if I could somehow either force the Table itself to update itself or have it update automatically whenever its selection changes, but at least there's some way to do it.

Is this supposed to work like this?

One thing that I wasn't really fond of in the default qBittorrent user interface is that you can't easily remove a torrent and delete its data at the same time, since you always have to separately click a checkbox to delete files. Hence, my client would have two separate actions: remove and remove + delete. And those actions should be possible with keyboard alone.

However, since it's quite risky to delete things without any confirmation, I decided to add a confirmation prompt before anything happens. Finder also does a very similar thing when you delete files on a network share, so it should feel pretty natural on macOS.

Thankfully this is fairly easy in SwiftUI. Just added a confirmationDialog to my view with a boolean binding that controls if the confirmation dialog is presented or not, with some customisations to boot.

.confirmationDialog(
    "Remove torrent and delete data?", isPresented: $showDeleteConfirmation
) {
    Button("Remove and delete data", role: .destructive) {
        // Delete logic here.
    }

    Button("Cancel", role: .cancel) {
        // Cancel logic here.
    }
}
.dialogIcon(Image(systemName: "trash.circle.fill"))
.dialogSeverity(.critical)

The end result is very similar to what you get from Finder, except that my button has too much text for the buttons to fit on one row so they're stacked. Had I opted for a shorter "Delete" instead, I'd get them on one row.

Unfortunately what was less simple was the keyboard shortcuts. You get Escape as the cancel button for free by just having your cancel button defined with a "cancel" role, but the main action button isn't free. From what I've gathered is that you're supposed to use Button.keyboardShortcut(.defaultAction) to define that a button is the main button and the default key binding for the given scenario should be used. For a regular confirmation dialog, that button is naturally Enter.

.confirmationDialog(...) {
    // This button is highlighted in blue and can be activated with Enter.
    Button("Delete") { ... }.keyboardShortcut(.defaultAction)

    // This button can be activated with Escape.
    Button("Cancel", role: .cancel) { ... }
}

But if you want to create a destructive confirmation dialog like what you find in Finder, there seemingly does not exist a keybinding that would allow you to press that button – even though Finder lets you do it with ⌘D. At least I couldn't find one, and even if I did, it'd be so obscure that no Mac user would actually stumble upon it.

.confirmationDialog(...) {
    // This button is highlighted in red and can be activated with ???.
    Button("Delete", role: .destructive) {
        ...
    }.keyboardShortcut(.defaultAction)

    // This button can be activated with Escape.
    Button("Cancel", role: .cancel) { ... }
}

I imagine that this is not intended, and that there should exist a .defaultAction shortcut that you can use with destructive confirmation dialogs. But since iOS is very light on keyboard shortcuts (I think you can use them if you have an iPad with a keyboard case?), I imagine that these sorts of things are just not a priority to Apple, as macOS is not where the money is. So after I'd determined that I probably wasn't just being dumb, I opted to just do the obvious thing instead:

.confirmationDialog(...) {
    // This button is highlighted in red and can be activated with Cmd+D.
    Button("Delete", role: .destructive) {
        ...
    }.keyboardShortcut(KeyEquivalent("D"), modifiers: .command)

    // This button can be activated with Escape.
    Button("Cancel", role: .cancel) { ... }
}

Hope that Apple never changes how the one in Finder works.

It it any good? What's it for?

After I'd spent a week or two working on this project, I opened up an old Storyboard-based project of mine since I had to make some updates to it. More specifically, I had to add a new text field to a settings section, which was to be located underneath an existing text field. And while I'd always considered the Xcode Interface Builder to be fairly easy and fun, aligning things and adding constraints manually did feel extremely silly after having worked in SwiftUI. Why am I doing all of this silly manual work, dragging and dropping fields and setting alignment constraints, when I could just do this?

Form {
    TextField(text: $old, prompt: Text("Old and busted")) {
        Text("Old and busted")
    }
    TextField(text: $new, prompt: Text("New hotness")) {
        Text("New hotness")
    }
}

Fast, simple, concise, automatic alignment. What's not to love?

On the other hand, whenever I did something that SwiftUI wasn't really designed to do, like have multiple lists in a NavigationSplitView sidebar, then it's a massive pain. Just getting this janky piece of shit solution working took a long while:

It's kinda hard to see why SwiftUI exists in the first place. I don't actually have any data but I'm pretty sure native desktop and mobile applications are on a steep decline, with companies electing to ship Electron or React Native apps instead of writing native applications for macOS, Windows, Linux, iOS and Android separately. Writing a truly native application is only for the biggest of companies, who can afford to write a handful of separate applications, or small boutique shops, who are targetting a niche. Perhaps Apple engineers are just building it for themselves after getting tired of having to deal with multiple different UI systems at the same time, since everything they make needs to ship on iOS, iPadOS and macOS. They even ship the Calculator app on the iPadOS nowadays.

Glad that it exists though. For all the frustrations I had during my time working with SwiftUI, I did actually find it pretty nice and fun. My gaming PC went unused for days during my busiest days building my app. I definitely want to try doing more native Apple development in the future. I want to try and port my client to iPadOS at some point in the future too, since it shouldn't be a massive undertaking with SwiftUI. Not that I'd be able to distribute it unless there's some drastic changes to iPadOS distribution coming, since I don't pay for Apple's developer program. No signing or notarisation for my apps.

Although the next time I try to build anything that requires HTTP, I'm going to start immediately with Alamofire instead of trying to make do with URLSession. Anything more than GET and you'll find yourself building another Alamofire.

The app

All in all, I did manage to actually make something: a very bare-bones qBittorrent client called Dreadnought. I even managed to make a shitty application icon for it! Progress has been extremely slow lately though, since I hit a point where I managed to make an application that does like 80% of the things I need in the qBittorrent Web UI, so everything else is more work for increasingly smaller gains.

I do use it daily though, so it definitely serves someone's purpose, but it's hard to recommend to anyone that is not me, myself nor I. Someone might even call this a feature, since building apps with no general use means getting no support requests. I also wouldn't be surprised if the state updates are expensive as hell and if it was leaking memory, since I have done basically no performance testing. Well if nothing else, at least it ships without a copy of Chromium bundled inside it.

The blog posts I haven't written (yet)

Austin Z. Henley's Blog — Sun, 21 Jul 2024 21:00:01 GMT

https://austinhenley.com/blog/notyetwritten.html

LLM Parameter Playground

smcleod.net — Sat, 20 Jul 2024 04:00:10 GMT

A web app to explore the effects of different inference parameters on LLMs.

Code, Chaos, and Copilots (AI/LLM Talk July 2024)

smcleod.net — Thu, 18 Jul 2024 04:00:10 GMT

Integrating AI into Daily Development

Links: No one wants to talk about death, deep takes on the human condition, and more!

The Story's Story — Wed, 17 Jul 2024 06:24:37 GMT

* “The Silence Doctors Are Keeping About Millennials’ Death.” Germane to me for obvious reasons; our culture denies both death and grief and seems not able to incorporate either, particularly when death is premature. My own end feels so close, particularly because my days, such as they are, are filled with pain and exhaustion. * “How […]

Understanding AI/LLM Quantisation Through Interactive Visualisations

smcleod.net — Wed, 17 Jul 2024 04:00:10 GMT

AI/LLM Quantisation Visualised

My Fave Podcasts 2024 - The Big Reset

swyx's site RSS Feed — Mon, 15 Jul 2024 23:16:06 GMT

4.5 years after my initial podcast inventory, with COVID and a career change under my belt, with my OPML file growing from 566 rss feeds in 2022 to 771 in 2024 (with quite a few of my fave NPR podcasts dying), and my backlog mounting to 43GB (according to this script, my outstanding episodes: 11915 (played: 22451)), I am now needing to declare podcast bankruptcy.

Story Points are Pointless, Measure Queues

Brightball Articles — Mon, 15 Jul 2024 13:05:55 GMT

Their creator has disavowed them. People cannot agree on what a story point even represents. The measure is different for every team that uses it. They sow confusion, create conflict, unreliable timelines, are easily gamed, demotivate and degrade the performance of your team.For everyone involved, this is a waste of time. Let's deep dive into why Story Points are so broken and how to avoid dealing with them ever again.

Can ChatGPT do data science?

Austin Z. Henley's Blog — Mon, 15 Jul 2024 00:00:01 GMT

https://austinhenley.com/blog/chatgptdatascience.html

Links: Slowness as a problem, kinds of conversations, curing cancer, and more!

The Story's Story — Sun, 14 Jul 2024 22:11:05 GMT

* “Pentagon ‘alarmingly slow’ at fielding new weapons.” China seems not to suffer this problem. Perhaps, given what’s happening with Taiwan, we should pay more attention. In other “China moves fast” news: “China State Shipbuilding Corporation is the world’s largest shipbuilder. It builds vessels for the People’s Liberation Army Navy and increasingly sophisticated dual-use commercial […]

The two-crisis update

The Story's Story — Thu, 11 Jul 2024 05:40:56 GMT

Five days ago I wrote about two simultaneous crises: one from swallowing and breathing poorly, and the second from PDL1V no longer holding back the tumors. Despite those beautiful scans in May showing extensive tumor-size reduction, the most recent scans show “Continued neoplastic disease progression throughout the neck as described. Of note tumor results in […]

Rejoining academia

Austin Z. Henley's Blog — Wed, 10 Jul 2024 23:00:01 GMT

https://austinhenley.com/blog/rejoiningacademia.html

The CP/M emulator is good enough, I think.

Steve Kemp's Blog — Tue, 09 Jul 2024 23:00:00 GMT

My previous post mentioned that I'd added some custom syscalls to my CP/M emulator and that lead to some more updates, embedding a bunch of binaries within the emulator so that the settings can be tweaked at run-time, for example running:

!DEBUG 1
!CTRLC 1
!CCP ccpz
!CONSOLE adm-3a

Those embedded binaries show up on A: even if they're not in the pwd when you launch the emulator.

Other than the custom syscalls I've updated the real BDOS/BIOS syscalls a bit more, so that now I can run things like the Small C compiler, BBC BASIC, and more. (BBCBasic.com used to launch just fine, but it turned out that the SAVE/LOAD functions didn't work. Ooops!)

I think I've now reached a point where all the binaries I care about run, and barring issues I will slow down/stop development. I can run Turbo Pascal, WordStar, various BASIC interpreters, and I have a significantly improved understanding of how CP/M works - a key milestone in that understanding was getting SUBMIT.COM to execute, and understanding the split between the BDOS and the BIOS.

I'd kinda like to port CP/M to a new (Z80-based) system - but I don't have such a thing to hand, and I guess there's no real need for it. Perhaps I can say I'm "done" with retro stuff, and go back to playing Super Mario Bros (1985) with my boy!

Optimizing a bignum library for fun

Austin Z. Henley's Blog — Tue, 09 Jul 2024 22:00:01 GMT

https://austinhenley.com/blog/bignum2.html

A reawakening of systems programming meetups

Notes on software development — Sun, 07 Jul 2024 03:00:00 GMT

This year has seen a resurgence in really high quality systems programming meetups. Munich Database Meetup, Berlin Systems Group, SF Distributed Systems Meetup, NYC Systems, Bengaluru Systems, to name a few.

This post summarizes a bit of disappointing recent tech meetup history, the new trend of excellent systems programming meetups, and ends with some encouragement and guidance for running your own systems programming events.

I will be a little critical in this post but I want to preface by saying: organizing meetups is really tough! It takes a lot of work and I have a huge amount of respect for meetup organizers even when their meetup style did not resonate with me.

Although much of this post talks about NYC Systems, the reason I think this post is worth writing is because so many other meetups in a similar vein popped up. I hope to encourage these other meetups and to encourage folks in other major metros (London, for example) to start similar meetups.

Meetups

I used to attend a bunch of meetups before the pandemic. But I quickly got disillusioned. Almost every meetup was varying degrees of startups pitching their product. The last straw for me was sitting through a talk at a JavaScript meetup that was by a devrel employee of a startup who literally gave a tutorial for their product.

There were also some pretty intelligent meetups like the New York Haskell Users Group and the New York Emacs Meetup. But not being an expert in either domain, and the attendees almost solely appearing to be experts, I didn't particularly enjoy going.

There were a couple of meetups that felt inclusive for various skill-levels of attendees yet still went into interesting depth. Specifically, New York Linux User Group and Papers We Love NYC.

These meetups were exceptional because they were language- and framework-agnostic, they would start broad to give you background, but then go deep into a topic. Maybe you only understood 50% of what was covered. But you get exposed to something new from an expert in that domain.

Unfortunately, the pandemic happened and these two excellent meetups basically have not come back.

A couple of students in Munich

The pandemic ended and I tried a couple of meetups I thought might be better quality. Rust and Go. But they weren't much better than I remembered. People would give a high level talk and brush over all the interesting concepts.

I had been thinking of doing an in-person talk series since 2022.

If I put together a systems/databases/distributed systems meetup in NYC (a physical meetup, not Zoom), who'd be interested (in attending, or presenting, or helping me organize, or donating space)?

No promises!
— Phil Eaton (@eatonphil) September 27, 2022

But I was busy with TigerBeetle until December of 2023 when I was messaged on LinkedIn by Georg Kreuzmayr, a graduate student at Technical University of Munich (TUM).

Georg and his friends, fellow graduate students at TUM, started a database club: TUMuchData. We got to talking about opportunities for collaboration and I started feeling a bit embarrassed that a graduate student had more guts than I had to get back onto the meetup organizer wagon.

A week later, with assurance from Justin Jaffray that at least he would show up with me if no one else did, I started the NYC Systems Coffee Club to bring together folks in NYC interested in any topic of systems programming (e.g. compilers, databases, web browser internals, distributed systems, formal methods, etc.). To bring them together in a completely informal setting for coffee at 9am in the morning in a public space in midtown Manhattan.

Trying something new! If you're a dev in NYC working
on (or interested in) systems programming, grab a coffee and come hang out at 1 Bryant Park (indoor space) this Thursday 9AM - 9:30AM.

See post for details and fill out the Google Form or DM me!https://t.co/A4bzcPGy6x pic.twitter.com/n1ECMd59ev
— Phil Eaton (@eatonphil) December 11, 2023

I set up that linked web page and started collecting subscribers to the club via Google Form. Once a month I'd send an email out to the list asking for RSVPs to this month's coffee club. The first 20 to respond would get a calendar invite.

And about the same time I started asking around on Twitter/LinkedIn if someone would be interested in co-organizing a new systems programming meetup in NYC. Angelo Saraceno immediately took me up on the idea and we met up.

NYC Systems

We agreed on the premise: this would be a language- and framework-agnostic meetup that was focused on engineering challenges, not product pitches. It would be 100% for the sake of corporate marketing, but corporate marketing of the engineering team, not the product.

NYC Systems was born!

We'd find speakers who could start broad and dive deep into some interesting aspect of databases, programming languages, distributed systems, and so on. Product pitches were necessary to establish a context, but the focus of the talk would be about some interesting recent technical challenge and how they dealt with it.

We'd schedule talks only every other month to ease our own burden in organizing and finding great speakers.

Once Angelo and I had decided to go forward, the next two challenges were finding speakers and finding a venue. Thanks to Twitter and LinkedIn, finding speakers turned out to be the easy part.

It was harder to find a venue. It was surprisingly challenging to find a company in NYC with a shared vision that the important thing about being associated with a meetup like this is to be associated with the quality of speakers and audience we can bring in by not allowing transparent product pitches.

Almost every company in Manhattan with space we spoke with had a requirement that they have their own speaker each night. That seemed like a bad idea.

I think it was especially challenging to find a company willing to relax about branding requirements like this because we were a new meetup.

It was pretty frustrating not to find a sympathetic company with space in Manhattan. And the only reason we didn't give up was because Angelo was so adament that this kind of meetup actually happen. It's always best to start something new with someone else for this exact reason. You can keep each other going.

In the end we went with the company that did not insist on their own speaker or their own branding. A Brooklyn-based company whose CEO immediately got in touch with me that they wanted to host us, Trail of Bits.

How it works

To keep things easy, I set up a web page on my personal site with information about the meetup. (Eventually we moved this to nycsystems.xyz.) I set up a Google Form to collect emails for a mailing list. And we started posting about the group on Twitter and LinkedIn.

Very pleased to share the first NYC Systems Talks are taking place next Thursday Feb 22nd 6PM. Hosted by @trailofbits, with @paulgb and @StefanKarpinski speaking.

Space is not infinite, fill out the Google Form if you can attend and would like an invite!https://t.co/jNssr5v1kJ
— Phil Eaton (@eatonphil) February 15, 2024

We published the event calendar in advance (an HTML table on the website) and announced each event's speakers a week in advance of the event. I'd send another Google Form to the mailing list taking RSVPs for the night. The first 60 people to respond got a Google Calendar invite.

It's a bit of work, sure, but I'd do anything to avoid Meetup.com.

It is interesting to see every new systems programming meetup also not pick Meetup.com. The only one that went with it, Munich Database Meetup, is a revival of an existing group, the Munich NoSQL Meetup and presumably they didn't want to give up their subscribers. Though most others use lu.ma.

The mailing list is now about 400+ people. And in each event RSVP we have a wait list of 20-30 people. Of course although 60 people say Yes initially, by the time of the event we have typically gotten about 50 people in attendance.

At each event, Trail of Bits provided screens, chairs, food, and drink. Angelo had recording equipment so he took over audio/video capturing (and later editing and publishing).

After each event we'd publish talk videos to our @NYCSystems Youtube.

Network effects

In March 2024, the TUMuchData folks joined Alex Petrov's Munich NoSQL Meetup to form the Munich Database Meetup. In May, Kaivalya Apte and Manish Gill started the Berlin Systems Group, inspired by Alex and the Munich Database Meetup.

I want to start a Berlin Database/Storage systems group, where we have regular meetups, discussions and talks.

WDYT? @mgill25 @mehd_io @ClickHouseDB @SnowflakeDB @awscloud @GoogleDE @TUBerlin

Can I get some support? Who else would be interested? #Databases

Thanks…
— Kaivalya Apte - The Geek Narrator (@thegeeknarrator) May 15, 2024

In May 2024, two PhD students in the San Francisco Bay Area, Shadaj Laddad and Conor Power, started the SF Distributed Systems meetup.

We’re super excited to be organizing a new SF Distributed Systems meetup NEXT WEEK! Our first meetup features @julianhyde and @conor_power23 presenting work on extending SQL and applying algebraic properties, sign up at https://t.co/d2lLDaQ5iJ
— Shadaj Laddad (@ShadajL) May 15, 2024

And in July 2024, Shraddha Agrawal, Anirudh Rowjee and friends kicked off the first Bengaluru Systems Meetup.

Are you ready, Systems Enthusiasts of Bengaluru?

Speaking at our first-ever meetup on 6th July, we have:@simsimsandy with "Learn about the systems that power GenAI applications" and @vivekgalatage with "The Browser Backstage: Performance vs Security"
(talks linked below!)
— Bengaluru Systems Meetup (@BengaluruSys) July 4, 2024

Suggestions

First off, don't pay for anything yourself. Find a company who will host. At the same time, don't feel the need to give in too much to the demands of the company. I'd be happy to help you think through how to talk about the event with companies. It is mutually beneficial for them to get to give a 5-minute hiring/product pitch and not need to do extensive branding nor to give a 30-minute product tutorial.

Second, keep a bit of pressure on speakers to not do an overview talk and not to do a product pitch. Suggest that they tell the story of some interesting recent bug or interesting recent feature. What happened? Why was it hard? What did you learn?

Focusing on these types of talks will help you get a really interesting audience.

I have been continuously surprised and impressed at the folks who show up for NYC Systems. It's a mix of technical founders in the systems space, pretty experienced developers in the systems space, graduate students, and developers of all sorts.

I am certain we can only get these kinds of folks to show up because we avoid product pitch-type talks.

Third, finding speakers is still hard! The best approach so far has been to individually message folks in industry and academia who hang out on Twitter. Sending out a public call is easy but doesn't often pan out. So keep an eye on interesting companies in the area.

Another avenue I've been thinking about is messaging VC connections to ask them if they know any engineers/technical founders/CTOs in the area who could give an interesting technical talk.

Fourth, speak with other organizers! I finally met Alex Petrov in person last month and we had a great time talking about the challenges and joys of organizing really high quality meetups.

I'm always happy to chat, DMs are open.

New post telling a bit of the history behind https://t.co/NEh1tm8v3Q; why it only exists due to folks like @georg_kreuzmayr and @ngeloxyz; the explosion of systems meetups around the world; and encouragement and suggestions for future organizers!https://t.co/dwe4TtmXKK pic.twitter.com/ZMLkVYdZDJ
— Phil Eaton (@eatonphil) July 7, 2024

Links: Good-tasting berries, other biology news, being the future you want to lead, and more!

The Story's Story — Sat, 06 Jul 2024 17:03:53 GMT

* “Why America’s Berries Have Never Tasted So Good: Driscoll’s had to figure out how to breed, produce and sell its most flavorful strawberries and raspberries. Now the strategy is starting to bear fruit” (wsj, $). Good news is underrated and seemingly hardly ever goes viral. I think that even though I’m very sick and […]

Very sick, and two crises hit at once

The Story's Story — Sat, 06 Jul 2024 02:39:42 GMT

I’ve been silent because I’ve been so very sick: swallowing has gotten much harder, and aspiration of food or liquid much more common, to the point that on July 1 I got a new PEG tube put in. Given how much removing the previous tube felt like a triumph, the new installation, although it’s keeping […]

Rating AI Tools

smcleod.net — Wed, 03 Jul 2024 04:00:10 GMT

Subjective ratings of the notable AI tools I've tried.

Benchmarking rust compilation speedups and slowdowns from sccache and -Zthreads

The NeoSmart Files — Mon, 01 Jul 2024 21:50:08 GMT

Just a PSA from one rust developer to another: if you use sccache, take a moment to benchmark a clean build1 of your favorite or current project and verify whether or not having RUSTC_WRAPPER=sccache is doing you any favors. I’ve … Continue reading →

The post Benchmarking rust compilation speedups and slowdowns from sccache and -Zthreads first appeared on The NeoSmart Files.

A write-ahead log is not a universal part of durability

Notes on software development — Mon, 01 Jul 2024 03:00:00 GMT

A database does not need a write-ahead log (WAL) to achieve durability. A database can write its long-term data structure durably to disk before returning to a client. Granted, this is a bad idea! And granted, a WAL is critical for durability by design in most databases. But I think it's helpful to understand WALs by understanding what you could do without them.

So let's look at what terrible design we can make for a durable database that has no write-ahead log. To motivate the idea of, and build an intuition for, a write-ahead log.

Thank you to Alex Miller for reviewing a version of this post.

But first, what is durability?

Durability

Durability happens in the context of a request a client makes to a data system (either an embedded system like SQLite or RocksDB or a standalone system like Postgres). Durability is a spectrum of guarantees the server provides when a client requests to write some data: that either the request succeeds and the data is safely written to disk, or the request fails and the client must retry or decide to do something else.

It can be difficult to set an absolute definition for durability since different databases have different concepts of what can go wrong with disks (also called a "storage fault model"), or they have no concept at all.

Let's start from the beginning.

An in-memory database

An in-memory database has no durability at all. Here is pseudo-code for an in-memory database service.

db = btree()

def handle_write(req):
  db.update(req.key, req.value)
  return 200, {}

def handle_read(req):
  value = db.read(req.key)
  return 200, {"value": value}

Throughout this post, for the sake of code brevity, imagine that the environment is concurrent and that data races around shared mutable values like db are protected somehow.

Writing to disk

If we want to achieve the most basic level of durability, we can write this database to a file.

f = open("kv.db")
db = btree.init_from_disk(f)

def handle_write(req):
  db.update(req.key, req.value)
  db.write_to_disk(f)
  return 200, {}

def handle_read(req):
  value = db.read(req.key)
  return 200, {"value": value}

btree.write_to_disk will call pwrite(2) under the hood. And we'll assume it does copy-on-write for only changed pages. So imagine we have a large database represented by a btree that takes up 10GiB on disk. With the btree algorithm, if we write a single entry to the btree, often only a single (often 4Kib) page will get written rather than all pages (holding all values) in the tree. At the same time, in the worst case, the entire tree (all 10GiB of data) may need to get rewritten.

But this code isn't crash-safe. If the virtual or physical machine this code is running on reboots, the data we wrote to the file may not actually be on disk.

fsync

File data is buffered by the operating system by default. By general consensus, writing data without flushing the operating system buffer is not considered durable. Every so often a new database will show up on Hacker News claiming to beat all other databases on insert speed until a commenter points out the new database doesn't actually flush data to disk.

In other words, the commonly accepted requirement for durability is that not only do you write data to a file on disk but you fsync(2) the file you wrote. This forces the operating system to flush to disk any data it has buffered.

f = open("kv.db")
db = btree.init_from_disk(f)

def handle_write(req):
  db.update(req.key, req.value)
  db.write_to_disk(f)
  f.fsync() # Force a flush
  return 200, {}

def handle_read(req):
  value = db.read(req.key)
  return 200, {"value": value}

Furthermore you must not ignore fsync failure. How you deal with fsync failure is up to you, but exiting immediately with a message that the user should restore from a backup is sometimes considered acceptable.

Databases don't like to fsync because it's slow. Many major databases offer modes where they do not fsync data files before returning a success to a client. Postgres offers this unsafe mode, though does not default to it and warns against it. MongoDB offers this unsafe mode but does not default to it.

An earlier version of this post said that MongoDB would unsafely flush on an interval. Daniel Gomez Ferro from MongoDB messaged me that while the docs are confusing, the default write concern "majority" does actually imply "j: true" which means data is synchronized (i.e. fsync-ed) before returning a success to a client.

Almost every database trades safety for performance in some regard. For example, few databases but SQLite and Cockroach default to Serializable Isolation. While it is commonly agreed that basically no level below Serializable Isolation (that all other databases default to) can be reasoned about. Other databases offer Serializable Isolation, they just don't default to it. Because it can be slow.

Group commit

But let's get back to fsync. One way to amortize the cost of fsync is to delay requests so that you write data from each of them and then fsync the data from all requests. This is sometimes called group commit.

For example, we could update the database in-memory but have a background thread serialize to disk and call fsync only every 5ms.

f = open("kv.db")
db = btree.init_from_disk(f)

group_commit_sems = []

@background_worker()
def group_commit():
  for:
    if clock() % 5ms == 0:
      db.write_to_disk(f)
      f.fsync() # Durably flush for the group
      for sem in group_commit_sems:
        sem.signal()

def handle_write(req):
  db.update(req.key, req.value)
  sem = semaphore()
  group_commit_sems.push(sem)
  sem.wait()
  return 200, {}

def handle_read(req):
  value = db.read(req.key)
  return 200, {"value": value}

It is critical that handle_write waits to return a success until the write is durable via fsync.

So to reiterate, the key idea for durability of a client request is that you have some version of the client message stored on disk durably with fsync before returning a success to a client.

From now on in this post, when you see "durable" or "durability", it means that the data has been written and fsync-ed to disk.

Optimizing durable writes

A key insight is that it's silly to serialize the entire permanent structure of the database to disk every time a user writes.

We could just write the user's message itself to an append-only log. And then only periodically write the entire btree to disk. So long as we have fsync-ed the append-only log file, we can safely return to the user even if the btree itself has not yet been written to disk.

The additional logic this requires is that on startup we must read the btree from disk and then replay the log on top of the btree.

f = open("kv.db", "rw")
db = btree.init_from_disk(f)

log_f = open("kv.log", "rw")
l = log.init_from_disk()
for log in l.read_logs_from(db.last_log_index):
  db.update(log.key, log.value)

group_commit_sems = []

@background_worker()
def group_commit():
  for:
    log_accumulator = log_page()
    if clock() % 5ms == 0:
      for (log, _) in group_commit_sems:
        log_accumulator.add(log)

      log_f.write(log_accumulator.page()) # Write out all log entries at once
      log_f.fsync() # Durably flush wal data
      for (_, sem) in group_commit_sems:
        sem.signal()

    if clock() % 1m == 0:
      db.write_to_disk(f)
      f.fsync() # Durably flush db data

def handle_write(req):
  db.update(req.key, req.value)
  sem = semaphore()
  log = req
  group_commit_sems.push((log, sem))
  sem.wait() # This time waiting for only the log to be written and flushed, not the btree.
  return 200, {}

def handle_read(req):
  value = db.read(req.key)
  return 200, {"value": value}

This is a write-ahead log!

Consider a few scenarios. One request writes the smallest key ever seen. And one request within the same millisecond writes the largest key ever seen. Writing these to disk on the btree means modifying at least two pages spread out in space on disk.

But if we only have to durably write these two messages to a log, they can likely both be included in the same log page. ("Likely" so long as key and values are small enough that multiple can fit into the same page.)

That is, it's cheaper to write only these small messages representing the client request to disk. And we save the structured btree persistence for a less frequent durable write.

Filesystem and disk bugs

Sometimes filesystems will write data to the wrong place. Sometimes disks corrupt data. A solution to both of these is to checksum the data on write, store the checksum on disk, and confirm the checksum on read. This combined with a background process called scrubbing to validate unread data can help you learn quickly when your data has been corrupted and you must recover from backup.

MongoDB's default storage engine WiredTiger does checksum data by default.

But some databases famous for integrity do not. Postgres does no data checksumming by default:

By default, data pages are not protected by checksums, but this can optionally be enabled for a cluster. When enabled, each data page includes a checksum that is updated when the page is written and verified each time the page is read. Only data pages are protected by checksums; internal data structures and temporary files are not.

SQLite likewise does no checksumming by default. Checksumming is an optional extension:

The checksum VFS extension is a VFS shim that adds an 8-byte checksum to the end of every page in an SQLite database. The checksum is added as each page is written and verified as each page is read. The checksum is intended to help detect database corruption caused by random bit-flips in the mass storage device.

But even this isn't perfect. Disks and nodes can fail completely. At that point you can only improve durability by introducing redundancy across disks (and/or nodes), for example, via distributed consensus.

Other reasons you need a WAL?

Some databases (like SQLite) require a write-ahead log to implement aspects of ACID transactions. But this need not be a requirement for ACID transactions if you do MVCC (SQLite does not). See my previous post on implementing MVCC for details.

Logical replication (also called change data capture (CDC)) is another interesting feature that requires a write-ahead log. The idea is that the log already preserves the exact order and changes that affect the database's "state machine". So we could copy these changes out of the system by tracking the write-ahead log, preserving change order, and apply these changes to a foreign system.

But again, just CDC is not about durability. It's an ancillary feature that write-ahead logs make simple.

Conclusion

A few key points. One, durability primarily matters if it is established before returning a success to the client. Second, a write-ahead log is a cheap way to get durability.

And finally, durability is a spectrum. You need to read the docs for your database to understand what it does and does not.

Here's a new post about durability and write-ahead logs. Write-ahead logs are used almost everywhere. But to build an intuition for why, it is helpful to imagine what you would do without a WAL. And to explore the meaning of durability.https://t.co/nzS2pMz22z pic.twitter.com/m1n9x8CNcp
— Phil Eaton (@eatonphil) July 1, 2024

How I screen CVs

Bytepawn - Marton Trencseni — Mon, 01 Jul 2024 01:00:00 GMT

Screening CVs is a critical step in the hiring process, used to determine which candidates progress further in the hiring funnel. While the approach to CV screening can vary by company and industry, the primary goal remains the same: to identify candidates with the potential to succeed in subsequent interview stages.

Organizing AI Engineer World's Fair 2024

swyx's site RSS Feed — Sat, 29 Jun 2024 04:52:27 GMT

> note... this only represents my half of running AIE - the intention was to have Ben also write his side on the logistics and sponsorships but we never got round to it and its been a year and a half so... you get what you get, this brain dump of stuff. sorry. 2025's version is https://www.latent.space/p/aiewf-2025-keynotes

Links: The longevity of electrics, improving health tech, building the future, and more!

The Story's Story — Wed, 26 Jun 2024 16:00:00 GMT

* “Electric Vehicles (EVs) Could Last Nearly Forever.” Buying or leasing a legacy internal combustion engine (ICE) car now or going forward is nuts because its value is going to drop to near-zero within a few years. Granted there are a lot of obvious things that people don’t see coming. * Making sense of honor culture. […]

Time Programming for Lawyers and Jurors

Speculative Branches — Wed, 26 Jun 2024 03:00:00 GMT

I often like to have streams of trials playing in the background when I am working, and the recent trial of interest was the trial of Karen Read, who was accused of murder. One issue in this case was a conflict between two timestamps logged by different apps on a potential suspect's phone. Apple health had logged the person climbing flights of stairs at a given time, and at the same timestamp, a log from the app Waze indicated that the person was driving. This seems impossible, but these apps used different time sources on the phone.

Debugging Slow Ubuntu Boot

jodavaho.io — Sun, 23 Jun 2024 03:00:00 GMT

Created with chatgpt + my own notes.

System Analysis

Check Boot Time:
- Use systemd-analyze to see the total boot time.
- Use systemd-analyze blame to see which services take the longest to start.
- Use systemd-analyze critical-chain to understand the critical path.
Check Boot Logs:
- Review the logs with journalctl -b to identify errors and warnings.

Hardware and BIOS/UEFI

Update BIOS/UEFI:
- Ensure your BIOS/UEFI is up to date.
- Check BIOS/UEFI settings for any boot-related options (e.g., Fast Boot).
Check Hardware:
- Ensure all hardware components (RAM, SSD/HDD) are functioning correctly.

Services and Applications

Disable Unnecessary Services:
- Use systemctl disable <service> to disable unneeded services.
- Use systemctl mask <service> to prevent services from starting.
Remove Unnecessary Applications:
- Use sudo apt-get remove <application> to uninstall unneeded applications.
- Use sudo apt-get purge <application> for a thorough cleanup.

File System and Storage

Check Disk Usage:
- Use df -h to check for any partitions that are almost full.
- Use du -sh /path/to/directory to find large files and directories.
Optimize File System:
- Use sudo e4defrag / for ext4 file system defragmentation.
- Use sudo tune2fs -O fast_commit /dev/sdX for ext4 optimizations.
Check for Disk Errors:
- Use sudo fsck /dev/sdX to check and repair filesystem errors.

Kernel and Drivers

Update Kernel and Drivers:
- Use sudo apt-get update and sudo apt-get upgrade to ensure you have the latest updates.
- Use ubuntu-drivers autoinstall to install the recommended drivers.

Startup and Grub

Optimize GRUB:
- Edit /etc/default/grub and set GRUB_TIMEOUT to a lower value (e.g., 2 seconds).
- Run sudo update-grub after making changes.
Reduce Kernel Load:
- In /etc/default/grub, add quiet splash to GRUB_CMDLINE_LINUX_DEFAULT for a quieter boot.

Swap and Memory

Optimize Swap Usage:
- Adjust vm.swappiness in /etc/sysctl.conf to a lower value (e.g., 10).
Check Memory Usage:
- Use free -h to check RAM usage.
- Use top or htop to monitor active processes and their memory consumption.

Network and Connectivity

Disable Unused Network Services:
- Disable services like NetworkManager-wait-online.service if not needed.
- Use systemctl disable NetworkManager-wait-online.service.
Optimize Network Settings:
- Edit /etc/network/interfaces to configure network interfaces for faster boot.

Miscellaneous

Disable Unused TTYs:
- Edit /etc/systemd/logind.conf and set NAutoVTs=1.
Use a Lightweight Desktop Environment:
- Consider switching to a lighter desktop environment like Xfce or LXDE.
Profile Your Boot:
- Use systemd-analyze plot > bootchart.svg to create a visual representation of the boot process.

Regular Maintenance

Clean Package Cache:
- Use sudo apt-get clean to clear the package cache.
Remove Old Kernels:
- Use sudo apt-get autoremove to remove old kernel versions.

2024 Midyear Project Update

Ben Overmyer's Site — Sun, 23 Jun 2024 03:00:00 GMT

2024 has been a hectic year so far. Most recently, we very nearly moved to western New York, but a series of unfortunate events and observations made us reconsider at the last second and decide to continue renting in Wilmington.

I haven't made any progress on reigniting my morning jogging habit. However, once my leg heals (dermatology thing, don't ask), I intend to start it up again. I completely halted my weight training after my doctor told me "it counts as exercise, I guess." That just took all the wind out of my sails.

My alcohol-free phases are on track. January and March were alcohol-free, and September will be the third for this year.

Silver Gryphon Games has stagnated. I finished the Aether second edition manuscript and got cover art for both it and Ingenium's second edition. However, Aether and Aethermancy are both stalled, waiting on Kevin. Considering how crazy his life is right now - as I'm writing this his hometown is flooded so bad he can't go home - I don't blame him one bit. I'll have to focus on getting Ingenium out the door.

Iron Arachne has had some minor tweaks to the star system and planet generators, and one new generator in the form of a Velgarth gift generator. Other than that, it hasn't had any new sci-fi generator updates. If I want to complete this goal, I'll need to work on something in the latter half of this year.

Two of my three planned trips are behind me. The resort vacation in Cancun was kind of like a light version of "The Hangover," so we'll just leave it at that. The eclipse-viewing party was awesome; I will probably never get to see an eclipse like that again in my lifetime.

The final planned trip for this year is in August. I also drove to New York to look at property and explore in the first half of June, and I will be flying up to Minnesota in late August for 2D Con. That's plenty of trips, I think! I didn't journal about any of them so far.

However, I have been consistent in journaling about my day job every Friday. I intend to turn this into a big year-end review, but that won't be public. At least, not in its entirety, anyway.

2024 Jun Summary

jodavaho.io — Tue, 18 Jun 2024 03:00:00 GMT

Wife and I are insanely busy. We both work 50+ hours a week. The kids don’t seem to mind - still hilarious, happy, and energetic. Our nanny is the GOAT.

I have not been playing many games lately. I’ve been doing a lot of yard work, and we’ve been spending a lot of time at the cabin. I’ve been working a lot, too. When I do play, I prefer Helldivers 2 by arrow head games. It’s insane. Check out some videos on youtube.

The back yard is coming together! We built the deck, landscaped the top of the terrace, and put in the playground. Next: build another terrace, plant 5 more trees, and put in a fire pit.

Work at Shield is pretty nuts. Busy, for sure. But mostly for other folks. I am still getting up to speed. Eventually I’ll manage the mission planning stack for real, but for now it’s that awkward phase where I am not the best person for any task, so until deadlines slack a little, I’ll let the cooks cook and pick up side tasks.

I’ve been writing a ton of Rust for side projects (attention HR: This does not involve Shield stuff). This marks 4 years of Rust for me.

One year of solo dev, wrapping up the grant-funded work

Weblog on marginalia.nu — Tue, 18 Jun 2024 03:00:00 GMT

A year ago I walked out of the office for the last time. I handed in my corpo laptop, said some good-byes, and since then I have been my own boss.

This first year has been funded by an NLnet grant, which I’m in the midst of wrapping up. As of now, the work is all done, the final request for payment has been sent.

There’s a similar last-day-of-school levity to both these events.

The limitations of LLMs, or why are we doing RAG?

Notes on software development — Mon, 17 Jun 2024 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Writing a simple Javascript async message queue server - Part III

Bytepawn - Marton Trencseni — Mon, 17 Jun 2024 01:00:00 GMT

In this final post on the toy Javascript async message queue server implementation, I make it compatible with the Python version. I use the library of unit tests developed previously to ensure identical behaviour between the two codebases.

ESPHome Server in Python

Pete Keen — Sat, 15 Jun 2024 03:00:00 GMT

Last year I installed holiday lights on my house (previously). At the time I chose a controller that had the following key qualities: a) cheap, b) worked with WLED, c) cheap.

After using it for a season I discovered that the reason why this controller is so cheap is because it uses an ESP8266, which is fine, but it doesn't play well with my setup. For example, if I enable the Home Assistant integration the controller falls over after a few hours. It also reboots for unknowable reasons sometimes and I would come home to find the lights in their default orange color.

I probably could have fixed this with a more powerful controller. I even bought a neat ESP32 controller with a built-in wide range voltage regulator but never got around to setting it up.

Loosely, what I want is:

Control the lights without random reboots
Easy Home Assistant integration
Easy customization
No Wi-Fi
Use hardware that I already have
Tailscale, ideally
Learn some stuff

I could have gotten the first two using a more powerful ESP32 module. Third could be done with ESPHome. Four and five are contradictory while staying within the constraint of an ESP32-based system.

Also last year, I built a little box that controls the power for my 3D printer with a Raspberry Pi Pico and a second Klipper instance (previously) so naturally I tried to get Klipper to fit in this non-3d-printer shaped hole. I tried so hard.

On the surface, Klipper appears to do everything that I want (control addressable LEDs, kind of customizable) but it makes no compromises in wanting to be a 3D printer controller. Most of the firmware is dedicated to running a motion controller, there's a lot of emphasis on scheduling things to happen in the near future, and there's a global printer object. Importantly for my purposes, there's no built-in way to set up a digital input without declaring it a button.

It's fine. Klipper is fine. It's just not built to be a generic IO platform.

So, what's a reasonable rational person to do?

Write an ESPHome Protocol Server

Of course.

There are essentially three ways to get arbitrary devices and entities to show up automatically in Home Assistant.

First, one can write a Home Assistant integration. This is fine and good but it doesn't work for me because my devices are far away from the VM that Home Assistant runs in.

Second, there's MQTT autodiscovery. I know this works because it's how my Zigbee devices integrate with HA, but I just could not make any of the existing generic autodiscovery libraries work consistently. Usually I would end up with a bunch of duplicate MQTT devices and then HA would get confused.

Third, there's ESPHome. ESPHome is a firmware for ESP modules (think: small devices with wifi like plugs, air quality monitors, etc). ESPHome belongs to the Open Home Foundation, same as Home Assistant, so it has commercial support and a first class HA integration. I already have a bunch of ESPHome devices running in my house, so it seems like a pretty natural fit.

The normal and ordinary way of using ESPHome is to write some YAML config that ESPHome compiles into a firmware for your device, then you flash the device and HA sets itself up to interact with the entities you described in YAML. What I want to do is just that last bit, the part where I can tell HA what entities I have and it sets up UI for me.

HA talks to ESPHome over what they call their "native API". The native API is a TCP-based streaming protocol where the ESPHome device is the server and Home Assistant is the client. They exchange protocol buffer encoded messages over either plain TCP or with a Noise-based encryption scheme.

Over the last week or so I built a Python implementation of that protocol named aioesphomeserver, bootstrapping off of the official aioesphomeapi client library that HA uses.

A Minimal Example

Here's a very simple example of what aioesphomeserver looks like:

import asyncio
        
        from aioesphomeserver import (
        Device,
        SwitchEntity,
        BinarySensorEntity,
        EntityListener,
        )
        
        class SwitchListener(EntityListener):
        async def handle(self, key, message):
        sensor = self.device.get_entity("test_binary_sensor")
        if sensor != None:
        await sensor.set_state(message.state)
        
        device = Device(
        name = "Test Device",
        mac_address = "AC:BC:32:89:0E:C9",
        )
        
        device.add_entity(
        BinarySensorEntity(
        name = "Test Binary Sensor",
        )
        )
        
        device.add_entity(
        SwitchEntity(
        name = "Test Switch",
        )
        )
        
        device.add_entity(
        SwitchListener(
        name="_listener",
        entity_id="test_switch"
        )
        )
        
        asyncio.run(device.run())

From the top, we import a bunch of stuff and then create a class that listens for messages from the device (the handle method). Then, we set up a device with a name and a fake MAC address. Device can generate a random one for you but it doesn't persist, so if you want this device to stick around in HA you should declare a static MAC.

We then add some entities to it: a binary sensor, a switch, and an instance of our switch listener configured for Test Switch.

Finally, we start the asyncio event loop.

With just that, you get the ESPHome web UI:

Adding the device to Home Assistant you'll see this:

AIO ESPHome Server Architecture

I tried to follow the spirit of ESPHome's architecture when writing the server.

The Device is a central registrar for Entitys and serves as a message hub. The native API server and web server are entities that plug into the message bus, as are things like SwitchEntity and BinarySensorEntity. Everything is async using Python's asyncio.

Any entity with a run method will automatically be scheduled as a task at startup.

A Production Example

The development case for this library has been driving the addressable LEDs on my house. I found a project named u2if that turns a Raspberry Pi Pico into a USB peripheral that provides a bunch of fun stuff: GPIO, I2C, SPI, PWM, ADC as well as an addressable LED driver for WS2812-compatible lights. The fun wrinkle of the light driver is that it offloads the bitstream generation to the Pico's PIO coprocessors.

I forked u2if and added a few things:

RGBW support, which was already in the codebase but not available
Support for the Pico clone boards I have (SparkFun Pro Micro RP2040)
A set of effects for Neopixel along with a console-mode simulator to use while developing
A Docker image that bundles the firmware and the Python library

This deployment consists of:

A Dell Wyse 3040 thin client running Alpine Linux that already handles Z-Wave for the garage
SparkFun Pro Micro RP2040 running the u2if firmware connected over USB
Two channels of RS485 transceivers so I can get the very fast, very unforgiving light control signals 40 feet from where the 3040 is mounted to the wall to where the light power injector lives.

Here is the full script that I'm using to drive the addressable lights on my house:

import asyncio
        
        from machine import WS2812B
        from neopixel.effects import StaticEffect, BlendEffect, TwinkleEffect
        
        from aioesphomeserver import (
        Device,
        LightEntity,
        LightStateResponse,
        EntityListener,
        )
        
        from aioesphomeapi import LightColorCapability
        
        class LightStrip(EntityListener):
        def __init__(self, *args, strings=[], effects={}, **kwargs):
        super().__init__(*args, **kwargs)
        self.strings = strings
        self.num_pixels = sum([s[1] for s in strings])
        self.effects = effects
        
        self.current_effect_name = None
        self.current_effect = StaticEffect(count=self.num_pixels)
        self.white_brightness = 0.0
        self.color_brightness = 0.0
        
        async def handle(self, key, message):
        if type(message) != LightStateResponse:
        return
        
        await self.device.log(1, f"message.effect: '{message.effect}'")
        
        if message.effect != "" and message.effect != self.current_effect_name:
        if message.effect in self.effects:
        self.current_effect_name = message.effect
        self.current_effect = self.effects[message.effect](self.num_pixels, message)
        self.current_effect.next_frame()
        
        if self.current_effect:
        self.current_effect.update(message)
        
        self.color_brightness = message.color_brightness
        self.white_brightness = message.brightness
        
        if message.state == False:
        self.color_brightness = 0.0
        self.white_brightness = 0.0
        
        def render(self):
        pixels = []
        
        for i in range(self.num_pixels):
        color = self.current_effect.pixels[i]
        
        pixel = [
        int(color[0] * 255.0 * self.color_brightness),
        int(color[1] * 255.0 * self.color_brightness),
        int(color[2] * 255.0 * self.color_brightness),
        int(color[3] * 255.0 * self.white_brightness),
        ]
        
        pixels.append(pixel)
        
        # partition strings
        # write to each string
        cur = 0
        for string, length in self.strings:
        last = cur + length - 1
        string.write(pixels[cur:last])
        cur = last + 1
        
        async def run(self):
        while True:
        self.current_effect.next_frame()
        self.render()
        await asyncio.sleep(1/24.0)
        
        
        device = Device(
        name = "Garage Stuff",
        mac_address = "7E:85:BA:7E:38:07",
        model = "Garage Stuff"
        )
        
        device.add_entity(LightEntity(
        name="Front Lights",
        color_modes=[LightColorCapability.ON_OFF | LightColorCapability.BRIGHTNESS | LightColorCapability.RGB | LightColorCapability.WHITE],
        effects=["Static", "Twinkle"],
        ))
        
        def make_twinkle_effect(count, state):
        return BlendEffect(
        TwinkleEffect(count=count, include_white_channel=True),
        StaticEffect(count=count, color=[state.red, state.green, state.blue, state.white], include_white_channel=True),
        mode='lighten',
        include_white_channel=True,
        )
        
        device.add_entity(LightStrip(
        name = "_front_lights_strip",
        entity_id = "front_lights",
        strings = [(WS2812B(23, rgbw = True, color_order="GRBW"), 20)],
        effects={
        "Static": lambda count, state: StaticEffect(count=count, color=[state.red, state.green, state.blue, state.white], include_white_channel=True),
        "Twinkle": make_twinkle_effect,
        },
        ))
        
        asyncio.run(device.run())

The structure is basically the same as the minimal example. We import some stuff, we set up an EntityListener class, and then we set up a Device with a LightEntity and an instance of the listener .

In this case, the listener listens for state responses from a Light entity and renders pixels according to a set of effects. It also has a run method that renders the current effect out every 1/24th of a second.

Should you use this?

I don't know!

If your constraints match mine, maybe it'd be helpful. If you want to expose a thing to Home Assistant and would rather have it show up as an ESPHome device rather than, say, writing your own HA integration or messing with MQTT or writing RESTful API handlers, this would probably be useful.

That said, I think if your use case fits within ESPHome proper you should use that. ESPHome has built in drivers for so many things and is going to be better supported (i.e. people are paid to work on it).

Pretty neat though, eh?

Writing unit tests for the async message queue server

Bytepawn - Marton Trencseni — Sat, 15 Jun 2024 01:00:00 GMT

I write a small library of unit tests for the message queue servers.

Confusion is a muse

Notes on software development — Fri, 14 Jun 2024 03:00:00 GMT

Some of the most interesting technical blog posts I read come from, and a common reason for posts I write is, confusion. You're at work and you start asking questions that are difficult to answer. You spend a few hours or a day trying to get to the bottom of things.

If you ask a question to very experienced and successful developers at work, they have a tendency not to give context and to simplify things down to a single answer. This may be a good way to make business decisions. (One can't afford to waste an eternity considering everything indefinitely.) But accepting an answer you don't understand is actively harmful for building intuition.

Certainly, sometimes not accepting an answer can be irritating. You'll have to figure that out.

But beyond "go along to get along", another reason we don't pursue what we're confused about is because we're embarrassed that we're confused in the first place. What's worse, the embarrassment we feel naturally grows the more experienced we get. "I've got this job title, I don't want to seem like I don't know what you mean."

But if you fight the embarrassment and pursue your confusion regardless, you'll likely figure something very interesting out. Moreover, you will probably not have been the only person who was confused. At least personally it is quite rare that I am confused about something and no one else is.

So pay attention when you get confused, and consider why it happened. What did you expect to be the case, and how did reality differ? Explore the angles and the options. When you finally understand, think about what led you to that understanding.

Write it down. Put it into an internal Markdown doc, an internal Atlassian doc, an internal Google Slides page, whatever. The medium doesn't matter.

This entire process doesn't come easily. We feel embarrassed. We aren't used to lingering on something we're confused by. We aren't used to writing things down.

But if you can make yourself pause every once in a while and think about what you (or someone around you) got confused by, and if you can force yourself to stop getting embarrassed by what you got confused by, and if you can write down the background and the reasoning that led to your ultimate understanding, you're going to have something pretty interesting to talk about.

You'll contribute to the growth and intuition of your colleagues. And you'll never run out of things to write about.

Confusion is embarrassing. But fight that feeling, and dig into why you're confused. And write it down.

You won't be the only one who was confused. And you'll tend to have something pretty interesting to talk about.https://t.co/IdX1nGBheR pic.twitter.com/KzTjqMxw6u
— Phil Eaton (@eatonphil) June 14, 2024

Making a bignum library for fun

Austin Z. Henley's Blog — Fri, 14 Jun 2024 01:00:01 GMT

https://austinhenley.com/blog/bignum1.html

Five Nine Problems

Speculative Branches — Thu, 13 Jun 2024 03:00:00 GMT

A guilty pleasure of mine is the pursuit of perfection. It is certainly a vice in most contexts, but there are some problems whose solutions demand a measure of perfection. These are problems that I will refer to as "5-9 problems": problems whose solutions need five 9's (or more) in some dimension. Usually, those nines are correctness of some kind, but they can also be availability or for some systems, speed.

Testing randomness extractors

Bytepawn - Marton Trencseni — Sun, 02 Jun 2024 01:00:00 GMT

I apply NIST's suite of statistical tests to my randomness extractor implementations from the previous posts.

Gollama: Ollama Model Manager

smcleod.net — Sat, 01 Jun 2024 04:00:10 GMT

Gollama is a Go-based client for Ollama for managing models.

How I run a software book club

Notes on software development — Thu, 30 May 2024 03:00:00 GMT

I've been running software book clubs almost continuously since last summer, about 12 months ago. We read through Designing Data-Intensive Applications, Database Internals, Systems Performance, and we just started Understanding Software Dynamics.

The DDIA discussions were in-person in NYC with about 5-8 consistent attendees. The rest have been over email with 300, 500, and 600 attendees.

This post is for folks who are interested in running their own book club. None of these ideas are novel. I co-opted the best parts I saw from other people running similar things. And hopefully you'll improve on my experience too, should you try.

Despite the length of this post running a book club takes almost no noticeable effort, other than when I need to select and confirm discussion leaders. It is the limited-effort-required to thank that I've kept up the book clubs so consistently.

Google Groups

I run the virtual book clubs over email. I create a Google Group and tell people to send me their email for an invite. I use a Google Form to collect emails since I get many. If you're doing a small group book club you can just collect member emails directly.

In the Google Form I ask people to volunteer to lead discussion for a chapter (or chapters). And I ask for a Twitter/GitHub/LinkedIn account.

When I've gotten enough responses I go through the list and check Twitter/GitHub/LinkedIn info to find people who might have a particularly interesting perspective to lead a discussion.

"Lead a discussion" sounds formal but I mean anything but. All I am looking for is someone to start a new Google Group thread each week and for them to share their thoughts.

For example a discussion leader might share:

What they liked about the chapter
Something new they learned from the chapter
A story about their work that the chapter reminded them of
A little project they hacked on, inspired by reading the chapter
A paper or YouTube video this chapter reminded them of
Something from the chapter that was confusing
Etc.

The "discussion leader" has no responsibility for remaining in the discussion after posting the thread. There just isn't an easy way to say "person who kicks off discussion" than to call them a "discussion leader".

By the way, I didn't do discussion leaders for the first book club, reading DDIA. And that book club took noticeably more effort. Because I organized it, I was effectively the discussion leader every time. Having discussion leaders disperses the effort of the book club. And I think it makes the club much more interesting.

SparkNotes-ification

One thing I noticed happening often was that the discussion leader might do a large summary of the chapter. I greatly appreciate and respect that effort, but I think this is not the ideal thing to happen. Of course you can't control what people do and maybe they really wanted to write a summary. But since noticing this happen I now try to discourage the discussion leader from summarizing since 1) it must be quite time-consuming and 2) it isn't as interesting as some of the above bullet points.

Confirming with leaders

When I have picked out folks who seem like they'd be fun discussion leaders, I bcc email them all asking them to confirm. At the same time I explain what being a discussion leader means. As I just explained it here above.

Each week's discussion gets a new Google Group thread. Discussion happens in responses to the thread.

I ask the discussion leaders to create the new discussion thread between Friday and Saturday their local time.

For folks who don't confirm, I email them one last time and then if they still haven't confirmed I find someone new.

I always lead the first week's discussion so that the discussion leaders can see what I do and so that I can establish the pattern.

Managing leaders

It takes a while to read a book. Sometimes the leaders forget to do their part. If it gets to be Sunday and the discussion leader for the week hasn't started discussion, I email them to gently ask if they are still available to kick off discussion. And if they are not, no worries, I can step in.

I have had to step in a few times to start discussion and it's no problem.

Managing non-leaders

Just as you need to clarify and set expectations for discussion leaders, you need to clarify and set expectations for everyone else.

When I invite people to the Google Group I typically also create an Intro thread where I explain the discussion format.

An annoying aspect of Google Groups is that I cannot limit who can create a thread without limiting who can respond to a thread.

It would simplify things for me if I could limit thread creation to discussion leaders. But since I cannot, I try to repeatedly and explicitly mention in the Intro thread that no one should start a new discussion thread unless they are a discussion leader. And that new threads will come out each weekend to discuss the previous chapter.

Setting the tone

One of the most important things to do in the Intro email is to set the tone. I try to clarify this is a friendly and encouraging group focused on learning and improving ourselves. We have experts in the group and we have noobs in the group and they are all welcome and will all come away with different things.

Why email?

Email seems to be the most time-friendly and demographic-friendly medium. Doing live discussion sounds stressful and difficult to schedule, although I believe Alex Petrov runs live discussions. Email forces you to slow down and think things through. And email is built around an inbox. If you didn't get to read some discussion, you can mark it unread. You can't do that in Discord or Slack.

Avoiding long-term commitments

When I pick a book, aside from picking books I think are likely to be exceptionally well-written, I try to avoid books that we could not finish within 3 months. It concerns me to try to get people to commit to something longer than that.

This has led to some distortion though. Systems Performance has only 16 chapters. One chapter a week means about 3 months in total. But each chapter is 100 pages long.

I was hesitant to do a reading of Understanding Software Dynamics because it has 28 chapters. But each chapter is only 10-15 pages long. So when I decided to go with it, I decided we'd read 2 chapters a week. Each discussion leader is responsible for 2 chapters at a time. That means we can finish within 3 months. And each week we read only 20-30 pages, which is still much more doable than 100 pages of Systems Performance.

On the other hand, we did make it through Systems Performance! Which gives me confidence to pick other books that are physically daunting, should they otherwise seem like a good idea.

A book ends

Many public book clubs go through a book a month and have no ending. That is totally fair. But what I love about the way I organize book clubs is that each reading is unrelated to the next. It's an entirely new signup for each book. You need only "commit" (I mean, you can drop off whenever and definitely people do) to a 3-month reading and then you can justly feel good about yourself and join again in the future or not.

In contrast a paper reading club has no obvious ending, unless you pick all the papers in advance and organize them around a school year or something. This has made running a paper reading club feel more concerning to me. Though I greatly appreciate folks like Aleksey Charapko and Murat Demirbas who do.

Most people don't actively contribute, but they still value it

In a group of 500 people, maybe 1-2% of those people actively contribute to discussion. 5-10 people. But I often hear from people who didn't participate that they still highly valued the group. And this high percentage of non-active-participants is part of why I keep allowing the group size to grow. There's little work I have to do and a bunch of people benefit.

Partipation dies off

People start out excited and chime in to introduce themselves but by the last chapter you're lucky if there are any responses to the discussion leader at all.

This is just how it goes. I don't really see anything wrong about this.

The only way I've tried to combat this is by reassuring everyone that they should keep up with discussion even if it means they must skim the chapter instead of really read it.

I also commonly hear from people that they stop reading because they got behind. So to combat that I also stress in the beginning that technical books are often forgiving and that you should just pick up with whatever chapter we're currently on even if it means skipping some chapters. You can always go back.

Doing it at your company likely won't go well

I wrote about this before. For some reason it's hard to get people who would otherwise join an external reading club to join a company-internal reading club.

Though perhaps I'm just doing it wrong because I hear of others like Elizabeth Garrett Christensen who run an internal software book club successfully.

Good luck, have fun!

That's all I've got. Send me questions if you've got any. But mostly, just give it a shot if you want to and you'll learn!

And if you still don't get it, you can of course just join one of my book clubs. :)

Since folks have asked, here's how I run a software book club.

But also, you could just join and see. :)https://t.co/tXBrLFYbvC pic.twitter.com/4iW8EfZCeY
— Phil Eaton (@eatonphil) May 30, 2024

Linux Command Line Computing book announcement

learnbyexample — Wed, 29 May 2024 03:00:00 GMT

Hello!

I am pleased to announce a new version of my Linux Command Line Computing ebook. This is the longest book I've published so far (204 pages) — it took me more than 7 months to complete the first version and another month for a minor revision.

This ebook aims to teach Linux command line tools and Shell Scripting for beginner to intermediate level users. The main focus is towards managing your files and performing text processing tasks. Plenty of examples are provided to make it easier to understand a particular tool and its various features. Exercises at the end of chapters will help you practice what you've learned and solutions are provided for reference. I hope this ebook would make it much easier for you to discover CLI tools, features and learning resources than my own blundering experience.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of the ebook for FREE till 9-June-2024. You can still pay if you wish ;)

Some of my bundles are on sale as well:

All books bundle is $12 (normal price $32) — all my 13 programming ebooks
Linux CLI Text Processing bundle is $7 (normal price $20) — Linux CLI tools, shell scripting, grep, sed, awk, perl and ruby one-liners

What's new?🔗

Some of the examples, exercises, descriptions and external links were updated/corrected
Book title changed to Linux Command Line Computing
New cover image

Videos🔗

Here's a short video about the Linux Command Line Computing ebook:

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Testimonials🔗

Ive only gotten through first pages but appears a good Unix/bash primer. I’ll probably recommend for new hires out of bootcamp because they’re usually weak here

— feedback on twitter

Nice book! I just started trying to get into linux today and you have some tips I haven’t found elsewhere and the text is an enjoyable read so far.

— feedback on reddit

Table of Contents🔗

Preface
Introduction and Setup
Command Line Overview
Managing Files and Directories
Shell Features
Viewing Part or Whole File Contents
Searching Files and Filenames
File Properties
Managing Processes
Multipurpose Text Processing Tools
Sorting Stuff
Comparing Files
Assorted Text Processing Tools
Shell Scripting
Shell Customization

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/cli-computing/.

GitHub repo🔗

Visit https://github.com/learnbyexample/cli-computing for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/cli-computing/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Feynman's Garden

Weblog on marginalia.nu — Sun, 26 May 2024 03:00:00 GMT

The best description of my problem solving process is the Feynman algorithm, which is sometimes presented as a joke where the hidden subtext is “be smart”, but I disagree. The “algorithm” is a surprisingly lucid description of how thinking works in the context of hard problems where the answer can’t simply be looked up or trivially broken down, iterated upon in a bottom-up fashion, or approached with similar methods.

Feynman’s thinking algorithm is described like this:

Randomness extractors: making fair coins out of biased coins

Bytepawn - Marton Trencseni — Sun, 26 May 2024 01:00:00 GMT

I discuss various models of biased bit sequences, and how to extract uniform random (or close to it) output bit sequences from them, illustrated with Python code.

The CP/M emulator is working well

Steve Kemp's Blog — Sat, 25 May 2024 15:00:00 GMT

In my recent posts I've talked about implementing BDOS and BIOS syscalls for my cp/m emulator. I've now implemented enough of the calls that I can run many of the standard binaries:

The Aztech C Compiler
Microsoft BASIC
Turbo Pascal
Wordstar
etc

Of course I've not implemented all the syscalls, so the emulation isn't 100% perfect and many binaries won't run. But I sent myself on a detour by implementing extra syscalls, custom syscalls.

Traditionally CP/M systems are "rebooted" by pressing Ctrl-C at the CCP prompt. I thought that was something I'd press by accident so I implemented the restart behaviour only when the user pressed Ctrl-C twice in a row. But then I added a custom syscall that lets you change hte value:

A>ctrlc
The Ctrl-C count is currently set to 2
A>ctrlc 1
The Ctrl-C count is currently set to 1
A>

So you can now change the value at runtime. Similarly there is support for switching CCP at runtime, and even changing the default output-device from ADM-3A to ANSI, or vice-versa. It's kinda neat to make these kind of extensions, and happily the traditional BIOS has two syscalls reserved for custom use so I just used one of those.

I've added support for testing whether a binary is running under my emulator, or not, using a custom syscall. So I can run:

A>test
This binary is running under cpmulator:

cpmulator unreleased
https://github.com/skx/cpmulator/

On another emulator I see this:

A>test
Illegal BIOS call 31
No, this binary is not running under cpmulator.

Anyway I'm happy with the current state of things, and I fixed a couple of bugs which means I now have support for SUBMIT.COM which is a real time-saver.

Confuddlement: Download Confluence Spaces as Markdown, Summarise with Ollama

smcleod.net — Thu, 23 May 2024 08:45:10 GMT

Download Confluence Spaces as Markdown, Summarise with Ollama

Abandoning Highfleet Optimization Tool

jodavaho.io — Thu, 23 May 2024 03:00:00 GMT

You know, it was fun while it lasted, but I just don’t have time to work on the Highfleet Optimizer anymore. It’s not like anyone was using it! We had grand ambitions though, and I learned a ton.

Here’s the stuff I do not regret at all about this project:

Writing a fast optimizer in C, and learning how to get that deployed to AWS Lambda
Getting quite good with FastAPI, Lambda, Cognito, and DynamoDB to build a fully functional, authenticated backend.
Learning to model the game’s mechanics regression and model fitting
Learning to use Lark to parse the game’s data files
Working with Altho

Here’s the stuff I do regret about this project:

Not many people are that interested in using computers to explore the trade space of video-game fleets. WHO KNEW.
Not seeing through the visualization part - the community would have loved that
Not getting the backend integrated with a discord bot - the search functionality alone would have been worth it.

I’ll leave it running as is. It cost me less than a buck a month to run the whole thing.

The reasons?

Well I’m super busy working at shield.ai, for one. For another, I just don’t play Highfleet anymore. I’ve moved on to Nebulous: Fleet Command, because it has a lot of the same focus on realism and immense attention to detail. I’m having a blast with that. Finally, I’ve got a few other projects that I’m cooking up, so stay tuned for those.

Writing a simple Python async message queue server - Part III

Bytepawn - Marton Trencseni — Wed, 22 May 2024 01:00:00 GMT

In this final post on the toy Python async message queue server implementation, I make further feature and code improvements.

ServerlessDays ANZ 2024 - All slides, notes and links

Ben E. C. Boyter — Mon, 20 May 2024 03:00:00 GMT

Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my ServerlessDaysANZ 2024 talk about “Abusing Go, AWS Lambda and bloom filters to make a true Australian serverless search engine”. Whatever the case may be, welcome!

For the slides visit https://boyter.org/static/serverlessdaysanz2024/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide. Press S to see the speaker notes, which are more or less the script I will be sticking to.

CodeAid: A classroom deployment of an LLM-based programming assistant

Austin Z. Henley's Blog — Sun, 19 May 2024 22:00:01 GMT

https://austinhenley.com/blog/codeaid.html

NVApi - Nvidia GPU Monitoring API

smcleod.net — Sat, 18 May 2024 14:30:03 GMT

Nvidia GPU Monitoring API

Experiment in Java native calls

Weblog on marginalia.nu — Thu, 16 May 2024 03:00:00 GMT

I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search.

The Marginalia Search engine is mainly written in Java, which is language that is good at many things, but not particularly pleasant to work with when it comes to low level systems programming.

Unfortunately, a part of building an internet search engine involves database-adjacent low level programming.

Implementing MVCC and major SQL transaction isolation levels

Notes on software development — Thu, 16 May 2024 03:00:00 GMT

In this post we'll build a database in 400 lines of code with basic support for five standard SQL transaction levels: Read Uncommitted, Read Committed, Repeatable Read, Snapshot Isolation and Serializable. We'll use multi-version concurrency control (MVCC) and optimistic concurrency control (OCC) to accomplish this. The goal isn't to be perfect but to explain the basics in a minimal way.

You don't need to know what these terms mean in advance. I did not understand them before doing this project. But if you've ever dealt with SQL databases, transaction isolation levels are likely one of the dark corners you either 1) weren't aware of or 2) wanted not to think about. At least, this is how I felt.

While there are many blog posts that list out isolation levels, I haven't been able to internalize their lessons. So I built this little database to demonstrate the common isolation levels for myself. It turned out to be simpler than I expected, and made the isolation levels much easier to reason about.

Thank you to Justin Jaffray, Alex Miller, Sujay Jayakar, Peter Veentjer, and Michael Gasch for providing feedback and suggestions.

All code is available on GitHub.

Why do we need transaction isolation?

If you already know the answer, feel free to skip this section.

When I first started working with databases in CRUD applications, I did not understand the point of transactions. I was fairly certain that transactions are locks. I was wrong about that, but more on that later.

I can't remember exact code I wrote, but here's something I could have written:

with database.transaction() as t:
  users = t.query("SELECT * FROM users WHERE group = 'admin';")
  ids = []
  for user in users:
    if some_complex_logic(user):
      ids.append(user.id)

  t.query("UPDATE users SET metadata = 'some value' WHERE id IN ($1);", ids)

I would have thought that all users that were seen from the initial SELECT who matched the some_complex_logic filter would be exactly the same that are updated in my second SQL statement.

And if I were using SQLite, my guess would have been correct. But if I were using MySQL or Postgres or Oracle or SQL Server, and hadn't made any changes to defaults, that wouldn't necessarily be true! We'll discover exactly why throughout this post.

For example, some other connection and transaction could have set a user's group to admin after the initial SELECT was executed. It would then be missed from the some_complex_logic check and from the subsequent UPDATE.

Or, again after our initial SELECT, some other connection could have modified the group for some user that previously was admin. It would then be incorrectly part of the second UPDATE statement.

These are just a few examples of what could go wrong.

This is the realm of transaction isolation. How do multiple transactions running at the same time, interacting with the same data, interact with each other?

The answer is: it depends. The SQL standard itself loosely prescribes four isolation levels. But every database implements these four levels slightly differently. Sometimes using entirely different algorithms. And even among the standard levels, the default isolation level for each database differs.

Funky bugs that can show up across databases and across isolation levels, often dependent on particular details of common ways of implementing isolation levels, create what are called "anomalies". Examples include intimidating terms like "dirty reads" and "write cycles" and G2-Item.

The topic is so complex that we've got decades of research papers critiquing SQL isolation levels, categorization of common isolation anomalies, walkthroughs of anomalies by Martin Kleppmann in Designing Data-Intensive Applications, Martin Kleppman's Hermitage project documenting common anomalies across isolation levels in major databases, and the ACIDRain paper showing isolation-related bugs in major open-source ecommerce projects.

These aren't just random links. They're each quite interesting. And particularly for practitioners who don't know why they should care, check out Designing Data-Intensive Applications and the last link on ACIDRain.

And this is only a small list of some of the most interesting research and writing on the topic.

So there's a wide variety of things to consider:

Not every database implements transaction isolation levels identically, resulting in different behavior
Not all researchers agree, and not all database developers agree, on what any given isolation level means
Not every database has the same default isolation level, and most developers tend not to change the default
Not every developer is correctly using the isolation level they pick (default or not)

Transaction isolation levels are basically vibes. The only truth for real projects is Martin Kleppmann's Hermitage project that catalogs behavior across databases. And a truth some people align with is Generalized Isolation Level Definitions.

So while all these linked works above are authoritative, and even though we can see that there might be some anomalies we have to worry about, the research can still be difficult to internalize. And many developers, my recent self included, do not have a great understanding of isolation levels.

Throughout this post we'll stick to informal definitions of isolation levels to keep things simple.

Let's dig in.

Locks? MVCC?

Historically, databases implemented isolation with locking algorithms such as Two-Phase Locking (not the same thing as Two-Phase Commit). Multi-version concurrency control (MVCC) is an approach that lets us completely avoid locks.

It's worthwhile to note that while we will validly not use locks (implementing what is called optimistic concurrency control or OCC), most MVCC databases do still use locks for certain things (implementing what is called pessimistic concurrency control).

But this is the story of databases in general. There are numerous ways to implement things.

We will take the simpler lockless route.

Consider a key-value database. With MVCC, rather than storing only the value for a key, we would store versions of the value. The version includes the transaction id (a monotonic incrementing integer) wherein the version was created, and the transaction id wherein the version was deleted.

With MVCC, it is possible to express transaction isolation levels almost solely as a set of different visibility rules for a version of a value; rules that vary by isolation level.

So we will build up a general framework first and discuss and implement each isolation level last.

Scaffolding

We'll build an in-memory key-value system that acts on transactions. I usually try to stick with only the standard library for projects like this but I really wanted a sorted data structure and Go doesn't implement one.

In main.go, let's set up basic helpers for assertions and debugging:

package main

import (
        "fmt"
        "os"
        "slices"

        "github.com/tidwall/btree"
)

func assert(b bool, msg string) {
        if !b {
                panic(msg)
        }
}

func assertEq[C comparable](a C, b C, prefix string) {
        if a != b {
                panic(fmt.Sprintf("%s '%v' != '%v'", prefix, a, b))
        }
}

var DEBUG = slices.Contains(os.Args, "--debug")

func debug(a ...any) {
        if !DEBUG {
                return
        }

        args := append([]any{"[DEBUG]"}, a...)
        fmt.Println(args...)
}

As mentioned previously, a value in the database will be defined with start and end transaction ids.

type Value struct {
        txStartId uint64
        txEndId   uint64
        value     string
}

Every transaction will be in an in-progress, aborted, or committed state.

type TransactionState uint8
const (
        InProgressTransaction TransactionState = iota
        AbortedTransaction
        CommittedTransaction
)

And we'll support a few major isolation levels.

// Loosest isolation at the top, strictest isolation at the bottom.
type IsolationLevel uint8
const (
        ReadUncommittedIsolation IsolationLevel = iota
        ReadCommittedIsolation
        RepeatableReadIsolation
        SnapshotIsolation
        SerializableIsolation
)

We'll get into detail about the meaning of the levels later.

A transaction has an isolation level, an id (monotonic increasing integer), and a current state. And although we won't make use of this data yet, transactions at stricter isolation levels will need some extra info. Specifically, stricter isolation levels need to know about other transactions that were in-progress when this one started. And stricter isolation levels need to know about all keys read and written by a transaction.

type Transaction struct {
        isolation  IsolationLevel
        id         uint64
        state      TransactionState

        // Used only by Repeatable Read and stricter.
        inprogress btree.Set[uint64]

        // Used only by Snapshot Isolation and stricter.
        writeset   btree.Set[string]
        readset    btree.Set[string]
}

We'll discuss why later.

Finally, the database itself will have a default isolation level that each transaction will inherit (for our own convenience in tests).

The database will have a mapping of keys to an array of value versions. Later elements in the array will represent newer versions of a value.

The database will also store the next free transaction id it will use to assign ids to new transactions.

type Database struct {
        defaultIsolation  IsolationLevel
        store             map[string][]Value
        transactions      btree.Map[uint64, Transaction]
        nextTransactionId uint64
}

func newDatabase() Database {
        return Database{
                defaultIsolation:  ReadCommittedIsolation,
                store:             map[string][]Value{},
                // The `0` transaction id will be used to mean that
                // the id was not set. So all valid transaction ids
                // must start at 1.
                nextTransactionId: 1,
        }
}

To be thread-safe: store, transactions, and nextTransactionId should be guarded by a mutex. But to keep the code small, this post will not use goroutines and thus does not need mutexes.

There's a bit of book-keeping when creating a transaction, so we'll make a dedicated method for this. We must give the new transaction an id, store all in-progress transactions, and add it to database transaction history.

func (d *Database) inprogress() btree.Set[uint64] {
        var ids btree.Set[uint64]
        iter := d.transactions.Iter()
        for ok := iter.First(); ok; ok = iter.Next() {
                if iter.Value().state == InProgressTransaction {
                        ids.Insert(iter.Key())
                }
        }
        return ids
}

func (d *Database) newTransaction() *Transaction {
        t := Transaction{}
        t.isolation = d.defaultIsolation
        t.state = InProgressTransaction

        // Assign and increment transaction id.
        t.id = d.nextTransactionId
        d.nextTransactionId++

        // Store all inprogress transaction ids.
        t.inprogress = d.inprogress()

        // Add this transaction to history.
        d.transactions.Set(t.id, t)

        debug("starting transaction", t.id)

        return &t
}

And we'll add a few more helpers for completing a transaction, for fetching a transaction by id, and for validating a transaction.

func (d *Database) completeTransaction(t *Transaction, state TransactionState) error {
        debug("completing transaction ", t.id)

        // Update transactions.
        t.state = state
        d.transactions.Set(t.id, *t)

        return nil
}

func (d *Database) transactionState(txId uint64) Transaction {
        t, ok := d.transactions.Get(txId)
        assert(ok, "valid transaction")
        return t
}

func (d *Database) assertValidTransaction(t *Transaction) {
        assert(t.id > 0, "valid id")
        assert(d.transactionState(t.id).state == InProgressTransaction, "in progress")
}

The final bit of scaffolding we'll set up is an abstraction for database connections. A connection will have at most associated one transaction. Users must ask the database for a new connection. Then within the connection they can manage a transaction.

type Connection struct {
        tx *Transaction
        db *Database
}

func (c *Connection) execCommand(command string, args []string) (string, error) {
        debug(command, args)

        // TODO
        return "", fmt.Errorf("unimplemented")
}

func (c *Connection) mustExecCommand(cmd string, args []string) string {
        res, err := c.execCommand(cmd, args)
        assertEq(err, nil, "unexpected error")
        return res
}

func (d *Database) newConnection() *Connection {
        return &Connection{
                db: d,
                tx: nil,
        }
}

func main() {
        panic("unimplemented")
}

And that's it for scaffolding. Now set up the go module and make sure this builds.

$ go mod init gomvcc
go: creating new go.mod: module gomvcc
go: to add module requirements and sums:
        go mod tidy
$ go mod tidy
go: finding module for package github.com/tidwall/btree
go: found github.com/tidwall/btree in github.com/tidwall/btree v1.7.0
$ go build
$ ./gomvcc
panic: unimplemented

goroutine 1 [running]:
main.main()
        /Users/phil/tmp/main.go:166 +0x2c

Great!

Transaction management

When the user asks to begin a transaction, we ask the database for a new transaction and assign it to the current connection.

 func (c *Connection) execCommand(command string, args []string) (string, error) {
         debug(command, args)

+       if command == "begin" {
+               assertEq(c.tx, nil, "no running transactions")
+               c.tx = c.db.newTransaction()
+               c.db.assertValidTransaction(c.tx)
+               return fmt.Sprintf("%d", c.tx.id), nil
+       }
+
         // TODO
         return "", fmt.Errorf("unimplemented")
 }

To abort a transaction, we call the completeTransaction method (which makes sure the database transaction history gets updated) with the AbortedTransaction state.

                return fmt.Sprintf("%d", c.tx.id), nil
        }

+       if command == "abort" {
+               c.db.assertValidTransaction(c.tx)
+               err := c.db.completeTransaction(c.tx, AbortedTransaction)
+               c.tx = nil
+               return "", err
+       }
+
         // TODO
         return "", fmt.Errorf("unimplemented")
 }

And to commit a transaction is similar.

                return "", err
        }

+       if command == "commit" {
+               c.db.assertValidTransaction(c.tx)
+               err := c.db.completeTransaction(c.tx, CommittedTransaction)
+               c.tx = nil
+               return "", err
+       }
+
         // TODO
         return "", fmt.Errorf("unimplemented")
 }

The neat thing about MVCC is that beginning, committing, and aborting a transaction is metadata work. Committing a transaction will get a bit more complex when we add support for Snapshot Isolation and Serializable Isolation, but we'll get to that later. Even then, it will not involve modifying any values we get, set, or delete.

Get, set, delete

Here is where things get fun. As mentioned earlier, the key-value store is actually map[string][]Value. With the more recent versions of a value at the end of the list of values for the key.

For get support, we'll iterate the list of value versions backwards for the key. And we'll call a special new isvisible method to determine if this transaction can see this value. The first value that passes the isvisible test is the correct value for the transaction.

                return "", err
        }

+       if command == "get" {
+               c.db.assertValidTransaction(c.tx)
+
+               key := args[0]
+
+               c.tx.readset.Insert(key)
+
+               for i := len(c.db.store[key]) - 1; i >= 0; i-- {
+                       value := c.db.store[key][i]
+                       debug(value, c.tx, c.db.isvisible(c.tx, value))
+                       if c.db.isvisible(c.tx, value) {
+                               return value.value, nil
+                       }
+               }
+
+               return "", fmt.Errorf("cannot get key that does not exist")
+       }
+
         // TODO
         return "", fmt.Errorf("unimplemented")
 }

I snuck in tracking which keys are read, and we'll also soon sneak in tracking which keys are written. This is necessary in stricter isolation levels. More on that later.

set and delete are similar to get. But this time when we walk the list of value versions, we will set the txEndId for the value to the current transaction id if the value version is visible to this transaction.

Then, for set, we'll append to the value version list with the new version of the value that starts at this current transaction.

                return "", err
        }

+       if command == "set" || command == "delete" {
+               c.db.assertValidTransaction(c.tx)
+
+               key := args[0]
+
+               // Mark all visible versions as now invalid.
+               found := false
+               for i := len(c.db.store[key]) - 1; i >= 0; i-- {
+                       value := &c.db.store[key][i]
+                       debug(value, c.tx, c.db.isvisible(c.tx, *value))
+                       if c.db.isvisible(c.tx, *value) {
+                               value.txEndId = c.tx.id
+                               found = true
+                       }
+               }
+               if command == "delete" && !found {
+                       return "", fmt.Errorf("cannot delete key that does not exist")
+               }
+
+               c.tx.writeset.Insert(key)
+
+               // And add a new version if it's a set command.
+               if command == "set" {
+                       value := args[1]
+                       c.db.store[key] = append(c.db.store[key], Value{
+                               txStartId: c.tx.id,
+                               txEndId:   0,
+                               value:     value,
+                       })
+
+                       return value, nil
+               }
+
+               // Delete ok.
+               return "", nil
+       }
+
        if command == "get" {
                c.db.assertValidTransaction(c.tx)

This time rather than modifying the readset we modify the writeset for the transaction.

And that is how commands get executed!

Let's zoom in to the core of the problem we have mentioned but not implemented: MVCC visibility rules and how they differ by isolation levels.

Isolation levels and MVCC visibility rules

To varying degrees, transaction isolation levels prevent concurrent transactions from messing with each other. The looser isolation levels prevent this almost not at all.

Here is what the 1999 ANSI SQL standard (page 84) has to say.

But as I mentioned in the beginning of the post, we're going to be a bit informal. And we'll mostly refer to Jepsen summaries of each isolation levels.

Read Uncommitted

According to Jepsen, the loosest isolation level, Read Uncommitted, has almost no restrictions. We can merely read the most recent (non-deleted) version of a value, regardless of if the transaction that set it has committed or aborted or not.

func (d *Database) isvisible(t *Transaction, value Value) bool {
        // Read Uncommitted means we simply read the last value
        // written. Even if the transaction that wrote this value has
        // not committed, and even if it has aborted.
        if t.isolation == ReadUncommittedIsolation {
                // We must merely make sure the value has not been
                // deleted.
                return value.txEndId == 0
        }

       assert(false, "unsupported isolation level")
       return false
}

Let's write a test that demonstrates this. We create two transactions, c1 and c2, and set a key in c1. The value set for the key in c1 should be immediately visible if c2 asks for that key. In main_test.go:

package main

import (
        "testing"
)

func TestReadUncommitted(t *testing.T) {
        database := newDatabase()
        database.defaultIsolation = ReadUncommittedIsolation

        c1 := database.newConnection()
        c1.mustExecCommand("begin", nil)

        c2 := database.newConnection()
        c2.mustExecCommand("begin", nil)

        c1.mustExecCommand("set", []string{"x", "hey"})

        // Update is visible to self.
        res := c1.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c1 get x")

        // But since read uncommitted, also available to everyone else.
        res = c2.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c2 get x")

        // And if we delete, that should be respected.
        res = c1.mustExecCommand("delete", []string{"x"})
        assertEq(res, "", "c1 delete x")

        res, err := c1.execCommand("get", []string{"x"})
        assertEq(res, "", "c1 sees no x")
        assertEq(err.Error(), "cannot get key that does not exist", "c1 sees no x")

        res, err = c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 sees no x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 sees no x")
}

Thank you to @glaebhoerl for pointing out that in an earlier version of this post, Read Uncommitted incorrectly made deleted values visible.

That's pretty simple! But also pretty useless if your workload has conflicts. If you can arrange your workload in a way where you know no concurrent transactions will ever read or write conflicting keys though, this could be pretty efficient! The rules will only get more complex (and thus potentially more of a bottleneck) from here on.

But for the most part, people don't use this isolation level. SQLite, Yugabyte, Cockroach, and Postgres don't even implement it. It is also not the default for any major database that does implement it.

Let's get a little stricter.

Read Committed

We'll pull again from Jepsen:

Read committed is a consistency model which strengthens read uncommitted by preventing dirty reads: transactions are not allowed to observe writes from transactions which do not commit.

This sounds pretty simple. In isvisible we'll make sure that the value has a txStartId that is either this transaction or a transaction that has committed. Moreover we will now begin checking against txEndId to make sure the value wasn't deleted by any relevant transaction.

                return value.txEndId == 0
        }

+       // Read Committed means we are allowed to read any values that
+       // are committed at the point in time where we read.
+       if t.isolation == ReadCommittedIsolation {
+               // If the value was created by a transaction that is
+               // not committed, and not this current transaction,
+               // it's no good.
+               if value.txStartId != t.id &&
+                       d.transactionState(value.txStartId).state != CommittedTransaction {
+                       return false
+               }
+
+               // If the value was deleted in this transaction, it's no good.
+               if value.txEndId == t.id {
+                       return false
+               }
+
+               // Or if the value was deleted in some other committed
+               // transaction, it's no good.
+               if value.txEndId > 0 &&
+                       d.transactionState(value.txEndId).state == CommittedTransaction {
+                       return false
+               }
+
+               // Otherwise the value is good.
+               return true
+       }
+
        assert(false, "unsupported isolation level")
        return false
 }

This begins to look useful! We will never read a value that isn't part of a committed transaction (or isn't part of our own transaction). Indeed this is the default isolation level for many databases including Postgres, Yugabyte, Oracle, and SQL Server.

Let's add a test to main_test.go. This is a bit long, but give it a slow read. It is thoroughly commented.

func TestReadCommitted(t *testing.T) {
        database := newDatabase()
        database.defaultIsolation = ReadCommittedIsolation

        c1 := database.newConnection()
        c1.mustExecCommand("begin", nil)

        c2 := database.newConnection()
        c2.mustExecCommand("begin", nil)

        // Local change is visible locally.
        c1.mustExecCommand("set", []string{"x", "hey"})

        res := c1.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c1 get x")

        // Update not available to this transaction since this is not
        // committed.
        res, err := c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        c1.mustExecCommand("commit", nil)

        // Now that it's been committed, it's visible in c2.
        res = c2.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c2 get x")

        c3 := database.newConnection()
        c3.mustExecCommand("begin", nil)

        // Local change is visible locally.
        c3.mustExecCommand("set", []string{"x", "yall"})

        res = c3.mustExecCommand("get", []string{"x"})
        assertEq(res, "yall", "c3 get x")

        // But not on the other commit, again.
        res = c2.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c2 get x")

        c3.mustExecCommand("abort", nil)

        // And still not, if the other transaction aborted.
        res = c2.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c2 get x")

        // And if we delete it, it should show up deleted locally.
        c2.mustExecCommand("delete", []string{"x"})

        res, err = c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        c2.mustExecCommand("commit", nil)

        // It should also show up as deleted in new transactions now
        // that it has been committed.
        c4 := database.newConnection()
        c4.mustExecCommand("begin", nil)

        res, err = c4.execCommand("get", []string{"x"})
        assertEq(res, "", "c4 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c4 get x")
}

Again this seems great. However! You can easily get inconsistent data within a transaction at this isolation level. If the transaction A has multiple statements it can see different results per statement, even if the transaction A did not modify data. Another transaction B may have committed changes between two statements in this transaction A.

Let's get a little stricter.

Repeatable Read

Again as Jepsen says, Repeatable Read is the same as Read Committed but with the following anomaly not allowed (quoting from the ANSI SQL 1999 standard):

P2 (“Non-repeatable read”): SQL-transaction T1 reads a row. SQL-transaction T2 then modifies or deletes that row and performs a COMMIT. If T1 then attempts to reread the row, it may receive the modified value or discover that the row has been deleted.

To support this, we will add additional checks for the Read Committed logic that make sure the value was not created and not deleted within a transaction that started before this transaction started.

As it happens, this is the same logic that will be necessary for Snapshot Isolation and Serializable Isolation. The additional logic (that makes Snapshot Isolation and Serializable Isolation different) happens at commit time.

                return true
        }

-       assert(false, "unsupported isolation level")
-       return false
+       // Repeatable Read, Snapshot Isolation, and Serializable
+       // further restricts Read Committed so only versions from
+       // transactions that completed before this one started are
+       // visible.
+
+       // Snapshot Isolation and Serializable will do additional
+       // checks at commit time.
+       assert(t.isolation == RepeatableReadIsolation ||
+               t.isolation == SnapshotIsolation ||
+               t.isolation == SerializableIsolation, "invalid isolation level")
+       // Ignore values from transactions started after this one.
+       if value.txStartId > t.id {
+               return false
+       }
+
+       // Ignore values created from transactions in progress when
+       // this one started.
+       if t.inprogress.Contains(value.txStartId) {
+               return false
+       }
+
+       // If the value was created by a transaction that is not
+       // committed, and not this current transaction, it's no good.
+       if d.transactionState(value.txStartId).state != CommittedTransaction &&
+               value.txStartId != t.id {
+               return false
+       }
+
+       // If the value was deleted in this transaction, it's no good.
+       if value.txEndId == t.id {
+               return false
+       }
+
+       // Or if the value was deleted in some other committed
+       // transaction that started before this one, it's no good.
+       if value.txEndId < t.id &&
+               value.txEndId > 0 &&
+               d.transactionState(value.txEndId).state == CommittedTransaction &&
+               !t.inprogress.Contains(value.txEndId) {
+               return false
+       }
+
+       return true
 }

 type Connection struct {

How do I derive these rules? Mostly by writing tests that should pass or fail and seeing what doesn't make sense. I tried to steal from existing projects but these rules were not so simple to discover. Which is part of what I hope makes this project particularly useful to look at.

Let's write a test for Repeatable Read. Again, the test is long but well commented.

func TestRepeatableRead(t *testing.T) {
        database := newDatabase()
        database.defaultIsolation = RepeatableReadIsolation

        c1 := database.newConnection()
        c1.mustExecCommand("begin", nil)

        c2 := database.newConnection()
        c2.mustExecCommand("begin", nil)

        // Local change is visible locally.
        c1.mustExecCommand("set", []string{"x", "hey"})
        res := c1.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c1 get x")

        // Update not available to this transaction since this is not
        // committed.
        res, err := c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        c1.mustExecCommand("commit", nil)

        // Even after committing, it's not visible in an existing
        // transaction.
        res, err = c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        // But is available in a new transaction.
        c3 := database.newConnection()
        c3.mustExecCommand("begin", nil)

        res = c3.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c3 get x")

        // Local change is visible locally.
        c3.mustExecCommand("set", []string{"x", "yall"})
        res = c3.mustExecCommand("get", []string{"x"})
        assertEq(res, "yall", "c3 get x")

        // But not on the other commit, again.
        res, err = c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        c3.mustExecCommand("abort", nil)

        // And still not, regardless of abort, because it's an older
        // transaction.
        res, err = c2.execCommand("get", []string{"x"})
        assertEq(res, "", "c2 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        // And again still the aborted set is still not on a new
        // transaction.
        c4 := database.newConnection()
        res = c4.mustExecCommand("begin", nil)

        res = c4.mustExecCommand("get", []string{"x"})
        assertEq(res, "hey", "c4 get x")

        c4.mustExecCommand("delete", []string{"x"})
        c4.mustExecCommand("commit", nil)

        // But the delete is visible to new transactions now that this
        // has been committed.
        c5 := database.newConnection()
        res = c5.mustExecCommand("begin", nil)

        res, err = c5.execCommand("get", []string{"x"})
        assertEq(res, "", "c5 get x")
        assertEq(err.Error(), "cannot get key that does not exist", "c5 get x")
}

Let's get stricter!

Snapshot Isolation

Back to Jepsen for a definition:

In a snapshot isolated system, each transaction appears to operate on an independent, consistent snapshot of the database. Its changes are visible only to that transaction until commit time, when all changes become visible atomically to any transaction which begins at a later time. If transaction T1 has modified an object x, and another transaction T2 committed a write to x after T1’s snapshot began, and before T1’s commit, then T1 must abort.

So Snapshot Isolation is the same as Repeatable Read but with one additional rule: the keys written by any two concurrent committed transactions must not overlap.

This is why we tracked writeset. Every time a transaction modified or deleted a key, we added it to the transaction's writeset. To make sure we abort correctly, we'll add a conflict check to the commit step. (This idea is also well documented in A critique of snapshot isolation. This paper can be hard to find. Email me if you want a copy.)

When a transaction A goes to commit, it will run a conflict test for any transaction B that has committed since this transaction A started.

Serializable Isolation is going to have a similar check. So we'll add a helper for iterating through all relevant transactions, running a check function for any transaction that has committed.

func (d *Database) hasConflict(t1 *Transaction, conflictFn func(*Transaction, *Transaction) bool) bool {
        iter := d.transactions.Iter()

        // First see if there is any conflict with transactions that
        // were in progress when this one started.
        inprogressIter := t1.inprogress.Iter()
        for ok := inprogressIter.First(); ok; ok = inprogressIter.Next() {
                id := inprogressIter.Key()
                found := iter.Seek(id)
                if !found {
                        continue
                }
                t2 := iter.Value()
                if t2.state == CommittedTransaction {
                        if conflictFn(t1, &t2) {
                                return true
                        }
                }
        }

        // Then see if there is any conflict with transactions that
        // started and committed after this one started.
        for id := t1.id; id < d.nextTransactionId; id++ {
                found := iter.Seek(id)
                if !found {
                        continue
                }

                t2 := iter.Value()
                if t2.state == CommittedTransaction {
                        if conflictFn(t1, &t2) {
                                return true
                        }
                }
        }

        return false
}

It was around this point that I decided I did really need a B-Tree implementation and could not just stick to vanilla Go data structures.

Now we can modify completeTransaction to do this check if the transaction intends to commit. If the current transaction A's write set intersects with any other transaction B committed since transaction A started, we must abort.

 func (d *Database) completeTransaction(t *Transaction, state TransactionState) error {
         debug("completing transaction ", t.id)

+
+       if state == CommittedTransaction {
+               // Snapshot Isolation imposes the additional constraint that
+               // no transaction A may commit after writing any of the same
+               // keys as transaction B has written and committed during
+               // transaction A's life.
+               if t.isolation == SnapshotIsolation && d.hasConflict(t, func(t1 *Transaction, t2 *Transaction) bool {
+                       return setsShareItem(t1.writeset, t2.writeset)
+               }) {
+                       d.completeTransaction(t, AbortedTransaction)
+                       return fmt.Errorf("write-write conflict")
+               }
+       }
+
         // Update transactions.
         t.state = state
         d.transactions.Set(t.id, *t)

Lastly, the definition of setsShareItem.

func setsShareItem(s1 btree.Set[string], s2 btree.Set[string]) bool {
        s1Iter := s1.Iter()
        s2Iter := s2.Iter()
        for ok := s1Iter.First(); ok; ok = s1Iter.Next() {
                s1Key := s1Iter.Key()
                found := s2Iter.Seek(s1Key)
                if found {
                        return true
                }
        }

        return false
}

Since Snapshot Isolation shares all the same visibility rules as Repeatable Read, the tests get to be a little simpler! We'll simply test that two transactions attempting to commit a write to the same key fail. Or specifically: that the second transaction cannot commit.

func TestSnapshotIsolation_writewrite_conflict(t *testing.T) {
        database := newDatabase()
        database.defaultIsolation = SnapshotIsolation

        c1 := database.newConnection()
        c1.mustExecCommand("begin", nil)

        c2 := database.newConnection()
        c2.mustExecCommand("begin", nil)

        c3 := database.newConnection()
        c3.mustExecCommand("begin", nil)

        c1.mustExecCommand("set", []string{"x", "hey"})
        c1.mustExecCommand("commit", nil)

        c2.mustExecCommand("set", []string{"x", "hey"})

        res, err := c2.execCommand("commit", nil)
        assertEq(res, "", "c2 commit")
        assertEq(err.Error(), "write-write conflict", "c2 commit")

        // But unrelated keys cause no conflict.
        c3.mustExecCommand("set", []string{"y", "no conflict"})
        c3.mustExecCommand("commit", nil)
}

Not bad! But let's get stricter.

Upon further discussion with Alex Miller, and after reviewing A Critique of ANSI SQL Isolation Levels, the difference I am trying to suggest (between Repeatable Read and Snapshot Isolation) likely does not exist. A Critique of ANSI SQL Isolation Levels mentions Repeatable Read must not exhibit P4 (Lost Update) anomalies. And it mentions that you must check for read-write conflicts to avoid these. Therefore it seems likely that you can't easily separate Repeatable Read from Snapshot Isolation when implemented using MVCC. The differences between Repeatable Read and Snapshot Isolation may more readily show up when implementing transactions the classical way with Two-Phase Locking.

To reiterate, with MVCC and optimistic concurrency control, correct implementations of Repeatable Read and Snapshot Isolation do not seem to be distinguishable. Both require write-write conflict detection.

Serializable Isolation

In terms of end-result, this is the simplest isolation level to reason about. Serializable Isolation must appear as if only a single transaction were executing at a time. Some systems, like SQLite and TigerBeetle, do Actually Serial Execution where only one transaction runs at a time. But few databases implement Serializable like this because it removes a number of fair concurrent execution histories. For example, two concurrent read-only transactions.

Postgres implements serializability via Serializable Snapshot Isolation. MySQL implements serializability via Two-Phase Locking. FoundationDB implements serializability via sequential timestamp assignment and conflict detection.

But the paper, A critique of snapshot isolation, provides a simple (though not necessarily efficient; I have no clue) approach via what they call Write Snapshot Isolation. In their algorithm, if any two transactions read and write set intersect (but not write and write set intersect), the transaction should be aborted. And this (plus Repeatable Read rules) is sufficient for Serializability.

I leave it to that paper for the proof of correctness. In terms of implementing it though it's quite simple and very similar to the Snapshot Isolation we already mentioned.

Inside of completeTransaction add:

                }) {
                        d.completeTransaction(t, AbortedTransaction)
                        return fmt.Errorf("write-write conflict")
+               }
+
+               // Serializable Isolation imposes the additional constraint that
+               // no transaction A may commit after reading any of the same
+               // keys as transaction B has written and committed during
+               // transaction A's life, or vice-versa.
+               if t.isolation == SerializableIsolation && d.hasConflict(t, func(t1 *Transaction, t2 *Transaction) bool {
+                       return setsShareItem(t1.readset, t2.writeset) ||
+                               setsShareItem(t1.writeset, t2.readset)
+               }) {
+                       d.completeTransaction(t, AbortedTransaction)
+                       return fmt.Errorf("read-write conflict")
                }
        }

And if we add a test for read-write conflicts:

func TestSerializableIsolation_readwrite_conflict(t *testing.T) {
        database := newDatabase()
        database.defaultIsolation = SerializableIsolation

        c1 := database.newConnection()
        c1.mustExecCommand("begin", nil)

        c2 := database.newConnection()
        c2.mustExecCommand("begin", nil)

        c3 := database.newConnection()
        c3.mustExecCommand("begin", nil)

        c1.mustExecCommand("set", []string{"x", "hey"})
        c1.mustExecCommand("commit", nil)

        _, err := c2.execCommand("get", []string{"x"})
        assertEq(err.Error(), "cannot get key that does not exist", "c2 get x")

        res, err := c2.execCommand("commit", nil)
        assertEq(res, "", "c2 commit")
        assertEq(err.Error(), "read-write conflict", "c2 commit")

        // But unrelated keys cause no conflict.
        c3.mustExecCommand("set", []string{"y", "no conflict"})
        c3.mustExecCommand("commit", nil)
}

We see it work! And that's it for a basic implementation of MVCC and major transaction isolation levels.

Production-quality testing

There are two major projects I'm aware of that help you test transaction implementations: Elle and Hermitage. These are probably where I'd go looking if I were implementing this for real.

This project took me long enough on its own and I felt reasonably comfortable with my tests that the gist of my logic was right that I did not test further. For that reason it surely has bugs.

Vacuuming and cleanup

One of the major things this implementation does not do is cleaning up old data. Eventually, older versions of values will be required by no transactions. They should be removed from the value version array. Similarly, eventually older transactions will be required by no transactions. They should be removed from the database transaction history list.

Even if we had the vacuuming process in place though, what about some extreme use patterns. What if a key's value was always going to be 1GB long. And what if multiple transactions made only small changes to the 1GB data. We'd be duplicating a lot of the value across versions.

It sounds less extreme when thinking about storing rows of data rather than key-value data. If a user has 100 columns and only updates one column a number of times, in our scheme we'd end up storing a ton of duplicate cell data for a row.

This is a real-world issue in Postgres that was called out by Andy Pavlo and the Ottertune folks. It turns out that Postgres alone among major databases stores the entire value for every version. In contrast, other major databases like MySQL store a diff.

Conclusion

This post only begins to demonstrate that database behavior differs quite a bit both in terms of results and in terms of optimizations. Everyone implements the ideas differently and to varying degrees.

Moreover, we have only begun to implement the behavior a real SQL database supports. For example, how do visibility rules and conflict detection work with range queries? What about sub-transactions, and save points? These will have to be covered another time.

Hopefully seeing this simple implementation of MVCC and visibility rules helps to clarify at least some of the basics.

Here's a new post walking through an implementation of MVCC and major SQL transaction isolation levels, in 400 lines of Go code.

These ideas might sound esoteric, but they impact almost every developer using any database.https://t.co/crFKM74R5h pic.twitter.com/o9awTPpvvx
— Phil Eaton (@eatonphil) May 16, 2024

Using build.rs to integrate rust applications with system libraries like a pro

The NeoSmart Files — Mon, 13 May 2024 19:24:30 GMT

I’m happy to announce the release of version 0.2 of the rsconf crate, with new support for informing Cargo about the presence of custom cfg keys and values (to work around a major change that has resulted in hundreds of … Continue reading →

The post Using build.rs to integrate rust applications with system libraries like a pro first appeared on The NeoSmart Files.

Writing a simple Javascript async message queue server - Part II

Bytepawn - Marton Trencseni — Sat, 11 May 2024 01:00:00 GMT

I write a somewhat more complicated, but still relatively simple async message queue server in Javascript.

CLI text processing with GNU Coreutils book announcement

learnbyexample — Wed, 08 May 2024 03:00:00 GMT

Hello!

I am pleased to announce a new version of my CLI text processing with GNU Coreutils ebook. Examples, descriptions and external links were updated/corrected and 100+ exercises were added.

You might be already aware of popular coreutils commands like head, tail, tr, sort and so on. This book will teach you more than twenty of such specialized text processing tools provided by the GNU coreutils package.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of the ebook for FREE till 10-April-2024. You can still pay if you wish ;)

The following bundles are heavily discounted:

All books bundle is $12 (normal price $32)
Linux CLI Text Processing bundle is $6 (normal price $20)

What's new?🔗

GNU coreutils package version updated to 9.1
Added 100+ exercises
In general, many of the examples, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to CLI text processing with GNU Coreutils
New cover image

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Testimonials🔗

In my opinion the book does a great job of quickly presenting examples of how commands can be used and then paired up to achieve new or interesting ways of manipulating data. Throughout the text there are little highlights offering tips on extra functionality or limitations of certain commands. For instance, when discussing the shuf command we're warned that shuf will not work with multiple files. However, we can merge multiple files together (using the cat command) and then pass them to shuf. These little gems of wisdom add a dimension to the book and will likely save the reader some time wondering why their scripts are not working as expected.

— book review by Jesse Smith on distrowatch.com

I discovered your books recently and they’re awesome, thank you! As a 20 year *nix they made me realize how much more there are to these rock solid and ancient tools, once you spend the time to actually learn the intricacies of them.

— feedback on reddit

Table of Contents🔗

Preface
Introduction
cat and tac
head and tail
tr
cut
seq
shuf
paste
pr
fold and fmt
sort
uniq
comm
join
nl
wc
split
csplit
expand and unexpand
basename and dirname
What next?

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/cli_text_processing_coreutils/introduction.html.

GitHub repo🔗

Visit https://github.com/learnbyexample/cli_text_processing_coreutils for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/cli_text_processing_coreutils/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Exploring 50 user interfaces for AI code suggestions

Austin Z. Henley's Blog — Wed, 08 May 2024 00:00:01 GMT

https://austinhenley.com/blog/intellicode.html

Using DuckDB to seamlessly query a large parquet file over HTTP

Weblog on marginalia.nu — Sun, 05 May 2024 03:00:00 GMT

A neat property of the parquet file format is that it’s designed with block I/O in mind, so that when you are interested in only parts of the contents of a file, it’s possible to some extent to only read that data. Many tools are aware of this property, and DuckDB is one of them. Depending on which circles you run in, a lesser known aspect of HTTP is range requests, where you specify which bytes in a file to be retrieved. It’s possible to combine this trio of properties to read remote parquet files directly in DuckDB.

The Rabbit R1 Pricing Myth

Dan Palmer — Sun, 05 May 2024 02:00:00 GMT

Two new “AI” devices have just launched, the Humane AI Pin, and the Rabbit R1. Both have received broadly negative reviews, criticising a lack of features, alongside the classic issues with modern AI systems – hallucination and confidently incorrect answers. The main distinguishing factor between the two devices has been price. Where the AI Pin sells for $700 with a $24 per month subscription, the R1 is “only” $199 with no subscription. This is however, a myth, and while it might be temporarily true, something is going to have to change.

Writing a simple Javascript async message queue server - Part I

Bytepawn - Marton Trencseni — Sun, 05 May 2024 01:00:00 GMT

I write a simple, bi-directional async message queue server in Javascript.

The CP/M emulator development continues

Steve Kemp's Blog — Sat, 04 May 2024 15:00:00 GMT

In my previous post I introduced a toy CP/M Emulator I'd been working on.

At the time it was capable of running the Infocom text-based adventure games, so I thought it was done. Of course I also wanted to run Microsoft's original BASIC and it turned out that was a challenge because the coding of their interpreter didn't use the standard CP/M entry-point for making syscalls (call 0x0005).

Instead of calling 0x0005 to invoke the BDOS/BIOS functions the BASIC interpreter used the single-byte CALL instructions which are available on the Z80 processor. There are a bunch of these instructions:

RST 00
RST 08
RST 10
RST 18
RST 20
RST 28
RST 30
RST 38

Each of those instructions is equivalent to a call instruction with a fixed offset, "call 0x0010", "call 0x0020", etc. I had to rework the emulator to cope with this approach, which causes repetition but nothing too surprising. The end result is that now my emulator can run Microsoft Basic, Tasty Basic, and some more programs.

Things work but a couple of the syscalls are of the form "Return true if there is a pending keystroke", or "wait until there is keyboard input present and return the first character". I have some busy-loops which peg the CPU, which sucks but works. On the downside running the code on a MacOS machine has some weird issues with repeated keys and similar. So I need to look into fixing that for my own sense of peace.

I put together a little repository of binaries for playing with though, and that's been helpful. My emulator has a special flag which treats sub-directories as "Drives". So A: points to A/, B: points to B/, etc. That makes distributing and working with things easy!

https://github.com/skx/cpm-dist
- Binaries for running on CP/M systems, organized by theme.

The Simply Complicated Automatic Sprinkler

Pete Keen — Thu, 02 May 2024 03:00:00 GMT

I've never owned a sprinkler before. I've just never had a need for one.

Until this past Monday, when I installed some grass seed and couldn't promise myself that I'd water it twice a day every day for the next month as one is supposed to do.

A normal person would just buy a mechanical or stand-alone timer device and be done with it. Not me. No way.

My first thought (a completely normal one, mind you) was to grab an esp32 and a relay board out of my parts bin and liberally apply ESPHome to the problem. This didn't happen because I recently cleaned my office and put those parts in a bin that I can't quite find right now.

cough. Anyway.

My fallback plan was to grab a Z-Wave relay that was more readily at hand and just ("just") use Home Assistant as the timer.

Parts List

Zooz Z-Wave ZEN17 Universal Relay
Orbit 3/4" inline sprinker valve
3/4" NPT to garden hose thread adapters
Generic 24VAC transformer
5 position terminal block
18AWG stranded two conductor silicone jacketed wire
some random crimp on butt connectors I had laying around
A sprinkler

In-ground sprinkler valves in the US pretty universally use 24V AC power. This is a very common signaling voltage for HVAC and other controls in residental settings so it makes sense. For example, non-smart doorbells are also generally 24-ish VAC.

The neatest thing about the Zooz ZEN17 (and it's friend the ZEN16) is that it can be powered by USB-C or 12-24 volt DC or AC. That means I can get away with just one power supply for the whole contraption without having to get any kind of fancy voltage regulator.

Sprinker valves come in a variety of sizes, none of which are directly compatible with garden hoses. In the US and Canada residental garden hoses have a universal connector size and thread: 3/4" NH (national hose) (or GHT (garden hose thread)), otherwise known as ANSI B1.20.7. So, I needed some thread adapters. Your local hardware store or supermegaglobal online bazaar named after a river in Brazil will provide you with many options.

Assembly

I didn't take any good pictures of the prototyping and assembly process. Hopefully these will be sufficient.

As you can see, the power supply is connected directly to the relay (via a small terminal block affixed to the relay with a command strip) and the red and black wires lead to the valve. One of the valve wires is connected directly to the power supply and the other is connected to the NO (normal open) terminal on one of the relays in the ZEN17. A short jumper wire is connected from the other leg of the power supply to the C (common) terminal on the same relay.

Here's a shot of the valve itself. You can see the green valve body, the black solenoid that activates the valve, and the brass thread adapters. You can also see the craptastic butt splice things I used, which will be replaced with proper waterproof junctions as soon as they arrive.

The grey and orange piece on the right is a quick release adapter. I have quick releases on the spigot and all of the hoses, which lets me easily install and remove the valve as needed.

Home Assistant

A generic relay isn't much good without something to control it, so let's turn to Home Assistant.

The central component of the sprinkler controller is a 5 minute timer helper which ties with two automations:

Sprinkler Start triggers at 8:30am and 5:30pm and starts the timer, but only if the forecast precipitation probability from OpenWeatherMap is less than 50%.
Sprinkler Relay triggers whenever the timer changes state.

Sprinkler Relay uses a Choice action with two branches:

Timer active and relay off -> turn the relay on
Timer idle and relay on -> turn the relay off

I also have a Time Pattern trigger on Sprinkler Relay that runs every minute. This acts as sort of a failsafe, so that if the timer ends during a Home Assistant restart, the sprinkler will shut off with at most an extra minute of runtime.

Here's Sprinkler Start in YAML:

alias: Sprinkler Start
        description: ""
        trigger:
        - platform: time
        at: "08:30:00"
        - platform: time
        at: "17:30:00"
        condition:
        - condition: numeric_state
        entity_id: sensor.openweathermap_forecast_precipitation_probability
        below: 50
        action:
        - service: timer.start
        metadata: {}
        data: {}
        target:
        entity_id: timer.sprinkler_runtime
        - service: notify.all_phones
        metadata: {}
        data:
        message: Sprinkler has started
        title: Sprinkler
        mode: single

and here's Sprinkler Relay:

alias: Sprinkler Relay
        description: ""
        trigger:
        - platform: state
        entity_id:
        - timer.sprinkler_runtime
        - platform: time_pattern
        minutes: "*"
        condition: []
        action:
        - choose:
        - conditions:
        - condition: state
        entity_id: timer.sprinkler_runtime
        state: idle
        - condition: state
        entity_id: switch.universal_relay_1
        state: "on"
        sequence:
        - service: switch.turn_off
        metadata: {}
        data: {}
        target:
        entity_id: switch.universal_relay_1
        - conditions:
        - condition: state
        entity_id: timer.sprinkler_runtime
        state: active
        - condition: state
        entity_id: switch.universal_relay_1
        state: "off"
        sequence:
        - service: switch.turn_on
        metadata: {}
        data: {}
        target:
        entity_id: switch.universal_relay_1
        mode: single

So, what do I have after all of that? I have a sprinkler valve that could be automated with a wrist watch but is actually automated by a Rube Goldberg-esque stack of technology, including radio protocols, ethernet, virtual machines, Python, Javascript, and Tailscale (because of course).

Simple!

Sloc Cloc and Code - ULOC Unique Lines of Code

Ben E. C. Boyter — Wed, 01 May 2024 03:00:00 GMT

I recently pushed out a new release of Sloc Cloc and Code (scc) with two main pieces of functionality. The first being accurate .gitignore support. The latter being the first new feature to hit the codebase in a long time in the form of a new metric you can access Unique Lines of Code or ULOC.

A few years ago exwhyz (who appears to have dropped off the internet) posted a feature request to include this new metric into scc’s outputs.

Paper review: A Comparison of Approaches to Advertising Measurement

Bytepawn - Marton Trencseni — Wed, 01 May 2024 01:00:00 GMT

Why are Randomized Controlled Trials (RCTs, known as A/B testing in much of the industry) testing is widely regarded as the golden standard of causal inference? What else can a Data Scientist do if A/B testing is not possible, and why are those alternatives inferior to A/B testing?

This papers shows, using 15 experiments (for ads on Facebook) where a RCT was conducted, that common observational methods (run on the Facebook data, by ignoring the control group) severely mis-estimate the true treatment life (as measured by the RCT), often by a factor of 3x or more. This is true, even though Facebook has (i) very large sample sizes, and, (ii) very high quality data (per-user feature vector) about its users which are used in the observational methods. This should be a major red flag for Data Scientists working on common marketing measurements (such as marketing campaigns) using observational methods.

Interactive GNU awk tutorial

learnbyexample — Tue, 30 Apr 2024 03:00:00 GMT

Know command line basics and want to learn the GNU awk command? Check out my interactive TUI app that gives a brief tour of this popular text processing command.

Installation🔗

This app is available on PyPI as awktutorial. Example installation instructions are shown below, adjust them based on your preferences and OS.

# virtual environment
$ python3 -m venv textual_apps
$ cd textual_apps
$ source bin/activate
$ pip install awktutorial

# launch the app
$ awktutorial

To run the app without having to enter the virtual environment again, add this alias to .bashrc (or equivalent):

# you'll have to change the path
alias awktutorial='/path/to/textual_apps/bin/awktutorial'

As an alternative to manually managing such virtual environments, you can use https://github.com/pypa/pipx instead:

$ pipx install awktutorial
$ awktutorial

As yet another alternative, you can install textual (see Textual documentation for more details), clone my TUI-apps repository repository and run the awk_tutorial.py file.

Adjust the terminal dimensions for the widgets to appear properly, for example 84x25 (characters x lines).

Ebook🔗

See my CLI text processing with GNU awk ebook to learn GNU awk with hundreds of examples and exercises.

Feedback🔗

I'd highly appreciate your feedback. Please file an issue if there are bugs, crashes, etc.

Hope you'll find this TUI app useful. Happy learning :)

How I would redesign Copilot Workspace

Austin Z. Henley's Blog — Tue, 30 Apr 2024 01:00:01 GMT

https://austinhenley.com/blog/copilotworkspace.html

Mistakes that data science students make

Austin Z. Henley's Blog — Sun, 28 Apr 2024 23:00:01 GMT

https://austinhenley.com/blog/datasciencemistakes.html

Barton Bike Lanes Letter

Pxtl.ca — Sun, 28 Apr 2024 03:00:00 GMT

So the other day I sent in an e-mail to the cycling office at the City of Hamilton (as well as the relevant councilors) to request a cycling accommodation in my area. I figured I may as well include the text of the letter here as well.

Open Source Summit: Bringing CI/CD Practices to Machine Learning with MLOps

tedious ramblings — Sat, 27 Apr 2024 04:00:00 GMT

Robert Hafner at the Open Source Summit: Bringing CI/CD Practices to Machine Learning with MLOps

“Reading” Articles via Podcast Software

Beetle Space — Fri, 26 Apr 2024 10:00:00 GMT

I wrote a while ago on reading articles offline by printing them on paper. The benefit was reduced eye strain and a superior reading experience, without being tethered to any device.

I continue to consume articles this way, with one exception: Long-form articles that are 20+ pages long. I often …

One hundred million little queries

Ben E. C. Boyter — Tue, 23 Apr 2024 03:00:00 GMT

I recently X’ed/Tweeted about how my searchcode.com since Nov 2022 using its custom index https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/ had processed over 50 million queries with a rolling average time around 80ms to process them.

That was a low-ball estimate though. I actually checked and its actually just over 100,000,000 queries. Which I find astonishing considering its running though a index I wrote.

However that number excludes all of the attempted queries. As you may expect any public search engine, especially one with a public unauthenticated API is pretty constantly hammered by bots. So what about those?

The Culture of Powerpoint

Bytepawn - Marton Trencseni — Thu, 18 Apr 2024 01:00:00 GMT

Last year, in 2023, Dennis Austin, one of the original developers of Powerpoint passed away. The news made it to the front page on Hacker News, and prompted a lively discussion of the merits of Powerpoint itself. I posted a top-level comment, which itself sparked a lively thread of responses. In this post I will expand and extend my points about Powerpoint in a business settings, after pointing to hard-learned lessons from Edward Tufte and Jeff Bezos.

Query Parsing and Understanding

Weblog on marginalia.nu — Wed, 17 Apr 2024 03:00:00 GMT

Been working on improving Marginalia Search query parsing and understanding. This is going to be a pretty long update, as it’s a few months’ work.

Apart from cleaning up the somewhat messy query parsing code, a problem I’m trying to address is that the search engine is currently only good at dealing with fairly focused queries, they don’t need to be short, but if you try to qualify a search that is too broad by adding more terms, it often doesn’t produce anything useful.

A simple golang CP/M emulator

Steve Kemp's Blog — Tue, 16 Apr 2024 00:00:00 GMT

A couple of years ago I wrote a simple text-based adventure game in Z80 assembly language, to amuse our child. The game was written for CP/M, because that is the operating system my single-board Z80-based computer runs upon.

Later I ported the game to the ZX Spectrum 48k.

Recently I went through a burst of enthusiasm and started to overhaul the code a little, adding word-wrapping and fixing a couple of bugs. That lead to a new release, and also a brief amount of (positive) feedback on hacker news.

After mulling it over I realized that the number of CP/M BIOS functions I was using was very minimal, almost only the minimum you'd expect:

Write a character to STDOUT.
Write a $-terminated string to STDOUT.
Read a character from STDIN.
Read a line from STDIN.

It crossed my mind that implementing those syscalls should be trivial, and if I bundled implementations with a Z80 emulator library I'd have a means of running the game without a real CP/M installation, and without using the ZX Spectrum port.

So I picked a golang-based Z80 emulator, and started hacking.

After a day I had a working system, and I added a few more syscalls:

Open File, Create File, Delete File, Close File.
Console I/O.
Read Record.

After that? I can now play Zork 1, Zork 2, Zork 3, and The Hitchhiker's guide to the Galaxy, from Infocom.

I suspect I'm "done" for now, though it might be nice to add WriteRecord and the other missing functions there's no obvious use for yet another CP/M, especially with a CCP.

Still CP/M. In Golang. For text-based adventures:

https://github.com/skx/cpmulator

Student paper: The Exciting Physics of an Excited Universe

Bytepawn - Marton Trencseni — Thu, 11 Apr 2024 01:00:00 GMT

Based on the book Compact Stars by N. K. Glendenning a general relativistic treatment of pressure and energy in compact stars is given, followed by simple models of neutron and quark stars. Hypothetical strange quark matter and strange stars are introduced highlighting the importance of sub-millisecond pulsar detections. (Note: this is a student paper I wrote sometime in 2008-2010.)

Deep Bug

Weblog on marginalia.nu — Wed, 10 Apr 2024 03:00:00 GMT

The project has been haunted by a mysterious bug since sometime February. It relates to the code that constructs the index, particularly the code that merges partial indices.

In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement.

You can conceptualize the revese index itself as two files, one with offset pointers into another file, which has sorted numbers. This code runs after each partition finishes crawling and processing its data, and has a run time of about 4 hours.

What makes a great technical blog

Notes on software development — Wed, 10 Apr 2024 03:00:00 GMT

I want to explain why the blogs in My favorite technical blogs are my favorite. That page is solely about non-corporate tech blogs. So this post is too. I'll have to make another list for favorite corporate tech blogs.

In short, they:

Tackle hard and confusing topics
Show working code
Make things simpler
Write regularly
Talk about tradeoffs and downsides
Avoid internet slang, memes, swearing, sarcasm, and ranting

Tackle hard and confusing topics

There are a number of problems in programming and computer science where otherwise knowledgeable programmers have to start mumbling about, or revert to cliches or group-think, because they aren't sure.

These are the best topics you can possibly dive deep into. And my favorite writers do exactly this.

They write about durability guarantees of disks and filesystems. They write about common pitfalls in benchmarking. They write about database consistency anomalies. They write about threading and IO models.

And they write about it by showing concrete examples and concrete logic so you can learn how to stop handwaving on the topic.

Their writing helps you come out with a useful mental model you can apply to your own problems.

And you know, sometimes it's not about the topic being obscure. Good writers have the ability to tackle a boring topic in an interesting light. Maybe by digging deeper into a root cause. Or showing you the history behind the scenes.

Moreover, my favorite writers don't know everything. But they also don't pretend to know everything. They're quick to admit they don't understand something and ask for help from their readers.

Show working code

I love to see complete working code in a post. In contrast there are many projects that start out simple and people write an article that covers the project at a high level. But they keep working on the project and it becomes more complex.

It's not always easy to follow commits over time.

Eli Bendersky and Serge Zaitsev are particularly great at developing small but meaningful projects in a single post or short series.

On the other hand, if people only did this, we wouldn't hear about the development of long-running projects like V8 or Postgres. So I guess this style has limits. And I don't penalize people talking about long-running projects for not showing working code.

Make things simpler

One of the marks of a good writer is that you can make complex topics simple. And not just by being reductive. Though sometimes even being reductive is useful for education.

In contrast I sometimes see articles by less experienced writers and I marvel how they make a simple topic so complex. I recognize this because I was absolutely like that 10 years ago, if not 5 years ago.

Write regularly

My favorite blogs typically get a new post at least once a month. Some people, like Murat, write once a week.

I think the practice probably does improve your writing but mostly it's that they keep my attention by publishing regularly!

Talk about tradeoffs and downsides

Nothing builds trust like talking about the issues with something you built. No project is perfect. And to ignore the downsides risks seeming like you don't know or understand them.

So the writers I like the most talk about decisions in context. They talk about the good and the bad.

Avoid internet slang, memes, swearing, sarcasm, and ranting

There's no way I can think of talking about this without sounding super lame.

One thing I've noticed, particularly among younger colleagues, is the use of memes or swearing or using 4chan slang or using sarcasm. I used to write like this 10 years ago too.

There is a chunk of your audience who won't care. The problem is that there's also a chunk of your (potential) audience who definitely does care. There's even a chunk of your audience who may not care but just won't understand (i.e. non-native English speakers).

I have friends and folks I respect who write very well. But that also are also overly, unnecessarily edgy when they write. I don't like sharing these posts because I don't want to unnecessarily offend or turn off people either.

Closing thoughts

It would be boring if everyone wrote the same way. I'm glad the internet is fun and weird. But I wanted to share a few things that go into my favorite technical blogs that I'm always happy to refer people to.

I wrote a short post on what I think makes a great technical blog.https://t.co/QRFtQyQyU5 pic.twitter.com/QpsQC90EX5
— Phil Eaton (@eatonphil) April 10, 2024

Paper: Monte Carlo Experiments of Network Effects in Randomized Controlled Trials

Bytepawn - Marton Trencseni — Tue, 09 Apr 2024 01:00:00 GMT

Connecting the dots

bayindirh — Mon, 08 Apr 2024 03:00:00 GMT

You can’t connect the dots looking forward; you can only connect them looking backwards. -- Steve Jobs

Normally, I’m not a quote person, and don’t get inspired by them much. Sometimes they provide great insights, but even in that case, I prefer to strain them through my own experience and see what remains.

However, in recent times, I’m experiencing a domino effect in my digital life and organization. I was able to realize almost all my dreams without realizing. This made me wonder how I did this in a short amount of time, at least for me.

I have a homepage (or landing page if you prefer the term) for a long time now. It all started with flavors.me, a now defunct service providing customizable landing/link pages which I used with a subdomain of a domain owned by a family member. After they closed shop, I continued to use the same subdomain directly. Then, one late night I decided to buy my own domain on a whim, and bayindirh.io was born.

The domain sat idle at first. Then I installed a web server to my “utility” cloud server which I used as an external eye to test things. This allowed to move my homepage there. Gotten bit by the self-hosting bug, I started to move services onto that server, streamlining tons of things in the process.

Next was my blog. I have experimented with Jekyll for a long time. While the idea was to generate “static” web pages, none of the useful templates were truly static, and making your own template was requesting an amount of time I didn’t have. As a result, the blog idea was frozen until which I found this platform, which allowed me to use my own domain, and being equally minimalist at the same time.

Same story happened for my digital garden: After finding a suitable solution which allowed to bring my own domain, everything clicked.

As it turned out, the act of buying a domain in the middle of a night because of me wanting a “permanent address” on the internet changed everything. Slowly in the beginning, but faster as I walked along the path. Now, that domain serves (directly or indirectly) all the things I wanted to have and grants me freedom to tinker, explore and do things by myself.

Without realizing, I have removed a great deal of friction from a lot of processes by buying that domain, and this allowed me to find and explore more services with the same frictionless nature and connect under my own domain, giving me a distributed yet interconnected infrastructure which allows me to create and communicate with the world.

While this process was a small step for humanity, it was a big one for me, and started a nice domino effect allowing me to realize my dreams and learn a lot, which I can only see when I look backwards.

Perhaps, Steve Jobs was right. Everybody should look back now and then. Because doing it makes you understand what happened for what reason, and makes finding the next dot easier. However, this is a subject for a future post.

Until next time,

Be kind.

Peak personal computing, or, my current workstation setup

Bytepawn - Marton Trencseni — Mon, 08 Apr 2024 01:00:00 GMT

Even since I've had a 386 in my room as a teenager, I've taken computers apart and modified and upgraded them. When getting a new workstation computer, I have never bought a pre-assembled one (like a Dell or HP), I always buy the components and build it myself. I've been doing this for 30+ years, so I'm used to it, it's part of my lifecycle. I built my current workstation in 2022, but due to reasons I describe in this post, I've been making a fair amount of modifications to it ever since. Finally, now in 2024 April, I feel that the system has come together and is perfect — I don't want to or plan to make any more changes to it.

Dieting and Differential Equations: Part 3, Tracking Calories using LLM

jodavaho.io — Sat, 06 Apr 2024 03:00:00 GMT

Executing the strategy

It’s really hard to know how much you’re eating.
It’s really annoying seeing weight go up, but noise around the signal is normal, using a “predictor” helps me ralize this
It’s beneficial to have a running estimate for when you’ll reach your goal.

I set out and used this to calculate a daily budget of food. I’ve been at it 40 days, and I’ve lost 11 lbs (average 0.275 lbs/day).

There are two very hard things about calorie restricted diets. The first actually knowing how many calories you’re eating. And the second is knowing if it’s working. Math is all well and good, but if it doesn’t match reality, it’s not useful.

Tracking calories

I’ve been using ChatGPT-4 to estimate my calorie intake from pictures, descriptions, and rough notes. It’s been amazing. Somehow, training on language has resulting in memorizing the calorie content of every food item in the world, or so it would seem.
There’s three things ChatGPT-4 does for me:

It answers adhoc questions about calorie estimates with prompts like “How many calories are in this plate of food (via picture), or “How many calories are in a McDonald’s Big Mac?”. These results help me order food.
It helps me track my daily intake. The initial prompt is below, then I fire off curt messages like “I ate a banana” or “I had a 20oz soda” or “new day: weight 212.2” and it accumulates data for me.
Finally, at the end of every day, I can pull out the summary, put it into a file, and run a plotter on it.

The prompt is:

Here is some json that represents what I need your help tracking. I will say things like "add a bowl of cheesits" and you will estimate the calories in that food item, add it to the running total for each day, and dump the revised json. I will also provide a weight once a day, and will occasionally provide exercises, for which you must also update the exercise field. The "net" field can be ignored but please include it with value zero. Is any part of this unclear? You're helping me track diet progress every day, with additional data from activities and meals throughout each day. 

{
  "Date": "2020-01-01",
  "Weight": 0,
  "Calories": 0,
  "Exercise": 0,
  "Pushups": 0,
  "Notes": "",
  "Net": 0 
}

And a typical interaction just happened as I was writing this, here it is with typos and all:

Me: 
Add a pb sandwich, and two chicken nuggest, target brand. 
ChatGPT:
Adding a peanut butter sandwich and two Target brand chicken nuggets to your day's intake:

1. Peanut Butter Sandwich:
   - Bread (2 slices): Approximately 160 calories for two slices of wheat bread.
   - Peanut Butter: For about 2 tablespoons, it's roughly 188 calories.

   Total for the peanut butter sandwich: approximately 348 calories.

2. Two Target Brand Chicken Nuggets: Target brand (such as Market Pantry) chicken nuggets typically contain about 45-50 calories each. For two nuggets, let's use 100 calories as an estimate.

Adding these items to the day's total:

- Peanut Butter Sandwich: 348 calories
- Two Chicken Nuggets: 100 calories

Total additional intake: 448 calories

Here's the updated information:

json [copy]
{
  "Date": "2024-04-01",
  "Weight": REDACTED,
  "Calories": 1008,
  "Exercise": 0,
  "Pushups": 0,
  "Notes": "",
  "Net": 0
}

How accurate is this? The chicken nuggets are about 80 each, and the brad is 70/ slice. But I was misleading. I actually ate Good and Gather, and Market Pantry actually is about 50-60 cal each. But for my purposes, it’s so easy to use that I don’t care if we’re off by 20%, since I am also back-estimating calories from my weight loss, as we’ll see next.

At this point, I’m content with the process. But I’d like a better answer to the accuracy question. To get this, we can do some statistical magic and estimate total calories and activity levels from my weight data (and the prior estimates). This might allow us to pin down the accuracy of the ChatGPT-4 estimates and my own exercise estimates.

That’s next time!

A paper reading club at work; databases and distributed systems research

Notes on software development — Fri, 05 Apr 2024 03:00:00 GMT

I started a paper reading club this week at work, focused on databases and distributed systems research. I posted in a general channel about the premise and asked if anyone was interested. I clarified the intended topic and that discussions would be asynchronous over email, run fortnightly.

Weekly seemed way too often. A month seemed way too infrequent. Fortnightly seemed decent.

I was nervous to do this because I've been here about 2 months. In the past I would have waited 6 months or a year to do this. But I don't know. If you see something you think should exist, why wait?

The only other consideration was past experiences I've written about having difficulty getting engagement with clubs at work. But EDB has near 1,000 employees. I figured there might at least be a couple interested.

Furthermore I figured if I only got a few people this entire idea would at least benefit myself, since I have been wanting to force myself to build a paper reading habit. And if no one responded, it would be only mildly embarassing and I'd not pursue it further.

But after a day, about 6 people showed interest. Which was better than I hoped! Folks from product management, support, development, and beyond.

So I opened a dedicated channel and asked people to start submitting papers and voting on them. One of my teammates started submitting some great papers on caches and reference counting.

I picked a first one, the Redshift paper, to get us started. Demonstrating the process to avoid confusion. And I made a calendar invite for everyone in the channel, the paper linked in the invite. I clarified in the invite that it was just a reminder and that the real discussion would still be async over email. (I've found it's best to repeatedly clarify process stuff.)

Once I had these first few folks interested I was able to post again in a broader company channel that a couple of us were starting this paper club. By the end of the day the dedicated channel was 29 folks. All in about 2 days.

Mailing lists are nicer than Slack or Discord in my opinion because they sort of force you to slow down, they are harder to miss (if someone starts a thread after you've seen a message in Slack or Discord, you tend to miss it), and easier to manage (read/unread).

Engineers often seem to get overwhelmed by a mass of Slack messages. Whereas they seem to be a bit more comfortable with email threads.

All of this is all the more important when you're running a global group. EDB has people everywhere.

Why do this?

Before I dropped out of college I did a research internship with a VLSI group at Harvard SEAS. And my favorite part was that they had a weekly (or biweekly?) Wednesday paper reading session where 15 people from the lab and adjacent labs would eat pizza after hours and discuss a paper.

I've been dying to recreate this at a company ever since. Since EDB is so distributed, we won't be discussing over pizza. But I'm still excited.

And I hope my experience serves as a blueprint for others.

I started a paper reading club at work, wrote about it as a possible blueprint for others.

I'm excited! I've wanted to have a gang at work with whom to read papers for a long time.https://t.co/vpwERj8pHe
— Phil Eaton (@eatonphil) April 6, 2024

I ❤️ open source, and, Cloud9 still rocks

Bytepawn - Marton Trencseni — Fri, 05 Apr 2024 01:00:00 GMT

For the past 8 years I've been using Cloud9 for writing code in the cloud, and I can wholeheartedly recommend it: it just works for me. 7 years ago AWS acquired the company, so the project is no longer developed. However, this is not a problem, since it still just works. The only problem I ever had was a minor UI bug, which I fixed in a couple of minutes, as detailed in this post. This was possible because Cloud9 is open-source. I ❤️ open source.

Dieting and Differential Equations: Part 2, Closed Loop diets

jodavaho.io — Thu, 04 Apr 2024 03:00:00 GMT

OK, let’s talk about my diet.

I’ve started a diet, with the goal of losing 32 lbs, or ~15% of my bodyweight, to get back to my circa 2000-2005 weight. I’m happy if I achieve 80% of that loss, ending up under 200. This is a long post, in which I describe the plan, and some misunderstandings I had about dieting and weight loss.

Along the way, I built an estimator that uses data to determine my real calorie intake, needs, and activity level. The results, for me, were surprising.

In practice, the hardest part about calorie deficits is tracking calories. Burying the lede a little, it turns out LLM / ChatGPT-4 is fantastic at counting calories for you from pictures, descriptions, or rough notes.

I’ll cover all that.

What’s a diet

My definition of diet is extended calorie deficit for the purposes of losing bodyfat. I do not differentiate types of food, but I prefer high protein for appetite control. I also don’t do time-based restriction, though I find that skipping breakfast and not eating after dinner is sustainable, and that’s roughly an 18/6.

From the first two parts of this series, we have a few equations to work with:

Mifflin-St Jeor¹ equation for Basal Metabolic Rate (BMR) is $ 10 \times weight(kg) + 6.25 \times height(cm) - 5 \times age(y) + G$, where $G$ is 5 for men, and -161 for women.
Total Daily Energy Expenditure (TDEE) is $BMR \times L$, where $L$ is a multiplier for your daily activity level with specific values ranging from “Sedentary” to “Extra Active”.
“Steady state weight” (SSW) = $\frac{1}{10}\left(cal - L \times [6.25 \times height - 5 \times age + G]\right)$
A weight loss model, $w(t) = SSW + (w_0 - SSW) e^{-\frac{10L}{B}(t)}$

Or, tidied up a bit:

$$ \begin{aligned} BMR =& 10 w + 6.25 H - 5A + G \\\\ &\text{ with } G \in \\{5, -161\\}\\\\ TDEE =&L \cdot BMR\\\\ &\text{ with } L \in \\{1.2, 1.375, 1.55, 1.725, 1.9\\}\\\\ SSW =& \frac{cal - L\cdot [6.25 H - 5 A + G]}{10} \\\\ w(t) =& SSW + (w_0 - SSW) e^{-\frac{10L}{B}(t)} \\\\ &\text{ with } B = 3500 | 7700 \text{ calories per unit lb|kg}\\\\ \end{aligned} $$

See previous posts for more details.

From this we’re accumulating python code:

from enum import Enum
import numpy as np

def bmr_m(weight, height, age):
    return 10 * weight + 6.25 * height - 5 * age + 5

def bmr_f(weight, height, age):
    return 10 * weight + 6.25 * height - 5 * age - 161

class Activity(Enum):
    SEDENTARY = 1.2
    LIGHT = 1.375
    MODERATE = 1.55
    VERY = 1.725
    EXTRA = 1.9

def calc_tdee(bmr, activity):
    """
    Calculate Total Daily Energy Expenditure
    """
    if (activity < 1.2 or activity > 1.9):
        raise ValueError("Activity must be between 1.2 and 1.9")
    return bmr * activity

def steady_state_weight(calories, activity, height, gender, age):
    """
    Reverse BMR to find weight given calories

    Accepts calories per day, activity level, height in cm, 
    gender='m'|'f', age in years

    Returns weight in kg
    """
    L = activity
    if gender=='m':
        wt = calories /  L - (6.25 * height - 5 * age + 5)
        wt = wt / 10
        return wt
    if gender == 'f':
        wt = calories /  L - (6.25 * height - 5 * age - 161)
        wt = wt / 10
        return wt
    raise ValueError('sex: m|f only')

def weight_over_time(days, calories, 
                     ht, age, gender, wt0, 
                     activity=Activity.SEDENTARY.value):
    """
    Given you want to consume `calories` per day, over the range of days, what
    will your weight be?

    Accepts wt0 in kg, ht in cm, gender='m'|'f', age in years
    Returns a list of weights over time, one per day, in kg
    """
    SSW = steady_state_weight(calories, activity, ht, gender, age)
    t = np.arange(0, days)
    exparr = np.exp(-10*activity/7700* t)
    res = SSW + exparr * (wt0-SSW)
    return res

We now have a vague sense that “in a few years” we’ll hit a steady state weight, and that we can lose weight faster or slower by choosing a different calorie intake.

We can also manually tweak the Activity enum to see how different activity levels affect our weight loss, or try different calorie intakes to try to hit a goal.

As hinted when we started, what people want is: “When will I hit my goal weight?” or “How many calories do I need to cut to hit my goal weight by a certain date?”

These are real optimization problems, and now we have the math to solve them.

Deriving calorie intake to hit a goal weight by a certain date

We derived this equation:

$$w(t) = SSW(cal,L)\cdot\left(1-e^{-\frac{10L}{B}(t)} \right)+ w_0 e^{-\frac{10L}{B}(t)} \text{} $$

Given above closed form equation, we can answer the question “If I want to lose 30 lbs by mid-summer, how many calories do I need to cut?” What we’re asking is: “Given t=days to end of diet, what is the value of calories that will result in w(t) = w_0 - <amount to lose>?”

To lose 10 kg in the next 90 days, form:

$$w_0 - 10 = SSW(x,L)\cdot\left(1-e^{-\frac{10L}{B}(90)} \right)+ w_0 e^{-\frac{10L}{B}(90)} \text{} $$

and solve for $x$.

The algorithm is essentially:

Solve $C_e = e^{-\frac{10L}{B}(90)}$
Calculate $C_p = (6.25 H - 5 A + G) / 10L$
Plug into $w_0 - 10 = \left(\frac{cal}{10L} - C_p\right)\cdot\left(1-C_e \right)+ w_0 C_e $
which reduces to $cal = 10L \left( w_0 - \frac{10}{1-C_e} + C_p\right) $

# import numpy as np # already imported above, but just remember
def predict_calories_to_lose(days, wt0, wt_target, ht, age, activity=Activity.SEDENTARY.value):
    """
    Given you want to lose `wt0 - wt_target` (kg!) in `days`, what should your
    daily calorie intake be?

    Accepts wt0 in kg, ht in cm, age in years
    Returns calories per day
    """
    C_e = np.exp(-10*activity/7700 * days)
    G = 5 # or -161 for biological females
    C_p = (6.25 * ht - 5 * age + G) / 10
    return 10 * activity * (wt0 - (wt0 - wt_target) / (1 - C_e) + C_p)

Determining arrival time at a given weight from a given strategy

You may also want to know when you’ll hit a given weight, given a strategy. This is no more difficult, we just solve for $t$ in the above equation.

$$\begin{aligned} w(t) &= SSW + (w_0 - SSW) e^{-\frac{10L}{B}(t)} &\text{} \\\\ w_{target} &= SSW(calories,L) + (w_0 - SSW(calories,L)) e^{-\frac{10L}{7700}(t)} &\text{} \\\\ w_{target} - SSW(calories,L) &= (w_0 - SSW(calories,L)) e^{-\frac{10L}{7700}(t)} &\text{} \\\\ \frac{w_{target} - SSW(calories,L)}{w_0 - SSW(calories,L)} &= e^{-\frac{10L}{7700}(t)} &\text{} \\\\ \ln\left(\frac{w_{target} - SSW(calories,L)}{w_0 - SSW(calories,L)}\right) &= -\frac{10L}{7700}(t) &\text{} \\\\ t &= -\frac{7700}{10L} \ln\left(\frac{w_{target} - SSW(calories,L)}{w_0 - SSW(calories,L)}\right) &\text{} \\\\ \end{aligned}$$

Note, the fraction $\frac{w_{target} - SSW(calories,L)}{w_0 - SSW(calories,L)}$ is the fraction of the way to the target weight, and, assuming you’re losing weight, is always less than 1. It somewhat curiously represents “target percent excess after diet” in a way.

This yields a simple algorithm:


def predict_days_to_weight(wt_target, calories, wt0, 
                           ht, gender, age, activity=Activity.SEDENTARY.value):
    """
    Given you want to hit `wt_target` (kg!) on a diet of `calories` per day and
    activity level `activity`, how many days will it take?

    Accepts wt0 in kg, ht in cm, age in years, gender='m'|'f'
    Returns days
    """
    SSW = steady_state_weight(calories, activity, ht, gender, age)
    numerator = wt_target - SSW
    denominator = wt0 - SSW
    return -7700/(10*activity) * np.log(numerator/denominator)

So there you have it! A mathy way to come up with your own activity / calorie balance that makes sense for you.

In case you’re curious I ran my situation as follows:

Start = to_kg(222)
End = to_kg(190)
Diff = End - Start
calsed = predict_calories_to_lose(158, Start, End, to_cm(77), 43, Activity.SEDENTARY.value)
calslight = predict_calories_to_lose(158, Start, End, to_cm(77), 43, Activity.LIGHT.value)
calsmod = predict_calories_to_lose(158, Start, End, to_cm(77), 43, Activity.MODERATE.value)

and got:

SED Calories to lose -32.0 lbs in 158 days: 1625.19
LIGHT Calories to lose -32.0 lbs in 158 days: 1964.75
MOD Calories to lose -32.0 lbs in 158 days: 2304.17

So, I’ve chosen 2000 as a target, revived the running and weightlifting, and I’m on my way.

In the next part, we’ll discuss how I track caloires, which is the hardest part, honestly.

But for now, a teaser on my progress, for which the goal arrival is updated daily. We’ll talk about this in the next post.

https://en.wikipedia.org/wiki/Basal_metabolic_rate ↩︎

Resolve "database is locked" with Go and SQLite

Ben E. C. Boyter — Thu, 04 Apr 2024 03:00:00 GMT

I have always been a big fan of SQLite and recently was using it with Go/Golang. However due to fact that goroutines are very convenient I quickly ran into the “database is locked” issue where multiple writes would try to update the database at any one time. Multiple reads worked without issue.

I could have solved this by throwing a mutex lock around the database writes, but thats something I could easily mess up by missing the mutex and locking the database at the wrong time. I could also have retries with a backoff which isn’t a bad idea for resilience, but overkill for my situation.

Dieting and Differential Equations: Part 1, Predicting Weight Loss

jodavaho.io — Tue, 02 Apr 2024 03:00:00 GMT

OK, let’s talk about my diet some more.

Along the way, I built an estimator that uses data to determine my real calorie intake, needs, and activity level. In practice, the hardest part about calorie deficits is tracking calories. Burying the lede a little, it turns out LLM / ChatGPT-4 is fantastic at counting calories for you from pictures, descriptions, or rough notes.

This is part two of a series. The first part disucsses basic math for calorie needs and steady state weight, and is here.

What’s a diet

From the first part, we have a few equations to work with:

Mifflin-St Jeor¹ $ BMR = 10 \times weight(kg) + 6.25 \times height(cm) - 5 \times age(y) + G$, where $G$ is 5 for men, and -161 for women.
$TDEE = BMR \times L$, where $L$ is a multiplier for your daily activity level.

Hereafter, shortened to

$$ \begin{aligned} BMR =& 10 w + 6.25 H - 5A + G \\\\ TDEE =&L \cdot BMR\\\\ &\text{ with } L \in \\{1.2, 1.375, 1.55, 1.725, 1.9\\} \end{aligned} $$

We also know we can calculate the steady state weight, which is the weight at which you will neither gain nor lose weight. This is a function of your activity, age and height, and the calories you consume.

$$ \begin{aligned} SSW &= \frac{cal - L\cdot [6.25 H - 5 A + G]}{10} \\\\ \end{aligned} $$

See part 0 for more details.

From this we’re accumulating python code:

from enum import Enum

def bmr_m(weight, height, age):
    return 10 * weight + 6.25 * height - 5 * age + 5

def bmr_f(weight, height, age):
    return 10 * weight + 6.25 * height - 5 * age - 161

class Activity(Enum):
    SEDENTARY = 1.2
    LIGHT = 1.375
    MODERATE = 1.55
    VERY = 1.725
    EXTRA = 1.9

def calc_tdee(bmr, activity):
    """
    Calculate Total Daily Energy Expenditure
    """
    if (activity < 1.2 or activity > 1.9):
        raise ValueError("Activity must be between 1.2 and 1.9")
    return bmr * activity

def steady_state_weight(calories, activity, height, gender, age):
    """
    Reverse BMR to find weight given calories
    """
    L = activity
    if gender=='m':
        wt = calories /  L - (6.25 * height - 5 * age + 5)
        wt = wt / 10
        return wt
    if gender == 'f':
        wt = calories /  L - (6.25 * height - 5 * age - 161)
        wt = wt / 10
        return wt
    raise ValueError('sex: m|f only')

We left of with the big question: how long does it take for the effects to be fully realized? Now, let’s figure that out, and, that will allow me to, answer, if I want to lose 30 lbs by mid-summer, how many calories do I need to cut?

Deriving a predictive model for weight loss

Alright, using only BMR / TDEE, what can we say about weight loss over time?

Given weight loss is a function of the difference between TDEE and calories, thinking it through, if we’re far from our target weight, the difference between TDEE and our calorie intake is large, so we should shed weight quickly. As we approach our target weight, the difference shrinks, and we should lose weight more slowly.

Let’s plot this. Assuming a 300 calorie deficit will shed $300/7700$ kgs, we have:

def wt_change(calories, tdee):
    """
    Given calories consumed, and calculated TDEE (using kg!), how much weight
    will you lose?

    Returns delta-weight in kg
    """
    calories_per_kg = 7700
    return (calories - tdee) / calories_per_kg

And so:

def predict_wt(days, ht, age, wt0, calorie_target, activity=Activity.SEDENTARY.value):
    """
    Given you want to cut calories by calorie_cut below your starting BMR, over
    the range of days, what will your weight be?

    Accepts wt0 in kg, ht in cm, age in years
    Returns a list of weights over time in kg
    """
    wts = [wt0]
    for i in range(1, days):
        current_wt = wts[-1]
        bmr = calc_bmr(current_wt, ht, age, 'm')
        tdee = activity * bmr
        dwt = wt_change(tdee, calorie_target)
        wts.append(wts[-1] + dwt)
    return wts

And if we plot out the weight loss over 3 years, we get:

def plotxy(x,y):
    import plotille as plt
    fig = plt.Figure()
    fig.width = 40
    fig.height = 10
    fig.plot(x, y)
    print(fig.show())

wts = predict_wt(365*3, to_cm(77), 43, to_kg(222), 2100)
plotxy(range(0, 365*3), wts)

And the result is:

Well, it definitely flattens out. And for my taste, that’s far too slow in the second or third year. When do we reach our SSW? Well, never, actually. We asymptotically approach it. So to get something like linear progression, you want to be the early stages, where the difference between TDEE and calories is large.

Light bulb #3: If you want to hit your target weight, you have to aim past it. If you want to hit it quickly, you have to aim far past it.

That gradual flattening out is a classic exponential decay. The equation for “weight over time” must certainly be of the form $w(t) \propto w_0 e^{-kt}$, where $w_0$ is the initial weight, $k$ is the decay constant, and $t$ is time.

I can happily continue plotting by iterating over days, but that’s not very fun, and I distrust my discrete simulation skills. Let’s derive the equation. Let’s start with the “Delta Weight” equation, and see if we can integrate that to find the weight over time.

$$ \frac{dw(t)}{dt} = \frac{calories - tdee(t)}{7700} $$

Expanding that out, we get (using the $TDEE$ equation from above):

$$ \frac{dw(t)}{dt} = \frac{cal - L (10 w(t) + 6.25 H - 5 A + G) }{7700} $$

Where $cal$ is the calories, $L$ is the activity level, $w(t)$ is the weight at time $t$, $H$ is height, $A$ is age, and $G$ is the “gender constant” from the BMR equation.

The fact that we have $dx/dt$ on one side and $x$ on the other is a hint that this is a first order linear differential equation. You can use automated tools to solve this, but it will generally be ugly if you don’t prepare the equation first.

So, we can prepare and solve it by separation of variables². We can break out $w(t)$ and tuck some constants away to get:

$$ \frac{dw(t)}{dt} = \frac{C}{B} - \frac{10L}{B} w(t) $$

where $C = cal - L [6.25 H - 5 A + G] $ and $B = 7700 $. This can be separated to:

$$\begin{aligned} &\left(\frac{1}{ \frac{C}{B} - \frac{10L}{B} w(t) }\right) dw(t) = dt\\\\ &\left(\frac{B}{ C - 10L w(t) }\right) dw(t) = dt \end{aligned}$$

Wolfram Alpha made me dig way back to Calc II u-substitution, with $u = C - 10L w(t)$, and $du = -10L dw(t) \implies dw(t) = \frac{1}{-10L} du$. Subbing those in, we get:

$$\begin{aligned} \left(\frac{B}{ u }\right)\frac{1}{-10L} &du(t) = dt \\\\ \left(-\frac{B}{ 10L }\right) &\frac{du(t)}{u} = dt \end{aligned}$$

And since $\int\frac{du}{u} = ln(u) + K$¹, we have $=ln(C-10Lw(t)) + K$ so we get:

$$\begin{aligned} -\frac{B}{10L} ln(C-10L w(t)) &= t + K \\\\ ln(C-10L w(t)) &= -\frac{10L}{B} (t + K)\\\\ C-10L w(t) &= e^{-\frac{10L}{B} (t + K)}\\\\ w(t) &= \frac{C - e^{-\frac{10L}{B} (t + K)}}{10L}\\\\ w(t) &= \frac{C}{10L} - e^{-\frac{10L}{B}(t)} \left(\frac{e^{-\frac{10LK}{B}}}{10L}\right) & \text{❓ ugly...} \\\\ \end{aligned}$$

Note that $K$, the integration constant, is hiding up there. Let’s use initial conditions of $w(0)$ to find solve the previous equation to eliminate arbitrary constants.

$$\begin{aligned} w(0) &= \frac{C}{10L} - e^{-\frac{10L}{B}(0)} \left(\frac{e^{-\frac{10LK}{B}}}{10L}\right) \\\\ w(0) &= \frac{C}{10L} - \frac{e^{-\frac{10LK}{B}}}{10L} \\\\ 10 L w(0) &= C - e^{-\frac{10LK}{B}}&\text{ } \\\\ e^{-\frac{10LK}{B}} &= C - 10 L w(0)&\text{✅ } \\\\ %e^{-\frac{10LK}{B}} &= cal - L[6.25 H - 5 A + G ] - 10 L w(0) \\\\ %e^{-\frac{10LK}{B}} &= cal - L [6.25 H - 5 A + G + 10 w(0)] \\\\ %e^{-\frac{10LK}{B}} &= cal - TDEE(0) \\\\ %-\frac{10LK}{B} &= ln(cal - TDEE(0)) \\\\ %K &= \frac{-B}{10L} \cdot \text{ln}(cal - TDEE(0))/(10L)&\text{✅ Solved K} \\\\ %\implies e^{-\frac{10LK}{B}} &= cal - TDEE &\text{✅ Simplifies!} \\\\ \end{aligned}$$

That’s nice! So subbing back, we have just solved $w(t)$ as:

$$\begin{aligned} w(t) &= \frac{C}{10L} - e^{-\frac{10L}{B}(t)} \left(\frac{e^{-\frac{10LK}{B}}}{10L}\right) &\text{} \\\\ w(t) &= \frac{C}{10L} - e^{-\frac{10L}{B}(t)} \left(\frac{C-10Lw_0}{10L}\right) & \text{} \\\\ \end{aligned}$$

Now, $C$ is a constant I introduced, with value $cal - L[6.25 H - 5 A + G]$, and $B$ is the 7700 calories per kg. This means:

$$ \begin{aligned} \frac{C}{10L} &= \frac{cal - L[6.25 H - 5 A + G]}{10L} &\text{} \\\\ \frac{C}{10L} &= SSW\text{} \\\\ \end{aligned} $$

Let’s expand, using $SSW$ for $C/10L$:

$$ \begin{aligned} w(t) &= \frac{C}{10L} - e^{-\frac{10L}{B}(t)} \left(\frac{C-10Lw_0}{10L}\right) & \text{} \\\\ w(t) &= SSW - e^{-\frac{10L}{B}(t)} \left(SSW -w_0\right) &\text{} \\\\ \\\\ w(t) &= SSW\cdot\left(1-e^{-\frac{10L}{B}(t)} \right)+ w_0 e^{-\frac{10L}{B}(t)} &\text{[✅] closed form!} \\\\ w(t) &= SSW + (w_0 - SSW) e^{-\frac{10L}{B}(t)} &\text{[✅] smaller form!} \\\\ &\text{w/ }SSW=cal - L[6.25 H - 5 A + G]/10L &\text{} \\\\ \end{aligned} $$

OK, so we have a closed form equation for weight over time. With time = 0:

$$ \begin{aligned} w(0) &= SSW + (w_0 - SSW) e^{-\frac{10L}{B}(0)} &\text{} \\\\ w(0) &= SSW + (w_0 - SSW)\cdot 1 &\text{} \\\\ w(0) &= w_0 &\text{} \\\\ \end{aligned} $$

… it gives us our starting weight, and as time goes to infinity:

$$ \begin{aligned} w(\infty) &= SSW + (w_0 - SSW) e^{-\frac{10L}{B}(\infty)} &\text{} \\\\ w(\infty) &= SSW + (w_0 - SSW) \cdot 0 &\text{} \\\\ w(\infty) &= SSW &\text{} \\\\ \end{aligned} $$

… it gives us our steady state weight. Perfect!


import numpy as np

def weight_over_time(days, calories, 
                     ht, age, wt0, 
                     activity=Activity.SEDENTARY.value):
    """
    Given you want to consume `calories` per day, over the range of days, what
    will your weight be?

    Accepts wt0 in kg, ht in cm, age in years
    Returns a list of weights over time, one per day, in kg
    """
    SSW = steady_state_weight(calories, activity, ht, 'm', age)
    t = np.arange(0, days)
    exparr = np.exp(-10*activity/7700* t)
    res = SSW + exparr * (wt0-SSW)
    return res

OK, let’s have a look! Given my weight loss goal, target calories, and starting weight, we can plot it out!

My current calorie goal / activity level goal combine to give a $SSW$ of 161 lbs. This is the weight I would asymptotically approach if I ate 2100 calories per day and maintained my current activity level, and represents a weight loss of 61 lbs.

Give my goal was to lose 32 lbs, I massively overshot my goal weight with this calorie / activity level. This is intentional, to lose weight quickly. After reaching ~190 or so, I’ll probably switch to a sustainable level of food/activity (if I even can). But who knows? Now I know.

This is probably not super healthy. I suspect this is what most people have in mind when they think “diet” - something that moves quick by massive over-compensation.

Light bulb #4: A more sustainable diet is to pick calories given steady-state-weight and activity level, and let years do the work

Note, given all the closed form equations above, it’s natural to start plugging in different parameters and solving for different things. Let’s do that next.

https://en.wikipedia.org/wiki/Basal_metabolic_rate ↩︎ ↩︎
https://en.wikipedia.org/wiki/First-order_differential_equation#Separation_of_variables I was not great at this in college, so it helps me feel better when I get to use it in real life. ↩︎

Learning to code with and without AI

Austin Z. Henley's Blog — Mon, 01 Apr 2024 06:45:01 GMT

https://austinhenley.com/blog/learningwithai.html

Dieting and Differential Equations: Part 0, Calories and Steady State Weight

jodavaho.io — Mon, 01 Apr 2024 03:00:00 GMT

OK, let’s talk about my diet.

After I stopped boxing, and biking to work, and in general doing anything, food became a problem. Snacking was constant, meals were added on when someone else made them, and treats were prevalent. Turns out, when you’re burning 2500 calories in roadwork every day, this isn’t a problem. No longer.

Along the way, I built an estimator that uses data to determine my real calorie intake, needs, and activity level. The results, for me, were surprising.

I’ll cover all that.

What’s a diet

First step: How many calories do I need?

A 222lb, tall biological male my age requires at least 2025 calories to maintain his weight. How do I know that, and what do I mean by at least? The value is given by the Basal Metabolic Rate equation, for which the most accurate form from Mifflin-St Jeor¹ $ BMR = 10 \times weight(kg) + 6.25 \times height(cm) - 5 \times age(y) + G$, where $G$ is 5 for men, and -161 for women.

Hereafter, shortened to

$$ BMR = 10 w + 6.25 H - 5A + G $$

Let’s get started calculating, using Python of course.

def bmr_m(weight, height, age):
    """
    Given weight (kg), height (cm), and age (years), return
    the 'sustaining calories' for males
    """
    return 10 * weight + 6.25 * height - 5 * age + 5

def bmr_f(weight, height, age):
    """
    Given weight (kg), height (cm), and age (years), return
    the 'sustaining calories' for females
    """
    return 10 * weight + 6.25 * height - 5 * age - 161

A close approximation of me, bmr_m(JoshWeight, JoshHeight, JoshAge)~=2000.

This is the amount of calories you need to maintain your weight if you were in a coma. Your daily activity increases this need. The activity factor varies from 1.2 (sedentary) to 1.9 (strenuous exercise daily).

Factoring in activity, we have “TDEE” (Total Daily Energy Expenditure), as:

$$ TDEE = BMR \cdot L \label{tdee}$$

Where $L$ is Activity Level, a constant from 1.2 to 1.9.

A helpful enum here (to match the literature) is:

from enum import Enum

class Activity(Enum):
    SEDENTARY = 1.2
    LIGHT = 1.375
    MODERATE = 1.55
    VERY = 1.725
    EXTRA = 1.9

def calc_tdee(bmr, activity):
    if (activity < 1.2 or activity > 1.9):
        raise ValueError("Activity must be between 1.2 and 1.9")
    return bmr * activity

If you’re like me, you jumped to 1.3 or so, thinking “well of course I’m not super active, but I do play with my kids and run around a bit, etc”. In a bit, we’ll build an estimator to bound these free-ish variables, so let’s proceed with SEDENTARY=1.2 and see how that shakes out.

Or, in my case, calc_tdee(2000, Activity.SEDENTARY.value) ~= 2400.

Losing weight on calorie restriction and “steady state” weight

OK, any amount of daily calories below your TDEE will result in weight loss, right? The generally accepted value is 3500 calories per pound of bodyfat, so if I cut 300 calories a day, I can expect to lose 1 lb every 12 days, right?

Just eat 2100 calories and lose 1 lb/12 days!

Wait a minute. If that’s true I’ll reach negative weights in about 7 years. There’s clearly something else going on. That something is the fact that your BMR is a function of your weight. So your “steady state” weight where you’ll stop losing weight is a function of gender, age, height, and calories. It’s the weight at which (ideally) bmr_m(wt,ht,age)*1.375 == 2100

Stated another way, there is a 6’ 6" male somewhere that eats 2100 calories /day , is not very active, and loses/gains no weight. That could be future me! So, BMR can tell us how much that man weighs.

Rearrange the TDEE equation to solve for weight:

$$ \begin{aligned} TDEE & = L [ BMR ] \\\\ cal &= L [ 10 w + 6.25 H - 5 A + G ]\\\\ \frac{cal}{L} &= 10 w + (6.25 H - 5 A + G) \\\\ 10 w &= \frac{cal}{L} - (6.25 H - 5 A + G) \\\\ w &= \frac{cal - L\left[6.25 H - 5 A + G\right]}{10} &\text{ ✅ } \\\\ \end{aligned} $$

def steady_state_weight(calories, activity, height, gender, age):
    """
    Reverse BMR to find weight given calories,
    Returns kg!
    """
    L = activity
    if gender=='m':
        wt = calories /  L - (6.25 * height - 5 * age + 5)
        wt = wt / 10
        return wt
    if gender == 'f':
        wt = calories /  L - (6.25 * height - 5 * age - 161)
        wt = wt / 10
        return wt
    raise ValueError('sex: m|f only')

Result for ~2100 calories given my person? ~171 lbs. Not bad! This uses 1.2 as activty. Look carefully at the above equation - the activity is in the denominator. This should sound alarm bells of the “That’s a sensitive arrangment” variety. What happens with “Light activity” values of 1.375?

The steady state weight drops 50 lbs. I think I knew this, from my boxing days, but I didn’t know it.²

Light bulb #1, adding a tiny bit of exercise will shed a ton of extra weight over time.

Let’s look at what a snack-a-day does to your SSW. Adding 10% to your intake (equivalent of a cookie, short IPA, or a 20oz soda /day) to an otherwise perfect diet, would raise my Sedentary SSW from 171 to 210 lbs.

That’s a gain of 40 lbs over time from a small treat daily.

Light bulb #2, a tiny bit of extra food will add a ton of extra weight over time.

That’s a lot of weight. Taking the stairs might burn 50 calories, which is how much you lose each decade from your BMR. Walking 30 extra minutes would burn 200, offsetting that 40 lb weight gain from the IPA. The battle of the bulge is won and lost in these tiny daily decisions. Anyone can tell you though, that the feedback is over years, not days, so your daily habits set in before you see the noticeable weight gain (or loss).

So the big question is, how long does it take for the effects to be fully realized? This goes back to the original question - how long does it take to lose 1 lb if I cut 300 calories / day?

Now, let’s figure that out, and, that will allow me to, answer, if I want to lose 30 lbs by mid-summer, how many calories do I need to cut?

That’s next.

https://en.wikipedia.org/wiki/Basal_metabolic_rate ↩︎
For reference, if my SSW was 205, and I was doing a couple hours of workouts / day and biking everywhere, my calories must have been ~3500-3800/day. ↩︎

The Upload (short story)

swyx's site RSS Feed — Sun, 31 Mar 2024 07:24:07 GMT

> My first attempt at bringing back the short story in 30 years.

Papers

Bytepawn - Marton Trencseni — Sat, 30 Mar 2024 01:00:00 GMT

A compilation of academic-style papers I have written over the years.

Trying out Rye

ブラック — Fri, 29 Mar 2024 02:00:00 GMT

The catch in the Rye.

One of my least popular takes as a Professional Python Developer™ seems to be that I don’t actually consider the Python ecosystem and its tooling to be "hot garbage". I also haven’t really bothered with project management tools like Poetry, since I think you can use virtual environments and pip and be just fine. Really, the only tools in my arsenal that I consider to be absolutely essential have been pipx, as distributing Python tools is kind of a pain, and pyenv, because Homebrew will absolutely break your Python installation without any mercy if you so much as blink. Of course, I do also use other tools too, but I don't consider them as essential as those two.

However, I do recognise that there is some value in one-stop-shop tools like Rust’s Cargo, which bakes in a lot more project management stuff into your Rust development workflow and is generally pretty nice to use. For this reason I was quite intrigued by Rye after watching Armin Ronacher’s demonstration video, since it seems like a version of Cargo (and to some degree Rustup) for the Python world.

On a conceptual level, it sounds very good. Instead of installing a bunch of tools and managing/using them individually, Rye would bundle together all sorts of tools and functionality, and let me write Python software with basically Rye alone. The number of different tools that I use and Rye would replace is actually rather long:

current tool	The Rye Way
pyenv	Rye automatically installs whatever Python version your project uses.
pip	Rye installs and pins your dependencies as you add them to your project.
pipx	Rye allows installing Python tools in isolated environments just like pipx.
VirtualFish	Rye automatically sets up a virtualenv in your project directory.
Black	Rye bundles Ruff for formatting (`rye fmt`).
Ruff	Rye bundles Ruff for linting (`rye lint`).

I started off by migrating all of my pipx installs to Rye and it worked great. All tools were ready and usable on my $PATH and I could even add optional dependencies with --extra-requirement flags. Not sure if there’s a handy way of adding those extra requirements after the fact like you can in pipx with pipx inject though. I imagine I could always just manually install them into the virtual environment but I’d prefer if there was a more straightforward and obvious way. Would make it even easier to use Rye as a drop-in replacement for pipx.

As for actual development, I decided to start off small and convert a tiny Python script I use for fetching data from an API to work with Rye. Since this was just a tiny script that I run in a Docker container, it wasn't exactly organized in the way that Rye would organize a script project, so I decided to also make it a bit more structured with a script entry-point and everything. This isn't actually a requirement but I figured that I should try to do it like Rye would want me to. For what it's worth, the default Rye-tastic way of organizing code is fairly logical and probably good starting point for people new to Python. Personally, I've never really used /src/ directories in Python but I don't hate it either.

For this tiny project, everything worked very nicely. Granted, there was only one dependency (requests), so it wasn't exactly a demanding test. But Rye would bootstrap whatever Python version I had in the .python-version text file, create a virtualenv out of it, install my dependencies into it and then allow me to run my script with just python like I didn't have a virtualenv at all.

And if I upgraded or downgraded my Python version by changing the value in the .python-version file, it would bootstrap everything again to make sure my development environment matched the requirements. This would make version upgrades a lot easier for projects with lots of different developers, as you could be pretty confident in everyone having an up-to-date development environment if they just run rye sync. I've had to work on projects that upgraded between major versions of Python, and you'd always have at least some issues bringing everyone up to speed.

Emboldened by the success of using Rye with this tiny script, I moved onto the next trial: using Rye to develop a Django project. And this is where the pain begins, and not (just) because I picked an example with an ancient Django 2.0 codebase.

I bootstrapped my Django project as a virtual package with Rye and managed to install all of the dependencies for my Django project that are required to run the application locally. Then I started to add the dependencies I need to actually serve it on the web and I step on the big fat LEGO brick lying in wait: I just can't install uWSGI. The reason for this is actually quite simple: uWSGI needs to compile C extensions, Rye uses static builds of Python, and static Python builds + compiling C extensions is a known house of pain.

The Rye FAQ even admits how the situation when it comes to C extensions is a problem, and one with no real solution at the moment. It does however offer a workaround: register a non-static Python build that you've obtained from somewhere as a Rye toolchain. Since Homebrew has decided to grace my Mac Studio with a Python 3.9 installation suitable for my purposes*, I decided to register that.

* Technically Django 2.0 doesn't actually support Python 3.9 but fortune favours the bold.

$ which python3.9
/opt/homebrew/bin/python3.9
$ rye toolchain register --name=homebrew /opt/homebrew/bin/python3.9
Registered /opt/homebrew/bin/python3.9 as homebrew@3.9.18

Obviously I would never recommend actually using Homebrew-supplied Python installations for any kind of development work, but I figured it'd be fine for testing. Now I can just pin my custom toolchain to my project and create the virtualenv from that.

$ rye pin homebrew@3.9.18
pinned homebrew@3.9.18 in project/.python-version
$ rye sync
Python version mismatch (found cpython@3.9.18, expected homebrew@3.9.18), recreating.
error: failed fetching toolchain ahead of sync

Caused by:
    unknown version homebrew-aarch64-macos@3.9.18

Oops.

Turns out that Rye is so set on managing your (static) Python toolchains that it will even attempt to download a toolchain already on your machine, and then fail because your custom toolchain doesn't actually exist beyond your machine. It won't even work even if you omit the custom name from rye toolchain register.

So yeah, Rye kinda sucks and is all sorts of broken if you ever need dependencies that compile C code. Obviously I didn't test out a wide array of C extensions, but considering that it's a very well-known problem with static builds, you'll probably trip up yourself sooner or later if you do a lot of Python development.

On one hand, I do understand that compiling software sucks because you need to hoard compile-time requirements and compiling software takes forever and makes your laptop burn your thighs and so on. However, I do think that compiling software like your Python development environment has some very significant upsides, such as the fact that your build of Python will reference paths actually present on your actual computer. Static Python builds are fine for some stuff but I feel like they're a bad idea to be the default and a truly awful idea to be the only option. The support for static compilation of Python just isn't there yet.

Even if the custom toolchain registering worked like it's supposed to, I find this design approach to completely water down the "one-stop-shop for all Python users" idea put forth by Rye, since I'm still required to install pyenv on my machine in order to have a toolchain to register in the first place. I think Rye would be a much better tool if it offered pyenv-style compilation of Python, and preferably made that the default option. Or at the very least ask me which style I prefer during the installation process. It already asks you how you want the python command work outside Rye-managed projects, so it's not unthinkable to have an option for compilation too.

In terms of ergonomics when it came to the Django project (the parts that worked), I also kinda miss the environment variable plugin that VirtualFish had, where it would set environment variables when you activated a virtualenv. As far as I can see, the way you're supposed to integrate environment variables as part of your Python development is by adding them to your Rye scripts. So if I want to set DJANGO_SETTINGS_MODULE for when I run any of the Django management commands, I'd chuck it in a separate file like .env and add this to my pyproject.toml file:

[tool.rye.scripts]
manage = { call = "manage", env-file = ".env" }

You could also add environment variables directly to the script definition but that sounds very not-portable if you work with other developers. But now I can run tests against my development settings module. I just need to use rye run manage to invoke the test management command instead of for example the manage.py script that Django will give you.

$ rye run manage test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
..................
---------------------------------------------------------------------
Ran 18 tests in 0.215s

OK
Destroying test database for alias 'default'...

I think this is totally reasonable and maybe juggling shell session environment variables isn't the job of a Python development tool anyway, but I do still kinda wish for there to be some kind of a better way for it.

All in all, I have mixed feelings about Rye. On one hand, many of the design ideas are good and I think I'd like to use a Cargo-but-for-Python to develop software, since I also like using Cargo. However, the insistence on using static Python builds leads to suffering and won't let me uninstall pyenv anyways. There's also some rough edges in implementation to cut yourself up further, like my attempt at getting Rye and Homebrew-Python to mingle shows. The fact that rye test is also just an alias to run pytest with no possibility for any other testing tool like the built-in unittest module, which is totally and absolutely a fine tool for testing that you get out of the box, seems very indicative of just how early in Rye's life we are at the moment.

If you have a working Python development environment, no matter how rudimentary its feature set might be, it might not be worth it to try out Rye now. Its best value at the moment would probably rather be for newbs, who can benefit from its easy setup and sane defaults for many things (like not using the system Python if you've got one). However, I think that there's also a good chance that Rye will actually be all-around great in like 12 months, so I also wouldn't invest too heavily in an alternative tool like Poetry if you're shopping around for project management tools.

Nevertheless, Python is still a fun programming language and the tooling is fine.

CLI text editing with ed

learnbyexample — Thu, 28 Mar 2024 02:00:00 GMT

I'm finally writing a post on the ed command. And I'm keeping it short so that I'll actually publish the post. The examples presented below will be easier to understand for those already familiar with Vim and sed. See the links at the end for learning resources.

Although I'm interested in getting to know ed better, I don't really find myself in situations where it'd help me. But, I have used it a few times to answer questions on stackoverflow.

Moving lines

Consider this sample input file:

$ cat ip.txt
apple
banana
cherry
fig
mango
pineapple

Suppose, you want to move the third line to the top. If you are using Vim, you can execute :3m0 where 3 is the input address, m is the move command and 0 is the target address. To do the same with ed:

$ printf '3m0\nwq\n' | ed -s ip.txt -

$ cat ip.txt
cherry
apple
banana
fig
mango
pineapple

The 3m0 part in the above ed command is identical to the Vim solution. After that, another command wq (write and quit) is issued to save the changes (again, Vim users would be familiar with this combination). The -s option suppresses diagnostics and other details. - is used to indicate that the ed script is passed via stdin.

You can also move lines based on a regexp match. Here's an example:

# move the first matching line containing 'an' to the top of the file
$ printf '/an/m0\nwq\n' | ed -s ip.txt -

$ cat ip.txt
banana
cherry
apple
fig
mango
pineapple

If you want to move all the matching lines, you can use the g command (same as Vim). Note that the first matching line will be moved first, then the next matching line and so on. So the order will be reversed after the move.

$ printf 'g/app/m0\nwq\n' | ed -s ip.txt -

$ cat ip.txt
pineapple
apple
banana
cherry
fig
mango

Here's the stackoverflow link that inspired the above examples. See this stackoverflow answer for more examples of moving lines. See this one to learn how to copy a particular line to the end of the file. See this unix.stackexchange answer for an example of moving a range of lines, where the same regex matches both the starting and ending lines.

Negative addressing

There are plenty of addressing features provided by the GNU sed command, but negative addressing isn't one. Here's an example of deleting the last but second line using ed:

$ cat colors.txt
red
green
blue
yellow
black

$ printf '$-2d\nwq\n' | ed -s colors.txt -
$ cat colors.txt
red
green
yellow
black

Resource links

My OneStream Journey So Far...

Subclassed — Wed, 27 Mar 2024 17:29:44 GMT

This blog has been quiet for some time. That's largely because I've done quite a bit of writing (and experimenting!) for my employer, OneStream. If you're interested in that sort of thing, here's a brief list of my posts on the company blog:

Blogging about actual work stuff has been a lot of fun! And if you go reading those posts, check out other ones as well - some of the stuff my colleagues can do is just amazing.

This said, the only constant in life is change. That means my contributions will probably slow down in the next few months, as I transition into a new role. Or maybe not! We'll see.

Brute force text search optimizations

Ben E. C. Boyter — Wed, 27 Mar 2024 02:00:00 GMT

One of the things I see commonly on any mockup is a search box, usually with vaguely defined requirements as to how this is meant to actually work. However most people tend to assume its an instant update as you type with a debounce option of 100ms or something.

The fastest way to implement this with a database is a horrible concatenated like query. However where you can store things in memory you can do a much better brute force search. A possible implementation in Go could look like the below.

Finding memory leaks in Postgres C code

Notes on software development — Wed, 27 Mar 2024 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Trust in SaaS

Dan Palmer — Sat, 23 Mar 2024 02:00:00 GMT

There’s a lie at the core of the SaaS ecosystem. In such a wide ranging category of services there are few commonalities, but arguably the main differentiator between SaaS and more traditional businesses is the notion of self-service. A customer can sign up, provide an email address and credit card, and be successful with the product. This is increasingly proving to be a lie, and customers are losing out because of it.

Organizing State of Diffusion++

swyx's site RSS Feed — Fri, 22 Mar 2024 19:09:15 GMT

I manifested a "State of Diffusion++" meetup yesterday that I felt was a success. 200 people signed up! Here are some photos!

SuperPrompter - Supercharge your text prompts for AI/LLM image generation

smcleod.net — Fri, 22 Mar 2024 02:00:00 GMT

SuperPrompter is a Python-based application that utilises the SuperPrompt-v1 model to generate optimised text prompts for AI/LLM image generation (for use with Stable Diffusion etc…) from user prompts.

See Brian Fitzgerald’s Blog for a detailed explanation of the SuperPrompt-v1 model and its capabilities / limitations.

Features

Utilises the SuperPrompt-v1 model for text generation.
A basic (aka ugly) graphical user interface built with tkinter.
Customisable generation parameters (max new tokens, repetition penalty, temperature, top p, top k, seed).
Optional logging of input parameters and generated outputs.
Bundling options to include or exclude pre-downloaded model files.
Unloads the models when the application is idle to free up memory.

Prebuilt Binaries

Check releases page to see if there are any prebuilt binaries available for your platform.

Stock returns of simple trading strategies

Bytepawn - Marton Trencseni — Fri, 22 Mar 2024 01:00:00 GMT

I clean up the code I wrote for earlier posts, and run simple trading experiments.

Llamalink - Ollama to LM Studio LLM Model Linker

smcleod.net — Thu, 21 Mar 2024 02:00:00 GMT

This is a simple command-line tool that allows you to easily link Ollama models to LM Studio's directory structure. It simplifies the process of symlinking Ollama models to LM Studio, making it convenient to use the models in both applications.

Three-Step Debugging: A Methodical Approach

Donat Studios — Thu, 21 Mar 2024 01:07:18 GMT

I've been working as a developer for almost twenty years now. In that time I have had the pleasure of helping many developers, and helping them debug code. Some were new to the field, had vastly more experience than me.

While some are naturally somewhat gifted at debugging, it's important to understand that the skill can be learned. In fact, even very experienced developers can struggle with it

I've known developers who stare at their code trying to reason their way through it. I've known developers who will change things at random until the problem goes away. I've known developers whose technique I describe as "I've tried nothing and I'm all out of ideas".

I take a very structured approach, and this is something I've meant to write down for years now. Solving 90% of the of the problems you will face comes down to three simple steps.

Identify Outputs
Identify Inputs
Repeat

Identify Outputs

There is a bug in your code; that much we know. It's causing something to not work as expected. The first step in debugging is to identify what that something is.

Determine what your expected output is, and what the actual output is.

Walk through the call stack to find where the output diverged from expected.

Identify Inputs

This step is crucial and often the most challenging.

In an ideal world we'd only be debug pure functional code. The only inputs would be parameters, and all calls within are deterministic.

We don't live in an ideal world.

We often find ourselves debugging code that inherits values and state from elsewhere. This can be self-fulfilling, such code is usually hardest to test and most prone to errors.

Sources of state include, but are not limited to:

Members of the containing class or its parents
Non-deterministic values and methods
- The current time
- Random number generators
- Potentially flakey network services
- Databases
- File systems
- User input
Global variables
Environment variables
Configuration files

This list is far from exhaustive and meant to give you an idea of the kinds of things you might need to consider.

While debugging, it's important to identify and note all these inputs. We then need to understand their state at the time of the error. This is often where I see developers struggle. Some part of their code can vary and they have failed to identify it.

Ways to identify their state logging, or by using a debugger to inspect the state of the system at the time of the error.

It comes down to a is a matter of personal preference as well as comfort with the tooling.

I know my way around my debugger, and yet I more often than not am inclined to just spit the state to standard out. I find this to be the most effective way of debugging. I know developers who are very good at using debuggers. I also know developers who spend a lot of time trying to make sense of their debugger.

I've never met a developer who is bad at using print statements.

The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.

– Brian Kernighan

Repeat

After identifying the code's outputs and inputs, use this information to track down the problem. You will need to repeat this process recursively through the call tree.

We work our way through the call stack, identifying the unexpected outputs and inputs as we go. Keep this up and we will locate the root cause.

Other Thoughts

Identifying outputs and inputs isn't limited to values of variables. It's about the state of the system as a whole - including where boundaries get crossed.

You need to understand how the system handles crossing boundaries and ensure that the handoff results as expected. Understanding how the system manages crossing these boundaries is crucial for identifying where errors might occur.

You also may need to understand the state of the larger system at the time of the error. "The system" is not limited to the code you're working on. It can also include the state of the database, the state of the file system, the state of the network, cookies, sessions, the end user's browser, etc.

When you've identified the issue, write a test that reproduces it. This confirms you've fixed it and understood it, as well as preventing its recurrence. If you don't already have a strong test suite, regression tests are a great place to start.

Conclusion

I hope this guide helps you improve your debugging skills. Debugging is a skill that you can learn and improve over time.

Debugging is not just about fixing errors. It's an opportunity to learn your codebase and how to enhance it.

And remember, careful thought, combined with well-placed print statements, remains the most effective debugging tool.

Leaving Amazon

The Cranky Developer on Crater Moon Development — Sat, 16 Mar 2024 20:12:09 GMT

Steps for leaving Amazon S3 for S3-compatible Backblaze B2

Zig, Rust, and other languages

Notes on software development — Fri, 15 Mar 2024 02:00:00 GMT

Having worked a bit in Zig, Rust, Go and now C, I think there are a few common topics worth having a fresh conversation on: automatic memory management, the standard library, and explicit allocation.

Zig is not a mature language. But it has made enough useful choices for a number of companies to invest in it and run it in production. The useful choices make Zig worth talking about.

Go and Rust are mature languages. But they have both made questionable choices that seem worth talking about.

All of these languages are developed by highly intelligent folks I personally look up to. And your choice to use any one of these is certainly fine, whichever it is.

The positive and negative choices particular languages made, though, are worth talking about as we consider what a systems programming language 10 years from now would look like. Or how these languages themselves might evolve in the next 10 years.

My perspective is mostly building distributed databases. So the points that I bring up may have no relevance to the kind of work you do, and that's alright. Moreover, I'm already aware most of these opinions are not shared by the language maintainers, and that's ok too. I am not writing to convince anyone.

Automatic memory management

One of my bigger issues with Zig is that it doesn't support RAII. You can defer cleanup to the end of a block; and this is half of the problem. But only RAII will allow for smart pointers and automatic (not manual) reference counting. RAII is an excellent option to default to, but in Zig you aren't allowed to. In contrast, even C "supports" automatic cleanup (via compiler extensions).

But most of the time, arenas are fine. Postgres is written in C and memory is almost entirely managed through nested arenas (called "memory contexts") that get cleaned up when some subset of a task finishes, recursively. Zig has builtin support for arenas, which is great.

Standard library

It seems regrettable that some languages have been shipping smaller standard libraries. Smaller standard libraries seem to encourage users of the language to install more transitively-unvetted third-party libraries, which increases build time and build flakiness, and which increases bitrot over time as unnecessary breaking changes occur.

People have been making jokes about node_modules for a decade now, but this problem is just as bad in Rust codebases I've seen. And to a degree it happens in Java and Go as well, though their larger standard libraries allow you to get further without dependencies.

Zig has a good standard library, which may be Go and Java tier in a few years. But one goal of their package manager seemed to be to allow the standard library to be broken up; made smaller. For example, JSON support moving out of the standard library into a package. I don't know if that is actually the planned direction. I hope not.

Having a large standard library doesn't mean that the programmer shouldn't be able to swap out implementations easily as needed. But all that is required is for the standard library to define an interface along with the standard library implementation.

The small size of the standard library doesn't just affect developers using the language, it even encourages developers of the language itself to depend on libraries owned by individuals.

Take a look at the transitive dependencies of an official Node.js package like node-gyp. Is it really the ideal outcome of a small standard library to encourage dependence in official libraries on libraries owned by individuals, like env-paths, that haven't been modified in 3 years? 68 lines of code. Is it not safer at this point to vendor that code? i.e. copy the env-paths code into node-gyp.

Similarly, if you go looking for compression support in Rust, there's none in the standard library. But you may notice the flate2-rs repo under the official rust-lang GitHub namespace. If you look at its transitive dependencies: flate2-rs depends on (an individual's) miniz_oxide which depends on (an individual's) adler that hasn't been updated in 4 years. 300 lines of code including tests. Why not vendor this code? It's the habits a small standard library builds that seem to encourage everyone not to.

I don't mean these necessarily constitute a supply-chain risk. I'm not talking about left-pad. But the pattern is sort of clear. Even official packages may end up depending on external party packages, because the commitment to a small standard library meant omitting stuff like compression, checksums, and common OS paths.

It's a tradeoff and maybe makes the job of the standard library maintainer easier. But I don't think this is the ideal situation. Dependencies are useful but should be kept to a reasonable minimum.

Hopefully languages end up more like Go than like Rust in this regard.

Explicit allocation

When folk discuss the Zig standard library's pattern of requiring an allocator argument for every method that allocates, they often talk about the benefit of swapping out allocators or the benefit of being able to handle OOM failures.

Both of these seem pretty niche to me. For example, in Zig tests you are encouraged to pass around a debug allocator that tells you about memory leaks. But this doesn't seem too different from compiling a C project with a debug allocator or compiling with different sanitizers on and running tests against the binary produced. In both cases you mostly deal with allocators at a global level depending on the environment you're running the code in (production or tests).

The real benefit of explicit allocations to me is much more trivial. You basically can't code a method in Zig without acknowledging allocations.

This is particularly useful for hotpath code. Take an iterator for example. It has a new() method, a next() method, and a done() method. In most languages, it's basically impossible at the syntax or compiler-level to know if you are allocating in the next() method. You may know because you know the behavior of all the code in next() by heart. But that won't happen all the time.

Zig is practically alone in that if you write the next() method and and don't pass an allocator to any method in the next() body, nothing in that next() method will allocate.

In any other language it might not be until you run a profiler that you notice an allocation that should have been done once in new() accidentally ended up in next() instead.

On the other hand, for all the same reasons, writing Zig is kind of a pain because everything takes an allocator!

Explicit allocation is not intrinsic to Zig, the language. It is a convention that is prevalent in the standard library. There is still a global allocator and any user of Zig could decide to use the global allocator. At which point you've got implicit allocation. So explicit allocation as a convention isn't a perfect solution.

But it, by default, gives you a level of awareness of allocations you just can't get from typical Go or Rust or C code, depending on the project's practices. Perhaps it's possible to switch off the Go, Rust and C standard library and use one where all functions that allocate do require an allocator.

But explicitly passing allocators is still sort of a visual hack.

I think the ideal situation in the future will be that every language supports annotating blocks of code as must-not-allocate or something along those lines. Either the compiler will enforce this and fail if you seem to allocate in a block marked must-not-allocate, or it will panic during runtime so you can catch this in tests.

This would be useful beyond static programming languages. It would be as interesting to annotate blocks in JavaScript or Python as must-not-allocate too.

Otherwise the current state of things is that you'd normally configure this sort of thing at the global level. Saying "there must not be any allocations in this entire program" just doesn't seem as useful in general as being able to say "there must not be any allocations in this one block".

Optional, not required, allocator arguments

Rust has nascent support for passing an allocator to methods that allocate. But it's optional. From what I understand, C++ STL is like this too.

These are both super useful for programming extensions. And it's one of the reasons I think Zig makes a ton of sense for Postgres extensions specifically. Because it was only and always ever built for running in an environment with someone else's allocator.

Praise for Zig, Rust, and Go tooling

All three of these have really great first-party tooling including build system, package management, test runners and formatters. The idea that the language should provide a great environment to code in (end-to-end) makes things simpler and nicer for programmers.

Meandering non-conclusion

Use the language you want to use. Zig and Rust are both nice alternatives to writing vanilla C.

On the other hand, I've been pleasantly surprised writing Postgres C. How high level it is. It's almost a separate language since you're often dealing with user-facing constructs, like Postgres's Datum objects which represent what you might think of as a cell in a Postgres database. And you can use all the same functions provided for Postgres SQL for working with Datums, but from C.

I've also been able work a bit on Postgres extensions in Rust with pgrx lately, which I hope to write about soon. And when I saw pgzx for writing Postgres extensions in Zig I was excited to spend some time with that too.

Wrote a post on my wishlist for Zig and Rust. Focused on automatic memory management, the standard library, and explicit allocation.https://t.co/dvynizU9V2 pic.twitter.com/iTXp5QVxj0
— Phil Eaton (@eatonphil) March 15, 2024

Sokoban

Kartik Agaram — Wed, 13 Mar 2024 09:00:00 GMT

The kids have been enjoying Baba is You, and watching them brought back pleasant memories for me of playing the classic crate-pushing game Sokoban. So I went looking and found a very nice project that has collected 300 classic publicly available Sokoban puzzles. Then of course I had to get it on my phone so I could play it anywhere. The result is the sokoban.love client.

video; 1 minute

On a technical level, with sokoban.love I've finally managed to figure out how to scale modifying programs on my phone beyond the tiny scripts Lua Carousel supports. Carousel treats each 'page' of the carousel as a separate script, and shares the screen between the code for the page and the drawings the page makes. When you switch between pages, Carousel saves and restores code for you so the script currently on screen is always the one currently drawing.

sokoban.love comes bundled with multiple pages of code (including 7000 lines for all the levels; those would be a pain to copy paste into Carousel). The pages all collaborate to create the app; switching pages changes nothing about the code that is running. The screen is also no longer shared between the app and its code editing environment. When you run the app the Carousel menu disappears, replaced by a single button to exit the app and edit its code.

This approach works well for editing on a phone. The trade-off I made is to jettison the live-editing experience. You can still get that with sokoban.love, but you'll need to get on a computer and connect driver.love to it like all my Freewheeling Apps.

As a bonus, sokoban.love includes a simple solver to eliminate some gruntwork for moving the player on touchscreens that you can see in action in this video. Tapping on the buttons along the edges moves the player a single square. Tapping on an empty square moves the player there if that is possible without moving any crates. Tapping on a crate and then an empty square will try to get the crate there if that is possible without moving any other crates.

Quickly reviewing your journal with bash

jodavaho.io — Mon, 11 Mar 2024 02:00:00 GMT

I’ve been journalling for 10+ years. In addition to forcing me to actually write down what happened that day, reviewing old entries provides a feeling of history that makes life feel soooo long and so rich. I can review my now-wife’s rocky courtship and feel grateful we made it. I looked back 3 years ago and found the day our children were conceived :D (we have twins). I can review the anxieties of my PhD years, etc etc.

For some reason, feeling like my past life has not been short helps me to feel that there’s so much life left to live. Looking back at the phenomenal changes of the last 5 years (or 10), shows me that I can do a lot with the next 5 years.

Novelty and so on may help to “slow” time, but for me the perception of the shortness of life is best fought by reminding yourself that it is not short, and there is so much change coming - more than you could imagine.

(and I’m horrible at doing it every day, maybe every week or so during slumps)

We’re at our computers all day every day. So I just lowered the barrier to entry with a few bash tricks. It’s helped me keep the habit up.

https://jodavaho.io/tags/bullet-journal.html

Now, I review 1, 5, and 10 years ago every day, to re-live my life from those years, so to speak:

# list dates from 10 years ago +/3 3 days
# get years and day range from args
echo "### $years years ago"
for i in `seq -w -$days $days`
do
  olddate=`date -d "$years years ago $i days" +%Y-%m-%d`  
  longdate=`date -d "$years years ago $i days" +%A\ %B\ %d,\ %Y`
  echo "### $olddate.md ($longdate)"
  cat $olddate.md
done

First month on a database team

Notes on software development — Mon, 11 Mar 2024 02:00:00 GMT

A little over a month ago, I joined EnterpriseDB on a distributed Postgres product (PGD). The process of onboarding myself has been pretty similar at each company in the last decade, though I think I've gotten better at it. The process is of course influenced by the team, and my coworkers have been excellent. Still, I wanted to share my thought process and personal strategies.

Avoid, at first, what is always challenging

Trickier things at companies are the people, organization, and processes. What code exists? How does it work together? Who owns what? How can I find easy code issues to tackle? How do I know what's important (so I can avoid picking it up and becoming a bottleneck).

But also, in the first few days or weeks you aren't exactly expected to contribute meaningfully to features or bugs. Your sprint contributions are not tracked too closely.

The combination of 1) what to avoid and 2) the sprint-freedom-you-have leads to a few interesting and valuable areas to work on on your own: the build process, tests, running the software, and docs.

But code need not be ignored either. Some frequent areas to get your first code contributions in include user configuration code, error messages, and stale code comments.

What follows are some little 1st day, 1st week, 1st month projects I went through to bootstrap my understanding of the system.

Build process

First off, where is the code and how do you build it? This requires you to have all the relevant dependencies. Much of my work is on a Postgres extension. This meant having a local Postgres development environment, having gcc, gmake (on mac), Perl, and so on. And furthermore, PGD is a pretty mature product so it supports building against multiple Postgres distributions. Can I build against all of them?

The easiest situation is when there are instructions for all of this, linked directly from your main repo. When I started, the instructions did exist but in a variety of places. So over the first week I started collecting all of what I had learned about building the system, with dependencies, across distributions, and with various important flags (debug mode on, asserts enabled, etc.). I finished the first week by writing a little internal blog post called "Hacking on PGD".

I hadn't yet figured out the team processes so I didn't want to bother anyone by trying to get this "blog post" committed anywhere yet as official internal documentation. Maybe there already was a good doc, I just hadn't noticed it yet. So I just published it in a private Confluence page and shared it in the private team slack. If anyone else benefited from it, great! Otherwise, I knew I'd want to refer back to it.

This is an important attitude I think. It can be hard to tell what others will benefit from. If you get into the habit of writing things down internally for your own sake, but making it available internally, it is likely others will benefit from it too. This is something I've learned from years of blogging publicly outside of work.

Moreover, the simple act of writing a good post creates yourself as something of an authority. This is useful for yourself if no one else.

Writing a good post

Let's get distracted here for a second. One of the most important things I think in documentation is documenting not just what does exist but what doesn't. If you had to take a path to get something to work, did you try other paths that didn't work? It can be extremely useful to figure out what exactly is required for something.

Was there a flag that you tried to build with but you didn't try building without it? Well try again without it and make sure it was necessary. Was there some process you executed where the build succeeded but you can't remember if it was actually necessary for the build to succeed?

It's difficult to explain why I think this sort of precision is useful but I'm pretty sure it is. Maybe because it builds the habit of not treating things as magic when you can avoid it. It builds the habit of asking questions (if only to yourself) to understand and not just to get by.

Static analysis? Dynamic analysis?

Going back to builds, another aspect to consider is static and dynamic analysis. Are there special steps to using gdb or valgrind or other analyzers? Are you using them already? Can you get them running locally? Has any of this been documented?

Maybe the answer to all of those is yes, or maybe none of those are relevant but there are likely similar tools for your ecosystem. If analysis tools are relevant and no one has yet explored them, that's another very useful area to explore as a newcomer.

Testing

After I got the builds working, I felt the obvious next step was to run tests. But what tests exist? Are there unit tests? Integration tests? Anything else? Moreover, is there test coverage? I was certain I'd be able to find some low-hanging contributions to make if I could find some files with low test coverage.

Alas, my certainty hit the wall in that there were in fact too many types of integration tests that all do provide coverage already. They just don't all report coverage.

The easiest ways to report coverage (with gcov) were only reporting coverage for certain integration tests that we run locally. There are more integration tests run in cloud environments and getting coverage reports there to merge with my local coverage files would have required more knowledge of people and processes, areas that I wanted not to be forced to think about too quickly.

So coverage wasn't a good route to go. But around this time, I noticed a ticket that asked for a simple change to user configuration code. I was able to make the change pretty quickly and wanted to add tests. We have our own test framework built on top of Postgres's powerful Perl test framework. But it was a little difficult to figure out how to use either of them.

So I copied code from other tests and pared it down until I got the smallest version of test code I could get. This took maybe a day or two of tweaking lines and rerunning tests since I didn't understand everything that was/wasn't required. Also it's Perl and I've never written Perl before so that took a bit of time and ChatGPT. (Arrays, man.)

In the end though I was able to collect my learnings into another internal confluence post just about how to write tests, how to debug tests, how to do common things within tests (for example, ensuring a Postgres log line was outputted), etc. I published this post as well and shared it in the team Slack.

Running

I had PGD built locally and was able to run integration tests locally, but I still hadn't gotten a cluster running. Nor played with the eventual consistency demos I knew we supported. We had a great quickstart that ran through all the manual steps of getting a two-node cluster up. This was a distillation, for devs, of a more elaborate process we give to customers in a production-quality script.

But I was looking for something in between a production-quality script and manually initializing a local cluster. And I also wanted to practice my understanding of our test process. So I ported our quickstart to our integration test framework and made a PR with this new test, eventually merging this into the repo. And I wrote a minimal Python script for bringing up a local cluster. I've got an open PR to add this script to the repo. Maybe I'll learn though that a simple script such as this does already exist, and that's fine!

Docs

The entire time, as I'd been trying to build and test and run PGD, I was trying to understand our terminology and architecture by going through our public docs. I had a lot of questions coming out of this I'd ask in the team channel.

Not to toot my horn but I think it's somewhat of a superpower to be able/willing to ask "dumb questions" in a group setting. That's how I frame it anyway. "Dumb question: what does X mean in this paragraph?" Or, "dumb question: when we say performance improvement because of Y, what is the intuition here?" Because of the time spent here, I was able to make a few more docs contributions as I read through the docs as well.

You have to balance where you ask your dumb questions though. Asking dumb questions to one person doesn't benefit the team. But asking dumb questions in too wide a group is sometimes bad politics. Asking "dumb questions" in front of your team seems to have the best bang for buck.

But maybe the more important contributions were, when I got more comfortable with the team, proposing to merge my personal, internal Confluence blog posts into the repo as docs. I think in a number of cases, what I wrote about indeed hadn't been concisely collected before and thus was useful to have as team documentation.

Even more challenging was trying to distill (a chunk of) the internal architecture. Only after following many varied internal docs and videos, and following through numerous code paths, was I able to propose an architecture diagram outlining major components and communication between them, with their differing formats (WAL records, internal enums, etc.) and means of communication (RPC, shared memory, etc.). This architecture diagram is still in review and may be totally off. But it's already helped at least me think about the system.

In most cases this was all information that the team had already written or explained but just bringing it together and summarizing provided a different useful perspective I think. Even if none of the docs got merged it still helped to build my own understanding.

Beyond the repo

Learning the project is just one aspect of onboarding. Beyond that I join the #cats channel, the #dogs channel, found some fellow New Yorkers and opened a NYC channel, and tried to find Zoom-time with the various people I'd see hanging around common team Slack channels. Trying to meet not just devs but support folk, product managers, marketing folk, sales folk, and anyone else!

Walking the line between scouring our docs and GitHub and Confluence and Jira on my own, and bugging people with my incessant questions.

I've enjoyed my time at startups. I've been a dev, a manager, a founder, a cofounder. But I'm incredibly excited to be back, at a bigger company, full-time as a developer hacking on a database!

And what about you? What do you do to onboard yourself at a new company or new project?

I've been having an absolute blast in my first month at EDB and I wanted to share a few of my strategies for onboarding myself on a database team. Strategies broadly applicable for devs on a new team/project.https://t.co/TS5qRLysuA pic.twitter.com/lvuxDBQJwx
— Phil Eaton (@eatonphil) March 12, 2024

Pensieve: Mar 8 2024 - on Dune 2

swyx's site RSS Feed — Sat, 09 Mar 2024 13:45:00 GMT

A collection of public thoughts that could be blogposts but i dont have time, so here, have the short form. I may upgrade these to full posts in future.

The volatility of simple trading strategy returns

Bytepawn - Marton Trencseni — Sat, 09 Mar 2024 01:00:00 GMT

I investigate the volatility of aggressive take-profit strategies on tech stocks.

Pensieve: Mar 7 2024 - on Agency

swyx's site RSS Feed — Fri, 08 Mar 2024 09:26:35 GMT

A collection of public thoughts that could be blogposts but i dont have time, so here, have the short form. I may upgrade these to full posts in future.

Installing direnv on macOS

Ben E. C. Boyter — Thu, 07 Mar 2024 02:00:00 GMT

Having had to install direnv multiple times over the last few days to get a team up and running im writing the method down because I keep having to look it up.

Install based on the following instructions https://github.com/direnv/direnv/blob/master/docs/installation.md Then hook it up to your shell via https://github.com/direnv/direnv/blob/master/docs/hook.md

Lastly to have it work with .env files create the following directory and file

~/.config/direnv/direnv.toml

and edit the contents to contain the following,

[global]
load_dotenv=true

You will need to restart your shell after doing this. Opening a new terminal is the easiest way to do this.

Acquired in 99 days

Austin Z. Henley's Blog — Wed, 06 Mar 2024 17:00:01 GMT

https://austinhenley.com/blog/acquiredin99days.html

A simple package for running many linters

Steve Kemp's Blog — Sun, 03 Mar 2024 15:00:00 GMT

I used to configure Emacs to run a linter when saving some specific type of files. For example I'd have a perl-utilities package to reformat perl code, and run the perl-linter on saving, then I'd have a hook to do the same thing for Dockerfiles, etc, etc.

It occurred to me recently that I should have a linter for both JSON and YAML files, since I have to edit those filetypes so damn often, and that there wasn't a great solution for those - Until it occurred to me I wrote sysbox which is a simple collection of tools in one binary, and that supports some validation commands:

sysbox validate-json /path/to/file
sysbox validate-yaml /path/to/file
sysbox validate-xml  /path/to/file

With that in mind it became obvious that what I want to do is pretty much always the same:

Run an external command, when the file is saved.
- If the exit-code of that command is "success" (i.e. zero):
  - Do nothing.
- If the exit-code is "failure" (i.e. non-zero):
  - Show the output.

And this process is the same for ANY of the linters I run. The only thing that changes is the command to run, based on the mode/type of file in question.

That lead to the following configuration:

(defvar save-check-config
      '(

        (:mode cperl-mode
         :exec "perl -wc -I. %f"
         :cond (executable-find "perl"))

        (:mode dockerfile-mode
         :exec "hadolint --no-color %f"
         :cond (executable-find "hadolint"))

        (:mode json-mode
         :exec "sysbox validate-json %f"
         :cond (executable-find "sysbox"))

        (:mode nxml-mode
         :exec "sysbox validate-xml %f"
         :cond (executable-find "sysbox"))

        (:mode perl-mode
         :exec "perl -wc -I. %s"
         :cond (executable-find "perl"))

        ;; This avoids creating .pyc files, which would happen if we had
        ;; used the more natural/obvious "python3 -m py_compile %s" approach
        (:mode python-mode
         :exec "python3 -c 'import ast; ast.parse(open(\"%f\").read())'"
         :cond (executable-find "python3"))

        (:mode sh-mode
         :exec "shellcheck %f"
         :cond (executable-find "shellcheck"))

        (:mode terraform-mode
         :exec "tflint --no-color --chdir %d"
         :cond (executable-find "tflint"))

        (:mode yaml-mode
         :exec "sysbox validate-yaml %f"
         :cond (executable-find "sysbox"))
        )
     )

Basically a list of things:

We have the mode of files to which the linter/validator applies.
We have the command to run
- %f is changed to the filename which has just been saved.
- %d is changed to the directory-name containing that file.
We add a :cond key to decide if we should run.
- Which basically is used for "if the binary is found .. run it, otherwise silently do nothing".

I'm quite pleased with how simple the package was to write, and now I have all my linting configuration in one-place.

https://github.com/skx/save-check.el/

I'd be tempted to do the same for "format on save", but to be honest with LSP most of the code I care about has that in-place already.

Should I rename to "multi-lint[er].el"? Probably, but I guess we'll see in the future.

The Yak Shave

Weblog on marginalia.nu — Wed, 28 Feb 2024 02:00:00 GMT

I set out a little over a week ago to add a service registry to Marginalia Search, primarily to reduce its dependence on docker. I would like it to be able to run on bare metal as well, which poses a problem since configuring the application manually is a bit of a headache with dozens of ports that need to be set up. It would also be desirable to be able to run multiple instances of important services in order elliminate downtime during upgrades.

2024–02–28: Support for WiFi on QuartzPro64

megi's PinePhone Development Log — Wed, 28 Feb 2024 01:00:00 GMT

Ideas

Donat Studios — Mon, 26 Feb 2024 20:43:34 GMT

A minimal functional language, maybe similar to Excel equations for sharing logic between unrelated languages. Share the same logic between a Go or PHP backend and a JavaScript front end.
mtpl minimum viable template language - like mustache but with less gotchas
A pkg.go.dev style service for PHP / composer packages.
Materialized JSON Parser - generate parsers ahead of time for specific JSON structures to read much faster
Raspberry PI Thermostat

2024 Feb Summary

jodavaho.io — Mon, 26 Feb 2024 02:00:00 GMT

I’ve been really into the Undaunted: Stalingrad game. Have been playing a lot less Hunt showdown, since I’ve been travelling, doing jury duty, and working more on this blog and other side projects.

I’m noodling a new personal finance side project, this one might actually be of general use. Highfleet Opt is still living, and we hope to do more work on it.

The kids are precocious, learning ownership and boundaries, trying to sound out letters for words (at our prompt, mostly), and being extremely energetic at home. It’s quite a workout.

Wife and I are running up to a 6mi goal for her marathon relay. We enjoyed a nice trip to Mexico lately for a wedding.

Work at Outrider is pretty nuts. Busy, for sure. Some friends have left, which is always a bummer. I am really proud of the work we did there to speed up the multi-agent planner by 100x (minutes to less than a second, max of 10s of minutes to a few seconds). I’d love to do more of that deep technical work. But 99% of all problems in software are interfaces, as in interfaces between teams and management and processes, so that’s always where the trouble will be.

I cannot believe we have no snow. None at all. Last year this time we had a least 18", with it piled up to 4’ on each side of my driveway.

Persona 3 Reload review

ブラック — Mon, 26 Feb 2024 02:00:00 GMT

Welcome back.

Like for many others, Persona 3 was my first foray into the Persona series, having played through FES back in 2011, and will always hold a special place in my heart. I have also been wanting for a Persona 3 remake since at least 2020, so I’ve been extremely excited to play through Persona 3 Reload ever since the rumours were confirmed to be real. And yet, with years of lofty expectations weighing down on it, I absolutely loved Persona 3 Reload.

Story-wise Persona 3 has always remained as my favourite one in the series. Granted, some of that might be nostalgia for my first Persona story, but it’s a feeling that’s stayed with me after Persona 4 and Persona 5. I just really enjoy the theme and overall feeling of the story. And now that I have experienced it again, I think I can still confidently claim it to be the superior of the three. Thankfully Reload manages to not alter the story even though it does ever-so-slightly expand it here and there. Not changing the story is also to its benefit since ATLUS’ efforts to add new story content for re-releases have often had very mixed results. And despite knowing the story, although fuzzy with some details after 13 years, the emotional highs still bit hard.

The gameplay side of things has a lot more changes, but it still very much feels like Persona 3. I was actually quite surprised that they didn’t relax the social link calendar to make it more like Persona 5 Royal. Getting all of my bonds maxed out in a single playthrough was definitely challenging with few days to spare. Granted, the social links are still considerably easier than in the original, but not to the point of feeling like the other games. The overall package does feel like a very good balance of the old experience and new quality of life.

The new Rewind system is excellent. I imagine it’s going to become a new staple feature in every single Persona game going forwards and reduce some of the save overwrite anxiety. Would’ve definitely saved me some time in prior titles. It’s also nice to see that despite trying to be faithful to the original game, they’re not beyond innovating when they have good new ideas, especially when they’re ones that do not dilute the original experience.

The combat side of things has definitely been Persona 5-ified – but actually less so than I imagined. I fully expected them to just clone the Royal combat system and call it a day, but they actually made a system unique to Reload. Familiar, but unique. And much like Persona 5’s combat system, it very much works. It’s fluid and fun. I always remembered Tartarus being the worst part of the original game, but somehow it doesn’t actually even as bad as I remembered in Reload. Either I am completely misremembering the experience or Reload has just improved it that much. Of course, it is still very repetitive labyrinth of procedural generation, but still a decent climb. The visual update also does help to make it less bleak and boring. The background music is however very faithful to the original, as in, not very inspiring. Might’ve listened to a couple of podcasts while I was at it.

The new visuals do look and feel excellent. It’s not exactly the sharpest edge of graphics technology and not all of the art assets are polished to the highest degree, but the style does make up for a lot of it. Especially when the only available versions of Persona 3 have been extremely low fidelity, Reload just feels extremely pretty in comparison. And I just absolutely love all of the character models. I wanted them to remake Persona 3 with the character models from Dancing in Moonlight when that came out, but Reload’s models are even better. And in a series first, there’s ray-traced reflections, which do look really nice.

The biggest downside might be the performance though – there’s some definite Unreal stutter happening even on powerful hardware. Maybe they’ll patch that out in due time, although I wouldn’t count on it. There’s definitely room for improvement on that department.

There’s still 2D-animated cutscenes, although some cutscenes that I remember being 2D animation in the original are now 3D live-rendered cutscenes. While the quality of the 2D animation has never been the highest, both in the original and Reload, I’m of two minds about this. The cutscenes in Reload are definitely good, but there’s just something irresistible about 2D cutscenes.

The voice acting in Persona 3 Reload is immaculate. First of all, almost all of the original cast from Persona 3 returns to redo their lines and they all sound exactly how I remember them sounding over a decade ago. The only exception that I am aware is Tanonaka, who passed away in 2010, and they got a fairly alright soundalike to voice Igor in his stead. And to make things even better, they got a bunch of voice actors for all of the social links, with every single rank of every single social link (bar the obvious silent one) being fully voiced! I really hope that we can expect all new Persona games to keep up this level of quality and quantity when it comes to voice acting.

The music side initially feels rather uncanny. Everything sounds so familiar to the songs I’ve had in my music library for the past ten years and yet are strikingly different. However, that feeling of uncanniness did wear off as I kept playing, and the music does feel like Persona 3 – even if served with the whiff of a bootleg recording. The couple of pivotal songs during critical parts of the game are also kept and sound very similar to the originals despite Kawamura Yumi being replaced with a new face in Takahashi Azumi. There are also a couple of original bangers, so I’ll definitely be adding Reload’s soundtrack to my usual music rotation in the future.

Yet I unfortunately can’t fully shake the feeling that maybe a small bit of Persona 3’s identity has been lost in the Persona 5-ification of the game. Not to a degree where I’d want to not play this or not recommend it to others, but just enough to make me think that there’s still a place in the universe for the original. I know that there’s definitely still a place for Portable, since the alternative perspective story for the female protagonist is not part of a Persona 3 remake. A lot of people were very vocal about this omission, but I don’t consider it critical: Portable always made it very clear that the female protagonist route was an alternative one. A new perspective, if you will. And who knows, maybe ATLUS will find the strength and courage to take our money again in the form of DLC for new perspectives.

Remakes are always a contentious topic; if it’s okay to replace part of the history with the new and shiny. However, I do think that Persona 3 Reload is a product of exquisite quality and that it’s definitely a great version of Persona 3 and more accessible for fans both new and old who’ve gotten used to the new way of things. However, that is what I think it should be considered as: a new and different version of Persona 3, and not as the new Persona 3 to replace the old. A great way to experience the original story, even if wrapped in a remixed experience. It’s not like hordes of people were going to dust off their PlayStation 2 to experience Persona 3, especially when the games are often going for what they retailed for originally – or possibly even more.

As an old fan of the series, I do highly recommend Persona 3 Reload. Whatever annoyance I felt before for the increased 70€ price tag has been replaced by the sheer joy that I got from my 100-hour journey into an old friend. I also imagine that people completely unfamiliar with Persona 3 will also enjoy it, as it is definitely a great game.

PS: Don't pay extra for the digital soundtrack. You'll hate yourself if you do. Just wait for a real soundtrack release.

Ruby One-Liners Guide book announcement

learnbyexample — Mon, 26 Feb 2024 02:00:00 GMT

Hello!

I am pleased to announce a new version of my Ruby One-Liners Guide ebook. Examples, exercises, solutions, descriptions and external links were added/updated/corrected.

When it comes to command line text processing, there are several well known tools like grep for filtering, sed for substitution and awk for field processing. Compared to such tools, Ruby has a feature rich regular expression engine, plenty of builtin modules and a thriving ecosystem. Another advantage is that Ruby is more portable.

This ebook will show examples for filtering and substitution features, field processing, using standard and third-party modules, multiple file processing, how to construct solutions that depend on multiple records, how to compare records and fields between two or more files, how to identify duplicates while maintaining input order and so on.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of Ruby One-Liners Guide for FREE till 29-February-2024. You can still pay if you wish ;)

Ruby Text Processing bundle is free as well:

So is the Magical one-liners bundle:

What's new?🔗

Command version updated to Ruby 3.3.0
Added more exercises
Long sections split into smaller ones
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to Ruby One-Liners Guide
New cover image

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Testimonials🔗

This Ruby one-liners cookbook is incredible. Pretty mind boggling all the stuff you can do.

— feedback on twitter

Table of Contents🔗

Preface
One-liner introduction
Line processing
Field separators
Record separators
Multiple file input
Processing multiple records
Two file processing
Dealing with duplicates
Processing structured data

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/learn_ruby_oneliners/.

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_ruby_oneliners for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_ruby_oneliners/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

LLMs on Hitler vs Elon

Blogity blog blog. Journal — Mon, 26 Feb 2024 01:10:00 GMT

So the question comes up via Twitter: who negatively impacted society more, elon tweeting memes or Hitler. When posted to Google’s Gemini LLM, it gets a hilaribad response back.

Gemini: Twitter ChatGPT: chat.openai.com Llama2-uncensored: Grok: Twitter and Grok on fun mode: Twitter

Marginalia: 3 Years

Weblog on marginalia.nu — Sun, 25 Feb 2024 02:00:00 GMT

It’s been three years since the inception of Marginalia Search, then a dinky experiment to find where the heck the cool Internet has gone, now my full time job.

While there’s always things that can be improved, it’s fair to say the search engine has never worked as well as it does right now.

A great number of milestones have been reached, perhaps biggest of all the search engine has moved out of my living room and into a proper enterprise server.

Introductory investigations into the stability of stock price volatility

Bytepawn - Marton Trencseni — Sun, 25 Feb 2024 01:00:00 GMT

I investigate the volatility of some securities, and its stability over time.

2024–02–24: Some upstreaming and Pinephone Pro modem audio

megi's PinePhone Development Log — Sat, 24 Feb 2024 01:00:00 GMT

rabbot.love

Kartik Agaram — Tue, 20 Feb 2024 09:00:00 GMT

rabbot.love is a little helper I whipped up to check the programs the kids were writing for a neat little paper computer.

video; 25 seconds

2024–02–17: New U-Boot build for Pinephone Pro

megi's PinePhone Development Log — Sat, 17 Feb 2024 01:00:00 GMT

KDA Tools

jodavaho.io — Thu, 15 Feb 2024 02:00:00 GMT

I play hunt showdown a lot. It’s very fun. It’s also insanely frustrating sometimes. The game has long matches, very frantic, quick battles, and a wide variety of meaningful character specialization and equipment options. It can take dozens of matches to determine if a loadout is worth it, and there are many loadouts, and a match takes an hour … In short, it is very hard to get feedback on what equipment loadouts, tactics, or friends are useful.

Furthermore, it’s hard to know if you’re doing well lately or doing better with this loadout or just play worse with your friend Justin. Luckily, I was once paid to solve these problems for real, so let’s solve them for fun.

This is the first of likely a bunch of posts on: designing a logging system (pen and paper), defining a math problem that will help us answer general questions, then writing some Rust to solve it. (Rust is not important here, it’s just what I play with when I’m making CLI tools). This is also my first longer-form post, and my first real entry in a series, so I’m going to have to come back and re-edit this until I get my writing habits in order. Finally, I am way behind on my actual writing, so this became a procrastination mechanism. But for now, caveat emptor and enjoy!

Skip to:

or just go see the final product

But first, if you appreciate this kind of content or would like to see me take on something else, let me know! You can reach me at hello@jodavaho.io.

Step 1 Get the data

The first step in any statistical analysis program is data gathering! Unfortunately, Hunt Showdown does not support a vibrant modding community, and uses encrypted telemetry, and so rather than get super technogeek here, we’ll just write down a few key stats for each game. This is ok, there are not many, and they are all available on the summary screen at the end of a match.

What we want to be able to do is say “Every time this happened, what seemed to happen a lot as well?”. For example “Every time I carried this gun, did I get more kills?” Or, “When I play with Justin, do we kill more bosses?”

So you have to actually write down everything you want to measure.

I kept a log on paper, writing down the date, the match number and the following things:

Kills
Deaths
Assists
Bosses killed
Bounties extracted
Main weapon (what weapon did I bring in)
Secondary weapon (what secondary)

I’d be suprised if it Itook me 30s to scribble this down. Later I kept a text editor open, alt-tabbed, and recorded one match per line as:

2021-10-11 k:3 d:1 b:2 a
2021-10-11 d

You can see a clear date, three kills, 1 death, two bounties extracted, and an assist. What a game! Then, of course, an immediate wipe. Hunt giveth, hunt taketh away.

By the way, this format is something I call “key value counts”¹. You can define any format you want, as long as you write a parser.

We should also record “loadouts”. I did often recorded actual weapons, or just “types”. For simplicity let’s assume I named a few loadouts that worked well together, like sniper, shotgun, rifle, carbine etc. So the log should look like:

2021-10-11 k:3 d:1 b:2 a sniper
2021-10-11 d shotgun

Later, I got curious about some additional “meta stats”. You can record these too! You’ll be able to correlate any variable later.

Ambushes (how many times did we set up a fight on an unaware opponent because we knew where they were going)
Raids (how many times did we rush an established defender, even if they were under attack already)
Defends (how many times did we fight off attackers while defending a bounty)
Wipes (how many times did we wipe another team)
Loss (how many times were we wiped)
Who I played with

A hunt-showdown actuary, maybe analyzing raid data? (credit: Midjourney)

Step 2: Form the problem

This is a classic treatment effect calculation. We want to know if a treatment (“Carry this gun instead of any other”) has an effect (“KDA rises or more bosses killed”).

So how do we model this? The punchline is I chose to assume that kills, deaths, bounties, etc, accumulated over time, were Poisson Processes².

That is, we assume that in the K-D-A-B space of all games, the number of kills, deaths, assists, and bounties per game, were essentially Poisson Random Variables³. The KDAB values are assumed independent of eachother, but not independnet of other variables like loadout.

Poisson Random Variables

Poisson Random Variables have a single parameter of interest, $\lambda$. That $\lambda$ is basically the expected number of times the “event” (like kills) should occur each “round”. Given a $\lambda$ you can calculate a distribution of possible event counts, each with a likelihood, for a given round. That just means we can plot how likely 3 kills was given we have an average number of kills per round of 2. We can also calculate how likeley getting more than 3 kills was, given we usually get 2.

If you average two kills per game, here’s the distribution of kills you’d usually see:

Poisson Distribution: Probability of seeing x events, given you usually see $\lambda=3$ events per round

Poisson Distribution: Probability of seeing x events, given you average $\lambda=1.2$ events per round

How can we use that? Well, we can slice the data! Suppose you wanted to compare kills when you use sniper. You can calculate the value of $\lambda$ given all the games with sniper ($\lambda_\text{sniper}$), versus the $\lambda$ for all games overall, and then ask “are these equal”? Since the best estimate of $\lambda$ for a given distribution is just the average rate observed, we have …

$$ \begin{aligned} \lambda_\text{sniper} &= \frac{\text{sum of all kills during sniper matches}}{\text{number of sniper matches}} \\\ \lambda_\text{all} &= \frac{\text{sum of all kills during all }}{\text{number of matches}} \\\ \end{aligned} $$

We’d like to test

$$ \begin{equation} \lambda_\text{sniper} \stackrel{?}{>} \lambda_\text{all} \tag{1} \label{test} \end{equation} $$

You could just compare them. Of course, this is not enough to say whether they are unequal⁴. Random variables are usually unequal by virtue of being random. What we want to know is, are they unequal more than would be expected? That is, “do we believe the choice of loadout was a differentiating factor on K, D, A, or B values”.

Poisson Statistical Tests

Let’s recall some statistics. We can transform Eq \eqref{test} to

$$ \begin{equation} \frac{\lambda_\text{sniper}}{ \lambda_\text{all} } \stackrel{?}{>} 1 \tag{2} \label{ratiotest} \end{equation} $$

This could be a “simple-simple likelihood ratio test”⁵, meaning that the distribution is known for both cases, and we want to know if the $\lambda$ parameter of the distributions are the same or not. We assume they are, and want to see if we have sufficient difference to determine that was a bad assumption.

It is extremely common to do the following, given a test like Eq \eqref{ratiotest}.

Assume they are equal (defined as the “null hypothesis”)
Contort the equations until they are combined into a ’test statistic'
Show that your ’test statistic’ has a nice distribution, which you can analytically bound by a value $p$.
Calculate when your ’test statistic’ is ‘out of bounds’, i.e., your “$p$ value” is too small, meaning you’ve observed something likely unexplained by the null hypothesis.

While the definition given in the prior paragraph appeared in 1933, it is suprisingly hard to find well-explained test statistics in literature for this type of test.⁶. But, some suprisingly recent results were helpful. Following [Gu2008] and [Ng2005] , we find a bunch of ways of doing steps 1-4 for Poisson ratios.⁷, and I just chose Section 2.4 in Gu’s paper, which is a bit of lovely prose that looks like:

lovely…

My absolutely favorite sub-subject in statistics is Experiment Design. I wrote a whole thesis on it. Experiment design is concerned with the question: “How many measurements should I make to ensure that the statistical test will pass if there is an effect?” This is of critical importance for budgetting trials, but also for bounding the time required to track targets using robots.

For this project, I will eventually sit down an add code that will tell you how many more games you should play to determine if the loadout is useful. [Ng2005] has these values in closed form for some test statistics we could be using here.

But for now, time to write some code.

A hunter unlocking the magic of mathematics. Credit: Midjourney

Step 3 Make the program

I am using the rust programming language. Rust is great. It’s like C, which I love, but also comes with a ton of modern niceties. I hate most of what people make for Rust (i.e., bloated libraries), but that’s not the language’s fault.

As much as I’d love to walk through the detailed programming, it’s just too much for one post. But I can present the high level steps and libraries used, and hope to write more on this later. The code is available online⁸, and is easily installed with cargo install kda-tools or by installing using dpkg as noted in docs.

First, we want code to read the game journal, above. If you don’t have one, here’s a sample journal from my actual play history.

2021-03-12 BAR+Scope pistol K K B alone
2021-03-12 BAR+Scope pistol K D D jb
2021-03-12 Short-Rifle Short-Shotgun K D jb
2021-03-12 BAR+Scope pistol D jp+jb
2021-03-13 BAR+Scope pistol jp D
2021-03-13 BAR+Scope pistol jp B D D A A
2021-03-13 Shotgun pistol jp D
2021-03-13 BAR+Scope pistol jp K
2021-03-14 Short-Rifle akimbo  alone
2021-03-17 LAR Sil pistol  alone
2021-03-17 pistol-stock akimbo  alone
2021-03-17 Short-Shotgun pistol-stock  alone

As mentioned, this is KVC⁹ format. So cargo add kvc. The documentation¹⁰ contains a simple function read_kvc_line_default which returns a vector of keys and counts as tuples. So you only need to read each line, parse, and we have our data!

To conduct a test, you want to go over the data, add up all rows with the factor of interest, and then using all rows as baseline. Then, calculate $\lambda_\text{h1}$ for the interesting ones, and calculate $\lambda_\text{h0}$ as the “null” hypothesis using all data, and plug into the above equation. Let’s do that for the practice data.¹¹

The “baseline” kills / match for the practice data above is a total of 5 Kills (sumH0=5) in 12 matches (nH0=12), for a value of $\lambda_\text{h0}=5/12$.

Suppose we want to test “BAR+Scope” gives more kills than usual.

Partitioning the data, we see:

2021-03-12 BAR+Scope pistol K K B alone
2021-03-12 BAR+Scope pistol K D D jb
2021-03-12 BAR+Scope pistol D jp+jb
2021-03-13 BAR+Scope pistol jp D
2021-03-13 BAR+Scope pistol jp B D D A A
2021-03-13 BAR+Scope pistol jp K

For a total of 4 kills (sumH1=4) in 6 matches (nH1=6). ($\lambda=2/3$)

We now need to plug these into the equations above. Luckily, there’s also code for that¹², which provides the function two_tailed_rates_equal, the example¹³ is straightforward.

let p = two_tailed_rates_equal(sumH1, nH1, sumH0, nH0);
//or
let p = two_tailed_rates_equal(4, 6, 5, 12);

In this case we get a conclusive answer, which is p=0.05! This tells use the probability of observing $\lambda=4/6$ given my usual playing record of 5 kills every 12 matches is very low. Meaning, something had a measureable effect!

The code is relatively straightfoward, but only if you closely follow the paper¹⁴

You can reproduce these steps with:

$ cat test | kda-explore 'K: BAR+Scope'
Processed. Read: 12 rows and 18 variables
Varibables found:
Date K alone pistol BAR+Scope B D jb Short-Rifle Short-Shotgun jp+jb jp A Shotgun akimbo LAR Sil pistol-stock 
Debug: processing: K: BAR+Scope
met    grp        n     M     rate  ~n    ~M    ~rate p     notes
K      BAR+Scope  4     6     0.67  1     6     0.17  0.05

Notice how we had to tell it K: BAR+Scope which means “Test kills / match using BAR+Scope vs normal performance”.

Now, there’s no reason you would necessarily have to prompt the user for “What would you like to know”. Given we just have to add up some variables and plug into an equation, we can do this really fast so why not just test everything?

That’s what I ended up doing. The program kda-explore will output the following on this journal:

$ cat journal | kda-explore
Processed. Read: 12 rows and 18 variables
Varibables found:
Date B alone K BAR+Scope pistol jb D Short-Rifle Short-Shotgun jp+jb jp A Shotgun akimbo LAR Sil pistol-stock 
Debug: processing: K D A : all
No matches found without grouping 'K', this test is useless. Skipping!
met    grp            n     M     rate  ~n    ~M    ~rate p     notes
K      BAR+Scope      4     6     0.67  1     6     0.17  0.05
K      jb             2     2     1.00  3     10    0.30  0.08
K      B              2     2     1.00  3     10    0.30  0.08
K      pistol         4     8     0.50  1     4     0.25  0.35
K      Short-Shotgun  1     2     0.50  4     10    0.40  0.78
K      Short-Rifle    1     2     0.50  4     10    0.40  0.78
K      alone          2     5     0.40  3     7     0.43  0.91
K      pistol-stock   0     2     0.00  5     10    0.50  0.74
K      akimbo         0     2     0.00  5     10    0.50  0.74
K      Sil            0     1     0.00  5     11    0.45  0.73
K      LAR            0     1     0.00  5     11    0.45  0.73
K      Shotgun        0     1     0.00  5     11    0.45  0.73
K      A              0     1     0.00  5     11    0.45  0.73
K      jp+jb          0     1     0.00  5     11    0.45  0.73
K      D              2     6     0.33  3     6     0.50  0.53
K      jp             1     4     0.25  4     8     0.50  0.35
No matches found without grouping 'D', this test is useless. Skipping!
met    grp            n     M     rate  ~n    ~M    ~rate p     notes
D      BAR+Scope      6     6     1.00  2     6     0.33  0.04
D      jb             3     2     1.50  5     10    0.50  0.05
D      A              2     1     2.00  6     11    0.55  0.05
D      pistol         7     8     0.88  1     4     0.25  0.05
D      jp             4     4     1.00  4     8     0.50  0.17
D      B              2     2     1.00  6     10    0.60  0.40
D      Shotgun        1     1     1.00  7     11    0.64  0.57
D      jp+jb          1     1     1.00  7     11    0.64  0.57
D      K              3     4     0.75  5     8     0.62  0.73
D      Sil            0     1     0.00  8     11    0.73  0.97
D      LAR            0     1     0.00  8     11    0.73  0.97
D      Short-Shotgun  1     2     0.50  7     10    0.70  0.64
D      Short-Rifle    1     2     0.50  7     10    0.70  0.64
D      pistol-stock   0     2     0.00  8     10    0.80  0.40
D      akimbo         0     2     0.00  8     10    0.80  0.40
D      alone          0     5     0.00  8     7     1.14  0.01
No matches found without grouping 'B', this test is useless. Skipping!
No matches found without grouping 'BAR+Scope', this test is useless. Skipping!
No matches found without grouping 'pistol', this test is useless. Skipping!
No matches found without grouping 'D', this test is useless. Skipping!
No matches found without grouping 'jp', this test is useless. Skipping!
No matches found without grouping 'A', this test is useless. Skipping!
met    grp            n     M     rate  ~n    ~M    ~rate p     notes
A      K              0     4     0.00  2     8     0.25  0.74
A      pistol-stock   0     2     0.00  2     10    0.20  0.66
A      akimbo         0     2     0.00  2     10    0.20  0.66
A      Short-Shotgun  0     2     0.00  2     10    0.20  0.66
A      Short-Rifle    0     2     0.00  2     10    0.20  0.66
A      jb             0     2     0.00  2     10    0.20  0.66
A      alone          0     5     0.00  2     7     0.29  0.48
A      Sil            0     1     0.00  2     11    0.18  0.33
A      LAR            0     1     0.00  2     11    0.18  0.33
A      Shotgun        0     1     0.00  2     11    0.18  0.33
A      jp+jb          0     1     0.00  2     11    0.18  0.33

You can see it complains “This test is useless”, when it cannot find that an event had occured. This is a corner case in how the test statistic is calculated. To get around that, and allow arbitrary tests, we can use other more modern (e.g., computationally painful) methods. That’s another one for next time.

Final Product

Let’s recap. First, you need to keep a journal of matches. Then, you can either calculate the $\lambda$ values from that journal, or write some tools to do it for you. If you’d like to use my tools, they are here: (see github.com/jodavaho/kda-tools )

kda-summary will summarize your K, D, and A values (and the usual KDA metric) over the entire journal.
kda-compare will run will look at the whole dataset and tell you if you’re doing significantly differently with different loadouts.
kda-explore will allow you to look at the conditional distributions of anything with and without anything else.

Get for debian / WSL

If you have cargo (apt install cargo), then cargo install kda-tools.

Otherwise, just grab one of the test debs in releases/ For example 1.3.0

Then, in bash/cli or wsl,

sudo dpkg -i kda-tools_1.3.0_amd64.deb

Conclusion

I’ve talked about how we can model Hunt Showdown statistics as Poisson Random Variables, how we can formulate a test to see how well we do with certain loadouts, and provided links to code which allows you to automated those tests.

I will, in the future, write up more on these tools, including discussing how non-analytical results are possible, how we can measure more than just Kills or Bounties (e.g., KDA), and how we can automate the whole thing. I’ll also dive into the code more, to provide examples of how it works.

Next, we’ll go over a more powerful tool kda-compare, which does automated testing of more than just things you can count. For example, it’ll measure “KDA” (i.e., $\frac{K+A}{D}$) and “Bounties per death” $\frac{B}{D}$, which existing analytical methods cannot do (to the best of my knowledge).

$ <journal.txt kda-compare
Processed. Read: 5 rows and 8 variables

[====================================================] 100.00 % 2696.14/s 
met    grp      n/d      val   N     n/d      ~val  M     p
kda    Sniper   5/1      5.00   3    2/2      1.00   2    0.06
kda    JP       2/0      inf    1    5/3      1.67   4    0.49
kda    Shotgun  2/2      1.00   2    5/1      5.00   3    0.77
kda    JB       2/2      1.00   2    5/1      5.00   3    0.78
b/d    Sniper   1/1      1.00   3    1/2      0.50   2    0.24
b/d    Shotgun  1/2      0.50   2    1/1      1.00   3    0.69
b/d    JB       2/2      1.00   2    0/1      0.00   3    NaN
b/d    JP       0/0      NaN    1    2/3      0.67   4    NaN

Just a teaser: The first row is met grp ... These are

the metric name (e.g., kda or bounties / death b/d)
item group (grp)
value counts (n)
deaths (d)
the value of the metric ‘val’ with the grp
number of matches where ‘grp’ was used (N)
value counts without the grp (n)
death counts without the grp (d)
the value of the metric ‘val’ without the grp
number of matches without the grp (M)
and the probability that we’d randomly see that ‘val’ given the distribution of the metric without the grp.

References

Ng2005 Ng, Hon Keung Tony, and Man‐Lai Tang. “Testing the equality of two Poisson means using the rate ratio.” Statistics in medicine 24, no. 6 (2005): 955-965.

Ractliffe1964 : Ractliffe, J. F. “The significance of the difference between two Poisson variables: an experimental investigation.” Applied Statistics (1964): 84-86.

Gu2008 : Gu, Kangxia, Hon Keung Tony Ng, Man Lai Tang, and William R. Schucany. “Testing the ratio of two poisson rates.” Biometrical Journal: Journal of Mathematical Methods in Biosciences 50, no. 2 (2008): 283-298.

See github.com/jodavaho/kvc for more information on key-value-counts ↩︎
Poisson Process: https://en.wikipedia.org/wiki/Poisson_point_process ↩︎
Poisson random varaibles: https://en.wikipedia.org/wiki/Poisson_distribution ↩︎
For example, getting 1, 2, 2 kills with shotgun (avg: 5/6) vs 2, 2, 2 kills with snipers (avg: 6/6) is different, but not different enough to care about. ↩︎
Likelihood Ratio Test: https://en.wikipedia.org/wiki/Likelihood-ratio_test# ↩︎
maybe a better statistician could have derived them? ↩︎
I really should have followed [Ng2005] , but live and learn ↩︎
github.com/jodavaho/kda-tools or cargo install kda-tools which pulls crates.io/crates/kda-tools ↩︎
kvc on github.com/jodavaho/kvc ↩︎
kvc on docs.rs ↩︎
The use of $H0$ to represent the null hypothesis and $H1$ to represent the “stuff that may have changed things” is mostly historical. ↩︎
poisson-rate-test crates.io or docs.rs ↩︎
two_tailed_rates_equal documentation on docs.rs ↩︎
See one_tailed code on poisson_rate_test/lib.rs lines 190-250 ↩︎

Blockchain: Earth has a gambling problem

jodavaho.io — Mon, 12 Feb 2024 02:00:00 GMT

All blockchain games are predicated on guessing random numbers in exchange for money. How is this not gambling?

Did you know that two percent of USA’s energy is just bitcoin mining?¹

This is dumb. Blockchain is automated gambling. I don’t mean “I’m gambling the cost of bitcoin will go up, so I buy some” I mean the entire system is:

Guess a number
Plug it into an equation, and see if the output matches a random number generated by another equation. The only difference between this coin or that is how the equations work.
if it matches, you win some ‘coin’! The number you guessed is ‘proof’ you earned the coin. Now you can sell it to someone who thinks this is valuable, and they’ll give you money².

(essentially, at an allegory level).

And for that, we run giant data centers and burn coal in third world countries³ to keep doing trillions of guesses per second to earn a little coin.

“Two percent of the US’s electricity generation now goes to bitcoin” https://arstechnica.com/science/2024/02/over-2-percent-of-the-uss-electricity-generation-now-goes-to-bitcoin/ ↩︎
Hey, it’s cool that “selling” means you change the equations of how the random numbers are generated, but so what? It doesn’t change the fundamental nature of gambling how you record who won. ↩︎
It happens here too: https://www.theverge.com/2024/2/6/24063773/the-epa-just-fined-a-bitcoin-mining-plant-for-violating-coal-ash-rules ↩︎

On Software Engineering

jodavaho.io — Mon, 12 Feb 2024 02:00:00 GMT

Reading

https://quiltro.org/notes/simplicity.html
mastering programming https://tidyfirst.substack.com/p/mastering-programming
https://matthewrocklin.com/index.html

Contribute

Writing

http://graphics.stanford.edu/~seander/bithacks.html Bithacks!

Persona 4 Golden review

ブラック — Mon, 12 Feb 2024 02:00:00 GMT

Everyone’s favourite Scooby-Doo game on the PlayStation Vita.

Y(o)u're a Japanese youth, kicked from the big city into the boonies, when shit kicks off and people start dying mysteriously in an otherwise quiet small town – and the only one who can reach out to the truth is you and your merry gang of superpowered high-school students.

A role-playing game lives or dies by its story and thankfully the story in Persona 4 Golden is good. Not maybe the most clever or through-provoking one, especially for supposedly being a murder mystery, but still a good experience. Granted, I knew the general gist of the story since I’ve played through Persona 4 on the PlayStation 2 over a decade ago. But despite knowing what generally will happen, it was still a fun time reading through. And since it’s been a while since I last read through it, and I’d already forgotten some parts of the Persona 4 story, I could experience some things with “fresh” eyes, which was also nice. I imagine any players without any knowledge about Persona 4 would enjoy it even further.

The chemistry between the character in the story is also good. By the end of the game, it really feels like you’ve made unbreakable bonds with your social links. I also enjoyed the humorous slice of life events that didn’t carry the story forwards. It’s just a fairly good time in general.

Most of the game is spent living the life of the most popular and proactive student in Japan by improving yourself and socialising with your friends and neighbours. These day-to-day activities can be quite overwhelming, as you might have half a dozen citizens of Inaba vying for your attention at any given moment. It does very much feel like there is not enough time to get everything done within a single playthrough, and I did actually fail to fully nurture two social links in a single playthrough. However, it’s also somehow so incredibly addicting for reasons completely oblivious to me. At times I found it hard to stop playing even though I really had to.

This addictive gameplay also extends to the actual dungeon crawling part of the game: it’s really fun and at times it was hard to put down my Steam Deck to go do other things. Maybe the only thing I have against it after Persona 5 is that the battle interface feels clunky and outdated. Not to a point where it would’ve stopped me from playing, but definitely had some more friction in the beginning. I also know that some people do not like the dungeons in Persona 4, but I actually didn’t have a problem with them. Most of them have pretty nice designs and soundtracks, so I can’t hate them, and since they’re randomly generated, you can’t just memorise and blast through them.

I can’t really remember exactly how all of the aspects in the original Persona 4 worked, but Golden does feel a lot more polished than the original. Easier too, which is seemingly a thing that always happens when they re-release any of the Persona games. However, the mechanics still feel quite stiff to all of the modernisations they stuck into Persona 5, so I definitely recommend people interested in the Persona series to start off with Persona 4 Golden instead of Persona 5 or Persona 5 Royal. The transition from P4G to P5R is definitely smoother than going in reverse.

Not completely sure about the third semester / epilogue though. It seems kinda of an unnecessary addition, and mainly revolves around a character who seems quite forcibly added to an existing and complete story and one that you’ll probably struggle to care about. That being said, it’s not exactly long to get through, so it kinda slipped past me anyway.

Unfortunately, Persona 4 Golden is showing its age in the graphics department though. We’re talking about a PlayStation 2 game that was polished up for the portable PlayStation Vita and is then rendered at a slightly higher resolution on the PC. That being said, it’s still absolutely fine. The art style does make up for a lot of the low fidelity and the 2D sprites are ageless. Still, the 15-year patina is another reason I’d recommend people to start off with Persona 4 Golden before moving onto flashier titles like Persona 5 Royal or Persona 3 Reload. It’s not like Persona games have any kind of interconnected stories, so you can freely choose what order you play them in.

Thankfully the soundtrack still absolutely holds up. Never has there been a Persona game with a bad soundtrack and you are definitely going to be listening to the soundtrack outside of the game too. Guaranteed.

And as you might expect from a PlayStation Vita title, this game feels tailor-made for the Steam Deck. The gameplay really feels at home on a platform where you can suspend and resume the game as you please. The only thing that gives off the feeling that it’s not a native title for the device are these small freezes that occur here and there. I’m talking like a half second freeze for half an hour of gameplay. Small hiccups, not exactly something to worry about. Otherwise the performance is rock-solid, even if I did often limit the game to 60 FPS instead of the possible 90 FPS (which was smooth and solid) for some battery life improvements. There’s even proper Steam Deck button graphics in the game!

Actually, the game looks downright silly on a 27-inch monitor, and really shows how this game was designed as a portable title. All of the user interface elements look XXL-sized and the low-fidelity assets are really laid out bare. I also can’t get the vibrant colours and inky blacks on my PC monitor as I could with my OLED Steam Deck screen, which made it feel like I was playing it on a supersized PlayStation Vita. Granted, I do have a pretty awful PC monitor, but so do a lot of people. I’d maybe even go as far as to recommend this game a little bit less if you cannot play it on a Steam Deck. Not that a Steam Deck is absolutely necessary – but I’d argue that it’s the best way to experience this game. Persona 4 Golden was a PlayStation Vita must-have, and now it’s a Steam Deck must-have.

I must also confession something: even though I had Persona 4 Golden on my Vita for the longest time, I never actually managed to finish it there. Kind of a PlayStation Vita sin, not finishing one of its greatest hits. I guess I was saving it for when I needed a portable time-killer and never actually got to the end despite starting the adventure. But maybe that was for the best – since I think that the ultimate way of enjoying this game is not on the poor old PlayStation Vita, but rather on the Steam Deck.

Persona 4 Golden is, despite its obvious veteran age, still a fantastic and fun game and it should probably be the first Persona game for newcomers for the series. An absolute must-have gem of a game.

A grab bag of articles I saved

jodavaho.io — Sun, 11 Feb 2024 02:00:00 GMT

Computer Science Blogs

Quality sites

Humanity

Why you, personally, should want a larger population

link: https://rootsofprogress.org/why-a-larger-population
description: Various reasons for above, including faster progress, more opportunities or options, deeper patterns (agglomeration effects)

Learning

Mathematics for the adventurous self-learner

link: https://www.neilwithdata.com/mathematics-self-learner
A set of books to read / work through to learn math

Meta-skills

All strength comes from repetition (of the basics)

link: https://dariusforoux.com/repetition/
The biggest mistake you can make is to ignore the basics in your profession.

“Long-term, we must begin to build our internal strengths. It isn’t just skills like computer technology. It’s the old-fashioned basics of self-reliance, self-motivation, self-reinforcement, self-discipline, self-command.” Steven Pressfield

Ten Simple Rules for Lifelong Learning, According to Hamming

link: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004020
docid: https://doi.org/10.1371/journal.pcbi.1004020
Rule 1. Cultivate Lifelong Learning as a “Style of Thinking” That Concentrates on Fundamental Principles Rather Than on Facts
Rule 2. Structure Your Learning to Ride the Information Tsunami Rather Than Drown in It
Rule 3. Be Prepared to Compete and Interact with a Greater and More Rapidly Increasing Number of Scientists Than at Any Time in the Past
Rule 4. Focus on the Future but Don’t Ignore the Past
Rule 5. Look for the Personal Angle “In that vein, we may infuse our learning with personal details, which some scientists publish.”
Rule 6. Learn from the Successes of Others
Rule 7. Use Trial and Error to Find the Style of Learning That Suits You
Rule 8. No Matter How Much Advice You Get and How Much Talent You Possess, It Is Still You Who Must Do the Learning and Put in the Time
Rule 9. Have a Vision to Give You a General Direction
Rule 10. Make Your Life Count: Struggle for Excellence

As Hamming says, because “there are so many ways of being wrong and so few of being right, studying successes is more efficient, and furthermore, when your turn comes you will know how to succeed rather than how to fail.” In addition, he notes that “vicarious learning from the experiences of others saves making errors yourself” - Hamming RW (1997) The Art of Doing Science and Engineering—Learning to Learn. Amsterdam: Gordon and Breach Science Publishers. pmid:25165803

An ongoing list of business, leadership, or management resources

jodavaho.io — Sun, 11 Feb 2024 02:00:00 GMT

Leadership and Teams

Turn the Ship Around homepage, amazon
Peopleware: Productive Projects and Teams amazon
- Related: “Let me tell you a secret” it’s the people
Five Dysfunctions of a Team summary, amazon
High output management amazon
Army Leadership Manual pdf, cached local copy
No Bullshit CTO /Tech lead guide web
How to start google web

Command line heaven

jodavaho.io — Sun, 11 Feb 2024 02:00:00 GMT

Tools

asciinema: Record and share your terminal sessions, the right way.

Web services

wttr.in: The right way to check the weather from your terminal.
qrenco.de: Generate QR codes from the command line.
cheat.sh: The right way to get help from the command line.
rate.sx: The right way to get currency exchange rates from the terminal.

Conferences, Journals, Etc

jodavaho.io — Sun, 11 Feb 2024 02:00:00 GMT

Multi-agent systems

AAMAS proceedings

Planning & Scheduling

ICAPS conf page, proceedings

Robotics

ICRA conf page
IROS conf page

MASSpace ‘24 conf page

Linux & OSS

Linux Foundation events
Open Source Summit events
Scale homepage
FOSSDEM homepage
DEFCON homepage

Learning

RustConf homepage
CppCon homepage
PyCon homepage

2024–02–11: Small tidbits and magnetometer driver upstreaming

megi's PinePhone Development Log — Sun, 11 Feb 2024 01:00:00 GMT

2024 Data Outlook

Bytepawn - Marton Trencseni — Sun, 11 Feb 2024 01:00:00 GMT

It is the beginning of the year — a good time to reflect on the previous year and make plans for the year ahead. I wrote this document for my team members in 2024 January to kick off the year. This is an abridged version with sensitive content removed.

Falsehoods I used to believe about shoes

Steve Kemp's Blog — Sat, 10 Feb 2024 15:00:00 GMT

Once upon a time I used to divide shoes, boots, and other footwear, into two categories:

Things that had good soles and a good grip.
Things that were slippery and treacherous.

Nowadays I know better. When walking on snow, ice, or slush it isn't the sole of the shoe that causes you to slip, slide, and fall. It is you.

The shoes do make a difference, which is why people use icebugs (i.e. shows with little nails sticking out) and similar things. But really it's all about walking "properly":

One leg directly below you.
Put your feet down flat.
Shorter strides.
Horizontal movement is dangerous.

TLDR; be like a penguin.

Also - I seem to no longer appear on "Planet Debian". Weird. A problem for another day.

The Computer Architecture of AI (in 2024)

Speculative Branches — Sat, 10 Feb 2024 02:00:00 GMT

Over the last year, as a person with a hardware background, I have heard a lot of complaints about Nvidia's dominance of the machine learning market and whether I can build chips to make the situation better. While the amount of money I would expect it to take is less than $7 trillion, hardware accelerating this wave of AI will be a very tough problem--much tougher than the last wave focused on CNNs--and there is a good reason that Nvidia has become the leader in this field with few competitors. While the inference of CNNs used to be a math problem, the inference of large language models has actually become a computer architecture problem involving figuring out how to coordinate memory and I/O with compute to get the best performance out of the system.

An intuition for distributed consensus in OLTP systems

Notes on software development — Thu, 08 Feb 2024 02:00:00 GMT

Distributed consensus in transactional databases (e.g. etcd or Cockroach) is a big deal these days. Most often under the hood are variations of log-based Paxos-like algorithms such as MultiPaxos, Viewstamped Replication, or Raft. While there are new variations that come out each year, optimizing for various workloads, these algorithms are fairly standard and well-understood.

In fact they are used in so many places, Kubernetes for example, that even if you don't decide to implement Raft (which is fun and I encourage it), it seems worth building an intuition for distributed consensus.

What happens as you tweak a configuration. What happens as the production environment changes. Or what to reach for as product requirements change.

I've been thinking about the basics of distributed consensus recently. There has been a lot to digest and characterize. And I'm only beginning to get an understanding.

This post is an attempt to share some of the intuition built up reading about and working in this space. Originally this post was also going to end with a walkthrough of my most recent Raft implementation in Rust. But I'm going to hold off on that for another time.

I was fortunate to have a few excellent reviewers looking at versions of this post: Paul Nowoczynski, Alex Miller, Jack Vanlightly, Daniel Chia, and Alex Petrov. Thank you!

Let's start with Raft.

Raft

Raft is a distributed consensus algorithm that allows you to build a replicated state machine on top of a replicated log.

A Raft library handles replicating and durably persisting a sequence (or log) of commands to at least a majority of nodes in a cluster. You provide the library a state machine that interprets the replicated commands. From the perspective of the Raft library, commands are just opaque byte strings.

For example, you could build a replicated key-value store out of SET and GET commands that are passed in by a client. You provide a Raft library state machine code that interprets the Raft log of SET and GET commands to modify or read from an in-memory hashtable. You can find concrete examples of exactly this replicated key-value store modeling in previous Raft posts I've written.

All nodes in the cluster run the same Raft code (including the state machine code you provide); communicating among themselves. Nodes elect a semi-permanent leader that accepts all reads and writes from clients. (Again, reads and writes are modeled as commands).

To commit a new command to the cluster, clients send the command to all nodes in the cluster. Only the leader accepts this command, if there is currently a leader. Clients retry until there is a leader that accepts the command.

The leader appends the command to its log and makes sure to replicate all commands in its log to followers in the same order. The leader sends periodic heartbeat messages to all followers to prolong its term as leader. If a follower hasn't heard from the leader within a period of time, it becomes a candidate and requests votes from the cluster.

When a follower is asked to accept a new command from a leader, it checks if its history is up-to-date with the leader. If it is not, the follower rejects the request and asks the leader to send previous commands to bring it up-to-date. It does this ultimately, in the worst case of a follower that has lost all history, by going all the way back to the very first command ever sent.

When a quorum (typically a majority) of nodes has accepted a command, the leader marks the command as committed and applies the command to its own state machine. When followers learn about newly committed commands, they also apply committed commands to their own state machine.

For the most part, these details are graphically summarized in Figure 2 of the Raft paper.

Availability and linearizability

Taking a step back, distributed consensus helps a group of nodes, a cluster, agree on a value. A client of the cluster can treat a value from the cluster as if the value was atomically written to and read from a single thread. This property is called linearizability.

However, with distributed consensus, the client of the cluster has better availability guarantees from the cluster than if the client atomically wrote to or read from a single thread. A single thread that crashes becomes unavailable. But some number f nodes can crash in a cluster implementing distributed consensus and still 1) be available and 2) provide linearizable reads and writes.

That is: distributed consensus solves the problem of high availability for a system while remaining linearizable.

Without distributed consensus you can still achieve high availability. For example, a database might have two read replicas. But a client reading from a read replica might get stale data. Thus, this system (a database with two read replicas) is not linearizable.

Without distributed consensus you can also try synchronous replication. It would be very simple to do. Have a fixed leader and require all nodes to acknowledge before committing, But the value here is extremely limited. If a single node in the cluster goes down the entire cluster is down.

You might think I'm proposing a strawman. We could simply designate a permanent leader that handles all reads and writes; and require a majority of nodes to commit a command before the leader responds to a client. But in that case, what's the process for getting a lagging follower up-to-date? And what happens if it is the leader who goes down?

Well, these are not trivial problems! And, beyond linearizability that we already mentioned, these problems are exactly what distributed consensus solves.

Why does linearizability matter?

It's very nice, and often even critical, to have a highly available system that will never give you stale data. And regardless, it's convenient to have a term for what we might naively think of as the "correct" way you'd always want to set and get a value.

So linearizability is a convenient way of thinking about complex systems, if you can use or build a system that supports it. But it's not the only consistency approach you'll see in the wild.

As you increase the guarantees of your consistency model, you tend to sacrifice performance. Going the opposite direction, some production systems sacrifice consistency to improve performance. For example, you might allow stale reads from any node, reading only from local state and avoiding consensus, so that you can reduce load on a leader and avoid the overhead of consensus.

There are formal definitions for lower consistency models, including sequential and read-your-writes. You can read the Jepsen page for more detail.

Best and worst case scenarios

A distributed system relies on communicating over the network. The worse the network, whether in terms of latency or reliability, the longer it will take for communication to happen.

Aside from the network, disks can misdirect writes or corrupt data. Or you could be mounted on a network filesystem such as EBS.

And processes themselves can crash due to low disk space or the OOM killer.

It will take longer to achieve consensus to commit messages these scenarios. If messages take longer to reach nodes, or if nodes are constantly crashing, followers will timeout more often, triggering leader election. And the leader election itself (which also requires consensus) will also take longer.

The best case scenario for distributed consensus is where the network is reliable and low-latency. Where disks are reliable and fast. And where processes don't often crash.

TigerBeetle has an incredible visual simulator that demonstrates what happens across ever-worsening environments. While TigerBeetle and this simulator use Viewstamped Replication, the demonstrated principles apply to Raft as well.

What happens when you add nodes?

Distributed consensus algorithms make sure that some minimum number of nodes in a cluster agree before continuing. The minimum number is proportional to the total number of nodes in the cluster.

A typical implementation of Raft for example will require 3 nodes in a 5-node cluster to agree before continuing. 4 nodes in a 7-node cluster. And so on.

Recall that the p99 latency for a service is at least as bad as the slowest external request the service must make. As you increase the number of nodes you must talk to in a consensus cluster, you increase the chance of a slow request.

Consider the extreme case of a 101-node cluster requiring 51 nodes to respond before returning to the client. That's 51 chances for a slower request. Compared to 4 chances in a 7-node cluster. The 101-node cluster is certainly more highly available though! It can tolerate 49 nodes going down. The 7-node cluster can only tolerate 3 nodes going down. The scenario where 49 nodes go down (assuming they're in different availability zones) seems pretty unlikely!

Horizontal scaling with distributed consensus? Not exactly

All of this is to say that the most popular algorithms for distributed consensus, on their own, have nothing to do with horizontal scaling.

The way that horizontally scaling databases like Cockroach or Yugabyte or Spanner work is by sharding the data, transparent to the client. Within each shard data is replicated with a dedicated distributed consensus cluster.

So, yes, distributed consensus can be a part of horizontal scaling. But again what distributed consensus primarily solves is high availability via replication while remaining linearizable.

This is not a trivial point to make. etcd, consul, and rqlite are examples of databases that do not do sharding, only replication, via a single Raft cluster that replicates all data for the entire system.

For these databases there is no horizontal scaling. If they support "horizontal scaling", they support this by doing non-linearizable (stale) reads. Writes remain a challenge.

This doesn't mean these databases are bad. They are not. One obvious advantage they have over Cockroach or Spanner is that they are conceptually simpler. Conceptually simpler often equates to easier to operate. That's a big deal.

Optimizations

We've covered the basics of operation, but real-world implementations get more complex.

Snapshots

Rather than letting the log grow indefinitely, most libraries implement snapshotting. The user of the library provides a state machine and also provides a method for serializing the state machine to disk. The Raft library periodically serializes the state machine to disk and truncates the log.

When a follower is so far behind that the leader no longer has a log entry (because it has been truncated), the leader transfers an entire snapshot to the follower. Then once the follower is caught up on snapshots, the leader can transfer normal log entries again.

This technique is described in the Raft paper. While it isn't necessary for Raft to work, it's so important that it is hardly an optimization and more a required part of a production Raft system.

Batching

Rather than limiting clients of the cluster to submitting only one command at a time, it's common for the cluster to accept many commands at a time. Similarly, many commands at a time are submitted to followers. When any node needs to write commands to disk, it can batch commands to disk as well.

But you can go a step beyond this in a way that is completely opaque to the Raft library. Each opaque command the client submits can also contain a batch of messages. In this scenario, only the user-provided state machine needs to be aware that each command it receives is actually a batch of messages that it should pull apart and interpret separately.

This latter techinque is a fairly trivial way to increase throughput by an order of magnitude or two.

Disk and network

In terms of how data is stored on disk and how data is sent over the network there is obvious room for optimization.

A naive implementation might store JSON on disk and send JSON over the network. A slightly more optimized implementation might store binary data on disk and send binary data over the network.

Similarly you can swap out your RPC for gRPC or introduce zlib for compression to network or disk.

You can swap out synchronous IO for libaio or io_uring or SPDK/DPDK.

A little tweak I made in my latest Raft implementation was to index log entries so searching the log was not a linear operation. Another little tweak was to introduce a page cache to eliminate unnecessary disk reads. This increased throughput for by an order of magnitude.

Flexible quorums

This brilliant optimization by Heidi Howard and co. shows you can relax the quorum required for committing new commands so long as you increase the quorum required for electing a leader.

In an environment where leader election doesn't happen often, flexible quorums can increase throughput and decrease latency. And it's a pretty easy change to make!

These are just a couple common optimizations. You can also read about parallel state machine apply, parallel append to disk, witnesses, compartmentalization, and leader leases. TiKV, Scylla, RedPanda, and Cockroach tend to have public material talking about this stuff.

There are also a few people I follow who are often reviewing relevant papers, if they are not producing their own. I encourage you to follow them too if this is interesting to you:

Safety and testing

The other aspect to consider is safety. For example, checksums for everything written to disk and passed over the network; or being able to recover from corruption in the log.

Testing is also a big deal. There are prominent tools like Jepsen that check for consistency in the face of fault injection (process failure, network failure, etc.). But even Jepsen has its limits. For example, it doesn't test disk failure.

FoundationDB made popular a number of testing techniques. And the people behind this testing went on to build a product, Antithesis, around deterministic testing of non-deterministic code while injecting faults.

And on that topic there's Facebook Experimental's Hermit deterministic Linux hypervisor that may help to test complex distributed systems. However, my experience with it has not been great and the maintainers do not seem very engaged with other people who have reported bugs. I'm hopeful for it but we'll see.

Antithesis and Hermit seem like a boon when half the trouble of working on distributed consensus implementations is avoiding flakey tests.

Another promising avenue is emitting logs during the Raft lifecycle and validating the logs against a TLA+ spec. Microsoft has such a project that has begun to see adoption among open-source Raft implementations.

Conclusion

Everything aside, consensus is expensive. There is overhead to the entire consensus process. So if you do not need this level of availability and can settle for some process via backups, it's going to have lower latency and higher throughput than if it had to go through distributed consensus.

If you do need high availability, distributed consensus can be a great choice. But consider the environment and what you want from your consensus algorithm.

Also, while MultiPaxos, Raft, and Viewstamped Replication are some of the most popular algorithms for distributed consensus, there is a world beyond. Two-phase commit, ZooKeeper Atomic Broadcast, PigPaxos, EPaxos, Accord by Cassandra. The world of distributed consensus also gets especially weird and interesting outside of OLTP systems.

But that's enough for one post.

Best SEO spam 2024 reddit

Weblog on marginalia.nu — Wed, 07 Feb 2024 02:00:00 GMT

One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time.

I’ve been noticing the search engine spam strategy of adding ‘reddit’ to page titles for a few years now, but it feels like it’s been growing a lot recently. I don’t think it’s actually working, but it’s so cute that they are trying.

Conscientiousness is (almost) all you need

jodavaho.io — Mon, 05 Feb 2024 02:00:00 GMT

Why do GPA and ACT scores matter?

Recently, there’s been some pushback against standardized tests and college entrance exams. It begs the question: What is all this testing good for? After all, well-to-do families and students can spend more time preparing for these exams, making them an unfair indicator of wealth moreso than IQ.

I propose: They are not a measure of ability but of rule-following, self-regulation, and preparadness. If this is true, the fact that you can study for these tests isn’t a bug, but a feature. If we were measuring IQ or “raw” ability for some kind of meritocratic system, we’d want a test you could not study for.

So why measure someone’s ability (and willingness) to study? Becasue then we’re measuring conscientiousness. And, it turns out conscientiousness is (almost) all you need.

Let’s rewind

Undergraduate GPA predicts lifetime earnings¹, incoming test scores and GPA are highly predictive of undergraduate GPA², advanced degrees (which increase earnings), and increased earnings within degrees ³, ⁴. I suggest these effects are because being a good student aka “eating books” is correlated with conscientiousness. They show up to lectures, prepare, and test well. And while test scores may be influenced by socioeconomic status, once that is removed, they are still an excellent predictor of grades².

And conscientiousness is very highly correlated with lifetime achievement, AND fufillment ⁵. So measuring conscientiousness, and signalling high conscientiousness is a really good idea.

IQ is great, but conscientiousness is how you get things done ⁶

Related: creation is our higher purpose

Latent Spoilers

Beetle Space — Sun, 04 Feb 2024 10:00:00 GMT

Why are entertainment UIs so bad?

Here are some examples:

You’re watching a recorded tennis match. At the end of the first set, you hit “Pause” to get a snack. The progress indicator pops up and shows you that you’re about 50% done with the video. You now …

Messing about with serious software

jodavaho.io — Sun, 04 Feb 2024 02:00:00 GMT

Messing about with serious software

In which we discuss setting up some spaceflight software from NASA inside a tiny webserver, inside an AWS lambda, and with Rust.

There is no reason you’d ever want to do this, but I was stuck at home sick and needed to scratch an itch: How does one know if the sun is shining on any particular part of the moon right now?

The solution is very complicated, actually.

Is the spot occluded by, e.g., being in a crater?
How high above the surface are you? Are you right on the surface?
Technically the sun goes out of line of sight about 8 minutes before the light stops hitting the spot - do we take that into account?
Does the atmosphere bend the light of the sun? Making it visible when it isn’t strictly in line of sight?
What does “now” mean? Does it mean that when we look at this spot the light is not hitting it (e.g., the reflection of sunlight from that spot is no longer reaching our eyes).
How do we know where the moon is relative to the sun precisely?
Doesn’t the moon wobble?
Suppose I wanted to plan the arrival of a spacecraft that will do distributed sensing, so now I need to know all this for future dates!

You can imagine that someone wanting to point a telescope at a spot on Mars would want to do similar calculations, lest they take a picture of a shadow-obscured region, or worse, something in darkness ¹.

Lunar terminator (creative commons)

One of the things I most appreciated about working at JPL was being surrounded by smart people who could think of all these details and solve them. So, I was exposed to NAIF SPICE system. SPICE is a C library for calculating observer geometry.

Putting this up into a webservice involved the following steps:

Figuring out if SPICE could solve this problem
Downloading and building the software
Finding and configuring the kernels (~180MB of them!)
Using Rust FFI to interact with the right SPICE apis
Configuring an API Gateway, Lambda, and domain name
Wrapping the rust code in Axum, and deploying

I’m definitely going to write about each of those steps … right after I get some sleep

Actually, taking images of the terminator can yield some interesting science opportunities, so now we really want to know where the sun is! ↩︎

Understanding Ruby Regexp book announcement

learnbyexample — Fri, 02 Feb 2024 02:00:00 GMT

Hello!

I just published a new version of the "Understanding Ruby Regexp" ebook. Corrected examples and descriptions for Atomic grouping, \G and \K features, improved examples, exercises and so on.

This book will help you learn Ruby Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises.

Ebook links🔗

You can download the PDF/EPUB versions of the book for free using the below links (you can also pay if you wish):

You can also read the book online here: https://learnbyexample.github.io/Ruby_Regexp/.

What's new?🔗

Ruby version updated to 3.3.0
Corrected examples and descriptions for Atomic grouping, \G and \K features
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to Understanding Ruby Regexp
New cover image
Images centered for EPUB format

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Table of Contents🔗

Preface
Why is it needed?
Regexp introduction
Anchors
Alternation and Grouping
Escaping metacharacters
Dot metacharacter and Quantifiers
Interlude: Tools for debugging and visualization
Working with matched portions
Character class
Groupings and backreferences
Interlude: Common tasks
Lookarounds
Modifiers
Unicode
Further Reading

GitHub repo🔗

Visit https://github.com/learnbyexample/Ruby_Regexp for markdown source, exercise solutions, sample chapters and other details related to the book.

See my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/Ruby_Regexp/issues
E-mail: learn by example.net@gmail.com (without the spaces)
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Supporting scc and my other free software tools

Ben E. C. Boyter — Wed, 31 Jan 2024 02:00:00 GMT

Sometime last year Justin Vincent of the techzing podcast suggests that I should add a license to Quote,

I “strongly” suggest that you wrap each product into a paid version - where you sell a “license” for usage with priority support.

New year means new energy and so I got around to do this. All the tools will remain free software, so there is no need to worry about a rug pull. However it does mean I am now offering proper email and if required face time support for every license sold for a year from the data of purchase.

Contexts, Friction and Distractions

Weblog on marginalia.nu — Tue, 30 Jan 2024 02:00:00 GMT

I get significantly more work done when I unplug my computer from the Internet. It’s not that my productive output drops to zero when I’m plugged in, but more like 70%.

Despite many of the tools that I use requiring a connection, and certainly the Internet containing a wealth of information that might expedite my work, these benefits are drastically outweighed by the wealth of distractions also available.

It’s very appealing, when the code is compiling or the docker containers restarting, to sneak open a browser tab with hacker news, or the Χ formerly known as Twitter, youtube, mastodon, a news site, or something similar to pass those minutes.

Highfleet Cogitator 1.0 Release

jodavaho.io — Mon, 29 Jan 2024 02:00:00 GMT

As of today, we’re at 1.0 for the Highfleet tools / cogitator site.

https://hfopt.jodavaho.io

We provide:

uploading ships, and getting a permalink for sharing.
creating collections of ships you have uploaded, which can also be shared
getting ship design hints from the optimizer

One nice thing is that all downloads are verified to be valid .seria files. We hope you find it useful - especially as a permanent archive of ships!

We have several improvements on the horizon:

Downloading collections (almost ready! for now you can download from a collection individually)
Ship stats and search
Stars / “save”
many more

Best part - it’s free. It was a passion project for a couple of us, and we wanted to give it to you.

Charlatans spreading misleading beginner advice are the evolutionary crabs of youtube content creators

Weblog on marginalia.nu — Mon, 29 Jan 2024 02:00:00 GMT

You have a hobby you’ve been into for a decade or more. You like talking about your hobby, and your friends and family, after listening to these things for as long as you’ve been into them, maybe aren’t as excited to always hear about it as you are about discussing them, so in an act of compassion you create a youtube channel where you can monologue about your passion instead.

Security Automation with Gitlab

Brightball Articles — Sat, 27 Jan 2024 11:38:13 GMT

I recently had the opportunity to speak to DEF CON 864 about the multiple layers of security automation within Gitlab, the open source tools that drive them and how the findings are managed and resolved.

Hype

bayindirh — Sat, 27 Jan 2024 02:00:00 GMT

Hype is a loaded word, both in meaning and energy. It’s moving masses in today’s consumerism oriented world, in every category possible, including technology. Both for consumers and professionals alike.

Hype is not inherently bad, but after a certain threshold, it’s nauseating and blinding, creating mindless followers and fanboys of the worst kind. When hype reaches these levels, Logical discourse becomes impossible to maintain, or even start. Worse, the effect is addicting, nudging people to follow the latest hype without even understanding the last thing they’re running after.

As I have written before, I’m a big believer of craftsmanship and mastery and the prefer the benefits it brings over changing tools too often.

In recent times, I have seen two examples of this over-the-top hype: Rust (the programming language) and generative AI models (GPT family, Dall-E and friends, the usual suspects). The hype is at self-sustaining levels because of what they bring to the table, but the hype around them is pumped up at the same time to increase their influence over their respective ecosystems, disregarding the damage they do to the circles they influence. I will try to analyze them one by one.

Rust is an interesting programming language. In short, it disallows you to do things which are potentially dangerous from memory safety perspective. The borrow checker is strict, directly halting compilation when something doesn’t fit to the security model at the slightest level. It’s even so strict that, Rust has an escape hatch called unsafe, which relaxes the security model to allow more things, however borrow checker has no switch. It’s always on. However, people are trying to bypass it for fun and profit.

The impact of Rust and its borrow checker is huge. You can’t have race conditions in your code, and it’s guaranteed to be memory safe with no leaks, no races and potentially no deadlocks. In essence, Rust’s borrow checker forces you to be mindful about how your data moves, and forces you to design your software around it, making it snug and secure. It increases development time somewhat, and makes you reinvent a couple of wheels in the process since some of your wheels doesn’t fit to Rust as is. That’s OK. Neat even.

Problem is, the excitement around Rust makes people blind. While not mapping one to one, the problems attacked by Rust are being worked on for a long time, solved with different approaches and trade-offs. However, with the hype, all this effort and other mechanisms and methods become invisible to these people. Moreover, pointing out the existence of other ways is dismissed without even understanding how they work or what they address. Discussions becomes scoffing matches, or worse, flamewars, leaving both ends bitter.

In my opinion, this drives people who want to understand Rust, but are veterans of other programming languages, out of the Rust ecosystem, because being able to objectively compare things and understand what they bring to the table is an essential part of the craft. Every software development starts with selection of correct tools for the job. Otherwise, your project becomes harder or impossible depending how off you are.

Another problem with this hype is “Rewrite it in Rust” movement, however it’s more nuanced, and deserves its own, separate blog post in the future.

There is a similar, if not bigger, hype cycle is happening in the Artificial Intelligence right now. With the advances in the hardware and computing capabilities, a couple of high impact generative models (GPT, DALL-E, Stable Diffusion) are developed and made accessible to the public. Then, everything spread like a wildfire.

This was in the making for a long time. AI, as a discipline, was not sleeping. Everybody was working hard for decades, but the computing power was not there, and with the power at hand, the barriers were removed and here we are.

However, the result is not a utopia, but something much more complex and nuanced. What we have needs to be understood and navigated. The developers of these models doesn’t want you see these parts of the story, but points you to the capabilities of these models. Summarizing web pages, analyzing reports, generating code on a whim, seeing a web page and duplicating it... The list goes on and on...

When one tries to lift the shiny lid, what there is a concoction of questionable things. Scraping of web regardless of how the content is licensed (ranging from copyleft to copyright, and everything in between), regardless of consent of the producers of said content, and doing various mental exercises to fit everything into fair use.

When it comes to image models, training or fine tuning models with known artists’ style brings another set of problems to the table, since the thing they refined is grabbed out of their hands, and their livelihood is being threatened. Moreover, this progress is even applauded by people making things harder both in the community and mentally for the affected artists.

Stripping a human being of their authenticity and telling that they are worthless is being derogatory, and traumatizing. Considering the time and effort spent by an artist to perfect their style and technique, this is one of the worst traumas to survive, and is hard to recover. Add in the passion required to endure this torturing path to mastery, and the outcome is pretty clear.

All in all, training generative models require immense amounts of material, and defending this material falls under fair use requires tons of mental gymnastics. For example, researchers in AI like to defend that their models learn like humans to claim that they’re “reading” the material like a person. However, same researchers claim that the model is not like a human and lacks conscience and other humanly traits, hence they can’t guarantee their models’ outputs’ correctness, or its honesty. As a topper, when pushed enough or correctly, many models can emit their training data verbatim, creating big problems in confidentiality, privacy, consent, and much more.

Similarly, energy consumption of these models both in training and inference is another problem, which requires its own blog post, because it’s again nuanced and needs space for discussion.

Point to these problems, and you’ll be again scoffed off, claiming being “anti-progress”, “close minded” and a “coward”. Some people claims that “AI companies are doing something amazing, hence they need no permission” to do what they are doing, and elimination of jobs and commoditization of art and other hard things are equalizing and democratizing.

Stopping people creating things by traumatizing, alienating, and forcing not to share what have they done is akin to poisoning the well you drink your water, but nobody is listening because of the hype.

There are other notable events as well, like OpenAI’s recent change of terms for allowing military use, and Eric Schmidt’s AI powered drones company for military applications. I leave these as a “reader’s exercise”, since this post is long enough in its current form.

Watching the backstage of AI research and engaging in discussions and reading about it is like seeing the sausage factory. This time, it’s way more unpleasant.

For a list of things discussed about AI, please see here.

Be aware of the hype, be mindful of what you say and do. Like everything, its overdose is deadly. Both for you and for the people around you.

Until next time,

Be kind.

Bring Back Short Stories

swyx's site RSS Feed — Wed, 24 Jan 2024 02:20:32 GMT

There was a time, before the screens took over, when people read these things called "books". Remarkably efficient things. Always-on display. Could survive dropping from a great height. Somewhat fragile but who cares when you could mass produce thousands of the things for hundreds of dollars.

add Whisper to your iOS Action Button

swyx's site RSS Feed — Tue, 23 Jan 2024 01:24:18 GMT

The new iPhones ditched the physical mute button with a software enabled action button, that allows some customization. the highest degree of customization is Shortcuts. Apple's default transcription is terrible. I got a new iPhone recently and so this is my chance.

How to achieve high performance, and, ratings

Bytepawn - Marton Trencseni — Sun, 21 Jan 2024 01:00:00 GMT

I wrote this document for my team members — Data Scientists and Data Engineers — to help them do a better on their annual performance reviews.

LLM Assisted Moderation

Beetle Space — Mon, 15 Jan 2024 10:00:00 GMT

Moderating online forums is labor intensive. Burnout is common. It’s almost a given that if a subreddit gets to a certain size, the quality will drop, even with active moderation - moderators simply cannot handle the scale.

Here are some ways LLMs can help out.

Enforcing Forum Rules

Whenever a …

VA-11 Hall-A review

ブラック — Sat, 13 Jan 2024 02:00:00 GMT

Time to mix drinks and change lives.

VA-11 Hall-A is basically a cyberpunk visual novel about a bartender and her clients, with a bartending minigame firmly attached to it. A relatively simple concept really. I'd actually cleared this game once already on the PlayStation Vita, but decided to get another playthrough in it for the yearly Daily VA-11 Hall-A, which I guess is already indicative of the game's quality.

While there is an underlying story in VA-11 Hall-A, it’s not really a plot game. Instead, it’s more of a character and slice of life game about a bar. And thankfully it does this part quite well. The characters are well written and it does generally feel like people are just trying to live their lives in the cyberpunkian shithole city they happen to live in. And to top it off, you do get a satisfactory jumping off point for the story – with some epilogue stuff sprinkled on top of it if you managed to do your day job properly.

There is also another strong point for VA-11 Hall-A: the art in this game is lovely. What it maybe lacks in resolution (since the game runs in a 1280x720 window) it more than enough makes up in style and atmosphere. Granted, there’s not really much event CG like you might have in over visual novels, but I don’t think it really matters here. There’s also a satisfying amount of animation for the characters, so it’s definitely an enjoyable visual experience.

The music is also pretty good, although I do have a problem with it: I don’t like the jukebox mechanic. Why am I in charge of picking the music? Can’t you pick appropriate music for me instead so I don’t have to pick something incredibly inappropriate for whatever worries my clientele carries in through the front door?

I do also have to criticise the Steam Deck experience too. Even though this game has been released on the PlayStation Vita and the Nintendo Switch, there’s no official controller layout. This means that it’s kind of a pain to play on the Steam Deck, and I couldn’t really find a controller layout that worked nicely. Touch screen controls for the bartending interface didn’t really work either. And even more annoyingly, even though the game advertises Steam Cloud support, it doesn’t sync save data between Windows and Linux versions. This game is far from a Steam Deck verification and anyone thinking of playing this on one should be aware that there are definitely issues here.

I also have to say that I hate the Model Warrior Julianne minigame and the fact that being it is part of getting all of the achievements. For a while I considered somehow cheating through it but thankfully I managed to eventually get through it by abusing every available cheese. I mean, it is an optional minigame so I can’t hate it too much, but the completionist in me compelled me to get through it and I definitely could’ve lived without it.

VA-11 Hall-A is a fairly short and simple slice of life bartending game, and an enjoyable one at that. If that sounds even remotely like an interesting concept to you, I have a hard time believing you wouldn’t get your money’s worth even at full price.

(Just don't get your hopes up for getting more of it, since the sequel seems to be stuck in development hell.)

You Should Have A Virtual Mailbox

swyx's site RSS Feed — Wed, 10 Jan 2024 02:18:08 GMT

If you fit one of these criteria:

Writing a minimal in-memory storage engine for MySQL/MariaDB

Notes on software development — Tue, 09 Jan 2024 02:00:00 GMT

I spent a week looking at MySQL/MariaDB internals along with ~80 other devs. Although MySQL and MariaDB are mostly the same (more on that later), I focused on MariaDB specifically this week.

Before last week I had never built MySQL/MariaDB before. The first day of this hack week, I got MariaDB building locally and made a code tweak so that SELECT 23 returned 213, and another tweak so that SELECT 80 + 20 returned 60. The second day I got a basic UDF in C working so that SELECT mysum(20, 30) returned 50.

The rest of the week I spent figuring out how to build a minimal in-memory storage engine, which I'll walk through in this post. 218 lines of C++.

It supports CREATE, DROP, INSERT, and SELECT for tables that only have INTEGER fields. It is explicitly not thread-safe because I didn't have time to understand MariaDB's lock primitives.

In this post I'll also talk about how the MariaDB custom storage API compares to the Postgres one, based on a previous hack week project I did.

All code for this post can be found in my fork on GitHub.

MySQL and MariaDB

Before we go further though, why do I keep saying MySQL/MariaDB?

MySQL is GPL licensed (let's completely ignore the commercial variations of MySQL that Oracle offers). The code is open-source. However, the development is done behind closed doors. There is a code dump every month or so.

MariaDB is a fork of MySQL by the creator of MySQL (who is no longer involved, as it happens). It is also GPL licensed (let's completely ignore the commercial variations of MariaDB that MariaDB Corporation offers). The code is open-source. The development is also open-source.

When you install "MySQL" in your Linux distro you are often actually installing MariaDB.

The two are mostly compatible. During this week, I stumbled onto that they evolved support for SELECT .. FROM VALUES .. differently. Some differences are documented on the MariaDB KB. But this KB is painful to browse. Which leads me to my next point.

The MySQL docs are excellent. Easy to read, browse; and they are thorough. The MariaDB docs are a work in progress. I'm sorry I can't be stoic: in just a week I've come to really hate using this KB. Thankfully, in some twisted way, it also doesn't seem to be very thorough either. It isn't completely avoidable though since there is no guarantee MySQL and MariaDB do the same thing.

Ultimately, I spent the week using MariaDB because I'm biased toward fully open projects. But I kept having to look at MySQL docs, hoping they were relevant.

Now that you understand the state of things, let's move on to fun stuff!

Storage engines

Mature databases often support swapping out the storage layer. Maybe you want an in-memory storage layer so that you can quickly run integration tests. Maybe you want to switch between B-Trees (read-optimized) and LSM Trees (write-optimized) and unordered heaps (write-optimized) depending on your workload. Or maybe you just want to try a third-party storage library (e.g. RocksDB or Sled or TiKV).

The benefit of swapping out only the storage engine is that, from a user's perspective, the semantics and features of the database stay mostly the same. But the database is magically faster for a workload.

You keep powerful user management, extension support, SQL support, and a well-known wire protocol. You modify only the method of storing the actual data.

Existing storage engines

MySQL/MariaDB is particularly well known for its custom storage engine support. The MySQL docs for alternate storage engines are great.

While the docs do warn that you should probably stick with the default storage engine, that warning didn't quite feel strong enough because nothing else seemed to indicate the state of other engines.

Specifically, in the past I was always interested in the CSV storage engine. But when you look at the actual code for the CSV engine there is a pretty strong warning:

First off, this is a play thing for me, there are a number of things
wrong with it:
  *) It was designed for csv and therefore its performance is highly
     questionable.
  *) Indexes have not been implemented. This is because the files can
     be traded in and out of the table directory without having to worry
     about rebuilding anything.
  *) NULLs and "" are treated equally (like a spreadsheet).
  *) There was in the beginning no point to anyone seeing this other
     then me, so there is a good chance that I haven't quite documented
     it well.
  *) Less design, more "make it work"

Now there are a few cool things with it:
  *) Errors can result in corrupted data files.
  *) Data files can be read by spreadsheets directly.

TODO:
 *) Move to a block system for larger files
 *) Error recovery, its all there, just need to finish it
 *) Document how the chains work.

-Brian

The difference between the seeming confidence of the docs and seeming confidence of the contributor made me chuckle.

The benefit of these diverse storage engines for me was that they give examples of how to implement the storage engine API. The csv, blackhole, example, and heap storage engines were particularly helpful to read.

The heap engine is a complete in-memory storage engine. Complete means complex though. So there seemed to be room for a stripped down version of an in-memory engine.

And that's we'll cover in this post! First though I want to talk a little bit about the limitations of custom storage engines.

Limitations

While being able to tailor a storage engine to a workload is powerful, there are limits to the benefits based on the design of the storage API.

Both Postgres and MySQL/MariaDB currently have a custom storage API built around individual rows.

Column-wise execution

I have previously written that custom storage engines allows you to switch between column- and row-oriented data storage. Two big reasons to do column-wise storage are 1) opportunity for compression, and 2) fast operations on a single column.

The opportunity for 1) compression on disk would still exist even if you needed to deal with individual rows at the storage API layer since the compression could happen on disk. However any benefits of passing around compressed columns in memory disappear if you must convert to rows for the storage API.

You'd also lose the advantage for 2) fast operations on a single column if the column must be converted into a row at the storage API whereupon it's passed to higher levels that perform execution. The execution would happen row-wise, not column-wise.

All of this is to say that while column-wise storage is possible, the benefit of doing so is not obvious with the current API design for both MySQL/MariaDB and Postgres.

Vectorization

An API built around individual rows also sets limits on the amount of vectorization you can do. A custom storage engine could still do some vectorization under the hood: always filling a buffer with N rows and returning a row from the buffer when the storage API requests a single row. But there is likely some degree of performance left on the table with an API that deals with individual rows.

Remember though: if you did batched reads and writes of rows in the custom storage layer, there isn't necessarily any vectorization happening at the execution layer. From a previous study I did, neither MySQL/MariaDB nor Postgres do vectorized query execution. This paragraph isn't a critique of the storage API, it's just something to keep in mind.

Storage versus execution

The general point I'm making here is that unless both the execution and storage APIs are designed in a certain way, you may attempt optimizations in the storage layer that are ineffective or even harmfull because the execution layer doesn't or can't take advantage of them.

Nothing permanent

The current limitations of the storage API are not intrinsic aspects of MySQL/MariaDB or Postgres's design. For both project there used to be no pluggable storage at all. We can imagine a future patch to either project that allows support for batched row reads and writes that together could make column-wise storage and vectorized execution more feasible.

Even today there have been invasive attempts to fully support column-wise storage and execution in Postgres. And there have also been projects to bring vectorized execution to Postgres.

I'm not as familiar with the MySQL landscape to comment about efforts at the moment their.

Debug build of MariaDB running locally

Now that you've got some background, let's get a debug build of MariaDB!

$ git clone https://github.com/MariaDB/server mariadb
$ cd mariadb
$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=Debug ..
$ make -j8

This takes a while. When I was hacking on Postgres (a C project), it took 1 minute on my beefy Linux server to build. It took 20-30 minutes to build MySQL/MariaDB from scratch. That's C++ for you!

Thankfully incremental builds of MySQL/MariaDB for a tweak after the initial build take roughly the same time as incremental builds of Postgres after a tweak.

Once the build is done, create a database.

$ ./build/scripts/mariadb-install-db --srcdir=$(pwd) --datadir=$(pwd)/db

And create a config for the database.

$ echo "[client]
socket=$(pwd)/mariadb.sock

[mariadb]
socket=$(pwd)/mariadb.sock

basedir=$(pwd)
datadir=$(pwd)/db
pid-file=$(pwd)/db.pid" > my.cnf

Start up the server.

$ ./build/sql/mariadbd --defaults-extra-file=$(pwd)/my.cnf --debug:d:o,$(pwd)/db.debug
./build/sql/mariadbd: Can't create file '/var/log/mariadb/mariadb.log' (errno: 13 "Permission denied")
2024-01-03 17:10:15 0 [Note] Starting MariaDB 11.4.0-MariaDB-debug source revision 3fad2b115569864d8c1b7ea90ce92aa895cfef08 as process 185550
2024-01-03 17:10:15 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2024-01-03 17:10:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2024-01-03 17:10:15 0 [Note] InnoDB: Number of transaction pools: 1
2024-01-03 17:10:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-01-03 17:10:15 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2024-01-03 17:10:15 0 [Note] InnoDB: Completed initialization of buffer pool
2024-01-03 17:10:15 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2024-01-03 17:10:15 0 [Note] InnoDB: End of log at LSN=57155
2024-01-03 17:10:15 0 [Note] InnoDB: Opened 3 undo tablespaces
2024-01-03 17:10:15 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2024-01-03 17:10:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2024-01-03 17:10:15 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2024-01-03 17:10:15 0 [Note] InnoDB: log sequence number 57155; transaction id 16
2024-01-03 17:10:15 0 [Note] InnoDB: Loading buffer pool(s) from ./db/ib_buffer_pool
2024-01-03 17:10:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-01-03 17:10:15 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-01-03 17:10:15 0 [Note] InnoDB: Buffer pool(s) load completed at 240103 17:10:15
2024-01-03 17:10:15 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-01-03 17:10:15 0 [Note] Server socket created on IP: '::'.
2024-01-03 17:10:15 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2024-01-03 17:10:15 0 [Note] ./build/sql/mariadbd: ready for connections.
Version: '11.4.0-MariaDB-debug'  socket: './mariadb.sock'  port: 3306  Source distribution

With that --debug flag, debug logs will show up in $(pwd)/db.debug. It's unclear why debug logs are treated separately from the console logs shown here. I'd rather them all be in one place.

In another terminal, run a client and make a request!

$ ./build/client/mariadb --defaults-extra-file=$(pwd)/my.cnf --database=test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 11.4.0-MariaDB-debug Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [test]> SELECT 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.001 sec)

Huzzah! Let's write a custom storage engine!

Where does the code go?

When writing an extension for some project, I usually expect to have the extension exist in its own repo. I was able to do this with the Postgres in-memory storage engine I wrote. And in general, Postgres extensions exist as their own repos.

I was able to create and build a UDF plugin outside the MariaDB source tree. But when it came to getting a storage engine to build and load successfully, I wasted almost an entire day (a large amount of time in a single hack week) getting nowhere.

Extensions for MySQL/MariaDB are most easily built via the CMake infrastructure within the repo. Surely there's some way to replicate that infrastructure from outside the repo but I wasn't able to figure it out within a day and didn't want to spend more time on it.

Apparently the normal thing to do in MySQL/MariaDB is to keep extensions within a fork of MySQL/MariaDB.

When I switched to this method I was able to very quickly get the storage engine building and loaded. So that's what we'll do.

Boilerplate

Within the MariaDB source tree, create a new folder in the storage subdirectory.

$ mkdir storage/memem

Within storage/memem/CMakeLists.txt add the following.

# Copyright (c) 2006, 2010, Oracle and/or its affiliates. All rights reserved.
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1335 USA

SET(MEMEM_SOURCES  ha_memem.cc ha_memem.h)
MYSQL_ADD_PLUGIN(memem ${MEMEM_SOURCES} STORAGE_ENGINE)

This hooks into MySQL/MariaDB build infrastructure. So next time you run make within the build directory we created above, it will also build this project.

The storage engine class

It would be nice to see a way to extend MySQL in C (for one, because it would then be easier to port to other languages). But all of the builtin storage methods use classes. So we'll do that too.

The class we must implement is an instance of handler. There is a single handler instance per thread, corresponding to a single running query. (Postgres gives each query its own process, MySQL gives each query its own thread.) However, handler instances are reused across different queries.

There are a number of virtual methods on handler we must implement in our subclass. For most of them we'll do nothing: simply returning immediately. These simple methods will be implemented in ha_memem.h. The methods with more complex logic will be implemented in ha_memem.cc.

Let's set up includes in ha_memem.h.

/* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; version 2 of the License.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335  USA */

#ifdef USE_PRAGMA_INTERFACE
#pragma interface /* gcc class implementation */
#endif

#include "thr_lock.h"
#include "handler.h"
#include "table.h"
#include "sql_const.h"

#include <vector>
#include <memory>

Next we'll define structs for our in-memory storage.

typedef std::vector<uchar> MememRow;

struct MememTable
{
  std::vector<std::shared_ptr<MememRow>> rows;
  std::shared_ptr<std::string> name;
};

struct MememDatabase
{
  std::vector<std::shared_ptr<MememTable>> tables;
};

Within ha_memem.cc we'll implement a global (not thread-safe) static MememDatabase* that all handler instances will query when requested. We need the definitions in the header file though because we'll store the table currently being queried in the handler subclass.

This is so that every call to write_row to write a single row or call to rnd_next to read a single row does not need to look up the in-memory table object N times within the same query.

And finally we'll define the subclass of handler and implementations of trivial methods.

class ha_memem final : public handler
{
  uint current_position= 0;
  std::shared_ptr<MememTable> memem_table= 0;

public:
  ha_memem(handlerton *hton, TABLE_SHARE *table_arg) : handler(hton, table_arg)
  {
  }
  ~ha_memem()= default;
  const char *index_type(uint key_number) { return ""; }
  ulonglong table_flags() const { return 0; }
  ulong index_flags(uint inx, uint part, bool all_parts) const { return 0; }
  /* The following defines can be increased if necessary */
#define MEMEM_MAX_KEY MAX_KEY     /* Max allowed keys */
#define MEMEM_MAX_KEY_SEG 16      /* Max segments for key */
#define MEMEM_MAX_KEY_LENGTH 3500 /* Like in InnoDB */
  uint max_supported_keys() const { return MEMEM_MAX_KEY; }
  uint max_supported_key_length() const { return MEMEM_MAX_KEY_LENGTH; }
  uint max_supported_key_part_length() const { return MEMEM_MAX_KEY_LENGTH; }
  int open(const char *name, int mode, uint test_if_locked) { return 0; }
  int close(void) { return 0; }
  int truncate() { return 0; }
  int rnd_init(bool scan);
  int rnd_next(uchar *buf);
  int rnd_pos(uchar *buf, uchar *pos) { return 0; }
  int index_read_map(uchar *buf, const uchar *key, key_part_map keypart_map,
                     enum ha_rkey_function find_flag)
  {
    return HA_ERR_END_OF_FILE;
  }
  int index_read_idx_map(uchar *buf, uint idx, const uchar *key,
                         key_part_map keypart_map,
                         enum ha_rkey_function find_flag)
  {
    return HA_ERR_END_OF_FILE;
  }
  int index_read_last_map(uchar *buf, const uchar *key,
                          key_part_map keypart_map)
  {
    return HA_ERR_END_OF_FILE;
  }
  int index_next(uchar *buf) { return HA_ERR_END_OF_FILE; }
  int index_prev(uchar *buf) { return HA_ERR_END_OF_FILE; }
  int index_first(uchar *buf) { return HA_ERR_END_OF_FILE; }
  int index_last(uchar *buf) { return HA_ERR_END_OF_FILE; }
  void position(const uchar *record) { return; }
  int info(uint flag) { return 0; }
  int external_lock(THD *thd, int lock_type) { return 0; }
  int create(const char *name, TABLE *table_arg, HA_CREATE_INFO *create_info);
  THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to,
                             enum thr_lock_type lock_type)
  {
    return to;
  }
  int delete_table(const char *name) { return 0; }

private:
  void reset_memem_table();
  virtual int write_row(const uchar *buf);
  int update_row(const uchar *old_data, const uchar *new_data)
  {
    return HA_ERR_WRONG_COMMAND;
  };
  int delete_row(const uchar *buf) { return HA_ERR_WRONG_COMMAND; }
};

A complete storage engine might seriously implement all of these methods. But we'll only seriously implement 7 of them.

To finish up the boilerplate, we'll switch over to ha_memem.cc and set up the includes.

/* Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved.

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; version 2 of the License.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335  USA */

#ifdef USE_PRAGMA_IMPLEMENTATION
#pragma implementation // gcc: Class implementation
#endif

#define MYSQL_SERVER 1
#include <my_global.h>
#include "sql_priv.h"
#include "unireg.h"
#include "sql_class.h"

#include "ha_memem.h"

Ok! Let's dig into the implementation.

Implementation

The global database

First up, we need to declare a global MememDatabase* instance. We'll also implement a helper function for finding the index of a table by name within the database.

// WARNING! All accesses of `database` in this code are thread
// unsafe. Since this was written during a hack week, I didn't have
// time to figure out MySQL/MariaDB's runtime well enough to do the
// thread-safe version of this.
static MememDatabase *database;

static int memem_table_index(const char *name)
{
  int i;
  assert(database->tables.size() < INT_MAX);
  for (i= 0; i < (int) database->tables.size(); i++)
  {
    if (strcmp(database->tables[i]->name->c_str(), name) == 0)
    {
      return i;
    }
  }

  return -1;
}

As I wrote this post I noticed that this code also assumes there's only a single database. That isn't how MySQL works. Everytime you call USE ... in MySQL you are switching between databases. You can query tables across databases. A real in-memory backend would need to be aware of the different databases, not just different tables. But to keep the code succinct we won't implement that in this post.

Next we'll implement plugin initialization and cleanup.

Plugin lifecycle

Before we register the plugin with MariaDB, we need to set up initialization and cleanup methods for it.

The initialization method will take care of initializing the global MememDatabase* database object. It will set up a handler for creating new instances of our handler subclass. And it will set up a handler for deleting tables.

static handler *memem_create_handler(handlerton *hton, TABLE_SHARE *table,
                                     MEM_ROOT *mem_root)
{
  return new (mem_root) ha_memem(hton, table);
}

static int memem_init(void *p)
{
  handlerton *memem_hton;

  memem_hton= (handlerton *) p;
  memem_hton->db_type= DB_TYPE_AUTOASSIGN;
  memem_hton->create= memem_create_handler;
  memem_hton->drop_table= [](handlerton *, const char *name) {
    int index= memem_table_index(name);
    if (index == -1)
    {
      return HA_ERR_NO_SUCH_TABLE;
    }

    database->tables.erase(database->tables.begin() + index);
    DBUG_PRINT("info", ("[MEMEM] Deleted table '%s'.", name));

    return 0;
  };
  memem_hton->flags= HTON_CAN_RECREATE;

  // Initialize global in-memory database.
  database= new MememDatabase;

  return 0;
}

The DBUG_PRINT macro is a debug helper MySQL/MariaDB gives us. As noted above, the output is directed to a file specified by the --debug flag. Unfortunately I couldn't figure out how to flush the stream this macro writes to. It seemed like occasionally when there was a segfault logs I expected to be there weren't there. And the file would often contain what looked like partially written logs. Anyway, as long as there wasn't a segfault the debug file will eventually contain the DBUG_PRINT logs.

The only thing the plugin cleanup function must do is delete the global database.

static int memem_fini(void *p)
{
  delete database;
  return 0;
}

Now we can register the plugin!

Plugin registration

The maria_declare_plugin and maria_declare_plugin_end register the plugin's metadata (name, version, etc.) and callbacks.

struct st_mysql_storage_engine memem_storage_engine= {
    MYSQL_HANDLERTON_INTERFACE_VERSION};

maria_declare_plugin(memem){
    MYSQL_STORAGE_ENGINE_PLUGIN,
    &memem_storage_engine,
    "MEMEM",
    "MySQL AB",
    "In-memory database.",
    PLUGIN_LICENSE_GPL,
    memem_init, /* Plugin Init */
    memem_fini, /* Plugin Deinit */
    0x0100 /* 1.0 */,
    NULL,                          /* status variables                */
    NULL,                          /* system variables                */
    "1.0",                         /* string version */
    MariaDB_PLUGIN_MATURITY_STABLE /* maturity */
} maria_declare_plugin_end;

That's it! Now we need to implement methods for writing rows, reading rows, and creating a new table.

Create table

To create a table, we make sure one by this name doesn't already exist, make sure it only has INTEGER fields, allocate memory for the table, and append it to the global database.

int ha_memem::create(const char *name, TABLE *table_arg,
                     HA_CREATE_INFO *create_info)
{
  assert(memem_table_index(name) == -1);

  // We only support INTEGER fields for now.
  uint i = 0;
  while (table_arg->field[i]) {
    if (table_arg->field[i]->type() != MYSQL_TYPE_LONG)
      {
    DBUG_PRINT("info", ("Unsupported field type."));
    return 1;
      }

    i++;
  }

  auto t= std::make_shared<MememTable>();
  t->name= std::make_shared<std::string>(name);
  database->tables.push_back(t);
  DBUG_PRINT("info", ("[MEMEM] Created table '%s'.", name));

  return 0;
}

Not very complicated. Let's handle INSERT-ing rows next.

Insert row

There is no method called when an INSERT starts. There is a table field on the handler parent class that is updated though when a SELECT or INSERT is going. So we can fetch the current table from that field.

Since we have a slot for a std::shared_ptr<MememTable> memem_table on the ha_memem class, we can check if it is NULL when we insert a row. If it is, we look up the current table and set this->memem_table to its MememTable.

But there's a bit more to it than just the table name. The const char* name passed to the create() method above seems to be a sort of fully qualified name for the table. By observation, when creating a table y in a database test, the const char* name value is ./test/y. The . prefix probably means that the database is local, but I'm not sure.

So we'll write a helper method that will reconstruct the fully qualified table name before looking up that fully qualified table name in the global database.

void ha_memem::reset_memem_table()
{
  // Reset table cursor.
  current_position= 0;

  std::string full_name= "./" + std::string(table->s->db.str) + "/" +
                         std::string(table->s->table_name.str);
  DBUG_PRINT("info", ("[MEMEM] Resetting to '%s'.", full_name.c_str()));
  assert(database->tables.size() > 0);
  int index= memem_table_index(full_name.c_str());
  assert(index >= 0);
  assert(index < (int) database->tables.size());

  memem_table= database->tables[index];
}

Then we can use this within write_row to figure out the current MememTable being queried.

But first, let's digress into how MySQL stores rows.

The MySQL row API

When you write a Postgres custom storage API, you are expected to basically read from or write to an array of Datum.

Totally sensible.

In MySQL, you read from and write to an array of bytes. That's pretty weird to me. Of course you can build your own higher level serialization/deserialization on top of it. But it's just strange to me everyone has to know this basically opaque API.

Certainly it's documented.

The handler class is the interface for dynamically loadable
storage engines. Do not add ifdefs and take care when adding or
changing virtual functions to avoid vtable confusion

Functions in this class accept and return table columns data. Two data
representation formats are used:
1. TableRecordFormat - Used to pass [partial] table records to/from
   storage engine

2. KeyTupleFormat - used to pass index search tuples (aka "keys") to
   storage engine. See opt_range.cc for description of this format.

TableRecordFormat
=================
[Warning: this description is work in progress and may be incomplete]
The table record is stored in a fixed-size buffer:

  record: null_bytes, column1_data, column2_data, ...

The offsets of the parts of the buffer are also fixed: every column has 
an offset to its column{i}_data, and if it is nullable it also has its own
bit in null_bytes.

In our implementation, we'll skip the support for NULL values. We'll only support INTEGER fields. But we still need to be aware that the first byte will be taken up. We'll also assume there won't be more than one byte of a NULL bitmap.

It is this opaque byte array that we'll read from in write_row(const uchar* buf) and write to in read_row(uchar* buf).

Insert row (take two)

To keep things simple we're going to store the row in MememTable the same way MySQL passes it around.

int ha_memem::write_row(const uchar *buf)
{
  if (memem_table == NULL)
  {
    reset_memem_table();
  }

  // Assume there are no NULLs.
  buf++;

  uint field_count = 0;
  while (table->field[field_count]) field_count++;

  // Store the row in the same format MariaDB gives us.
  auto row= std::make_shared<std::vector<uchar>>(
      buf, buf + sizeof(int) * field_count);
  memem_table->rows.push_back(row);

  return 0;
}

Which makes reading the row quite simple too!

Read row

The only slight difference between reading and writing a row is that MySQL/MariaDB will tell us when the SELECT scan for a table starts.

We'll use that opportunity to reset the current_row cursor and reset the memem_table field. Since, again, handler classes are only used once per query but they are reused for queries running at other times.

int ha_memem::rnd_init(bool scan)
{
  reset_memem_table();
  return 0;
}

int ha_memem::rnd_next(uchar *buf)
{
  if (current_position == memem_table->rows.size())
  {
    // Reset the in-memory table to make logic errors more obvious.
    memem_table= NULL;
    return HA_ERR_END_OF_FILE;
  }
  assert(current_position < memem_table->rows.size());

  uchar *ptr= buf;
  *ptr= 0;
  ptr++;

  // Rows internally are stored in the same format that MariaDB
  // wants. So we can just copy them over.
  std::shared_ptr<std::vector<uchar>> row= memem_table->rows[current_position];
  std::copy(row->begin(), row->end(), ptr);

  current_position++;
  return 0;
}

And we're done!

Build and test

Go back into the build directory we created within the source tree root and rerun make -j8.

Kill the server (you'll need to do something like killall mariadbd since the server doesn't respond to Ctrl-c). And restart it.

For some reason this plugin doesn't need to be loaded. We can run SHOW PLUGINS; in the MariaDB CLI and we'll see it.

$ ./build/client/mariadb --defaults-extra-file=/home/phil/vendor/mariadb/my.cnf --database=test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 11.4.0-MariaDB-debug Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [test]> SHOW PLUGINS;
+-------------------------------+----------+--------------------+-----------------+---------+
| Name                          | Status   | Type               | Library         | License |
+-------------------------------+----------+--------------------+-----------------+---------+
| binlog                        | ACTIVE   | STORAGE ENGINE     | NULL            | GPL     |
...
| MEMEM                         | ACTIVE   | STORAGE ENGINE     | NULL            | GPL     |
...
| BLACKHOLE                     | ACTIVE   | STORAGE ENGINE     | ha_blackhole.so | GPL     |
+-------------------------------+----------+--------------------+-----------------+---------+
73 rows in set (0.012 sec)

There we go! To create a table with it we need to set ENGINE = MEMEM. For example, CREATE TABLE x (i INT) ENGINE = MEMEM.

Let's create a script to try out the memem engine, in storage/memem/test.sql.

drop table if exists y;
drop table if exists z;

create table y(i int, j int) engine = MEMEM;
insert into y values (2, 1029);
insert into y values (92, 8);
select * from y where i + 8 = 10;

create table z(a int) engine = MEMEM;
insert into z values (322);
insert into z values (8);
select * from z where a > 20;

And run it.

$ ./build/client/mariadb --defaults-extra-file=$(pwd)/my.cnf --database=test --table --verbose < storage/memem/test.sql
--------------
drop table if exists y
--------------

--------------
drop table if exists z
--------------

--------------
create table y(i int, j int) engine = MEMEM
--------------

--------------
insert into y values (2, 1029)
--------------

--------------
insert into y values (92, 8)
--------------

--------------
select * from y where i + 8 = 10
--------------

+------+------+
| i    | j    |
+------+------+
|    2 | 1029 |
+------+------+
--------------
create table z(a int) engine = MEMEM
--------------

--------------
insert into z values (322)
--------------

--------------
insert into z values (8)
--------------

--------------
select * from z where a > 20
--------------

+------+
| a    |
+------+
|  322 |
+------+

What you see there is the power of storage engines! It supports the full SQL language even while we implemented storage somewhere completely different than the default.

In-memory is boring

Certainly, I'm getting bored doing the same project over and over again on different databases. However, it's minimal projects like this that make it super easy to then go and port the storage engine to something else.

The goal here is to be minimal but meaningful. And I've accomplished that for myself at least!

On ChatGPT

As I've written before, this sort of exploration wouldn't be possible within the time frame I gave myself if it weren't for ChatGPT. Specifically, the paid tier GPT4.

Neither the MySQL nor the MariaDB docs were so helpful that I could immediately figure out things like how to get the current table name within a scan (the table member of the handler class).

With ChatGPT you can ask questions like: "In a MySQL C++ plugin, how do I get the name of the table from a handler class as a C string?". Sometimes it's right and sometime's it's not. But you can try out the code and if it builds it is at least somewhat correct!

Wrote a post walking you through building a super minimal in-memory storage engine for MySQL/MariaDB in 218 lines of C++.

And took time again to reflect on the limitations of custom storage engines and how MySQL compares to Postgres internally here.https://t.co/nImUC36DPs pic.twitter.com/1Oj2Lcua8O
— Phil Eaton (@eatonphil) January 9, 2024

Visualizing SQLAlchemy Databases as Mermaid and Dot Diagrams using Paracelsus

tedious ramblings — Mon, 08 Jan 2024 16:30:00 GMT

SQLAlchemy is a powerful tool that lets Python developers define their database schemas using simple classes. I made Paracelsus to turn those SQLAlchemy defined databases into diagrams that can be used to visualize the schema.

Using a Static JSON File in Home Assistant

Pete Keen — Mon, 08 Jan 2024 02:00:00 GMT

Recently I found myself needing to bring some JSON from a file into a Home Assistant sensor. Specifically, the electricity rates for my power company are woefully out of date on OpenEI so I decided I could just maintain the data myself.

Home Assistant doesn't have a direct way to read JSON data from a file into a sensor. There's the File platform which has a promising name but is actually a trap. File is meant for use cases where something writes to, say, a CSV file continuously and you just want to read the most recent line. It specifically does not read the whole file.

After a lot of searching I came across the Command Line platform. The integration does a number of things, but for our purposes it lets you periodically run a command within the context of the Home Assistant container and bring the output back into Home Assistant as a sensor.

Let's say you have a JSON file named rate.json in your Home Assistant configuration directory:

{
        "name": "Base Rate",
        "rate": 0.15
        }

You can bring that into a sensor with the following snippet in your configuration.yaml file:

command_line:
        - sensor:
        name: "Electrity Rate"
        command: 'cat rate.json',
        value_template: "{{ value_json['rate'] }}"
        unit_of_measurement: "USD/kWh",
        json_attributes:
        - name
        - rate

This config does a couple things. The command key specifies what command HA should run, in this case cat to read the file to stdout. value_template extracts the rate key from the file into the sensor's value. The json_attributes list pulls the list of keys into attributes in the sensor, which you can later access from a template using state_attr(). I have also specfied unit_of_measurement here just because the Energy reporting system needs that if you want to use this as an input.

So, the above is great if you have one static set of attributes to bring in, but sensor values can be at most 255 characters. What if you have a bigger file that you need to pull just a little data out of?

Let's say we have this slightly bigger file rates.json:

[
        {
        "id": "d1-11_summer_on_peak",
        "name": "Summer Peak",
        "months": [6, 7, 8, 9],
        "days": [1, 2, 3, 4, 5],
        "hours": [15, 16, 17, 18],
        "rate": 0.23525,
        "peak": true
        },
        {
        "id": "d1-11_summer_off_peak",
        "name": "Summer Off-Peak",
        "months": [6, 7, 8, 9],
        "days": [0, 1, 2, 3, 4, 5, 6],
        "hours": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "rate": 0.17859,
        "peak": false
        },
        {
        "id": "d1-11_winter_on_peak",
        "name": "Winter Peak",
        "months": [1, 2, 3, 4, 5, 10, 11, 12],
        "days": [1, 2, 3, 4, 5],
        "hours": [15, 16, 17, 18],
        "rate": 0.17879,
        "peak": true
        },
        {
        "id": "d1-11_winter_off_peak",
        "name": "Winter Off-Peak",
        "months": [1, 2, 3, 4, 5, 10, 11, 12],
        "days": [0, 1, 2, 3, 4, 5, 6],
        "hours": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "rate": 0.1658,
        "peak": false
        }
        ]

This file describes the rate structure that's in effect at my house, DTE rate D1.11. Each entry in the array is a rate, and the first matching rate based on month, weekday, and hour is the effective rate.

Ordinarily one might reach for a HACS integration or something, but the Home Assistant container has another trick up it's sleeve: it bundles jq.

jq is a tool for querying and manipulating JSON streams. This essay isn't meant to be a jq tutorial so we're not going to go in depth into what this query in rate_filter.jq does, but in broad strokes it picks the first matching rate from the input file and extracts just the name, rate, and peak keys.

map(select(
        (.months[] | contains($ARGS.positional[0] | tonumber))
        and (.days[] | contains($ARGS.positional[1] | tonumber))
        and (.hours[] | contains($ARGS.positional[2] | tonumber))
        ))[0] | {name, rate, peak}

Here's a modified command line sensor that runs jq appropriately:

command_line:
        - sensor:
        name: "DTE Rate"
        command: 'jq -f rate_filter.jq rates.json --args {{ now().month }} {{ now().isoweekday() % 7 }} {{ now().hour }}'
        value_template: "{{ value_json['rate'] }}"
        unit_of_measurement: USD/kWh
        json_attributes:
        - name
        - rate
        - peak

The relevant change here is to the command key, which now invokes jq with the -f argument to pass the filter as a file rather than trying to quote everything properly within HA, then passes the actual rates.json file, then treats the rest of the arguments as positional args. These are accessed within rate_filter.jq as $ARGS.positional[0] etc.

With this set up I can access the current electric rate within my Home Assistant in a way that is compatible with the Energy dashboard, which is completely local, and which should be easy to maintain in the future.

The above isn't at all specific to electric rates, by the way. This technique should work for any data that you need in HA but is more complicated than a plain input can work with.

DAPLs: Domain Agnostic Programming Languages

metablog — Wed, 03 Jan 2024 11:29:07 GMT

Every once in a while, you get an insight that hits you like a truck. Or maybe a ton of bricks. Or a truck carrying a ton of bricks. Developing Objective-S has delivered a bunch of these, but one of the biggest was that our General Purpose Programming Languages are nothing of the sort. They are Domain Specific Languages for the domain of algorithms. See also: ALGOL. To move forward, programming languages will have to support more than just this one architectural style.

Alas, communicating this insight has been...challenging. One method was branding Objective-S as "the first general purpose programming language". This did not always go over well.

This doesn't really sound much like a discussion...

I understand you're trying to make a fun rhetorical point with the framing, but I have to say it just makes me uninterested in any serious engagement.

Makes it seem like you don't care to even acknowledge existing PL work. =/
— Chandler Carruth (@chandlerc1024) May 10, 2021

It looks interesting. But stuff like this:
> Objective-S is the first general purpose programming language
Those kind of statements annoy me to be honest. Is it really true? Or is it over-the-top marketing hype? For me – and I'm sure I'm not the only person who feels this way – it creates a negative first impression.

skissane on May 9, 2021 on: Objective-S: architecture-oriented language based ...

Of course, there is no such thing as bad publicity, but this had a bit too much of a lunatic-fringe vibe, no matter how correct the insight, and no matter how ill-fitting the moniker "general purpose language" really is for our call/return-oriented algorithm-DSLs.

As Richard Feynman once put it, "One of the miseries of life is that everyone names everything a litte bit wrong, and so it makes everything a little harder to understand in the world than it would be if it were named differently". Calling our algorithm-DSLs "general purpose" implies that we have solved the problem of generality, when we have not, and that the only real alternative is to be more specific, hence DSLs. But DSLs also don't really work that well, because the successful ones almost invariable grow non-domain-specific features, just in a haphazard way. Or they need to be combined to cover different fields, so we get language workbenches that allow us to define lots of little DSLs and combine them.

This all points to the fact that our problem is not being too general, but too specific. Our algorithm-DSLs just aren't very good at covering a lot of the problems programmers have to solve, though of course they are Turing-complete and can get us there, somehow.

Riffing off those ideas, and leaving aside the minefield of incorrect but entrenched terminology, I propose the term Domain Agnostic Programming Language. Because any sufficiently powerful DSL can be bent out of shape sufficiently for any purpose, just like our algorithm-DSLs can. They just aren't a good fit. And so Objective-S is not the first general purpose language, it is the first, and almost certainly the worst, DAPL. And hopefully its programming environment will be dapper.

My Fave New Podcasts of 2023

swyx's site RSS Feed — Wed, 03 Jan 2024 09:11:43 GMT

As someone who does a lot of my learning via podcasts, I've been putting up picks lists for 4 years straight (see main 2019 list, then my 2020 and 2021 and 2022 diffs), so it's time to do year 5(!)

Looking Ahead to 2024

Ben Overmyer's Site — Mon, 01 Jan 2024 02:00:00 GMT

This year I want to continue my introduction to bodybuilding and round out my fitness regimen with regular cardio. For January and February, I'll finish the "Start From Scratch" series of workouts. I will also start running three days a week again. I'll be taking progress photos. By July, I want to see distinct muscle definition in the mirror. I also want to try and get my VO2 max consistently up to 38 in the same timeframe.

Last year January and November were alcohol-free months. This year, I'll go alcohol-free for three months out of the year. January is one of them. I might make March and August the other two, but I'll leave that up to planning later.

For Silver Gryphon Games, my sole objective this year is to get Aethermancy released. Once that's done, I'll try and get the second editions of Aether and Ingenium out, but those are bonuses.

For Iron Arachne, I want to really push on the sci-fi side of generators. I will expand the details for the planet, star nation, and star system generators.

I have three trips planned for this year so far. One is a trip to Mexico with college friends. The second is an eclipse viewing party. The third is a father-son cruise in the Mediterranean. For each of these I want to make sure I journal about them.

Finally, I'm going to try and set a journaling habit. Some of this will turn into blog posts, but the idea is to record more data about my life and use that for reflection later.

2023: year in perspective

learnbyexample — Mon, 01 Jan 2024 02:00:00 GMT

TL;DR: Updated six programming ebooks, created four interactive TUI apps for exercises, wrote blog posts, recorded YouTube videos, newsletter prospered, read 100+ novels, and so on. Had a great year in terms of ebook sales despite worries over AI tools 😇

Books updated🔗

This year I focused on updating my existing ebooks instead of working on a new one. I managed to revise 6 out of my 13 published works so far. Examples and exercises were added and improved. Typos were corrected, sections added for new features (if any), new book covers, promo videos and so on.

Understanding Python re(gex)? — Learn Python Regular Expressions step-by-step from beginner to advanced levels with 300+ examples
CLI text processing with GNU grep and ripgrep — Example based guide to mastering GNU grep and ripgrep
CLI text processing with GNU sed — Example based guide to mastering GNU sed
CLI text processing with GNU awk — Example based guide to mastering GNU awk one-liners
Perl One-Liners Guide — Example based guide for text processing with Perl from the command line
Understanding JavaScript RegExp — Learn JavaScript Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises

TUI apps🔗

Last year, I had learned a bit of Textual. My aim was to create interactive apps for practicing exercises from my ebooks. I wrote the following apps:

Python re(gex)? exercises — 100+ exercises for Python Regular Expressions
- Python re(gex)? playground — interactive playground, also includes a cheatsheet
Grep Exercises — 50+ exercises for GNU grep (or alternate implementations like ripgrep)
Sed Exercises — 50+ exercises for GNU sed
Awk Exercises — 80+ exercises for GNU awk

And I also added more exercises for the Linux CLI Text Processing Exercises app.

Blog posts🔗

Most of my blog posts this year were related to book and interactive app announcements. So, not really a choice to pick favorites from:

I also posted some weekly programming tips (Python, Linux, Vim).

Book sales🔗

Revenue from ebook sales were about 10% lower than last year. At the start of the year, I'd have been satisfied even if it had been 50% lower. I wasn't writing new ebooks and AI tools were all the rage on social media. Somehow, I got lucky with self-promotion posts for my GNU awk ebook and the rest of the months weren't too shabby. Here's my Gumroad revenue chart for 2023:

You can clearly see when the GNU awk ebook was updated. Sales on Gumroad was actually just a bit higher than last year. It was on Leanpub that sales were much lower, almost half compared to last year. Profits reduced more than 10% since Gumroad increased their fees. Overall, I'm still earning more than I need and I'm hoping that next year wouldn't see too much drop in sales.

I started a newsletter, learnbyexample weekly, two years back. I've managed to send an email every Friday without fail so far and I'm proud of that. Sometimes I had to schedule issues weeks ahead. Total subscriber count crossed 1000 earlier this month and some readers are even paying me monthly despite this being a free newsletter.

Fictional reading🔗

I enjoy reading fantasy and science-fiction novels. I read 100+ SFF books this year despite aiming for less than 100! Anyway, I wrote a post listing my favorites here.

I even participated in NaNoWriMo. I only wrote 20K words, but I did have some fun. The novel went nowhere though and it is languishing now. Not sure if I'd get back to it someday.

Goals for 2024🔗

There are seven more books I need to update. Hopefully I get them done in a year, though I won't be pushing hard. If I crave to write some new books instead, I'd switch over to them. Or even do something else entirely. After more than six years writing tutorials and books, I sure can do with a break.

Here's wishing you a very happy, healthy and prosperous 2024 👍 😇