Hacker News Personal Blogs 2023 | Top 100 Blogs

2023 in Retrospective

bayindirh — Fri, 29 Dec 2023 02:00:00 GMT

The year is ending in a couple of days, and I wanted to leave a public note about how it went for me.

2023 was an interesting year. I spent a lot of time getting used to my new life, and to be honest, it went well. I’m happier than ever, and learning to handle my life in this new chapter.

The biggest change was how I allocate my time during the day. This nudges me to learn to downscale and manage my time better or be more efficient in how I accomplish things in general. This part of my transition is not over yet, I’m actively downscaling in both physical and digital ways. For example, I don’t have a personal desktop computer anymore, but living with a single laptop. I have almost completed my move to cloud storage, converging and simplifying it in the process. I’m trying to reduce the number of my physical belongings, however that part is going slower than I dreamt of. 2023 brought minimalism with all its peculiarities, but it’s not a failure by any means.

Another small yet important change was finishing transitioning to fountain pens for note-taking. I have disposable pens, but new ones are not being added much. In the end, I’ll need to have a couple because I use rulers sometimes, and there are special-purpose ones, like pigment markers, that I’ll need to stock, but all the grunt work is handled by fountain pens, which bring joy and do not create waste except the glass bottles you empty, which can be recycled. I can say that I’m more mindful of the amount of plastic I consume, and I try to actively recycle it (whether it’s done or not is another issue). I’ll be talking about note-taking and fountain pens in the soonish future.

I continued learning Go. I wanted to be able to have something between Python and C++ and Go filled that gap perfectly. Simpler than C++, compiled, garbage collected, and fast enough. It’s open source and has a GCC implementation, which is crucial for me. All in all, I love the language and its ergonomics. It proved itself to be a great companion for my automation and downscaling efforts. I may write what I think about Go in a future blog post.

I continued implementing and almost finished one of the tools I wanted to have with my newly acquired Go knowledge: Nudge, a small command line tool to interface with Pushover for sending push notifications to devices from the Linux command line, which is a crucial requirement for “lights-out operations” which I want to move towards for the mundane things I routinely do in front of a computer because I want to have more time for myself and people I care.

Lastly, with the AI revolution, I became way more mindful of how these systems are trained, and its effects on our privacy and free software. As a result, changed my trajectory towards services and systems which do not devour your things for their profit without your consent. During that transition, I also left Google search for Kagi, which is something you pay for to use. However, this payment makes sure that you don’t become the product during the process, at least for now. I’m pretty happy about this decision, and I recommend you to give Kagi a try.

2024 will bring the next step in many of these endeavors and will add new ones on top of it. I’m hopeful that the new year will bring better things upon us since 2023 was enough both for me and our planet as a whole.

Until next time,

Be kind.

A Hobby Coding Biography

Weblog on marginalia.nu — Thu, 28 Dec 2023 02:00:00 GMT

This is a bit of a retrospective of every project I’ve worked on, as far as I remember them. I’ve tried to unearth any artifacts that remain. Far from everything is flattering and resounding success, but then again, maybe that’s good. There are definitely patterns in the things that didn’t pan out. Earliest Traces I was definitely programming stuff, but I don’t think it ever amounted to anything tangible. It was more like playing house, I built GUIs that looked like real applications but didn’t really do anything but look cool to me.

What I learned during Advent of Code 2023

Linux Engineer's random thoughts — Wed, 27 Dec 2023 21:00:00 GMT

Advent of Code is an Advent calendar of small programming puzzles. I participated in this year's edition, finishing it for the second time in row. The puzzles of all editions are always accessible.

The principle is to read the problem, get a puzzle input (more or less tailored to your …

Make your own way

Notes on software development — Wed, 27 Dec 2023 02:00:00 GMT

Over the years, I have repeatedly felt like I missed the timing for a meetup or an IRC group or social media in general. I'd go to a meetup every so often but I'd never make a meaningful connection with people, whereas everyone else knew each other. I'd join an IRC group and have difficulty catching up with what seemed to be the flow of conversation.

I hadn't thought much about this until the pandemic when I started a Discord group for software internals and a virtual tech talk series called Hacker Nights. Since 2021 the Discord reached around 1,500 members and ~20 fairly active members. And the Meetup peaked at about 300 members with about 10-20 showing up each Meetup.

After the pandemic receded I started an NYC-based book club over 2 months with about 5-8 active attendees. I ran a virtual hack week on Discord where I got ~100 devs into a temporary Discord server and we talked about Postgres internals and shared resources. Ultimately around 5 of us wrote blog posts and built new projects to explore Postgres.

I started a virtual, async email book club (that is still ongoing) with 300 devs from November 2023 to Feb 2024. There have been around 20 active members of the club. And each week the discussion is kicked off by one of the members, not myself.

And I felt like there wasn't enough community opportunity for folks in systems programming in NYC so I started an Manhattan-based Systems Coffee Club. Around 15 people showed up to the first meeting and seemed even more excited about it than I was. (And I was excited!) We'll see where it goes from here.

Organizing people to do this stuff doesn't come easy to me. I enjoy doing it to a degree, but every night before an event I have trouble sleeping. Worried about embarrassing myself. When the event happens though, and people are happy to be there to chat with everyone else, as they invariably have been, it makes it worthwhile.

Everyone want community

Something I realized along the way is that people (maybe devs especially, I don't know) are looking for community. And when I have noticed there seems to be a missing flashpoint (a topic, a career focus, a book, etc.) for community, it's been pretty easy to get people together around it.

The lifecycle of groups

Groups, meetups, naturally live and die. Organizers get burnt out. I don't see this as a problem. It's just the way it is.

At some point I'll get burnt out too. Or I'll get pickier. For example, I've been avoiding starting a systems programming meetup in NYC because I know it will be a big effort. So I've done lower effort groups like book clubs and coffee clubs.

Don't worry about signing yourself up for indefinite work. Just do whatever you'd like to and don't feel bad if you have to stop. Someone else will eventually start the next great group, even if it comes in a different medium or flavor.

Community is contagious

There are great communities out there that have inspired me.

Aleksey Charapko's and Murat Demirbas's virtual Distributed Systems Reading Group
Alex Petrov's database paper reading group
Andy Pavlo's database interview series
Paul Butler's BrowserTech meetup
Eric Zhang's New York Systems Reading Group

And this year I've been hearing about more.

TU Munich students started a Student Database Group
A group of developers starting a Türkiye-language CS reading group

There are yet a few more systems programming groups I've heard rumors about being started on the US West Coast and Stockholm.

Do whatever you want!

If you feel like you can't find the right group or that you don't fit in with existing groups or that you're missing a moment, there are surely other folks in the same boat. Waiting for a new group to join. You may be the catalyst.

There's enormous potential for getting people together and doing something interesting and there isn't necessarily anyone telling you you should. Things you try may work and they may not. The more you try the more you'll learn what works and what doesn't. I've had a few years of making mistakes organizing to hone the sense.

The only boring thing to do is to necessarily limit yourself to the sort of thing others have done before! Run a browser meetup instead of a React meetup. Interview hardware developers to teach software developers something. Get software developers with 20 years of experience in niche fields to teach the rest of us something. Read books beyond SICP or Clean Code. Try difficult programming projects.

Whatever you want though, don't let me deter you. If you think something should exist, give it a shot!

I used to struggle to get much out of meetups, couldn't pick up the flow of IRC. Some point I stopped trying solely to fit in. Instead to do what I thought was interesting. And to my surprise, folks were interested in coming along too!

Make your own wayhttps://t.co/tVEa2ndiZm pic.twitter.com/piWSsv14lj
— Phil Eaton (@eatonphil) December 27, 2023

A Frivolous Feature

Weblog on marginalia.nu — Fri, 22 Dec 2023 02:00:00 GMT

Marginalia Search very recently gained the ability to filter results by Autonomous System, not only searching by ASN but by the organization information for that AS. At a glance this seems like a somewhat frivolous feature, but it has interesting effects. Autonomous Systems are part of the Internet’s routing infrastructure. If your mental model of an IP number is that they are the phone number of the computer, this is something akin to a postal code.

Podcast: IaC in depth with Robert Hafner

tedious ramblings — Thu, 21 Dec 2023 16:00:00 GMT

Robert Hafner on The IaC Podcast.

How To Read An Article On The Internet

Weblog on marginalia.nu — Wed, 20 Dec 2023 02:00:00 GMT

A simple guide to reading in 9 simple steps Navigate to the desired article. Dismiss the GDPR banner It may seem safe to start reading, but you need to wait about 10 seconds as the various ad auctions resolve and scripts load in Wait while the article is populated with ads. While the article is in front of you, there is no point to starting to read yet, as the minute’s worth of layout shift will make you lose your place.

WARC'in the crawler

Weblog on marginalia.nu — Wed, 20 Dec 2023 02:00:00 GMT

The Marginalia Crawler has seen improvements! A long term problem with the crawler design is that if for whatever reason the crawler shuts down, then it needs to re-start fetching whatever domains it was currently traversing during the termination from zero. This isn’t fantastic, since not only does crawling a website take a fair bit of time, it’s a nuisance for the server admins to re-crawl stuff that was already fetched, and a real liability for ending up in robots.

Dwell and the Tools for Thought Hackathon

swyx's site RSS Feed — Wed, 13 Dec 2023 22:47:31 GMT

I worked on a little hackathon project at AGI House with Sasha. We won!

Go Libraries/Packages I Like

Ben E. C. Boyter — Wed, 13 Dec 2023 02:00:00 GMT

In no particular order a list of Go libraries/packages I really like and some reasons why.

zerolog I know that the changes to slog make it far closer to being what zero log is now, but I still prefer it for my own use.
requests Anyone who works with python for a while learns about the requests library for making HTTP calls. This is similar but for Go. It has a few sharp edges when dealing with websites that don’t follow standards, but otherwise is amazing. Coupled with the next library in the list you end up with a pretty solid way to call endpoints.
trip HTTP client middleware that allows automated retries, with backoff’s among other things such as authorization. Coupled with requests it really cleans up your HTTP calls.
lo If you know of the wonderful underscore.js library then you probably have an idea of what this is. Similar idea, providing a lot of helper functions that allow dealing with some of Go’s sharp edges more easily. It does abstract away things that you might need to know about if you need absolute performance, but for most things it just makes life easier.
templ HTML template language for Go. There is nothing you can do in this that you cannot in Go templates, but you get compile time checks of the code rather than actually having to run it. It also means you don’t need to worry about embedding template (although this is fairly easy these days anyway). Works really well with HTMX.
gjson A simple way to get values out of JSON without needing to unmarshall into a struct. Really useful for tests, as well as annoying API’s that have mixed data types.
mux The inbuilt Go router isn’t horrible, but is painful to use. This one supports path variables, and you can specify which http methods it uses.
sqlc Generate type safe Go code to access your database. Of all the DB layer abstractions I have used this is best. Its similar to something I wrote in PHP over a decade ago, although much better.
xurls The number of times I have gone looking for a URL regex only to find it misses some edge case… This solves a lot of that pain, especially when it comes to extracting them out of text.

The below are things I am exploring, but not sold on yet.

Using the BlueSky API to change your Handle to your Domain Name

Pxtl.ca — Wed, 13 Dec 2023 02:00:00 GMT

Personally in the post-Muskening twitter world, I'm pulling for Mastodon, but Bluesky is a somewhat interesting platform too. One feature of Bluesky that stands out is that you can set your name to be based on a domain-name you own. I own Pxtl.ca, so I thought I'd take a crack at leveraging that feature.

What follows is a guide on how to do this in Powershell, which is a programming language you already have installed if you use Windows.

Optimizing Ship Load-outs for Highfleet: Problem Setup

jodavaho.io — Wed, 13 Dec 2023 02:00:00 GMT

Over-engineering Highfleet load-outs using Integer Programming

I built a highfleet ship optimizer by bundling a fast C optimizer, binding to a higher level language, and providing a web API. Then, Altho built a front-end. Try it at hfopt.jodavaho.io/optimizer

This post is about setting up the problem. See related posts.

Dieselpunk warship schematic. Made with Midjourney.

Highfleet is an incredibly fun game. You take control of a small set of ships, raid and pillage and revolt your way to glory. Along the way you must carefully manage your resources to keep your ships repaired, and customize them to the situation at hand. Lovely time.

The most addicting part, as is common in this genre¹, is the ship design and customization. You can fully tailor your fleet to your desired play style! Instead of rave about the gameplay, let’s talk about how you design ships.

Ship Design Problem statement

In Highfleet, you can “pre-design” ships before starting a campaign. You have a limited starting budget, so you need to balance power, weight, engines, guns, ammo types, layout, crew capacity, sensing, etc, to get the desired range, speed, firepower, survivability, and combat time, while minimizing cost. This problem is masterfully posed by the developers, and has hooked a small community for years.

I’m interested in automating this, to a certain extent. Can we at least come up with an optimal module set - and leave the layout / cosmetics to the creative user? Staring at the above paragraph, one might see this is a classic optimization problem². Words like “minimize”, “select”, “all while …”, “such that”, etc are clues.

We can re-state above as:

Select zero or more whole modules from a list
Such that the modules have all their requirements satisfied (e.g., engines have fuel, guns have ammo)
all while minimizing cost
and satisfying some desired final ship stats

Easy. Formulating the problem is half the battle.

Solving

To solve a problem like this you need the defining equations:

A “cost” function. In this case, it’s literally the dollar value of each module - easily obtained by writing down the values from the game. This is also called the “Objective Function”.
The “feasibility constraints”, which essentially answer: Is this a valid ship at all? Without these, we could choose modules that seem to satisfy constraints, but would never fly in-game. For example: “Do we have enough ammo feeds to keep the guns working”, this again we can copy down from in-game stats
The “trade space equations”. In this case, we want to know the following: How does the selection of modules affect the desired final ship stats like Range, Speed, Firepower, etc. This turns out to be very hard, but doable with help.
We also want the “desired stats”, which are just “constraints” we impose on the trade space. We’ll get this from the users as a set of valid ranges (e.g., speed is greater than X, weight is less than Y, blah blah)

Great - all we need to do is reverse-engineer how the in-game stats are calculated. easy right? Instead, I reached out to the game developer, and r/highfleet, to see what they could share. Everyone had some models, and Koshutin didn’t reply. So, I had to do it myself for the most part.

Reverse Engineering the Game Stats

For completeness, here is the list of relevant stats, obtained by parsing the ship files (which deserves its own post).

aavalue, airspeed, ammobox_need, ammobox_total, at_left, at_right, b.x, b.y, combatvalue, crew_capacity, crew_need, dynamic, dynamic_landing, dynamic_map, dynamic_map_repaired, firepower_guns, fss_capacity, fss_total, fuel, fuel_capacity, fuel_need, fuel_need_cr, fuel_on, hp, hp_integral, hp_max, mass, parts, parts_integral, power, power_need, power_real, power_total, power_total_repaired, price, r, signature, signature_ir, signature_rd, thrust_left, thrust_left_landing, thrust_left_map, thrust_left_map_repaired, thrust_right, thrust_right_landing, thrust_right_map, thrust_right_map_repaired, weapons_on

Some we can throw right out. b.x, b.y for example make no sense to me. Some (weapons_on) probably have more to do with the campaign mode than the performance. Some (airspeed, r) are exactly what we want to reverse engineer and optimize for.

My explorations were fairly crude. I’d download many of the fan-made ships from reddit or discord, check the stats, make a spreadsheet, and start plotting to see if the relationships were obvious. Some guesses were obvious (airspeed should be a function of thrust/weight, which it was). Some were not (range (r?), in particular).

For forming the equations, some unit analysis really paid off. For example, range ultimately (sensibly) was essentially:

$$ r \approx C_1 \cdot \frac{ \text{fuel_capacity} }{ \text{fuel_consumption} } \cdot \text{airspeed} + C_2 $$

which has units meters- note the unknown constants though³. Determining the constants is a job for linear regression! I won’t plot out the details, you can see them here.

At this repo you can see how it worked. Note this file with ship data, hand crafted! It has a sparses set of stats manually pulled from ships. Just enough to confirm the relationships. Once I had those I could pull up any ship from reddit or discord, and verify it. It was always close enough. Of course, I have to re-do this every time the game balances, but soon, I’ll automate this - another post to write.

from study notebook, showing combat time regression

I did discover some hidden features this way, did you know generators require fuel? Makes sense, but at the time this was undocumented.

So, that’s how I was able to pin down the relationshps for the “desired stats” from the game files. Next, we can formulate the optimizer.

Mr. Koshutin, if you are reading this, thank you for letting the community enjoy your game in this way. You certainly didn’t have to!

There is a pseudo-genre of such games, and highfleet stands alone for its unique setting and fun combat. ↩︎
There are some softer constraints, too: Does it look cool? Does it land well? These are subjective and not well posed for optimization, so I ignore those (for now!). ↩︎
One might imagine that the constants fall out of the in-game stats, but they do not. I suspect this is because “fudge factors” are applied. ↩︎

We don't need no stinking bastion host!

Ben E. C. Boyter — Sun, 10 Dec 2023 02:00:00 GMT

So on occasion, I find myself wanting to run SQL against customer databases. Where this gets annoying is that because we are often in AWS using lambda I have no ability to connect to them. Occasionally im able to bully the friendly devops’ and security people to allow me access to a bastion host, which I can then connect to the SQL database.

However this is often a painful process if allowed at all, and so with permission I sometimes add a special route into the application allowing me to run arbitrary SQL as needed.

Cleaning Up Speech Recognition with GPT

Beetle Space — Thu, 07 Dec 2023 10:00:00 GMT

My friends often ask for use cases for GPT. Below I provide one recent example.

I attend a weekly educational seminar where real estate professionals present the work they do. I take notes on paper, and later transcribe them to my computer, which is a pain. As a result, I …

What I'm up to - December 2023

Philip I. Thomas — Mon, 04 Dec 2023 17:17:11 GMT

This is my monthly newsletter about what I'm up to, which I send in place of social media.

What I did in November

Started writing some essays about online work, dependability, tools, and craft on the Contraption Company site. Subscribe there if you want to follow them. First essays: Booklet's Architecture, One Year of Dependable Software, and The Transition from Indiehacking to Micro Companies.
Had a great food-focused trip through Europe with a friend. I started up north in Oslo, and ended south in Barcelona. I went to some old favorites such as Noma, April, Popl, and Dishoom, and found some new favorites such as Disfrutar and Silo (whose book I read in April). Highlights included jumping in the Oslofjord from a floating sauna, eating a food that's legal in only three countries, walking an F1 track, and seeing the Pope's old residence. I also got to meet customers of Contraption Company products along the way.
Started a Postcard for some photography experiments - here's my favorite so far.
Continuing to work hard on Booklet. A lot of new communities and members joined in November, and I'm working hard on some new features.

Things to share

Articles: Maybe don't drive into Manhattan. A Coder Considers the Waning Days of the Craft. SHARPE is FAANG for defense. Anti-tank Vespa. The Indie Era of Startups.
Podcasts: Tim Cook on Dua Lipa's podcast. Brian Chesky's new playbook. Craig Mod on Longform. Lenny Rachitsky on How I Write.
Music: MØ
Apps: Ruter, the official transportation app for Oslo, is fantastic.
Coffee I'm drinking: Los Pirineos Pacamara – Christmas Edition

Plans for December

Rolling out some new Booklet features.
Working on more essays.
Thawing in California.

Where I'll be

22-23 Dec: Cleveland, OH
23-27 Dec: Joshua Tree, CA
27 Dec - 3 Jan: Los Angeles

If you're reading this, let me know if we overlap!

Bouldering

swyx's site RSS Feed — Wed, 29 Nov 2023 10:16:46 GMT

My first introduction to bouldering was in 2017, in DUMBO Boulders in Brooklyn Bridge Park (now closed):

Festive offers for books on Python, Linux, Regular Expressions and more

learnbyexample — Sat, 25 Nov 2023 02:00:00 GMT

Hello!

Here are some exciting deals for my programming ebooks as well as from other creators.

My ebooks🔗

Offers valid till 30-Nov-2023:

All 13 Books Bundle — $10 (normal price $32)
Learn by example Python bundle — $4 (normal price $15)
Understanding Python re(gex)? — FREE (normal price $10)

Indie creators🔗

Python Problem-Solving Bootcamp — 40% off or purchasing power parity discount whichever is greater
Python books by Michael Driscoll and Teach Me Python Membership — 33% off with black23 discount code
Ebooks on Django and Git — 50% off, plus purchasing power parity if applicable
- see also author's blog post for links to other Django-related deals
The Python Coding Place Membership — 70% off
Python Morsels Membership — lifetime access for the price of 2 years
- see also author's blog post for links to other Python deals
Python To Projects - 5 Week Online Course — 25% off
Python books by Reuven Lerner — 40% off
wizard zines — 50% off on PDFs, 30% off on print versions
Level up with Tailwind CSS and Complete Guide to CSS Flex and Grid — 50% off

Miscellaneous🔗

NoStarch Press — 35% off with DEALS4DAYS code
The Pragmatic Bookshelf — 40% off on all ebooks and audio books
Manning Publications — save 50% when you buy 2 or more MEAPs, eBooks, pBooks, liveProjects, or liveVideos
InfoSec Hack Friday — InfoSec related software/tools
The Cyber Plumber's Lab Guide and Interactive Access — 33% OFF
Leanpub Monthly Sale and Leanpub Weekly Sale — offers for programming books, bundles and courses
Huge list of awesome deals — tools, productivity, books, courses, etc
blackfridaydeals.dev — Hottest Black Friday Deals for Developers
Black Friday Deals — Savings on Tech Books and Courses

Happy learning :)

(fake) Football Leagues

infrequent oscillations — Thu, 23 Nov 2023 14:01:32 GMT

In which we fake some leagues for fun.

Lua Carousel

Kartik Agaram — Thu, 23 Nov 2023 09:00:00 GMT

I finally decided to hang up a shingle on itch.io. My first app there is not a game. Lua Carousel is a lightweight environment for writing small, throwaway Lua and LÖVE programs. With many thanks to Mike Stein who helped me figure out how to get it working on iOS, this is my first truly cross-platform app, working on Windows, Mac, Linux, iOS and Android.

repo

Carousel has its own devlog/notebook. I try to post little scripts that are easy for someone to copy to their clipboard, paste into Carousel and run. Some examples:

Hilbert curve and Lindenmayer tree fractal (70 lines each)
rectangle art (40 lines)
simple example using the physics engine. (100 lines)
Chromacline: a war between colors in Game of Life world (100 lines)
simple UI examples; simple voice recorder (<50 lines each)
sprite editor (100 lines)
Pong wars (70 lines)
equation plotting over an infinite surface with pan and pinch-zoom (150 lines)
Lots of different kinds of charts (200 lines)

Starfield review

ブラック — Wed, 22 Nov 2023 02:00:00 GMT

Mr. Howard's Mild Ride.

The very first impression I had with Starfield was that the world design looked really good. The art direction of Starfield is great. I'm a big fan of the "NASA punk". However, the further I progressed in the game, the less impressed I got with the world. While the individual aspects of it might look good, it's way too repetitive. You land on a single wasteland planet and you've really landed on all of them. And as far as I can tell, the few points of interest in them are repeated throughout the known universe.

The cities are pathetic too. There's a couple of different cities with vastly different designs. They do however have one thing in common: the scale. All of the cities feel woefully small. It feels like I'm walking through a movie set than an actual city. And despite the lack of scale, there's loading screens absolutely everywhere. Get out of your space ship – loading screen. Go to a small residential area – loading screen. Go to the weapon store inside the small residential area – loading screen. Thankfully the missions aren't as full of loading screens but it seems strange that these small cities are in a big-budget 2023 release.

Traversal between places is also deeply unsatisfying. I had to go from a city on a planet, to a bandit camp on the same planet. I opened the extremely poor planet map, selected the bandit camp, and tried to transport there. Sorry, the game said, you can't fast travel to a place you haven't discovered yet. Then it forced me to fast travel to my spaceship and then land my spaceship at the bandit camp. And what does "landing your spaceship" actually entail? A loading screen and a jump cut to the spaceship landing, which all takes around 10 seconds. Almost all traversal in this game is fast travel and yet it still does this idiotic gatekeeping for what it thinks "fast travel" is.

The space bit in Starfield doesn't really improve the situation at all. You can either talk to whatever NPCs happen to be around you, or engage them in battle. That's it. Despite the lack of anything, you'll still be going to space very often since you need to spend at least a few seconds there between fast travels. Actually, the NPCs either reaching out or attacking got very tiresome since I just wanted to travel to my destination and not play tag with pirates.

The repetition in the world design also comes across in the story campaign. The story had me go look for Andreja, and I found her in an enemy-infested mine, and at the end of the mine, I found an artefact. Then later in the story, I was tasked to go look for more artefacts, one of which was at the end of an enemy-infested mine. A mine with the exact same layout, except this one was on a different planet and lacked one Andreja. Both of these mines also had a locked locker room, with contraband for the taking after you picked open the lock. While I can understand that the planets look similar when you have such a vast number of them, I expected a lot more from the main story locations in a AAA game.

Really feels like Bethesda is just taking the piss with the repetition considering missions are actually named "Into the Unknown" and "Further Into the Unknown". A bit on the nose there Todd.

As for the story itself, I didn't really care for it. First of all, I found the premise of how you became an adventurer quite contrived. And while I think they want to make Constellation feel like some sort of an elite gentleman explorer's club, most of the story missions made me feel like their glorified errand girl instead. For every interesting mission, like Unearthed or Entangled, there were two or three missions where I just had to fly from one remote planet to another, literally digging for quest items. Surely there must've been a more interesting story to write in the Starfield universe, right? And despite the fact that the story involves life and death situations, the writing is just so devoid of any edge or character. The whole thing just feels like such a lifeless and watered down experience.

Guess it didn't help that I couldn't care for any of the characters, including my own. The only (human) member of Constellation that I didn't have any active contempt towards was Walter. Vasco was cool too, and got promoted to my primary companion after Sarah got very angry at me for a heinous murder where I shot a man after he and his friends unloaded every bullet they had towards our direction. I guess Sarah was looking to end her miserable existence and I rudely interrupted it by defending myself.

I've heard that some of the side content is much superior but unfortunately I seem to have never encountered it. I guess I did mainly focus on the main quest so maybe that's on me. However, I did encounter side content that I didn't care for.

One example is a mission where settlers task me to drive out a corp from a planet that they consider their birthright and one that they seem willing to fight for. I go talk to the corp and they want me to either blow up the settlers or convince them to become serfs on their luxury resort planet. I decided to not blow up these poor settlers and instead opt to tell them about this indentured servitude proposal. Considering how defiant the settler captain acted, I expected this to lead into a war where they take the planet by force. Nope, the settlers turn heel and agree to become slaves. They they asked me to bring them a bunch of materials, presumably so they can make their own leg irons to really enjoy their bondage. I considered blowing them up for their own good but settled to just leave instead.

I did enjoy the ending though, at least on a conceptual and storytelling level. Unfortunately the gameplay side of it is extremely herky-jerky, and is not at all enhanced by the space combat or waves of trash mobs you also have to deal with. And right at the last second, the story really just raises more questions than it answers. At least the ending was pretty.

As a game, Starfield is a rather unfocused affair. It's so chock-full of different systems that it feels very overwhelming when you get started. Maybe veterans to Bethesda RPGs have already learned to juggle these systems in prior games, but as a newcomer, you need to ingest a lot of (unhelpful) tutorials at the beginning. You have ground traversal, space traversal, ground combat, ship combat, ship building, outpost building, cooking, crafting, spacesuit upgrading, weapon customisation, mystical powers, lockpicking, persuasion, scanning, mining, stealth, encumbrance, commerce, status effects and so on. "A mile wide and an inch deep" is the best quote I have heard to describe this game. Many of these systems are very badly taught to you, and even when you learn them, they're still not that great. Quite possibly why it also took me around ten hours to actually get into the game properly.

The ground combat feels fine. There's a decent selection of firearms available to you and they do feel pretty distinct from each other. It's definitely not very flashy, and there's very limited amount of specialisation you can do with the combat but I did generally enjoy engaging the enemy. The boost pack also livens up the movement for the combat sections too.

The mystical space powers feel like a complete waste of time, at least for enhancing the combat. I really only ever used the unlimited stamina power and otherwise ignored them. And the way that you obtain all of these various powers is so needlessly repetitive. You have to literally jump through hoops for them, again and again and again. It feels more like an elaborate prank than game design.

Space combat is less fine. First of all, whenever you engage in ship combat, you were most likely just on your way to do something completely different until someone decided to harass you. And when they do, they usually do it with their pals. In three-dimensional space. I feel like most of the time in combat is spent around doing loops in order to just find one of the three hostile ships so you can blast them, while trying to keep their two friends from doing the same to you. Even though I rarely struggled, I still did not enjoy it.

Ship building seemed pretty cool, at least for someone that enjoys that kind of a thing. I mainly used it pretty pragmatically, so instead of trying to replicate famous designs or come up with my own, I just used it to upgrade an existing design. It does take some getting used to, since the game doesn't really teach you how to do it and I had to get supplementary lessons from Google on how I can get ship pieces to connect how I want. And unfortunately changing your ship in any way completely destroys any interior decor you might have going on (for understandable technical reasons), so it definitely kept me from doing it just for fun. Maybe people that are heavily into this would find more to criticise about it but I thought that the ship building was cool overall.

Outpost building is also something that is not extremely well taught to you. And it doesn't really feel like you would ever need to do it. However, once I learned a bit on how it works, I got a frenzy of finding resource-heavy planets, building outposts on them all, and connecting them all together to a master outpost. I got pretty absorbed into it until I had all of my outposts connected. Then I realised that I didn't really have any kind of use for it. It just kinda is there, filling up my inventory and few resource crates with a bunch of minerals that I don't really have an use for. I did my outpost fairly early in my run and had no reason to touch it afterwards.

Inventory management is also subpar. For some reason Bethesda decided that the majority of the screen should be taken by 3D representations of your items instead of useful information. While the gun models look nice, I don't find myself that interested in knowing what copper ore looks like. I'd much rather have a side-by-side view of my inventory and a vendor's inventory instead of having to flip between two or three different inventories as I unload my stuff. The vendors will also instantly run out of money when you go sell things, so you need to go do your tedious inventory management with every single vendor in the game. At least I did get used to the inventory, so the user interface is not a constant source of pain, but I still pine for something better.

The lockpicking minigame is actually rather fun but it takes home the prize for the worst explained system in the game. I had no clue on how to do it and was just randomly trying to get through it. Then I watched a short tutorial video on YouTube and got it. Now I'm pretty good at picking open the locks and can actually enjoy some momentary tinkering with locks. What a shame the game doesn't want to teach you about it.

Not a big fan of character creation either. It feels like some of the aspects of it are very varied and some are barely varied at all. I would've also enjoyed having a voice for my character. There's one part in the story that actually makes me feel like it would've been very reasonable and feasible thing even. And I really don't understand the point of the traits. I remember people criticising the lifepaths in Cyberpunk 2077 for having little impact on the game, but I feel like the traits in Starfield have even less of an impact. I was a Neon Street Rat and it just gave me a couple of new choices in one particular part of the game.

Don't really like the skill trees either, especially how it locks some basic abilities in the game behind a skill point. Or many skill points for things in the advanced tiers of the skill tree. I'd expect to be able to do at least some stealth without investing points to a stealth skill but alas. You also need to level up your skills, so don't even think about reaching rank 4 in Medicine without spending time harming yourself and then ingesting health kits by the dozen.

Gatekeeping parts of the game by hiding them in the skill tree are also why I never really got into the crafting and upgrading aspect of the game. You have a few things open at the start of the game but anything beyond that requires investing your few skill points. A shame, since I would've liked to upgrade my weapons more, but Weapon Engineering is an advanced Science skill, so I'd need to invest and level up my science skills before I even got a single point in.

Technical aspects of the game are a bit of a mixed bag. The game was surprisingly stable and bug-free for a new AAA release, although neither aspect was perfect. My crew sank into my spaceship more times than I remember to count and when the game decided to crash, it screwed me out of a lot of progress. At least Cyberpunk 2077 was polite enough autosave very frequently to make up for its lack of stability, and wouldn't make me lose 30 minutes of gameplay with a single hard crash. The performance seems relatively poor considering the lack of ray tracing. Considering how Cyberpunk 2077 looks a lot better with crazy ray tracing, Starfield doesn't deliver that much of a higher frame rate.

It does seem like they did manage to improve it a bit with the recent patch though, but I've yet to clock any serious amount of time in it to make a judgement. Fingers crossed!

I tend to avoid pre-release marketing since I want to temper my expectations for new stuff, but I'm still left feeling disappointed. It's not that Starfield is an awful game. I did manage to beat it after all. It just feels really mediocre and undaring. Perhaps it's worth taking a stab at if you were already paying for Game Pass, or found it in a bargain bin, but for its pain points and overwhelming sense of mediocrity, it's easy to skip and hard to recommend.

The Knight Capital Disaster

Speculative Branches — Wed, 22 Nov 2023 02:00:00 GMT

This account comes from several publicly available sources as well as accounts from insiders who worked at Knight Capital Group at the time of the issue. I am telling it second- or third-hand.

On August 1, 2012, Knight Capital fell on its sword. It experienced a software glitch that literally bankrupted the company. Between 9:30 am and 10:15 am EST, the employees of Knight capital watched in disbelief and scrambled to figure out what went wrong as the company acquired massive long and short positions, largely concentrated in 154 stocks, totaling 397 million shares and $7.65 billion. At 10:15, the kill switch was flipped, stopping the company's trading operations for the day. By early afternoon, many of Knight Capital's employees had already sent out resumes, expecting to be unemployed by the end of the week.

Wild Speculation Regarding the OpenAI Fiasco

Tilting at Windmills — Tue, 21 Nov 2023 02:00:00 GMT

First, I feel I need to make one thing clear: I'm nobody and have no special inside information. I'm just a random terminally-online guy who…

Open source, locally hosted AI powered Siri replacement

smcleod.net — Mon, 20 Nov 2023 03:00:00 GMT

Using an ESP32 S3 Box 3 with Willow, Home Assistant and Large Language Models (LLMs) create a locally hosted, offline, AI powered Siri / Alexa replacement.

Exploring a Postgres query plan

Notes on software development — Sun, 19 Nov 2023 02:00:00 GMT

I learned this week that you can intercept and redirect Postgres query execution. You can hook into the execution layer so you're given a query plan and you get to decide what to do with it. What rows to return, if any, and where they come from.

That's very interesting. So I started writing code to explore execution hooks. However, I got stuck interpreting the query plan. Either there's no query plan walking infrastructure or I just didn't find it.

So this post is a digression into walking a Postgres query plan. By the end we'll be able to run psql -c 'SELECT a FROM x WHERE a > 1' and reconstruct the entire SQL string from a Postgres QueryDesc object, the query plan object Postgres builds.

With that query plan walking infrastructure in place, we'll be in a good state to not just print out the query plan while walking it but instead to translate the query plan or evaluate it in our own way (e.g. over column-wise data, or vectorized execution over row-wise data).

Code for this project is available on Github.

What is a query plan?

If you're familiar with parsers and compilers, a query plan is like an intermediate representation (IR) of a program. It is not as raw as an abstract syntax tree (AST); it has already been optimized.

If that doesn't mean anything to you, think of a query plan as a structured and optimized version of the SQL query you submit to your database. It isn't text anymore. It is probably a tree.

Check out another Justin Jaffray article on the subject for more detail.

Development environment

Before we get to walking the query plan, let's set up the infrastructure to intercept query execution where we can eventually add in our print debugging of the query plan reconstructed as a SQL string.

Once you've got Postgres build dependencies, build and install a debug version of Postgres:

$ git clone https://github.com/postgres/postgres && cd postgres
$ # Make sure you're on the same commit I'm on, just to be safe.
$ git checkout b218fbb7a35fcf31539bfad12732038fe082a2eb
$ ./configure --enable-cassert --enable-debug CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer"
$ make -j8
$ # Installs to to /usr/local/pgsql/bin.
$ sudo make install

I'm not going to cover Postgres extension infrastructure in detail. I wrote a bit about it in my last post. You need only read the first half, if at all; not the actual Table Access Method implementation.

It will be even simpler in this post because Postgres hooks are extensions but not extensions you install with CREATE EXTENSION. If you want to read about the different kinds of Postgres extensions, check out this article by Steven Miller.

The minimum we need, aside from the hook code itself, is a Makefile that uses PGXS:

MODULES = pgexec

PG_CONFIG = /usr/local/pgsql/bin/pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

The MODULES value there corresponds to the C file we'll create shortly, pgexec.c.

This pg_config binary path is important because you might have different versions of Postgres installed, for example by your package manager. It is important that the extension is built against the same version of Postgres which will load the extension.

Now we're ready for some hook code.

Intercepting query execution

You can find the basic structure of a hook (and which hooks are available) in Tamika Nomara's unofficial Postgres hooks docs.

There is no official central place describing all hooks I could find in Postgres docs. Some hooks are described in various places throughout the docs though.

Based on that page, we can write a bare minimum hook that will intercept queries, log when we've done so, and pass control back to the standard execution path for the actual query. In pgexec.c:

#include "postgres.h"
#include "fmgr.h"
#include "executor/executor.h"

PG_MODULE_MAGIC;

static ExecutorRun_hook_type prev_executor_run_hook = NULL;

static void print_plan(QueryDesc* queryDesc) {
  elog(LOG, "[pgexec] HOOKED SUCCESSFULLY!");
}

static void pgexec_run_hook(
  QueryDesc* queryDesc,
  ScanDirection direction,
  uint64 count,
  bool execute_once
) {
  print_plan(queryDesc);
  return prev_executor_run_hook(queryDesc, direction, count, execute_once);
}

void _PG_init(void) {
  prev_executor_run_hook = ExecutorRun_hook;
  if (prev_executor_run_hook == NULL) {
    prev_executor_run_hook = standard_ExecutorRun;
  }
  ExecutorRun_hook = pgexec_run_hook;
}

void _PG_fini(void) {
  ExecutorRun_hook = prev_executor_run_hook;
}

You can discover the standard_ExectutorRun function from a quick git grep ExecutorRun_hook in the Postgres source which leads to src/backend/executor/execMain.c#L306:

void
ExecutorRun(QueryDesc *queryDesc,
            ScanDirection direction, uint64 count,
            bool execute_once)
{
    if (ExecutorRun_hook)
        (*ExecutorRun_hook) (queryDesc, direction, count, execute_once);
    else
        standard_ExecutorRun(queryDesc, direction, count, execute_once);
}

So our hook will just log and pass back execution to the existing execution hook. Let's build and install the extension.

$ make
$ sudo make install

Now we'll create a new database and tell it to load the extension.

$ /usr/local/pgsql/bin/initdb test-db
$ echo "shared_preload_libraries = 'pgexec'" > test-db/postgresql.conf

Remember, hooks are not CREATE EXTENSION extensions. As far as I can tell they can't be dynamically loaded (without some additional dynamic loading infrastructure one could potentially write). So every time you make a change you need to rebuild the extension, reinstall it, and restart the Postgres server.

And start the server in the foreground:

$ /usr/local/pgsql/bin/postgres \
  --config-file=$(pwd)/test-db/postgresql.conf \
  -D $(pwd)/test-db
  -k $(pwd)/test-db
2023-11-18 19:35:16.680 GMT [3215547] LOG:  starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1), 64-bit
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv6 address "::1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-11-18 19:35:16.682 GMT [3215550] LOG:  database system was shut down at 2023-11-18 19:20:16 GMT
2023-11-18 19:35:16.684 GMT [3215547] LOG:  database system is ready to accept connections

Keep an eye on this foreground process since this is where elog(LOG, ...) calls will show up.

Now in a new window, create a test.sql script that we can use to exercise the hook:

DROP TABLE IF EXISTS x;
CREATE TABLE x (a INT);
INSERT INTO x VALUES (309);
SELECT a FROM x WHERE a > 1;

Run psql so we can trigger the hook:

$ /usr/local/pgsql/bin/psql -h localhost postgres -f test.sql
DROP TABLE
CREATE TABLE
INSERT 0 1
  a
-----
 309
(1 row)

And in the postgres foreground process you should see a log:

2023-11-19 17:42:03.045 GMT [3242321] LOG:  [pgexec] HOOKED SUCCESSFULLY!
2023-11-19 17:42:03.045 GMT [3242321] STATEMENT:  INSERT INTO x VALUES (309);
2023-11-19 17:42:03.045 GMT [3242321] LOG:  [pgexec] HOOKED SUCCESSFULLY!
2023-11-19 17:42:03.045 GMT [3242321] STATEMENT:  SELECT a FROM x WHERE a > 1;

That's our hook! Interestingly only the INSERT and SELECT statements show up, not the DROP and CREATE.

Now let's see if we can reconstruct the query text from that first argument, the QueryDesc* that pgexec_run_hook receives. And let's simplify things for ourselves and only worry about reconstructing a SELECT query.

`Node`s and `Datum`s

But first, let's talk about two fundemental ways data in Postgres (code) is organized.

Postgres code is extremely dynamic and, maybe relatedly, fairly object-oriented. Almost every entity in Postgres is a Node. While values in Postgres that are exposed to users of Postgres are Datums.

Each node has a type, NodeTag, that we can switch on to decide what to do. In contrast, Datum has no type. The type of the Datum must be known by context before using one of the transform functions like DatumGetBool to retrieve a C value from a Datum.

A table is a Node. A query plan is a Node. A sequential scan is a Node. A join is a Node. A literal in a query is a Node. The value for the literal in a query is a Datum.

Here is how The Internals of PostgreSQL book visualizes a query plan for example:

Every box in that image is a Node.

And all Nodes in code I've seen share a common definition prefix like this:

typedef struct SomeThing {
  pg_node_attr(abstract) // If the node is indeed abstract in the OOP sense.
  NodeTag type;
}

Many Nodes you'll see are abstract, like Plan. But by printing out NodeTag and checking the value printed in src/include/nodes/nodetags.h, you can find the concrete type of the Node.

src/include/nodes/nodetags.h is generated during a preprocessing step. (Don't look if regex in Perl worries you).

We'll get back to Nodes later.

What's in a `QueryDesc`?

Let's take a look at the QueryDesc struct:

typedef struct QueryDesc
{
    /* These fields are provided by CreateQueryDesc */
    CmdType     operation;      /* CMD_SELECT, CMD_UPDATE, etc. */
    PlannedStmt *plannedstmt;   /* planner's output (could be utility, too) */
    const char *sourceText;     /* source text of the query */
    Snapshot    snapshot;       /* snapshot to use for query */
    Snapshot    crosscheck_snapshot;    /* crosscheck for RI update/delete */
    DestReceiver *dest;         /* the destination for tuple output */
    ParamListInfo params;       /* param values being passed in */
    QueryEnvironment *queryEnv; /* query environment passed in */
    int         instrument_options; /* OR of InstrumentOption flags */

    /* These fields are set by ExecutorStart */
    TupleDesc   tupDesc;        /* descriptor for result tuples */
    EState     *estate;         /* executor's query-wide state */
    PlanState  *planstate;      /* tree of per-plan-node state */

    /* This field is set by ExecutorRun */
    bool        already_executed;   /* true if previously executed */

    /* This is always set NULL by the core system, but plugins can change it */
    struct Instrumentation *totaltime;  /* total time spent in ExecutorRun */
} QueryDesc;

The PlannedStmt field looks interesting. Let's take a look:

typedef struct PlannedStmt
{
    pg_node_attr(no_equal, no_query_jumble)

    NodeTag     type;

    CmdType     commandType;    /* select|insert|update|delete|merge|utility */

    uint64      queryId;        /* query identifier (copied from Query) */

    bool        hasReturning;   /* is it insert|update|delete RETURNING? */

    bool        hasModifyingCTE;    /* has insert|update|delete in WITH? */

    bool        canSetTag;      /* do I set the command result tag? */

    bool        transientPlan;  /* redo plan when TransactionXmin changes? */

    bool        dependsOnRole;  /* is plan specific to current role? */

    bool        parallelModeNeeded; /* parallel mode required to execute? */

    int         jitFlags;       /* which forms of JIT should be performed */

    struct Plan *planTree;      /* tree of Plan nodes */

    List       *rtable;         /* list of RangeTblEntry nodes */

    List       *permInfos;      /* list of RTEPermissionInfo nodes for rtable
                                 * entries needing one */

    /* rtable indexes of target relations for INSERT/UPDATE/DELETE/MERGE */
    List       *resultRelations;    /* integer list of RT indexes, or NIL */

    List       *appendRelations;    /* list of AppendRelInfo nodes */

    List       *subplans;       /* Plan trees for SubPlan expressions; note
                                 * that some could be NULL */

    Bitmapset  *rewindPlanIDs;  /* indices of subplans that require REWIND */

    List       *rowMarks;       /* a list of PlanRowMark's */

    List       *relationOids;   /* OIDs of relations the plan depends on */

    List       *invalItems;     /* other dependencies, as PlanInvalItems */

    List       *paramExecTypes; /* type OIDs for PARAM_EXEC Params */

    Node       *utilityStmt;    /* non-null if this is utility stmt */

    /* statement location in source string (copied from Query) */
    int         stmt_location;  /* start location, or -1 if unknown */
    int         stmt_len;       /* length in bytes; 0 means "rest of string" */
} PlannedStmt;

The struct Plan* planTree field in there looks like what we'd want. But Plan is abstract:

typedef struct Plan
{
    pg_node_attr(abstract, no_equal, no_query_jumble)

    NodeTag     type;

So let's try printing out the planTree->type field and find the Node it is concretely. In pgexec.c change the definition of print_plan:

static void print_plan(QueryDesc* queryDesc) {
  elog(LOG, "[pgexec] HOOKED SUCCESSFULLY! %d", queryDesc->plannedstmt->planTree->type);
}

Rebuild and reinstall the extension, and restart Postgres:

$ make
$ sudo make install
$ /usr/local/pgsql/bin/postgres \
  --config-file=$(pwd)/test-db/postgresql.conf \
  -D $(pwd)/test-db
  -k $(pwd)/test-db
2023-11-18 19:35:16.680 GMT [3215547] LOG:  starting PostgreSQL 17devel on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1), 64-bit
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv6 address "::1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-11-18 19:35:16.681 GMT [3215547] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-11-18 19:35:16.682 GMT [3215550] LOG:  database system was shut down at 2023-11-18 19:20:16 GMT
2023-11-18 19:35:16.684 GMT [3215547] LOG:  database system is ready to accept connections

And in another window run psql:

$ /usr/local/pgsql/bin/psql -h localhost postgres -f test.sql

And check the logs from the postgres process we just started and you should notice:

2023-11-19 17:46:18.834 GMT [3242495] LOG:  [pgexec] HOOKED SUCCESSFULLY! 322
2023-11-19 17:46:18.834 GMT [3242495] STATEMENT:  SELECT a FROM x WHERE a > 1;

So 322 is the NodeTag for the Plan. If we look that up in Postgres's src/include/nodes/nodetags.h (remember, this is generated after ./configure && make so I can't link you to it):

$ grep ' = 322' src/include/nodes/nodetags.h
        T_SeqScan = 322,

Hey, that makes sense! A SELECT without any indexes definitely sounds like a sequential scan!

Walking a sequential scan

Let's take a look at the SeqScan struct:

typedef struct SeqScan
{
    Scan        scan;
} SeqScan;

Ok, that's not very interesting. Let's look at Scan then:

typedef struct Scan
{
    pg_node_attr(abstract)

    Plan        plan;
    Index       scanrelid;      /* relid is index into the range table */
} Scan;

That's interesting! scanrelid represents the table we're scanning. I don't know what "range table" means exactly. But there was a field on the PlannedStmt called rtable that seems relevant.

rtable was described as a List of RangeTblEntry nodes. And browsing around the file where List is defined we can see some nice methods for working with Lists, like list_length().

Let's print out the scanrelid and let's check out the length of the rtable and see if it's filled out. Let's also restrict our print_plan code to only look at SeqScan nodes. In pgexec.c:

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;

  elog(LOG, "[pgexec] relid: %d, rtable length: %d", scan->scan.scanrelid, list_length(queryDesc->plannedstmt->rtable));
}

Rebuild and reinstall the extension, and restart Postgres. (You can find the instructions for this above if you've forgotten.) Re-run the test.sql script. And check the Postgres server logs. You should see:

2023-11-19 18:00:34.184 GMT [3244438] LOG:  [pgexec] relid: 1, rtable length: 1
2023-11-19 18:00:34.184 GMT [3244438] STATEMENT:  SELECT a FROM x WHERE a > 1;

Awesome! So rtable does have data in it. There's only one table in this query so its length makes sense to be 1. The scanrelid being 1 also though is weird. Let's fetch the nth value from the rtable list using scanrelid-1 as the index.

For the RangeTblEntry itself, let's take a look:

typedef enum RTEKind
{
    RTE_RELATION,               /* ordinary relation reference */
    RTE_SUBQUERY,               /* subquery in FROM */
    RTE_JOIN,                   /* join */
    RTE_FUNCTION,               /* function in FROM */
    RTE_TABLEFUNC,              /* TableFunc(.., column list) */
    RTE_VALUES,                 /* VALUES (<exprlist>), (<exprlist>), ... */
    RTE_CTE,                    /* common table expr (WITH list element) */
    RTE_NAMEDTUPLESTORE,        /* tuplestore, e.g. for AFTER triggers */
    RTE_RESULT,                 /* RTE represents an empty FROM clause; such
                                 * RTEs are added by the planner, they're not
                                 * present during parsing or rewriting */
} RTEKind;

typedef struct RangeTblEntry
{
    pg_node_attr(custom_read_write, custom_query_jumble)

    NodeTag     type;

    RTEKind     rtekind;        /* see above */

    /*
     * XXX the fields applicable to only some rte kinds should be merged into
     * a union.  I didn't do this yet because the diffs would impact a lot of
     * code that is being actively worked on.  FIXME someday.
     */

    /*
     * Fields valid for a plain relation RTE (else zero):
     *
     * rellockmode is really LOCKMODE, but it's declared int to avoid having
     * to include lock-related headers here.  It must be RowExclusiveLock if
     * the RTE is an INSERT/UPDATE/DELETE/MERGE target, else RowShareLock if
     * the RTE is a SELECT FOR UPDATE/FOR SHARE target, else AccessShareLock.
     *
     * Note: in some cases, rule expansion may result in RTEs that are marked
     * with RowExclusiveLock even though they are not the target of the
     * current query; this happens if a DO ALSO rule simply scans the original
     * target table.  We leave such RTEs with their original lockmode so as to
     * avoid getting an additional, lesser lock.
     *
     * perminfoindex is 1-based index of the RTEPermissionInfo belonging to
     * this RTE in the containing struct's list of same; 0 if permissions need
     * not be checked for this RTE.
     *
     * As a special case, relid, relkind, rellockmode, and perminfoindex can
     * also be set (nonzero) in an RTE_SUBQUERY RTE.  This occurs when we
     * convert an RTE_RELATION RTE naming a view into an RTE_SUBQUERY
     * containing the view's query.  We still need to perform run-time locking
     * and permission checks on the view, even though it's not directly used
     * in the query anymore, and the most expedient way to do that is to
     * retain these fields from the old state of the RTE.
     *
     * As a special case, RTE_NAMEDTUPLESTORE can also set relid to indicate
     * that the tuple format of the tuplestore is the same as the referenced
     * relation.  This allows plans referencing AFTER trigger transition
     * tables to be invalidated if the underlying table is altered.
     */
    Oid         relid;          /* OID of the relation */
    char        relkind;        /* relation kind (see pg_class.relkind) */
    int         rellockmode;    /* lock level that query requires on the rel */
    struct TableSampleClause *tablesample;  /* sampling info, or NULL */
    Index       perminfoindex;

In SELECT a FROM x, x should be a plain relation RTE (to use the terminology there). So we can add a guard that validates that. But we don't get a Relation. (You might remember from my previous post that Relation is where we can finally see the table name.)

We get an Oid for the Relation. So we need to find a way to lookup a Relation from an Oid. And by grepping around in Postgres (or via judicious use of ChatGPT, I confess), we can notice RelationIdGetRelation that takes an Oid and returns a Relation. Notice also that the comment says we should close the relation when we're done with RelationClose.

So putting it altogether (and again, reusing some code from that previous post), we can print out the table name.

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  RangeTblEntry* rte = NULL;
  Relation relation = {};
  char* tablename = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;
  rte = list_nth(queryDesc->plannedstmt->rtable, scan->scan.scanrelid-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[pgexec] Unsupported FROM type: %d.", rte->rtekind);
    return;
  }

  relation = RelationIdGetRelation(rte->relid);
  tablename = NameStr(relation->rd_rel->relname);

  elog(LOG, "[pgexec] SELECT [todo] FROM %s", tablename);

  RelationClose(relation);
}

You'll also need to add a new #include for utils/rel.h.

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 18:36:03.986 GMT [3246777] LOG:  [pgexec] SELECT [todo] FROM x
2023-11-19 18:36:03.986 GMT [3246777] STATEMENT:  SELECT a FROM x WHERE a > 1;

Fantastic! Before we get into walking the SELECT columns and the (optional) WHERE clause, let's do some quick refactoring.

A string builder

Let's add a little string builder library so we can emit a single string we build up to a single elog() call.

I wrote this ahead of time and won't explain it here since the details aren't relevant.

Just copy this and paste near the top of pgexec.c:

typedef struct {
  char* mem;
  size_t len;
  size_t offset;
} PGExec_Buffer;

static void buffer_init(PGExec_Buffer* buf) {
  buf->offset = 0;
  buf->len = 8;
  buf->mem = (char*)malloc(sizeof(char) * buf->len);
}

static void buffer_resize_to_fit_additional(PGExec_Buffer* buf, size_t additional) {
  char* new = {};
  size_t newsize = 0;

  Assert(additional >= 0);

  if (buf->offset + additional < buf->len) {
    return;
  }

  newsize = (buf->offset + additional) * 2;
  new = (char*)malloc(sizeof(char) * newsize);
  Assert(new != NULL);
  memcpy(new, buf->mem, buf->len * sizeof(char));
  free(buf->mem);
  buf->len = newsize;
  buf->mem = new;
}

static void buffer_append(PGExec_Buffer*, char*, size_t);

static void buffer_appendz(PGExec_Buffer* buf, char* c) {
  buffer_append(buf, c, strlen(c));
}

static void buffer_append(PGExec_Buffer* buf, char* c, size_t chars) {
  buffer_resize_to_fit_additional(buf, chars);
  memcpy(buf->mem + buf->offset, c, chars);
  buf->offset += chars;
}

static void buffer_appendf(
  PGExec_Buffer *,
  const char* restrict,
  ...
) __attribute__ ((format (gnu_printf, 2, 3)));

static void buffer_appendf(PGExec_Buffer *buf, const char* restrict fmt, ...) {
  // First figure out how long the result will be.
  size_t chars = 0;
  va_list arglist;
  va_start(arglist, fmt);
  chars = vsnprintf(0, 0, fmt, arglist);
  Assert(chars >= 0); // TODO: error handling.

  // Resize to fit result.
  buffer_resize_to_fit_additional(buf, chars);

  // Actually do the printf into buf.
  va_end(arglist);
  va_start(arglist, fmt);
  chars = vsprintf(buf->mem + buf->offset, fmt, arglist);
  Assert(chars >= 0); // TODO: error handling.
  buf->offset += chars;
  va_end(arglist);
}

static char* buffer_cstring(PGExec_Buffer* buf) {
  char zero = 0;
  const size_t prev_offset = buf->offset;

  if (buf->offset == buf->len) {
    buffer_append(buf, &zero, 1);
    buf->offset--;
  } else {
    buf->mem[buf->offset] = 0;
  }

  // Offset should stay the same. This is a fake NULL.
  Assert(buf->offset == prev_offset);

  return buf->mem;
}

static void buffer_free(PGExec_Buffer* buf) {
  free(buf->mem);
}

Next we'll modify print_plan() in pgexec.c to use it, and add stubs for printing the SELECT columns and WHERE clauses.

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  buffer_appendz(buf, " [where todo]");
}

static void buffer_print_select_columns(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  buffer_appendz(buf, "[columns todo]");
}

static void print_plan(QueryDesc* queryDesc) {
  SeqScan* scan = NULL;
  RangeTblEntry* rte = NULL;
  Relation relation = {};
  char* tablename = NULL;
  Plan* plan = queryDesc->plannedstmt->planTree;
  PGExec_Buffer buf = {};

  if (plan->type != T_SeqScan) {
    elog(LOG, "[pgexec] Unsupported plan type.");
    return;
  }

  scan = (SeqScan*)plan;
  rte = list_nth(queryDesc->plannedstmt->rtable, scan->scan.scanrelid-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[pgexec] Unsupported FROM type: %d.", rte->rtekind);
    return;
  }

  buffer_init(&buf);

  relation = RelationIdGetRelation(rte->relid);
  tablename = NameStr(relation->rd_rel->relname);

  buffer_appendz(&buf, "SELECT ");
  buffer_print_select_columns(&buf, queryDesc, plan);
  buffer_appendf(&buf, " FROM %s", tablename);
  buffer_print_where(&buf, queryDesc, plan);

  elog(LOG, "[pgexec] %s", buffer_cstring(&buf));

  RelationClose(relation);
  buffer_free(&buf);
}

Now we just need to implement the buffer_print_where and buffer_print_select_columns functions and our walking infrastructure will be done! For now. :)

Walking the `WHERE` clause

If you remember back to the SeqScan and Scan nodes, they were both basically empty. They had a Plan and a scanrelid. So the rest of the SELECT info must be in the Plan since it wasn't in the Scan.

Let's look at Plan again. One part that stands out is:

    /*
     * Common structural data for all Plan types.
     */
    int         plan_node_id;   /* unique across entire final plan tree */
    List       *targetlist;     /* target list to be computed at this node */
    List       *qual;           /* implicitly-ANDed qual conditions */
    struct Plan *lefttree;      /* input plan tree(s) */
    struct Plan *righttree;
    List       *initPlan;       /* Init Plan nodes (un-correlated expr
                                 * subselects) */

qual kinda looks like a WHERE clause. (And targetlist kinda looks like the columns the SELECT pulls).

Lists just contain void pointers, so we can't tell what the type of qual or targetlist children are. But I'm going to make a wild guess they are Nodes.

There's even a nice helper that casts void pointers to Node* and pulls out the type, nodeTag().

And reading around pg_list.h shows some interesting helper utilities like foreach that we can use to iterate the list.

Let's try printing out the type of qual's members.

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  ListCell* cell = NULL;
  bool first = true;
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  foreach(cell, plan->qual) {
    if (!first) {
      buffer_appendz(buf, " AND ");
    }

    first = false;
    buffer_appendf(buf, "[node: %d]", nodeTag(lfirst(cell)));
  }
}

Notice any vestiges of LISP?

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:17:00.879 GMT [3250850] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [node: 15]
2023-11-19 19:17:00.879 GMT [3250850] STATEMENT:  SELECT a FROM x WHERE a > 1;

Well, our code didn't crash! So the guess about qual List entries being Nodes seems right. Let's look up that node type in the Postgres repo:

$ grep ' = 15,' src/include/nodes/nodetags.h
        T_OpExpr = 15,

Woot! That is exactly what I'd expect the WHERE clause here to be.

Now that we know qual is a List of Nodes, let's do a bit of refactoring since targetlist will probably also be a List of Nodes. Back in pgexec.c:

static void buffer_print_expr(PGExec_Buffer*, Node*);
static void buffer_print_list(PGExec_Buffer*, List*, char*);

static void buffer_print_opexpr(PGExec_Buffer* buf, OpExpr* op) {
  buffer_appendf(buf, "[opexpr: todo]");
}

static void buffer_print_expr(PGExec_Buffer* buf, Node* expr) {
  switch (nodeTag(expr)) {
  case T_OpExpr:
    buffer_print_opexpr(buf, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

static void buffer_print_list(PGExec_Buffer* buf, List* list, char* sep) {
  ListCell* cell = NULL;
  bool first = true;

  foreach(cell, list) {
    if (!first) {
      buffer_appendz(buf, sep);
    }

    first = false;
    buffer_print_expr(buf, (Node*)lfirst(cell));
  }
}

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  buffer_print_list(buf, plan->qual, " AND ");
}

And let's check out OpExpr!

Walking `OpExpr`

Take a look at the definition of OpExpr:

typedef struct OpExpr
{
    Expr        xpr;

    /* PG_OPERATOR OID of the operator */
    Oid         opno;

    /* PG_PROC OID of underlying function */
    Oid         opfuncid pg_node_attr(equal_ignore_if_zero, query_jumble_ignore);

    /* PG_TYPE OID of result value */
    Oid         opresulttype pg_node_attr(query_jumble_ignore);

    /* true if operator returns set */
    bool        opretset pg_node_attr(query_jumble_ignore);

    /* OID of collation of result */
    Oid         opcollid pg_node_attr(query_jumble_ignore);

    /* OID of collation that operator should use */
    Oid         inputcollid pg_node_attr(query_jumble_ignore);

    /* arguments to the operator (1 or 2) */
    List       *args;

    /* token location, or -1 if unknown */
    int         location;
} OpExpr;

The important fields are opno, the Oid of the operator, and args. args looks like another List of Nodes so we already know how to handle that.

But how do we find the string name of the operator? Presumably there's infrastructure like RelationIdGetRelation that takes an Oid and gets us an operator object.

Well I got stuck here as well. Again, thankfully, ChatGPT gave me some suggestions. There's no great story for how I got it working. So here's buffer_print_opexpr.

static void buffer_print_op(PGExec_Buffer* buf, OpExpr* op) {
  HeapTuple opertup = SearchSysCache1(OPEROID, ObjectIdGetDatum(op->opno));

  buffer_print_expr(buf, lfirst(list_nth_cell(op->args, 0)));

  if (HeapTupleIsValid(opertup)) {
    Form_pg_operator operator = (Form_pg_operator)GETSTRUCT(opertup);
    buffer_appendf(buf, " %s ", NameStr(operator->oprname));
    ReleaseSysCache(opertup);
  } else {
    buffer_appendf(buf, "[Unknown operation: %d]", op->opno);
  }

  // TODO: Support single operand operations.
  buffer_print_expr(buf, lfirst(list_nth_cell(op->args, 1)));
}

And add the following two includes to the top of pgexec.c:

#include "catalog/pg_operator.h"
#include "utils/syscache.h"

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:42:52.916 GMT [3252974] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [Unknown: 6] > [Unknown: 7]
2023-11-19 19:42:52.916 GMT [3252974] STATEMENT:  SELECT a FROM x WHERE a > 1;

And we continue to make progress! Let's look up the type of these two unknown nodes.

$ grep ' = 6,' src/include/nodes/nodetags.h
        T_Var = 6,
$ grep ' = 7,' src/include/nodes/nodetags.h
        T_Const = 7,

Let's deal with Const first.

Walking `Const`

If we take a look at the Const definition:

typedef struct Const
{
    pg_node_attr(custom_copy_equal, custom_read_write)

    Expr        xpr;
    /* pg_type OID of the constant's datatype */
    Oid         consttype;
    /* typmod value, if any */
    int32       consttypmod pg_node_attr(query_jumble_ignore);
    /* OID of collation, or InvalidOid if none */
    Oid         constcollid pg_node_attr(query_jumble_ignore);
    /* typlen of the constant's datatype */
    int         constlen pg_node_attr(query_jumble_ignore);
    /* the constant's value */
    Datum       constvalue pg_node_attr(query_jumble_ignore);
    /* whether the constant is null (if true, constvalue is undefined) */
    bool        constisnull pg_node_attr(query_jumble_ignore);

    /*
     * Whether this datatype is passed by value.  If true, then all the
     * information is stored in the Datum.  If false, then the Datum contains
     * a pointer to the information.
     */
    bool        constbyval pg_node_attr(query_jumble_ignore);

    /*
     * token location, or -1 if unknown.  All constants are tracked as
     * locations in query jumbling, to be marked as parameters.
     */
    int         location pg_node_attr(query_jumble_location);
} Const;

It looks like we need to switch on the consttype (an Oid) to figure out how to interpret the constvalue (a Datum). Remember I mentioned earlier that how to interpret a Datum is dependent on context. consttype is the context here.

In this case, although consttype is an Oid and we had to use Postgres infrastructure to look up the Oid's corresponding object, there are some builtin types and the literals we've queried with are among them.

We can simply check if consttype == INT4OID and the interpret the Datum as an int32 if so. DatumGetInt32 will get us that int32 in that case.

To support the new Const type, we'll add a case in buffer_print_expr to look for a T_Const.

static void buffer_print_expr(PGExec_Buffer* buf, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

And add a new function, buffer_print_const:

static void buffer_print_const(PGExec_Buffer* buf, Const* cnst) {
  switch (cnst->consttype) {
  case INT4OID:
    int32 val = DatumGetInt32(cnst->constvalue);
    buffer_appendf(buf, "%d", val);
    break;

  default:
    buffer_appendf(buf, "[Unknown consttype oid: %d]", cnst->consttype);
  }
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 19:53:47.922 GMT [3253746] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE [Unknown: 6] > 1
2023-11-19 19:53:47.922 GMT [3253746] STATEMENT:  SELECT a FROM x WHERE a > 1;

Great! Now we just have to tackle T_Var.

Walking `Var`

Let's take a look at the definition of Var:

typedef struct Var
{
    Expr        xpr;

    /*
     * index of this var's relation in the range table, or
     * INNER_VAR/OUTER_VAR/etc
     */
    int         varno;

    /*
     * attribute number of this var, or zero for all attrs ("whole-row Var")
     */
    AttrNumber  varattno;

    /* pg_type OID for the type of this var */
    Oid         vartype pg_node_attr(query_jumble_ignore);
    /* pg_attribute typmod value */
    int32       vartypmod pg_node_attr(query_jumble_ignore);
    /* OID of collation, or InvalidOid if none */
    Oid         varcollid pg_node_attr(query_jumble_ignore);

    /*
     * RT indexes of outer joins that can replace the Var's value with null.
     * We can omit varnullingrels in the query jumble, because it's fully
     * determined by varno/varlevelsup plus the Var's query location.
     */
    Bitmapset  *varnullingrels pg_node_attr(query_jumble_ignore);

    /*
     * for subquery variables referencing outer relations; 0 in a normal var,
     * >0 means N levels up
     */
    Index       varlevelsup;

    /*
     * varnosyn/varattnosyn are ignored for equality, because Vars with
     * different syntactic identifiers are semantically the same as long as
     * their varno/varattno match.
     */
    /* syntactic relation index (0 if unknown) */
    Index       varnosyn pg_node_attr(equal_ignore, query_jumble_ignore);
    /* syntactic attribute number */
    AttrNumber  varattnosyn pg_node_attr(equal_ignore, query_jumble_ignore);

    /* token location, or -1 if unknown */
    int         location;
} Var;

It looks like this refers to a relation in the range table list again. So this means we need to have access to the full PlannedStmt so we can read its rtable field again to find the table. Then we need to look up the Relation for the table and then we can use the Var's varattno field to pick the nth attribute from the relation and get its string representation.

However, ChatGPT found a slightly higher-level function: get_attname() that takes a relation Oid and an attribute index and returns the string name of the column.

So here's what buffer_print_var looks like:

static void buffer_print_var(PGExec_Buffer* buf, PlannedStmt* stmt, Var* var) {
  char* name = NULL;
  RangeTblEntry* rte = list_nth(stmt->rtable, var->varno-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[Unsupported relation type for var: %d].", rte->rtekind);
    return;
  }

  name = get_attname(rte->relid, var->varattno, false);
  buffer_appendz(buf, name);
  pfree(name);
}

You'll also need to add another #include for utils/lsyscache.h.

Let's add the case T_Var: check in buffer_print_expr, and also feed the PlannedStmt* through all the necessary buffer_print_X functions:

static void buffer_print_expr(PGExec_Buffer*, PlannedStmt*, Node*);
static void buffer_print_list(PGExec_Buffer*, PlannedStmt*, List*, char*);

static void buffer_print_opexpr(PGExec_Buffer* buf, PlannedStmt* stmt, OpExpr* op) {
  HeapTuple opertup = SearchSysCache1(OPEROID, ObjectIdGetDatum(op->opno));

  buffer_print_expr(buf, stmt, lfirst(list_nth_cell(op->args, 0)));

  if (HeapTupleIsValid(opertup)) {
    Form_pg_operator operator = (Form_pg_operator)GETSTRUCT(opertup);
    buffer_appendf(buf, " %s ", NameStr(operator->oprname));
    ReleaseSysCache(opertup);
  } else {
    buffer_appendf(buf, "[Unknown operation: %d]", op->opno);
  }

  // TODO: Support single operand operations.
  buffer_print_expr(buf, stmt, lfirst(list_nth_cell(op->args, 1)));
}

static void buffer_print_const(PGExec_Buffer* buf, Const* cnst) {
  switch (cnst->consttype) {
  case INT4OID:
    int32 val = DatumGetInt32(cnst->constvalue);
    buffer_appendf(buf, "%d", val);
    break;

  default:
    buffer_appendf(buf, "[Unknown consttype oid: %d]", cnst->consttype);
  }
}

static void buffer_print_var(PGExec_Buffer* buf, PlannedStmt* stmt, Var* var) {
  char* name = NULL;
  RangeTblEntry* rte = list_nth(stmt->rtable, var->varno-1);
  if (rte->rtekind != RTE_RELATION) {
    elog(LOG, "[Unsupported relation type for var: %d].", rte->rtekind);
    return;
  }

  name = get_attname(rte->relid, var->varattno, false);
  buffer_appendz(buf, name);
  pfree(name);
}

static void buffer_print_expr(PGExec_Buffer* buf, PlannedStmt* stmt, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_Var:
    buffer_print_var(buf, stmt, (Var*)expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, stmt, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

static void buffer_print_list(PGExec_Buffer* buf, PlannedStmt* stmt, List* list, char* sep) {
  ListCell* cell = NULL;
  bool first = true;

  foreach(cell, list) {
    if (!first) {
      buffer_appendz(buf, sep);
    }

    first = false;
    buffer_print_expr(buf, stmt, (Node*)lfirst(cell));
  }
}

static void buffer_print_where(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->qual == NULL) {
    return;
  }

  buffer_appendz(buf, " WHERE ");
  buffer_print_list(buf, queryDesc->plannedstmt, plan->qual, " AND ");
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 20:03:14.351 GMT [3254458] LOG:  [pgexec] SELECT [columns todo] FROM x WHERE a > 1
2023-11-19 20:03:14.351 GMT [3254458] STATEMENT:  SELECT a FROM x WHERE a > 1;

Huzzah!

Walking the column list

Let's get rid of [columns todo]. We already had the idea that List* targetlist on the Plan struct was a list of expression Nodes. Let's try it.

static void buffer_print_select_columns(PGExec_Buffer* buf, QueryDesc* queryDesc, Plan* plan) {
  if (plan->targetlist == NULL) {
    return;
  }

  buffer_print_list(buf, queryDesc->plannedstmt, plan->targetlist, ", ");
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and you should see:

2023-11-19 20:12:48.091 GMT [3255398] LOG:  [pgexec] SELECT [Unknown: 53] FROM x WHERE a > 1
2023-11-19 20:12:48.091 GMT [3255398] STATEMENT:  SELECT a FROM x WHERE a > 1;

Hmm. Let's look up Node 53 in Postgres:

$ grep ' = 53,' src/include/nodes/nodetags.h
        T_TargetEntry = 53,

Based on the definition of TargetEntry, it looks like we can ignore most of the fields (because we don't need to handle SELECT a AS b yet) and just proxy the child expr field.

Let's add a case T_TargetEntry to buffer_print_expr:

static void buffer_print_expr(PGExec_Buffer* buf, PlannedStmt* stmt, Node* expr) {
  switch (nodeTag(expr)) {
  case T_Const:
    buffer_print_const(buf, (Const*)expr);
    break;

  case T_Var:
    buffer_print_var(buf, stmt, (Var*)expr);
    break;

  case T_TargetEntry:
    buffer_print_expr(buf, stmt, (Node*)((TargetEntry*)expr)->expr);
    break;

  case T_OpExpr:
    buffer_print_opexpr(buf, stmt, (OpExpr*)expr);
    break;

  default:
    buffer_appendf(buf, "[Unknown: %d]", nodeTag(expr));
  }
}

Rebuild and reinstall the extension, and restart Postgres. Re-run the test.sql script. Check the Postgres server logs and:

2023-11-19 20:17:51.114 GMT [3257827] LOG:  [pgexec] SELECT a FROM x WHERE a > 1
2023-11-19 20:17:51.114 GMT [3257827] STATEMENT:  SELECT a FROM x WHERE a > 1;

We did it!

Variations

Let's try out some other queries to make sure this wasn't just luck.

$ /usr/local/pgsql/bin/psql -h localhost postgres -c 'SELECT a + 1 FROM x'
 ?column?
----------
      310
(1 row)

$ /usr/local/pgsql/bin/psql -h localhost postgres -c 'SELECT a + 1 FROM x WHERE 2 > a'
 ?column?
----------
(0 rows)

And back in the Postgres server logs:

2023-11-19 20:19:28.057 GMT [3257874] LOG:  [pgexec] SELECT a + 1 FROM x
2023-11-19 20:19:28.057 GMT [3257874] STATEMENT:  SELECT a + 1 FROM x
2023-11-19 20:19:30.474 GMT [3257878] LOG:  [pgexec] SELECT a + 1 FROM x WHERE 2 > a
2023-11-19 20:19:30.474 GMT [3257878] STATEMENT:  SELECT a + 1 FROM x WHERE 2 > a

Not bad!

Next steps

Printing out the statement here isn't incredibly useful. But it establishes a basis for future work that might avoid Postgres's query execution engine and do the execution ourselves, or to proxy execution to another system.

Postscript: On ChatGPT

My recent Postgres explorations would have been basically impossible if it weren't for being able to ask ChatGPT simple, stupid questions like "How do I get from a Postgres Var to a column name".

It isn't always right. It doesn't always give great code. Actually, it normally gives pretty weird code. But it's been extremely useful for quick iteration when I get stuck.

The only other place the information exists is in small blog posts around the internet, the Postgres mailing lists (that so far for me hasn't been super responsive), and the code itself.

I've been on a Postgres roll. Let's dig into interpreting a Postgres query plan in preparation for future work on completely diverting the flow of Postgres query execution using execution hooks!https://t.co/EZrgoIiTuX pic.twitter.com/7S6d6olPX8
— Phil Eaton (@eatonphil) November 19, 2023

Github CI

Pxtl.ca — Fri, 17 Nov 2023 02:00:00 GMT

Well, it took longer than I would've liked but I got Github CI working with this blog! Finally learned how to use Github workflows.

Workflow files are here

sum-grid.love

Kartik Agaram — Thu, 16 Nov 2023 09:00:00 GMT

A little sudoku-like app for helping first-graders practice addition. This attempt at situated software for schooling got a little more use than spell-cards.love.

video; 25 seconds

rustcred, a git credentials helper

infrequent oscillations — Wed, 15 Nov 2023 15:54:31 GMT

In which we store some credentials with example code.

CHAOS;HEAD NOAH review

ブラック — Wed, 15 Nov 2023 02:00:00 GMT

Press any key for schizophrenia.

CHAOS;HEAD is a really interesting story. On the surface it appears to be a murder mystery story but it's not really that. There are murders and mysteries, but they don't meet in the same way that they would in a whodunit. I guess I'd rather call it a psychological horror, even if it's not really outright horror. And much like STEINS;GATE, the entry in the series that people are probably the most familiar with, it is rife with conspiracies and larger implications. Really enjoyed myself as I read through it for the first ending.

The meat of the story is really in the common route, which then unlocks the other routes and endings after having completed it once. The character-specific routes aren't that special even if they do flesh out some of the characters a bit more and have some good elements. The most outstanding one of them is the Rimi route, which gives the most exposition for the common route, although I did also enjoy the Sena one too as a Sena fan. The endings also felt a bit lacking compared to the emotions I felt from the common route. Even the true ending didn't really change things up that much or be more detailed than the base ending. Thankfully the common route was interesting so I don't feel soured by these issues.

The protagonist Takumi is a flawed but interesting character. An anti-social, delusional otaku hiding from the world in his shipping container home. And because of those delusions, he becomes an unreliable narrator for the story, making the reader unsure what parts of the story are real and what are delusions. He enters a delusion, then something unexpected happens and you can't know if it's still a delusion or not. Or he encounters a stranger that supposedly knows him and you can't know which is true. The unreliability of his delusions really fits the whole mysterious and disturbing theme of the story very well.

The delusion triggers are pretty interesting system that allow having a positive delusion, a negative delusion, or no delusion. The delusions will then trigger either a small scene with no relevance to the plot, or change the actual routing (at least if you answer a quiz correctly). While the game does have an explanation as to why this would happen, it's still pretty hard to figure out why having a negative delusion in a given scenario would result in a different ending. Not quite as opaque of a routing system as the text message system in STEINS;GATE was, but still mixed up a bit. You'll probably want to look up a route guide for this one.

Art in CHAOS;HEAD is just about alright. Some of the character sprites and CG images have a derpy feeling to them, with the characters looking like their facial features are trying to escape their faces. I didn't find the art fundamentally flawed but it did definitely stick out, even in some quite pivotal parts of the story. The characters also have animated mouths but there's not much animation beyond that and some camera pans. And while the CG did have some quality issues, the amount of it was just fine. I guess this isn't really the most visual of packages.

But what was actually superb was the sound design. The sound design in this game is fantastic. Takumi is fully voiced and Yoshino does a superb job at voicing his uncertainties, fears, delusions, paranoia and other ailments. Maybe his performance was actually too good, since it overshadowed some of the side characters, even though there were also good performances there. Like for example Takahashi Chiaki's performance as Yua. The music and sound effects are also to the point when it comes to producing a creepy, supernatural and mysterious atmosphere. There's very few times when I'm impressed to this point by just the sound design in a visual novel. You'll definitely want to have audio on for this experience. Bonus points for an excellent insert song.

You’ll also want to have audio on because it’s very hard to figure out who’s talking without it. The user interface has no indication on who the speaker at the moment is, or if the text is just narration. If you’re deaf, you can go get fucked as far as CHAOS;HEAD is concerned. The interface is also kinda ugly in general and the text display is kinda rough. This is probably a function of the game’s age, as it’s already 15 years old. Not a dealbreaker but something I did definitely note during it.

For what it's worth, installing the Committee of Zero patch does clean up the text display so maybe consider doing that.

Whilst it's largely overshadowed by its excellent younger sibling STEINS;GATE, CHAOS;HEAD is still an interesting story and recommended reading despite some of its flaws.

Maybe I should now check why everyone hates the anime adaptation…

Fun with Makefiles - Dynamic Menu Generation

smcleod.net — Tue, 14 Nov 2023 02:00:00 GMT

This reusable snippet will generate a menu of targets from the Makefile. It will use `fzf` if it is installed, otherwise it will use a numbered menu.

MBA Washing

smcleod.net — Mon, 13 Nov 2023 11:00:00 GMT

"MBA Washing" refers to the phenomenon where individuals or companies, often with a strong theoretical or academic background but limited recent practical experience, adopt and reinterpret industry-specific terminology and cultural movements.

Turning a project into a product

Stefan Schüller — Sat, 11 Nov 2023 10:37:16 GMT

Foto by Foto-Press

A step by step account of how I turned a project of mine into a polished product that will be available for purchase over at stationdisplay.com.

GopherConSyd 2023 - All slides, notes and links

Ben E. C. Boyter — Thu, 09 Nov 2023 02:00:00 GMT

Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my GopherConSyd 2023 talk about “Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code”. Whatever the case may be, welcome!

For the slides visit https://boyter.org/static/gophercon-syd-presentation-2023/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide. Press S to see the speaker notes, which are more or less the script I will be sticking to.

My go-to 'make code go faster' ideas

jodavaho.io — Thu, 09 Nov 2023 02:00:00 GMT

I’m not an optimization guru by any means. It’s never been something I’ve been allowed to focus on at work, sadly. At some jobs, performance is secondary to correctness and robustness, and at others, it’s secondary to flashy features.

But, I’ve used the following tricks in hotloops

Dimension reduction (esp via convolution)
Branchless calculation
SIMD

SIMD gets a lot of love, but it’s a constant-factor improvement and can be tough to coax out of the compiler (unless you use a library).

Branchless calculation is getting less relevant as a “trick”. A really dumb example is if you have to do this:

// Example 1
if (x>=0) {
    return x*y;
} else {
    return x*b;
}

Then you can speed it up by doing this:

// Example 2:
int c = (x>0);
return x*(c*y + (1-c)*b); // either x*y or x*b;
//better: x*(c*(y-b) + (b));

And if you can combine this with unrolling and SIMD, then you’re really well off. The second example works just as well if those are Eigen::VectorXd length 1000, and now you’re really cranking out the speed.

Something like this:

// Example 3
for ( i .. N )
{
  if (x[i]>=0) {
      out[i] = x[i]*y[i];
  } else {
      out[i] x[i]*b[i];
  }
}

Becomes trivial, and looks exactly like //Example 2

But even more fundamentally, those usually need to come after:

Using the right data structures
Using the right algorithms
Reframing the problem for speed, e.g., giving up a little to go a lot faster
Memoization/caching/pre-calculating

These can usually get me 80% of the way there. When they all come together, you’re staring at your machine wondering how it is done already.

I rarely see great returns on multi-threaded / multi-process, unless you’re doing rote calculations on a large dataset in a batch.

Anchor Tags

Weblog on marginalia.nu — Tue, 07 Nov 2023 02:00:00 GMT

I’ve been working on getting anchor tag keywords into the search engine, basically using link texts to complement the keywords on a webpage. The problem I’m attempting to address is that many websites don’t really describe themselves particularly well. As Steve Ballmer’s stage performance once illustrated, merely repeating a word doesn’t on its own make what you’re saying relevant to the term. Another good example of how it falls short is PuTTY’s website, which will be used as a pilot case to improve.

What I'm up to - November 2023

Philip I. Thomas — Mon, 06 Nov 2023 15:32:33 GMT

This is my monthly newsletter about what I'm up to, which I send in place of social media.

What I did in October

Switched my daily driver Aeropress coffee recipe to this one by Jonathan Gagné. It produces delicious results. (His book Physics of Filter Coffee is excellent.)
Witnessed the aftermath of a bike crash, did some activism, and got featured in an article on Curbed.
Booklet, the modern email software I make, as a bunch of new features.
Postcard, the personal website software I make, continues to grow as people leave the site formerly known as Twitter.
Made some light updates to the Contraption Company website.

Things to share

Articles: How Linear builds products. "What I tell employees about negative press." How Not To Sort By Average Rating.
Music: nadie sabe lo que va a pasar mañana
Podcasts: Charlie Munger on Acquired.
Apps: Cursor, an AI-first code editor, has replaced VSCode as my daily development tool. Dutch Cycling Lifestyle is a fun use of AI.
Trend: While chat is the best-known product from OpenAI, the Embeddings feature is powerful and being incorporated into many products in ways that are not obviously "AI."
Videos: Magnus Midtbø did some videos with Norwegian military that gave a documentary-like glimpse into their lives.
Coffees I'm drinking: Tim Wendelboe Caballero Geisha.

Plans for November

Continuing to build Booklet features.
Spending a couple of weeks in Europe for a mix of work and fun.
Working on essays for Contraption Company.

Where I'll be

12-14 Nov: Oslo
14-16 Nov: Copenhagen
16-19 Nov: London
19-24 Nov: Nice, FR
24-26 Nov: Barcelona

Let me know if we overlap.

Being a minor AI public figure

swyx's site RSS Feed — Fri, 03 Nov 2023 22:21:53 GMT

I was recently involved in moderating a chat with Kanjun Qiu of Imbue at the MIT AI conf: https://photos.google.com/share/AF1QipNJ9i78ICeg2YuemyAXmtStKvqr9l0Tao3xQWxmeAVjBALHn_NnFvPXFlNSkdMfqA?pli=1&key=dTFRRHBTLVRZTEVCem0zal8tNVkxblh0V3k4VXhR

Writing a storage engine for Postgres: an in-memory Table Access Method

Notes on software development — Wed, 01 Nov 2023 02:00:00 GMT

With Postgres 12, released in 2019, it became possible to swap out Postgres's storage engine.

This is a feature MySQL has supported for a long time. There are at least 8 different built-in engines you can pick from. MyRocks, MySQL on RocksDB, is another popular third-party distribution.

I assume there will be a renaissance of Postgres storage engines. To date, the efforts are nascent. OrioleDB and Citus Columnar are two promising third-party table access methods being actively developed.

Why alternative storage engines?

The ability to swap storage engines is useful because different workloads sometimes benefit from different storage approaches. Analytics workloads and columnar storage layouts go well together. Write-heavy workloads and LSM trees go well together. And some people like in-memory storage for running integration tests.

By swapping out only the storage engine, you get the benefit of the rest of the Postgres or MySQL infrastructure. The query language, the wire protocol, the ecosystem, etc.

Why not foreign data wrappers?

Very little has been written about the difference between foreign data wrappers (FDWs) and table access methods. Table access methods seems to be the lower-level layer where presumably you get better performance and cleaner integration. But there is clearly overlap between these two extension options.

For example there is a FDW for ClickHouse so when you create tables and rows and query the tables you are really creating and querying rows in a ClickHouse server. Similarly there's a FDW for RocksDB. And Citus's columnar engine works either as a foreign data wrapper or a table access method.

The Citus page draws the clearest distinction between FDWs and table access methods, but even that page is vague. Performance doesn't seem to be the main difference. Closer integration, and thus the ability to look more like vanilla Postgres from the outside, seems to be the gist.

In any case, I wanted to explore the table access method API.

Digging in

I haven't written Postgres extensions before and I've never written C professionally. If you're familiar with Postgres internals or C and notice something funky, please let me know!

It turns out that almost no one has written how to implement the minimal table access methods for various storage engine operations. So after quite a bit of stumbling to get the basics of an in-memory storage engine working, I'm going to walk you through my approach.

This is prototype-quality code which hopefully will be a useful base for further exploration.

All code for this post is available on GitHub.

A debug Postgres build

First off, let's make a debug build of Postgres.

$ git clone https://github.com/postgres/postgres
$ # An arbitrary commit from `master` after Postgres 16 I am on
$ git checkout 849172ff4883d44168f96f39d3fde96d0aa34c99
$ cd postgres
$ ./configure --enable-cassert --enable-debug CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer"
$ make -j8
$ sudo make install

This will install Postgres binaries (e.g. psql, pg_ctl, initdb, pg_config) into /usr/local/pgsql/bin.

I'm going to reference those absolute paths throughout this post because you might have a system (package manager) install of Postgres already.

Let's create a database and start up this debug build:

$ /usr/local/pgsql/bin/initdb test-db
$ /usr/local/pgsql/bin/pg_ctl -D test-db -l logfile start

Extension infrastructure

Since we installed Postgres from scratch, /usr/local/pgsql/bin/pg_config will supply all of the infrastructure we need.

The "infrastructure" is basically just PGXS: Postgres Makefile utilities.

It's convention-heavy. So in a new Makefile for this project we'll specify:

MODULES: Any C sources to build, without the .c file extension
EXTENSION: Extension metadata file, without the .control file extension
DATA: A SQL file that is executed when the extension is loaded, this time with the .sql extension

MODULES = pgtam
EXTENSION = pgtam
DATA = pgtam--0.0.1.sql

PG_CONFIG = /usr/local/pgsql/bin/pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

The final three lines set up the PGXS Makefile library based on the particular installed Postgres build we want to build the extension against and install the extension to.

PGXS gives us a few important targets like make distclean, make, and make install we'll use later on.

`pgtam.c`

A minimal C file that registers a function capable of serving as a table access method is:

#include "postgres.h"
#include "fmgr.h"
#include "access/tableam.h"

PG_MODULE_MAGIC;

const TableAmRoutine memam_methods = {
  .type = T_TableAmRoutine,
};

PG_FUNCTION_INFO_V1(mem_tableam_handler);
Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
  PG_RETURN_POINTER(&memam_methods);
}

If you want to read about extension basics without the complexity of table access methods, you can find a complete, minimal Postgres extension I wrote to validate the infrastructure here. Or you can follow a larger tutorial.

The workflow for registering a table access method is to first run CREATE EXTENSION pgtam. This assumes pgtam is an extension that has a function that returns a TableAmRoutine struct instance, a table of table access methods.

Then you must run CREATE ACCESS METHOD mem TYPE TABLE HANDLER mem_tableam_handler. And finally you can use the access method when creating a table with USING mem: CREATE TABLE x(a INT) USING mem.

`pgtam.control`

This file contains extension metadata. At a minimum, the version of the extension and the filename for the extension where it should be installed.

default_version = '0.0.1'
module_pathname = '$libdir/pgtam'

`pgtam--0.0.1.sql`

Finally, in pgtam--0.0.1.sql (which is executed when we call CREATE EXTENSION pgtam), we register the handler function as a foreign function, and then we register the function as an access method.

CREATE OR REPLACE FUNCTION mem_tableam_handler(internal)
RETURNS table_am_handler AS 'pgtam', 'mem_tableam_handler'
LANGUAGE C STRICT;

CREATE ACCESS METHOD mem TYPE TABLE HANDLER mem_tableam_handler;

Build

Now that we've got all the pieces in place, we can build and install the extension.

$ make
$ sudo make install

Let's add a test.sql script to exercise the extension:

DROP EXTENSION IF EXISTS pgtam CASCADE;
CREATE EXTENSION pgtam;
CREATE TABLE x(a INT) USING mem;

And run it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
DROP EXTENSION
CREATE EXTENSION
psql:test.sql:3: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:3: error: connection to server was lost

Ok, so psql crashed! Let's look at the server logs. When we started Postgres with pg_ctl we specified the log file as logfile in the directory where we ran pg_ctl.

If we look through it we'll spot an assertion failure:

$ grep Assert logfile
TRAP: failed Assert("routine->scan_begin != NULL"), File: "tableamapi.c", Line: 52, PID: 2906922

That's a great sign! This is Postgres's debug infrastructure helping to make sure the table access method is correctly implemented.

Table access method stubs

The next step is to add function stubs for all the non-optional methods of the TableAmRoutine struct.

I've done all the work for you already so you can just copy this over the existing pgtam.c. It's a big file, but don't worry. There's nothing to explain. Just a bunch of blank functions returning default values when required.

#include "postgres.h"
#include "fmgr.h"
#include "access/tableam.h"
#include "access/heapam.h"
#include "nodes/execnodes.h"
#include "catalog/index.h"
#include "commands/vacuum.h"
#include "utils/builtins.h"
#include "executor/tuptable.h"

PG_MODULE_MAGIC;

const TableAmRoutine memam_methods;

static const TupleTableSlotOps* memam_slot_callbacks(
  Relation relation
) {
  return NULL;
}

static TableScanDesc memam_beginscan(
  Relation relation,
  Snapshot snapshot,
  int nkeys,
  struct ScanKeyData *key,
  ParallelTableScanDesc parallel_scan,
  uint32 flags
) {
  return NULL;
}

static void memam_rescan(
  TableScanDesc sscan,
  struct ScanKeyData *key,
  bool set_params,
  bool allow_strat,
  bool allow_sync,
  bool allow_pagemode
) {
}

static void memam_endscan(TableScanDesc sscan) {
}

static bool memam_getnextslot(
  TableScanDesc sscan,
  ScanDirection direction,
  TupleTableSlot *slot
) {
  return false;
}

static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {
  return NULL;
}

static void memam_index_fetch_reset(IndexFetchTableData *scan) {
}

static void memam_index_fetch_end(IndexFetchTableData *scan) {
}

static bool memam_index_fetch_tuple(
  struct IndexFetchTableData *scan,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot,
  bool *call_again,
  bool *all_dead
) {
  return false;
}

static void memam_tuple_insert(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate
) {
}

static void memam_tuple_insert_speculative(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate,
  uint32 specToken) {
}

static void memam_tuple_complete_speculative(
  Relation relation,
  TupleTableSlot *slot,
  uint32 specToken,
  bool succeeded) {
}

static void memam_multi_insert(
  Relation relation,
  TupleTableSlot **slots,
  int ntuples,
  CommandId cid,
  int options,
  BulkInsertState bistate
) {
}

static TM_Result memam_tuple_delete(
  Relation relation,
  ItemPointer tid,
  CommandId cid,
  Snapshot snapshot,
  Snapshot crosscheck,
  bool wait,
  TM_FailureData *tmfd,
  bool changingPart
) {
  TM_Result result = {};
  return result;
}

static TM_Result memam_tuple_update(
  Relation relation,
  ItemPointer otid,
  TupleTableSlot *slot,
  CommandId cid,
  Snapshot snapshot,
  Snapshot crosscheck,
  bool wait,
  TM_FailureData *tmfd,
  LockTupleMode *lockmode,
  TU_UpdateIndexes *update_indexes
) {
  TM_Result result = {};
  return result;
}

static TM_Result memam_tuple_lock(
  Relation relation,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot,
  CommandId cid,
  LockTupleMode mode,
  LockWaitPolicy wait_policy,
  uint8 flags,
  TM_FailureData *tmfd)
{
  TM_Result result = {};
  return result;
}

static bool memam_fetch_row_version(
  Relation relation,
  ItemPointer tid,
  Snapshot snapshot,
  TupleTableSlot *slot
) {
  return false;
}

static void memam_get_latest_tid(
  TableScanDesc sscan,
  ItemPointer tid
) {
}

static bool memam_tuple_tid_valid(TableScanDesc scan, ItemPointer tid) {
  return false;
}

static bool memam_tuple_satisfies_snapshot(
  Relation rel,
  TupleTableSlot *slot,
  Snapshot snapshot
) {
  return false;
}

static TransactionId memam_index_delete_tuples(
  Relation rel,
  TM_IndexDeleteOp *delstate
) {
  TransactionId id = {};
  return id;
}

static void memam_relation_set_new_filelocator(
  Relation rel,
  const RelFileLocator *newrlocator,
  char persistence,
  TransactionId *freezeXid,
  MultiXactId *minmulti
) {
}

static void memam_relation_nontransactional_truncate(
  Relation rel
) {
}

static void memam_relation_copy_data(
  Relation rel,
  const RelFileLocator *newrlocator
) {
}

static void memam_relation_copy_for_cluster(
  Relation OldHeap,
  Relation NewHeap,
  Relation OldIndex,
  bool use_sort,
  TransactionId OldestXmin,
  TransactionId *xid_cutoff,
  MultiXactId *multi_cutoff,
  double *num_tuples,
  double *tups_vacuumed,
  double *tups_recently_dead
) {
}

static void memam_vacuum_rel(
  Relation rel,
  VacuumParams *params,
  BufferAccessStrategy bstrategy
) {
}

static bool memam_scan_analyze_next_block(
  TableScanDesc scan,
  BlockNumber blockno,
  BufferAccessStrategy bstrategy
) {
  return false;
}

static bool memam_scan_analyze_next_tuple(
  TableScanDesc scan,
  TransactionId OldestXmin,
  double *liverows,
  double *deadrows,
  TupleTableSlot *slot
) {
  return false;
}

static double memam_index_build_range_scan(
  Relation heapRelation,
  Relation indexRelation,
  IndexInfo *indexInfo,
  bool allow_sync,
  bool anyvisible,
  bool progress,
  BlockNumber start_blockno,
  BlockNumber numblocks,
  IndexBuildCallback callback,
  void *callback_state,
  TableScanDesc scan
) {
  return 0;
}

static void memam_index_validate_scan(
  Relation heapRelation,
  Relation indexRelation,
  IndexInfo *indexInfo,
  Snapshot snapshot,
  ValidateIndexState *state
) {
}

static bool memam_relation_needs_toast_table(Relation rel) {
  return false;
}

static Oid memam_relation_toast_am(Relation rel) {
  Oid oid = {};
  return oid;
}

static void memam_fetch_toast_slice(
  Relation toastrel,
  Oid valueid,
  int32 attrsize,
  int32 sliceoffset,
  int32 slicelength,
  struct varlena *result
) {
}

static void memam_estimate_rel_size(
  Relation rel,
  int32 *attr_widths,
  BlockNumber *pages,
  double *tuples,
  double *allvisfrac
) {
}

static bool memam_scan_sample_next_block(
  TableScanDesc scan, SampleScanState *scanstate
) {
  return false;
}

static bool memam_scan_sample_next_tuple(
  TableScanDesc scan,
  SampleScanState *scanstate,
  TupleTableSlot *slot
) {
  return false;
}

const TableAmRoutine memam_methods = {
  .type = T_TableAmRoutine,

  .slot_callbacks = memam_slot_callbacks,

  .scan_begin = memam_beginscan,
  .scan_end = memam_endscan,
  .scan_rescan = memam_rescan,
  .scan_getnextslot = memam_getnextslot,

  .parallelscan_estimate = table_block_parallelscan_estimate,
  .parallelscan_initialize = table_block_parallelscan_initialize,
  .parallelscan_reinitialize = table_block_parallelscan_reinitialize,

  .index_fetch_begin = memam_index_fetch_begin,
  .index_fetch_reset = memam_index_fetch_reset,
  .index_fetch_end = memam_index_fetch_end,
  .index_fetch_tuple = memam_index_fetch_tuple,

  .tuple_insert = memam_tuple_insert,
  .tuple_insert_speculative = memam_tuple_insert_speculative,
  .tuple_complete_speculative = memam_tuple_complete_speculative,
  .multi_insert = memam_multi_insert,
  .tuple_delete = memam_tuple_delete,
  .tuple_update = memam_tuple_update,
  .tuple_lock = memam_tuple_lock,

  .tuple_fetch_row_version = memam_fetch_row_version,
  .tuple_get_latest_tid = memam_get_latest_tid,
  .tuple_tid_valid = memam_tuple_tid_valid,
  .tuple_satisfies_snapshot = memam_tuple_satisfies_snapshot,
  .index_delete_tuples = memam_index_delete_tuples,

  .relation_set_new_filelocator = memam_relation_set_new_filelocator,
  .relation_nontransactional_truncate = memam_relation_nontransactional_truncate,
  .relation_copy_data = memam_relation_copy_data,
  .relation_copy_for_cluster = memam_relation_copy_for_cluster,
  .relation_vacuum = memam_vacuum_rel,
  .scan_analyze_next_block = memam_scan_analyze_next_block,
  .scan_analyze_next_tuple = memam_scan_analyze_next_tuple,
  .index_build_range_scan = memam_index_build_range_scan,
  .index_validate_scan = memam_index_validate_scan,

  .relation_size = table_block_relation_size,
  .relation_needs_toast_table = memam_relation_needs_toast_table,
  .relation_toast_am = memam_relation_toast_am,
  .relation_fetch_toast_slice = memam_fetch_toast_slice,

  .relation_estimate_size = memam_estimate_rel_size,

  .scan_sample_next_block = memam_scan_sample_next_block,
  .scan_sample_next_tuple = memam_scan_sample_next_tuple
};

PG_FUNCTION_INFO_V1(mem_tableam_handler);

Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
  PG_RETURN_POINTER(&memam_methods);
}

Let's build and test it!

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE

Hey we're getting somewhere! It successfully created the table with our custom table access method.

Querying rows

Next, let's try querying the table by adding a SELECT a FROM x to test.sql and running it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

This time there's nothing in logfile that helps:

$ tail -n15 logfile
2023-11-01 18:43:32.449 UTC [2906199] LOG:  database system is ready to accept connections
2023-11-01 18:58:32.572 UTC [2907997] LOG:  checkpoint starting: time
2023-11-01 18:58:35.305 UTC [2907997] LOG:  checkpoint complete: wrote 28 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=2.712 s, sync=0.015 s, total=2.733 s; sync files=23, longest=0.004 s, average=0.001 s; distance=128 kB, estimate=150 kB; lsn=0/15F88E0, redo lsn=0/15F8888
2023-11-01 19:08:14.485 UTC [2906199] LOG:  server process (PID 2908242) was terminated by signal 11: Segmentation fault
2023-11-01 19:08:14.485 UTC [2906199] DETAIL:  Failed process was running: SELECT a FROM x;
2023-11-01 19:08:14.485 UTC [2906199] LOG:  terminating any other active server processes
2023-11-01 19:08:14.486 UTC [2906199] LOG:  all server processes terminated; reinitializing
2023-11-01 19:08:14.508 UTC [2908253] LOG:  database system was interrupted; last known up at 2023-11-01 18:58:35 UTC
2023-11-01 19:08:14.518 UTC [2908253] LOG:  database system was not properly shut down; automatic recovery in progress
2023-11-01 19:08:14.519 UTC [2908253] LOG:  redo starts at 0/15F8888
2023-11-01 19:08:14.520 UTC [2908253] LOG:  invalid record length at 0/161DE70: expected at least 24, got 0
2023-11-01 19:08:14.520 UTC [2908253] LOG:  redo done at 0/161DE38 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2023-11-01 19:08:14.521 UTC [2908254] LOG:  checkpoint starting: end-of-recovery immediate wait
2023-11-01 19:08:14.532 UTC [2908254] LOG:  checkpoint complete: wrote 35 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.010 s, total=0.012 s; sync files=27, longest=0.003 s, average=0.001 s; distance=149 kB, estimate=149 kB; lsn=0/161DE70, redo lsn=0/161DE70
2023-11-01 19:08:14.533 UTC [2906199] LOG:  database system is ready to accept connections

This was the first place I got stuck. How on earth do I figure out what methods to implement? I mean, it's clearly one or more of these methods from the struct. But there are so many methods.

I tried setting a breakpoint in gdb on the process returned by SELECT pg_backend_pid() for a psql session, but the breakpoint never seemed to be hit for any of my methods.

So I did the low-tech solution and opened a file, /tmp/pgtam.log, turned off buffering on it, and added a log to every method on the TableAmRoutine struct:

@@ -12,9 +12,13 @@

 const TableAmRoutine memam_methods;

+FILE* fd;
+#define DEBUG_FUNC() fprintf(fd, "in %s\n", __func__);
+
 static const TupleTableSlotOps* memam_slot_callbacks(
   Relation relation
 ) {
+  DEBUG_FUNC();
   return NULL;
 }

@@ -26,6 +30,7 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
+  DEBUG_FUNC();
   return NULL;
 }

@@ -37,9 +42,11 @@
   bool allow_sync,
   bool allow_pagemode
 ) {
+  DEBUG_FUNC();
 }

 static void memam_endscan(TableScanDesc sscan) {
+  DEBUG_FUNC();
 }

 static bool memam_getnextslot(
@@ -47,17 +54,21 @@
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

 static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {
+  DEBUG_FUNC();
   return NULL;
 }

 static void memam_index_fetch_reset(IndexFetchTableData *scan) {
+  DEBUG_FUNC();
 }

 static void memam_index_fetch_end(IndexFetchTableData *scan) {
+  DEBUG_FUNC();
 }

 static bool memam_index_fetch_tuple(
@@ -68,6 +79,7 @@
   bool *call_again,
   bool *all_dead
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -78,6 +90,7 @@
   int options,
   BulkInsertState bistate
 ) {
+  DEBUG_FUNC();
 }

 static void memam_tuple_insert_speculative(
@@ -87,6 +100,7 @@
   int options,
   BulkInsertState bistate,
   uint32 specToken) {
+  DEBUG_FUNC();
 }

 static void memam_tuple_complete_speculative(
@@ -94,6 +108,7 @@
   TupleTableSlot *slot,
   uint32 specToken,
   bool succeeded) {
+  DEBUG_FUNC();
 }

 static void memam_multi_insert(
@@ -104,6 +119,7 @@
   int options,
   BulkInsertState bistate
 ) {
+  DEBUG_FUNC();
 }

 static TM_Result memam_tuple_delete(
@@ -117,6 +133,7 @@
   bool changingPart
 ) {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -133,6 +150,7 @@
   TU_UpdateIndexes *update_indexes
 ) {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -148,6 +166,7 @@
   TM_FailureData *tmfd)
 {
   TM_Result result = {};
+  DEBUG_FUNC();
   return result;
 }

@@ -157,6 +176,7 @@
   Snapshot snapshot,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -164,9 +184,11 @@
   TableScanDesc sscan,
   ItemPointer tid
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_tuple_tid_valid(TableScanDesc scan, ItemPointer tid) {
+  DEBUG_FUNC();
   return false;
 }

@@ -175,6 +197,7 @@
   TupleTableSlot *slot,
   Snapshot snapshot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -183,6 +206,7 @@
   TM_IndexDeleteOp *delstate
 ) {
   TransactionId id = {};
+  DEBUG_FUNC();
   return id;
 }

@@ -193,17 +217,20 @@
   TransactionId *freezeXid,
   MultiXactId *minmulti
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_nontransactional_truncate(
   Relation rel
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_copy_data(
   Relation rel,
   const RelFileLocator *newrlocator
 ) {
+  DEBUG_FUNC();
 }

 static void memam_relation_copy_for_cluster(
@@ -218,6 +245,7 @@
   double *tups_vacuumed,
   double *tups_recently_dead
 ) {
+  DEBUG_FUNC();
 }

 static void memam_vacuum_rel(
@@ -225,6 +253,7 @@
   VacuumParams *params,
   BufferAccessStrategy bstrategy
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_scan_analyze_next_block(
@@ -232,6 +261,7 @@
   BlockNumber blockno,
   BufferAccessStrategy bstrategy
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -242,6 +272,7 @@
   double *deadrows,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -258,6 +289,7 @@
   void *callback_state,
   TableScanDesc scan
 ) {
+  DEBUG_FUNC();
   return 0;
 }

@@ -268,14 +300,17 @@
   Snapshot snapshot,
   ValidateIndexState *state
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_relation_needs_toast_table(Relation rel) {
+  DEBUG_FUNC();
   return false;
 }

 static Oid memam_relation_toast_am(Relation rel) {
   Oid oid = {};
+  DEBUG_FUNC();
   return oid;
 }

@@ -287,6 +322,7 @@
   int32 slicelength,
   struct varlena *result
 ) {
+  DEBUG_FUNC();
 }

 static void memam_estimate_rel_size(
@@ -296,11 +332,13 @@
   double *tuples,
   double *allvisfrac
 ) {
+  DEBUG_FUNC();
 }

 static bool memam_scan_sample_next_block(
   TableScanDesc scan, SampleScanState *scanstate
 ) {
+  DEBUG_FUNC();
   return false;
 }

@@ -309,6 +347,7 @@
   SampleScanState *scanstate,
   TupleTableSlot *slot
 ) {
+  DEBUG_FUNC();
   return false;
 }

And then in the entrypoint, initialize the file for logging.

@@ -369,5 +408,9 @@
 PG_FUNCTION_INFO_V1(mem_tableam_handler);

 Datum mem_tableam_handler(PG_FUNCTION_ARGS) {
+  fd = fopen("/tmp/pgtam.log", "a");
+  setvbuf(fd, NULL, _IONBF, 0); // Prevent buffering
+  fprintf(fd, "\n\nmem_tableam handler loaded\n");
+
   PG_RETURN_POINTER(&memam_methods);
 }

Let's give it a shot!

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

And let's check our log file:

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks

Now we're getting somewhere!

I later realized elog() is the way most people log within Postgres/within extensions. I didn't know that when I was getting started though. This separate logging was a simple way to get the info out.

`slot_callbacks`

Since the request crashes and the last logged function is memam_slot_callbacks, it seems like that is where we should concentrate. The table access method docs suggest looking at the default heap access method for inspiration.

Its version of slot_callbacks returns &TTSOpsBufferHeapTuple:

static const TupleTableSlotOps *
heapam_slot_callbacks(Relation relation)
{
    return &TTSOpsBufferHeapTuple;
}

I have no idea what that means, but since it is defined in src/backend/executor/execTuples.c it doesn't seem to be tied to the heap access method implementation. Let's try it.

While it works initially, I noticed later on that TTSOpsBufferHeapTuple turns out not to be the right choice here. TTSOpsVirtual seems to be the right implementation.

@@ -19,7 +19,7 @@
   Relation relation
 ) {
   DEBUG_FUNC();
-  return NULL;
+  return &TTSOpsVirtual;
 }

 static TableScanDesc memam_beginscan(

Build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
psql:test.sql:6: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:test.sql:6: error: connection to server was lost

It still crashes. But this time in /tmp/pgtam.log we made it into a new method!

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan

`scan_begin`

The function signature is:

TableScanDesc heap_beginscan(
  Relation relation,
  Snapshot snapshot,
  int nkeys,
  ScanKey key,
  ParallelTableScanDesc parallel_scan,
  uint32 flags
);

Since we just implemented stub versions of all the methods, we've been returning NULL. Since we're failing in this function, maybe we should try returning something that isn't NULL.

By looking at the definition of TableScanDesc, we can see it is a pointer to the TableScanDescData struct defined in src/include/access/relscan.h.

Let's malloc a TableScanDescData, free it in endscan, and return the TableScanDescData instance in beginscan:

@@ -30,8 +30,12 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
+  TableScanDescData* scan = {};
   DEBUG_FUNC();
-  return NULL;
+
+  scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
+
+  return (TableScanDesc)scan;
 }

 static void memam_rescan(
@@ -87,6 +87,7 @@

 static void memam_endscan(TableScanDesc sscan) {
   DEBUG_FUNC();
+  free(sscan);
 }

Build and run (you can do it on your own). No difference.

I got stuck for a while here too. Clearly something must be filled out in this struct but it could be anything. Through trial and error I realized the one field that must be filled out is scan->rs_rd.

@@ -34,6 +34,7 @@
   DEBUG_FUNC();

   scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
+  scan->rs_rd = relation;

   return (TableScanDesc)scan;
 }

We build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
 a
---
(0 rows)

And it works! It doesn't return anything but that's correct. There's nothing to return.

So what if we actually want to return something? Let's check our logs in /tmp/pgtam.log.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_endscan

Ok, I'm getting the gist of the API. A full table scan (which this is, because there are no indexes at play) starts with an initialization for a slot, then the scan begins, then getnextslot is called for each row, and then endscan is called to allow for cleanup.

So let's try returning a row in getnextslot.

`getnextslot`

The getnextslot signature is:

bool memam_getnextslot(
  TableScanDesc sscan,
  ScanDirection direction,
  TupleTableSlot *slot
);

So the sscan should be what we returned from beginscan and the interface docs say the current row gets stored in slot.

The return value seems to indicate whether or not we've reached the end of the scan. However, the scan will still end even if you return true if the slot is not filled out correctly. If the slot is filled out correctly and you unconditionally return true, you will crash the process.

Let's take a look at the definition of TupleTableSlot:

typedef struct TupleTableSlot
{
    NodeTag     type;
#define FIELDNO_TUPLETABLESLOT_FLAGS 1
    uint16      tts_flags;      /* Boolean states */
#define FIELDNO_TUPLETABLESLOT_NVALID 2
    AttrNumber  tts_nvalid;     /* # of valid values in tts_values */
    const TupleTableSlotOps *const tts_ops; /* implementation of slot */
#define FIELDNO_TUPLETABLESLOT_TUPLEDESCRIPTOR 4
    TupleDesc   tts_tupleDescriptor;    /* slot's tuple descriptor */
#define FIELDNO_TUPLETABLESLOT_VALUES 5
    Datum      *tts_values;     /* current per-attribute values */
#define FIELDNO_TUPLETABLESLOT_ISNULL 6
    bool       *tts_isnull;     /* current per-attribute isnull flags */
    MemoryContext tts_mcxt;     /* slot itself is in this context */
    ItemPointerData tts_tid;    /* stored tuple's tid */
    Oid         tts_tableOid;   /* table oid of tuple */
} TupleTableSlot;

tts_values is an array of Datum (which is a Postgres value). So that sounds like the actual values of the row. The tts_isnull field also looks important since that seems to be whether each value in the row is null or not. And tts_nvalid sounds important too since presumably it's the length of the tts_isnull and tts_values arrays.

The rest of it may or may not be important. Let's try filling out these three fields though and see what happens.

Datum

Back in the Postgres C extension documentation, we can see some simple examples of converting between C types and Postgres's Datum type.

For example:

Datum
add_one(PG_FUNCTION_ARGS)
{
    int32   arg = PG_GETARG_INT32(0);

    PG_RETURN_INT32(arg + 1);
}

If we look at the definition of PG_RETURN_INT32 in src/include/fmgr.h, we see:

#define PG_RETURN_INT32(x)   return Int32GetDatum(x)

So Int32GetDatum() is the function we'll use to set a Datum for a cell in a row.

@@ -54,13 +54,26 @@
   DEBUG_FUNC();
 }

+static bool done = false;
 static bool memam_getnextslot(
   TableScanDesc sscan,
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
   DEBUG_FUNC();
-  return false;
+
+  if (done) {
+    return false;
+  }
+
+  slot->tts_nvalid = 1;
+  slot->tts_values = (Datum*)malloc(sizeof(Datum) * slot->tts_nvalid);
+  slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
+  slot->tts_isnull = (bool*)malloc(sizeof(bool) * slot->tts_nvalid);
+  slot->tts_isnull[0] = false;
+  done = true;
+
+  return true;
 }

 static IndexFetchTableData* memam_index_fetch_begin(Relation rel) {

The goal is that we return a single row and then exit the scan. It will have one 32-bit integer cell (remember we created the table CREATE TABLE x (a INT); INT is shorthand for INT4 which is a 32-bit integer) that will have the value 314.

But if we build and run this, we get no rows.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
 a
---
(0 rows)

I got stuck for a while here. Plugging my getnextslot code into ChatGPT helped. One thing it gave me to try was calling ExecStoreVirtualTuple on the slot. I noticed that the built-in heap access method also called a function like this in getnextslot.

And I realized that tts_nvalid is already set up and the memory for tts_values and tts_isnull is already allocated. So the code became a little simpler.

@@ -66,11 +66,9 @@
     return false;
   }

-  slot->tts_nvalid = 1;
-  slot->tts_values = (Datum*)malloc(sizeof(Datum) * slot->tts_nvalid);
   slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
-  slot->tts_isnull = (bool*)malloc(sizeof(bool) * slot->tts_nvalid);
   slot->tts_isnull[0] = false;
+  ExecStoreVirtualTuple(slot);
   done = true;

   return true;

Build and run:

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
  a
-----
 314
(1 row)

Fantastic!

Creating a table

Now that we've proven we can return random data, let's set up infrastructure for storing tables in memory.

@@ -15,6 +15,41 @@
 FILE* fd;
 #define DEBUG_FUNC() fprintf(fd, "in %s\n", __func__);

+
+struct Column {
+  int value;
+};
+
+struct Row {
+  struct Column* columns;
+  size_t ncolumns;
+};
+
+#define MAX_ROWS 100
+struct Table {
+  char* name;
+  struct Row* rows;
+  size_t nrows;
+};
+
+#define MAX_TABLES 100
+struct Database {
+  struct Table* tables;
+  size_t ntables;
+};
+
+struct Database* database;
+
+static void get_table(struct Table** table, Relation relation) {
+  char* this_name = NameStr(relation->rd_rel->relname);
+  for (size_t i = 0; i < database->ntables; i++) {
+    if (strcmp(database->tables[i].name, this_name) == 0) {
+      *table = &database->tables[i];
+      return;
+    }
+  }
+}
+
 static const TupleTableSlotOps* memam_slot_callbacks(
   Relation relation
 ) {

Based on what we logged in /tmp/pgtam.log it seems like memam_relation_set_new_filelocator is called when a new table is created. So let's handle adding a new table there.

@@ -233,7 +268,16 @@
   TransactionId *freezeXid,
   MultiXactId *minmulti
 ) {
+  struct Table table = {};
   DEBUG_FUNC();
+
+  table.name = strdup(NameStr(rel->rd_rel->relname));
+  fprintf(fd, "Created table: [%s]\n", table.name);
+  table.rows = (struct Row*)malloc(sizeof(struct Row) * MAX_ROWS);
+  table.nrows = 0;
+
+  database->tables[database->ntables] = table;
+  database->ntables++;
 }

 static void memam_relation_nontransactional_truncate(

Finally, we'll initialize the in-memory Database* when the handler is loaded.

@@ -428,5 +472,11 @@
   setvbuf(fd, NULL, _IONBF, 0); // Prevent buffering
   fprintf(fd, "\n\nmem_tableam handler loaded\n");

+  if (database == NULL) {
+    database = (struct Database*)malloc(sizeof(struct Database));
+    database->ntables = 0;
+    database->tables = (struct Table*)malloc(sizeof(struct Table) * MAX_TABLES);
+  }
+
   PG_RETURN_POINTER(&memam_methods);
 }

If we build and run, we won't notice anything new.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
  a
-----
 314
(1 row)

But we should see a message in /tmp/pgtam.log about the new table being created.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

And there it is! Creation looks good.

Inserting rows

Let's add INSERT INTO x VALUES (23), (101); to test.sql and run the SQL script.

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
 314
(1 row)

And let's check the log to see what method is called when we try to INSERT.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_slot_callbacks
in memam_tuple_insert
in memam_tuple_insert
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

tuple_insert seems to be the method! Looks like it gets called once for each row to insert. Perfect.

The signature for tuple_insert is:

void memam_tuple_insert(
  Relation relation,
  TupleTableSlot *slot,
  CommandId cid,
  int options,
  BulkInsertState bistate
);

We can get the table name from relation, and instead of writing to slot we can read from slot->tts_values instead.

@@ -141,7 +141,38 @@
   int options,
   BulkInsertState bistate
 ) {
+  TupleDesc desc = RelationGetDescr(relation);
+  struct Table* table = NULL;
+  struct Column column = {};
+  struct Row row = {};
+
   DEBUG_FUNC();
+
+  get_table(&table, relation);
+  if (table == NULL) {
+    elog(ERROR, "table not found");
+    return;
+  }
+
+  if (table->nrows == MAX_ROWS) {
+    elog(ERROR, "cannot insert more rows");
+    return;
+  }
+
+  row.ncolumns = desc->natts;
+  Assert(slot->tts_nvalid == row.ncolumns);
+  Assert(row.ncolumns > 0);
+
+  row.columns = (struct Column*)malloc(sizeof(struct Column) * row.ncolumns);
+  for (size_t i = 0; i < row.ncolumns; i++) {
+    Assert(desc->attrs[i].atttypid == INT4OID);
+    column.value = DatumGetInt32(slot->tts_values[i]);
+    row.columns[i] = column;
+    fprintf(fd, "Got value: %d\n", column.value);
+  }
+
+  table->rows[table->nrows] = row;
+  table->nrows++;
 }

 static void memam_tuple_insert_speculative(

Build and run and again we won't notice anything new.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
 314
(1 row)

But if we check the logs, we should see the two column values we inserted, one for each row.

$ cat /tmp/pgtam.log


mem_tableam handler loaded


mem_tableam handler loaded
in memam_relation_set_new_filelocator
Created table: [x]


mem_tableam handler loaded
in memam_relation_needs_toast_table


mem_tableam handler loaded
in memam_slot_callbacks
in memam_tuple_insert
Got value: 23
in memam_tuple_insert
Got value: 101
in memam_estimate_rel_size
in memam_slot_callbacks
in memam_beginscan
in memam_getnextslot
in memam_getnextslot
in memam_endscan

Woohoo!

Un-hardcoding the scan

The final thing we need to do is drop the hardcoded 314 we returned from getnextslot and instead we need to look up the current table and return rows from it. This also means we need to keep track of which row we're on. So beginscan will also need to change slightly.

@@ -57,6 +56,14 @@
   return &TTSOpsVirtual;
 }

+
+struct MemScanDesc {
+  TableScanDescData rs_base; // Base class from access/relscan.h.
+
+  // Custom data.
+  uint32 cursor;
+};
+
 static TableScanDesc memam_beginscan(
   Relation relation,
   Snapshot snapshot,
@@ -65,11 +72,13 @@
   ParallelTableScanDesc parallel_scan,
   uint32 flags
 ) {
-  TableScanDescData* scan = {};
-  DEBUG_FUNC();
+  struct MemScanDesc* scan;

-  scan = (TableScanDescData*)malloc(sizeof(TableScanDescData));
-  scan->rs_rd = relation;
+  DEBUG_FUNC();
+
+  scan = (struct MemScanDesc*)malloc(sizeof(struct MemScanDesc));
+  scan->rs_base.rs_rd = relation;
+  scan->cursor = 0;

   return (TableScanDesc)scan;
 }
@@ -89,23 +97,26 @@
   DEBUG_FUNC();
 }

-static bool done = false;
 static bool memam_getnextslot(
   TableScanDesc sscan,
   ScanDirection direction,
   TupleTableSlot *slot
 ) {
+  struct MemScanDesc* mscan = (struct MemScanDesc*)sscan;
+  struct Table* table = NULL;
   DEBUG_FUNC();

-  if (done) {
+  ExecClearTuple(slot);
+
+  get_table(&table, mscan->rs_base.rs_rd);
+  if (table == NULL || mscan->cursor == table->nrows) {
     return false;
   }

-  slot->tts_values[0] = Int32GetDatum(314 /* Some unique-looking value */);
+  slot->tts_values[0] = Int32GetDatum(table->rows[mscan->cursor].columns[0].value);
   slot->tts_isnull[0] = false;
   ExecStoreVirtualTuple(slot);
-  done = true;
-
+  mscan->cursor++;
   return true;
 }

Let's try it out.

$ make && sudo make install
$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to table x
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
INSERT 0 2
  a
-----
  23
 101
(2 rows)

And there we have it. :)

Awesome SQL power

So we tried one table and we tried a SELECT without anything else.

What happens if we use more of SQL? Let's create another table and try some more complex queries. Edit test.sql:

DROP EXTENSION IF EXISTS pgtam CASCADE;
CREATE EXTENSION pgtam;
CREATE TABLE x(a INT) USING mem;
CREATE TABLE y(b INT) USING mem;
INSERT INTO x VALUES (23), (101);
SELECT a FROM x;
SELECT a + 100 FROM x WHERE a = 23;
SELECT a, COUNT(1) FROM x GROUP BY a ORDER BY COUNT(1) DESC;
SELECT b FROM y;

Run it:

$ /usr/local/pgsql/bin/psql postgres -f test.sql
psql:test.sql:1: NOTICE:  drop cascades to 2 other objects
DETAIL:  drop cascades to table x
drop cascades to table y
DROP EXTENSION
CREATE EXTENSION
CREATE TABLE
CREATE TABLE
INSERT 0 2
  a
-----
  23
 101
(2 rows)

 ?column?
----------
      123
(1 row)

  a  | count
-----+-------
  23 |     1
 101 |     1
(2 rows)

 b
---
(0 rows)

Pretty sweet!

Next steps

It would be neat to build a storage engine that reads from and writes to a CSV a la MySQL's CSV storage engine. Or a storage engine that uses RocksDB.

It would also be good to figure out how indexes work, how deletions work, how updates and DDL beyond CREATE works.

And I should probably contribute some of this to the table access method docs which are pretty sparse at the moment.

I've been working this week to understand Postgres Table Access Methods for alternative storage engines.

Especially challenging because the documentation is pretty sparse and few minimal implementations exist.

I wrote up my approach!https://t.co/LQGglRkev5 pic.twitter.com/v0MeOu4Hbr
— Phil Eaton (@eatonphil) November 2, 2023

SDXL LoRA Training

smcleod.net — Mon, 30 Oct 2023 22:00:00 GMT

A talk I gave to some peers on creating your own SDXL LoRA models from my tinkering around over the last few weeks.

Partitioning The Index

Weblog on marginalia.nu — Mon, 30 Oct 2023 02:00:00 GMT

So a bit of an update on what I’ve been working on. This will be adapted into release notes in a while, but I haven’t quite wrapped a bow on the change set yet. Still, it has certainly been a few weeks. Didn’t quite land how busy I’ve been until I set down to draft this post. Them’s some changes, and I’m skipping a few to keep this meandering post at a sane length.

Hyperdimension Neptunia Re;Birth1 review

ブラック — Sat, 28 Oct 2023 03:00:00 GMT

I can't believe it's not kusoge!

The story of Hyperdimension Neptunia Re;Birth1 isn't really anything special. It's ultimately just kind of a generic save-the-world adventure story and the actual substance is in the (comedic) character interactions more than anything else. I don't think I can claim to have loved the story, but I did find it charming in its own way. At least I don't regret reading through it. The English translation seems a bit liberal, but not to a degree that it'd completely ruin the entire thing either.

The combat is surprisingly good. There's kind of a good rhythm to the whole thing, where you first try to get a sneak attack in the overworld, then take turns in the battle to position yourself optimally, chip away at the enemy's shields and then deal heavy damage to them. During boss fights you would also look to optimise the mileage you get out of your powerful EX skills too. It's also fairly fast-paced, with the possibility to skip most of the animations, so you can really charge through your battles.

And boy is it a good thing the combat feels as good as it does since this game leans very heavily on grinding to pad out the game. I don't think there's any way to get through the story without grinding. You just won't get the levels to clear all the level check bosses. I had trouble beating a couple of the bosses, both early in the game and late in the game, and when I searched for tips in case I missed something, most of the advice on the Internet was "grind more". So if there's a way to beat the game without grinding, it went unnoticed by me and seemingly the rest of the Internet too.

The grinding also extends to the achievements. About half of the achievements for this game require grinding, be it for character levels or money. The amount of grinding is pretty mental. Really has this "low budget, must stretch out the game somehow" thing written all over it.

Not to say that there's no aspect for battle tactics and gameplay. For example, I lost to a boss, restarted the fight and then won the second bout. No grinding needed in the between the attempts. However, I'm not sure if the victory game down to skills or just luck. I feel like there's a limited amount of things you can do in regards to your tactics when the boss has the ability to insta-kill two members from your three-member party. It's quite possible that the boss just unleashed less of its wave of death area-of-effect attacks on my party on the latter attempt.

The rest of the game mechanics are kinda lame. The quests feel like they were added to check a checkbox, same as with the colosseum. And the share balancing act that you need to do in order to unlock the true ending or to unlock the little sisters is also not amazing, especially since it'll involve those quests and colosseum. Not massively time consuming thankfully, but still a bit of a mindless grind, doing the same things again and again to shift some percentages around.

Technically the game isn't the best. While I do like the art, despite the fact that the game reuses assets like it's going out of fashion, the graphics felt so bad and blurry at the beginning that I felt the need to install the Neptastic Mod on my Steam Deck. That did certainly help the game look nicer, although the FPS would not necessarily always be a stable 60 FPS. It also felt like the FPS would gradually get worse as I kept the game open as the Steam Deck was asleep. I think I went from around 40 FPS to 60 FPS in the same dungeon by just simply restarting the game. Deck users beware.

But in general, Hyperdimension Neptunia Re;Birth1 does run pretty well on a Steam Deck. Maybe not as well as you'd expect considering that this is originally a PlayStation Vita game, but well enough even with the modding. Combined with the fast-paced combat and heavy focus on grinding, it makes for a pretty decent portable time killer when you have a couple of minutes to do something. Not really sure if I'd ever want to actually sit in front of my gaming PC and play through this game. It just doesn't feel like that kind of a game (quite possibly due to its Vita roots), which is why I played it from start to finish on the Steam Deck. Fairly certain even that I beat most of the game while sitting on the toilet.

Hyperdimension Neptunia Re;Birth1 is a bit of a messy game with good core gameplay loop and enough charm to keep you around. For the couple of euro I paid for it, it did make for some good portable entertainment. Just don't really expect much beyond that from it.

The contest of wills begins

--Mike-- — Fri, 27 Oct 2023 14:53:03 GMT

So, this morning when I click on my morning link to Peter Zeihan, I've been greeted with this, instead of him talking from some new location.

We've entered the contest of wills. Google/YouTube thinks that they can force ads down my throat in service of their customers(the advertisers), and is willing to block my ad blocker in an attempt to force my compliance.

I'm sure there will eventually be a different platform to watch Peter's videos on... and the wait begins. We programmers/technicians have a superpower, we're willing to put up with a ton of grief to bend things to our will, it's how we route around the normal rules and get shit done.

I curate my sources, and Peter is one of them, I'm not willing to pollute that stream of information with the Ads that Google wants to force feed me. It'll be interesting to see how this all works out.

JavaScript regular expressions cheatsheet and examples

learnbyexample — Fri, 27 Oct 2023 03:00:00 GMT

Above diagram created using Regulex

This blog post gives an overview of regular expression syntax and features supported by JavaScript. Examples have been tested on the Chrome/Chromium console and includes features not available in other browsers and platforms. This post is an excerpt from my Understanding JavaScript RegExp book.

Elements that define a regular expression🔗

Note	Description
MDN: Regular Expressions	MDN reference for JavaScript regular expressions
`/pat/`	a RegExp object
`const pet = /dog/`	save regexp in a variable for reuse, clarity, etc
`/pat/.test(s)`	check if the pattern is present anywhere in the input string
	returns `true` or `false`
`i`	flag to ignore case when matching alphabets
`g`	flag to match all occurrences
`new RegExp('pat', 'i')`	construct RegExp from a string
	optional second argument specifies flags
	use backtick strings with `${}` for interpolation
`source`	property to convert a RegExp object to a string
	helps to insert a RegExp inside another RegExp
`flags`	property to get flags of a RegExp object
`s.replace(/pat/, 'repl')`	method for search and replace
`s.search(/pat/)`	gives the starting location of the match or `-1`
`s.split(/pat/)`	split a string based on regexp

Anchors	Description
`^`	restricts the match to the start of string
`$`	restricts the match to the end of string
`m`	flag to match the start/end of line with `^` and `$` anchors
	`\r`, `\n`, `\u2028` and `\u2029` are line separators
	DOS-style files use `\r\n`, may need special attention
`\b`	restricts the match to the start and end of words
	word characters: alphabets, digits, underscore
`\B`	matches wherever `\b` doesn't match

^, $ and \ are metacharacters in the above table, as these characters have a special meaning. Prefix a \ character to remove the special meaning and match such characters literally. For example, \^ will match a ^ character instead of acting as an anchor.

Feature	Description
`pat1\|pat2\|pat3`	multiple regexp combined as conditional OR
	each alternative can have independent anchors
`(pat)`	group pattern(s), also a capturing group
`a(b\|c)d`	same as `abd\|acd`
`(?:pat)`	non-capturing group
`(?<name>pat)`	named capture group
`.`	match any character except line separators
`s`	flag to match line separators as well
`[]`	character class, matches one character among many

Alternation precedence: pattern which matches earliest in the input gets higher priority. Tie-breaker is left-to-right if matches have the same starting location.

Greedy Quantifiers	Description
`?`	match `0` or `1` times
`*`	match `0` or more times
`+`	match `1` or more times
`{m,n}`	match `m` to `n` times
`{m,}`	match at least `m` times
`{n}`	match exactly `n` times
`pat1.*pat2`	any number of characters between `pat1` and `pat2`
`pat1.pat2\|pat2.pat1`	match both `pat1` and `pat2` in any order

Greedy here means that the above quantifiers will match as much as possible that'll also honor the overall regexp. Appending a ? to greedy quantifiers makes them non-greedy, i.e. match as minimally as possible. Quantifiers can be applied to literal characters, groups, backreferences and character classes.

Character class	Description
`[ae;o]`	match any of these characters once
`[3-7]`	range of characters from `3` to `7`
`[^=b2]`	negated set, match other than `=` or `b` or `2`
`[a-z-]`	`-` should be the first/last or escaped using `\` to match literally
`[+^]`	`^` shouldn't be the first character or escaped using `\`
`[\]\\]`	`]` and `\` should be escaped using `\`
	`[` doesn't need escaping, but `\[` can also be used
`\w`	similar to `[A-Za-z0-9_]` for matching word characters
`\d`	similar to `[0-9]` for matching digit characters
`\s`	similar to `[ \t\n\r\f\v]` for matching whitespace characters
	use `\W`, `\D`, and `\S` for their opposites respectively
`u`	flag to enable unicode matching
`v`	superset of `u` flag, enables additional features
`\p{}`	Unicode character sets
`\P{}`	negated Unicode character sets
	see MDN: Unicode character class escape for details
`\u{}`	specify Unicode characters using codepoints

Lookarounds	Description
lookarounds	create custom positive/negative assertions
	zero-width like anchors and not part of matching portions
`(?!pat)`	negative lookahead assertion
`(?<!pat)`	negative lookbehind assertion
`(?=pat)`	positive lookahead assertion
`(?<=pat)`	positive lookbehind assertion
	variable length lookbehind is allowed
`(?!pat1)(?=pat2)`	multiple assertions can be specified next to each other in any order
	as they mark a matching location without consuming characters
`((?!pat).)*`	Negates a regexp pattern

Matched portion	Description
`m = s.match(/pat/)`	assuming the `g` flag isn't used and regexp succeeds,
	returns an array with the matched portion and 3 properties
	`index` property gives the starting location of the match
	`input` property gives the input string `s`
	`groups` property gives dictionary of named capture groups
`m[0]`	for the above case, gives the entire matched portion
`m[N]`	matched portion of the Nth capture group
`d`	flag to get the starting and ending locations of the matching portions via the `indices` property
`s.match(/pat/g)`	returns only the matched portions, no properties
`s.matchAll(/pat/g)`	returns an iterator containing details for each matched portion and its properties
Backreference	gives the matched portion of the Nth capture group
	use `$1`, `$2`, `$3`, etc in the replacement section
	`$&` gives the entire matched portion
	$` gives the string before the matched portion
	`$'` gives the string after the matched portion
	use `\1`, `\2`, `\3`, etc within the regexp definition
`$$`	insert `$` literally in the replacement section
`$0N`	same as `$N`, allows to separate backreference and other digits
`\N\xhh`	allows to separate backreference and digits in the regexp definition
`(?<name>pat)`	named capture group
	use `\k<name>` for backreferencing in the regexp definition
	use `$<name>` for backreferencing in the replacement section

Regular expression examples🔗

test() method

> let sentence = 'This is a sample string'

> /is/.test(sentence)
< true
> /xyz/.test(sentence)
< false

> if (/ring/.test(sentence)) {
      console.log('mission success')
  }
< mission success

new RegExp() constructor

> new RegExp('dog', 'i')
< /dog/i

> new RegExp('123\\tabc')
< /123\tabc/

> let greeting = 'hi'
> new RegExp(`${greeting.toUpperCase()} there`)
< /HI there/

string and line anchors

// string anchors
> /^cat/.test('cater')
< true
> ['surrender', 'newer', 'door'].filter(w => /er$/.test(w))
< ['surrender', 'newer']

// use 'm' flag to match at the start/end of each line
> /^par$/m.test('spare\npar\nera\ndare')
< true

// escape metacharacters to match them literally
> /b\^2/.test('a^2 + b^2 - C*3')
< true

replace() method and word boundaries

> let items = 'catapults\nconcatenate\ncat'
> console.log(items.replace(/^/gm, '* '))
< * catapults
  * concatenate
  * cat

> let sample = 'par spar apparent spare part'
// replace 'par' only at the start of word
> sample.replace(/\bpar/g, 'X')
< 'X spar apparent spare Xt'
// replace 'par' at the end of word but not whole word 'par'
> sample.replace(/\Bpar\b/g, 'X')
< 'par sX apparent spare part'

alternations and grouping

// replace either 'cat' at the start of string or 'cat' at the end of word
> 'catapults concatenate cat scat'.replace(/^cat|cat\b/g, 'X')
< 'Xapults concatenate X sX'

// same as: /\bpark\b|\bpart\b/g
> 'park parked part party'.replace(/\bpar(k|t)\b/g, 'X')
< 'X parked X party'

MDN: Regular Expressions Guide provides the escapeRegExp() function, useful to automatically escape metacharacters.
- See also XRegExp, provides handy methods like XRegExp.escape() and XRegExp.union(). The union method has additional functionality of allowing a mix of string and RegExp literals and also takes care of renumbering backreferences.

> function escapeRegExp(string) {
    return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
  }

> function unionRegExp(arr) {
    return arr.map(w => escapeRegExp(w)).join('|')
  }

> new RegExp(unionRegExp(['c^t', 'dog$', 'f|x']), 'g')
< /c\^t|dog\$|f\|x/g

dot metacharacter and quantifiers

// matches character '2', any character and then character '3'
> '42\t35'.replace(/2.3/, '8')
< '485'
// 's' flag will allow line separators to be matched as well
> 'Hi there\nHave a Nice Day'.replace(/the.*ice/s, 'X')
< 'Hi X Day'

// same as: /part|parrot|parent/g
> 'par part parrot parent'.replace(/par(en|ro)?t/g, 'X')
< 'par X X X'

> ['abc', 'ac', 'abbc', 'xabbbcz'].filter(w => /ab{1,4}c/.test(w))
< ['abc', 'abbc', 'xabbbcz']

match() method

// entire matched portion
> 'too soon a song snatch'.match(/so+n/)[0]
< 'soon'
// matched portion of the second capture group
> let purchase = 'coffee:100g tea:250g sugar:75g chocolate:50g'
> purchase.match(/:(.*?)g.*?:(.*?)g.*?chocolate:(.*?)g/)[2]
< '250'

// starting location of the matching portion
> 'cat and dog'.match(/dog/).index
< 8
// start and end+1 location of the matching portion
> 'awesome'.match(/so/d).indices[0]
< [3, 5]

// get all matching portions with 'g' flag
// no properties or group portions
> 'par spar apparent spare part'.match(/\bs?par[et]\b/g)
< ['spare', 'part']

// useful for debugging purposes as well
> 'green:3.14:teal::brown:oh!:blue'.match(/:.*?:/g)
< [':3.14:', '::', ':oh!:']

matchAll() method

// same as: match(/so*n/g)
> Array.from('song too soon snatch'.matchAll(/so*n/g), m => m[0])
< ['son', 'soon', 'sn']
// get the starting index for each match
> Array.from('song too soon snatch'.matchAll(/so*n/g), m => m.index)
< [0, 9, 14]

// get only the capture group portions as an array for each match
> Array.from('2023/04,1986/Mar,'.matchAll(/(.*?)\/(.*?),/g), m => m.slice(1))
< (2) [Array(2), Array(2)]
  0: (2) ['2023', '04']
  1: (2) ['1986', 'Mar']
  length: 2
  [[Prototype]]: Array(0)

function/dictionary in the replacement section

> function titleCase(m, g1, g2) {
        return g1.toUpperCase() + g2.toLowerCase()
  }
> 'aBc ac ADC aBbBC'.replace(/(a)(.*?c)/ig, titleCase)
< 'Abc Ac Adc Abbbc'

> '1 42 317'.replace(/\d+/g, m => m*2)
< '2 84 634'

> let swap = { 'cat': 'tiger', 'tiger': 'cat' }
> 'cat tiger dog tiger cat'.replace(/cat|tiger/g, k => swap[k])
< 'tiger cat dog cat tiger'

split() method

// split based on one or more digit characters
> 'Sample123string42with777numbers'.split(/\d+/)
< ['Sample', 'string', 'with', 'numbers']
// include the portion that caused the split as well
> 'Sample123string42with777numbers'.split(/(\d+)/)
< ['Sample', '123', 'string', '42', 'with', '777', 'numbers']

// split based on digit or whitespace characters
> '**1\f2\n3star\t7 77\r**'.split(/[\d\s]+/)
< ['**', 'star', '**']

// use non-capturing group if capturing is not needed
> '123handed42handy777handful500'.split(/hand(?:y|ful)?/)
< ['123', 'ed42', '777', '500']

backreferencing with normal/non-capturing/named capture groups

// remove any number of consecutive duplicate words separated by space
// use \W+ instead of space to cover cases like 'a;a<-;a'
> 'aa a a a 42 f_1 f_1 f_13.14'.replace(/\b(\w+)( \1)+\b/g, '$1')
< 'aa a 42 f_1 f_13.14'

// add something around the entire matched portion
> '52 apples and 31 mangoes'.replace(/\d+/g, '($&)')
< '(52) apples and (31) mangoes'

// duplicate the first field and add it as the last field
> 'fork,42,nice,3.14'.replace(/,.+/, '$&,$`')
< 'fork,42,nice,3.14,fork'

// use non-capturing groups when backreferencing isn't needed
> '1,2,3,4,5,6,7'.replace(/^((?:[^,]+,){3})([^,]+)/, '$1($2)')
< '1,2,3,(4),5,6,7'

// named capture groups, same as: replace(/(\w+),(\w+)/g, '$2,$1')
> 'good,bad 42,24 x,y'.replace(/(?<fw>\w+),(?<sw>\w+)/g, '$<sw>,$<fw>')
< 'bad,good 24,42 y,x'

examples for lookarounds

// change 'cat' only if it is not followed by a digit character
// note that the end of string satisfies the given assertion
// 'catcat' has two matches as the assertion doesn't consume characters
> 'hey cats! cat42 cat_5 catcat'.replace(/cat(?!\d)/g, 'dog')
< 'hey dogs! cat42 dog_5 dogdog'

// change whole word only if it is not preceded by : or --
> ':cart apple --rest ;tea'.replace(/(?<!:|--)\b\w+/g, 'X')
< ':cart X --rest ;X'

// extract digits only if it is preceded by - and followed by ; or :
> '42 apple-5, fig3; x-83, y-20: f12'.match(/(?<=-)\d+(?=[;:])/g)
< ['20']

// words containing all lowercase vowels in any order
> let words = ['sequoia', 'questionable', 'exhibit', 'equation']
> words.filter(w => /(?=.*a)(?=.*e)(?=.*i)(?=.*o).*u/.test(w))
< ['sequoia', 'questionable', 'equation']

// replace only the third occurrence of 'cat'
> 'cat scatter cater scat'.replace(/(?<=(cat.*?){2})cat/, 'X')
< 'cat scatter Xer scat'

// match if 'do' is not there between 'at' and 'par'
> /at((?!do).)*par/.test('fox,cat,dog,parrot')
< false

u and v flags

// extract all consecutive letters, use \P{L} to invert the set
> 'fox:αλεπού,eagle:αετός'.match(/\p{L}+/gu)
< ['fox', 'αλεπού', 'eagle', 'αετός']

// extract all consecutive Greek letters
> 'fox:αλεπού,eagle:αετός'.match(/\p{sc=Greek}+/gu)
< ['αλεπού', 'αετός']

// extract whole words not surrounded by punctuation marks
> 'tie. ink east;'.match(/(?<!\p{P})\b\w+\b(?!\p{P})/gu)
< ['ink']

// remove all punctuation characters except . ! and ?
> let para = '"Hi", there! How *are* you? All fine here.'
> para.replace(/[\p{P}--[.!?]]+/gv, '')
< 'Hi there! How are you? All fine here.'

Debugging and Visualization tools🔗

As your regexp gets complicated, it can get difficult to debug when you run into issues. Building your regexp step by step from scratch and testing against input strings will go a long way in correcting the problem. To aid in such a process, you could use various online regexp tools.

regex101 is a popular site to test your regexp. You'll have to first choose the flavor as JavaScript. Then you can add your regexp, input strings, choose flags and an optional replacement string. Matching portions will be highlighted and explanation is offered in separate panes. There's also a quick reference and other features like link sharing, code generator, quiz, cheatsheet, etc.

Another useful tool is jex: regulex which converts your regexp to a railroad diagram, thus providing a visual aid to understanding the pattern.

Understanding JavaScript RegExp book🔗

Visit my repo learn_js_regexp for details about the book I wrote on JavaScript regular expressions. The ebook uses plenty of examples to explain the concepts from the basics and includes exercises to test your understanding. The cheatsheet and examples presented in this post are based on the contents of this book.

Understanding JavaScript RegExp book announcement

learnbyexample — Thu, 26 Oct 2023 03:00:00 GMT

Hello!

I just published a new version of "Understanding JavaScript RegExp" ebook. Added examples for d and v flags, corrected many mistakes, improved examples, exercises and so on.

This book will help you learn JavaScript Regular Expressions step-by-step from beginner to advanced levels with hundreds of examples and exercises.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of Understanding JavaScript RegExp for FREE till 05-Nov-2023. You can still pay if you wish ;)

All Books Bundle is just $12 (normal price $32) — includes all my 13 programming ebooks.

What's new?🔗

Examples and exercises added for d and v flags
Strings in code snippets changed to be uniformly represented in single quotes
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to Understanding JavaScript RegExp
New cover image
Images centered for EPUB format

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Testimonials🔗

Literally was having a mini-breakdown about not understanding Regex in algorithm solutions the other day and now I'm feeling so much better, so thank YOU! I genuinely feel like I'm developing the skill for spotting when and where to use them after so much practice!

— feedback on twitter

Table of Contents🔗

Preface
Why is it needed?
RegExp introduction
Anchors
Alternation and Grouping
Escaping metacharacters
Dot metacharacter and Quantifiers
Interlude: Tools for debugging and visualization
Working with matched portions
Character class
Groupings and backreferences
Interlude: Common tasks
Lookarounds
Unicode
Further Reading

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/learn_js_regexp/.

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_js_regexp for markdown source, exercise solutions, sample chapters and other details related to the book.

See my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

I would highly appreciate if you'd let me know how you felt about this book. It could be anything from a simple thank you, Gumroad rating, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_js_regexp/issues
E-mail: learn by example.net@gmail.com (without the spaces)
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Kenmore Dryer Repair

Pxtl.ca — Tue, 24 Oct 2023 03:00:00 GMT

Once again, everything is made of garbage.

Embed only the video from another post on X or Twitter

The NeoSmart Files — Sun, 22 Oct 2023 20:33:56 GMT

Twitter has a new-ish feature that lets you embed only the video from another post or tweet in a post/tweet of your own (without quote-replying the source tweet itself). Only the video is then embedded in your post, and a … Continue reading →

The post Embed only the video from another post on X or Twitter first appeared on The NeoSmart Files.

Four layers of self-improvement

jodavaho.io — Sun, 22 Oct 2023 03:00:00 GMT

I’ve been tracking my OKR’s since 2014. Sometimes I accomplish what I set out for, sometimes I don’t. Sometimes I accomplish what I put in OKRs but didn’t accomplish what I wanted.

Here’s what I’ve learned.

There are usually four types of OKRs, and not differentiating types makes it really hard to get what you wanted
Not only are there four, but they are roughly speaking prerequisites of each other.
Not only that, but looking at all the self-help books I’ve read, it actually seems like most the books also fit into these four categories, and just reading them out of order makes no sense.

These categories are:

Intent-building (actions that create action). Perhaps these are better known as “values”. When lost, you feel listless.
Foundational stuff (actions that everyone needs to do to be “better”). Basic self care is here. When lost, you feel unhealthy and “stuck” even if you have good intent.
Focus activities (the stuff you need to realize your specific vision of self). Family, job duties, social activities, etc are here. These “round out” an otherwise healthy person.
Specialized activities (the stuff that really defines who you are). These are the things you are proud of. Resume-level or hobby-level things like “I love boxing” or “I love making cabinets”.

So my strategy going forward is to work from the bottom up. If I find a day where I can’t get anything done, I’ll at least work on intent. Maybe it seems obvious, but just focusing on what I want and planning a better tomorrow makes tomorrow much easier. And focusing on intent allows you to remember throughout the day to work on your goals. Lately, just taking a break from work during the day to move an inch forward is a huge help. For example 10 pullups between meetings.

Anyway, I will shortly analyze 3rd quarter, but for now, this idea of layering on improvements is something I’m noodling as I redefine what success looks like.

io_uring basics: Writing a file to disk

Notes on software development — Thu, 19 Oct 2023 03:00:00 GMT

King and I wrote a blog post about building an event-driven cross-platform IO library that used io_uring on Linux. We sketched out how it works at a high level but I hadn't yet internalized how you actually code with io_uring. So I strapped myself down this week and wrote some benchmarks to build my intuition about io_uring and other IO models.

I started with implementations in Go and ported them to Zig to make sure I had done the Go versions decently. And I got some help from King and other internetters to find some inefficiencies in my code.

This post will walk through my process, getting increasingly efficient (and a little increasingly complex) ways to write an entire file to disk with io_uring, from Go and Zig.

Notably, we're not going to fsync() and we're not going to use O_DIRECT. So we won't be testing the entire IO pipeline from userland to disk hardware but just how fast IO gets to the kernel. The focus of this post is more on IO methods and using io_uring, not absolute numbers.

All code for this post is available on GitHub.

This code is going to indirectly show some differences in timing between Go and Zig. I could care less about benchmarketing. And I hope something about Zig vs Go is not what you take away from this post either.

The goal is to build an intuition and be generally correct. Observing the same relative behavior between implementations across two languages helps me gain confidence what I'm doing is correct.

io_uring

With normal blocking syscalls you just call read() or write() and wait for the results. io_uring is one of Linux's more powerful asynchronous IO offerings. Unlike epoll, you can use io_uring with both files and network connections. And unlike epoll you can even have the syscall run in the kernel.

To interact with io_uring, you register a submission queue for syscalls and their arguments. And you register a completion queue for syscall results.

You can batch many syscalls in one single call to io_uring, effectively turning up to N (4096 at most) syscalls into just one syscall. The kernel still does all the work of the N syscalls but you avoid some overhead.

As you check the completion queue and handle completed submissions, the submission queue is also freed all or somewhat, and you can now add more submissions.

For a more complete understanding, check out the kernel document Efficient IO with io_uring.

io_uring vs liburing

io_uring is a complex, low-level interface. Shuveb Hussain has an excellent series on programming io_uring. But that was too low-level for me as I was trying to figure out how to just get something working.

Instead, most people use liburing or a ported version of it like the Zig standard library's io_uring.zig or Iceber's iouring-go.

io_uring started clicking for me when I tried out the iouring-go library. So we'll start there.

Boilerplate

First off, let's set up some boilerplate for the Go and Zig code.

In main.go add:

package main

import (
  "bytes"
  "fmt"
  "os"
  "time"
)

func assert(b bool) {
  if !b {
    panic("assert")
  }
}

const BUFFER_SIZE = 4096

func readNBytes(fn string, n int) []byte {
  f, err := os.Open(fn)
  if err != nil {
    panic(err)
  }
  defer f.Close()

  data := make([]byte, 0, n)

  var buffer = make([]byte, BUFFER_SIZE)
  for len(data) < n {
    read, err := f.Read(buffer)
    if err != nil {
      panic(err)
    }

    data = append(data, buffer[:read]...)
  }

  assert(len(data) == n)

  return data
}

func benchmark(name string, data []byte, fn func(*os.File)) {
  fmt.Printf("%s", name)
  f, err := os.OpenFile("out.bin", os.O_RDWR | os.O_CREATE | os.O_TRUNC, 0755)
  if err != nil {
    panic(err)
  }

  t1 := time.Now()

  fn(f)

  s := time.Now().Sub(t1).Seconds()
  fmt.Printf(",%f,%f\n", s, float64(len(data))/s)

  if err := f.Close(); err != nil {
    panic(err)
  }

  assert(bytes.Equal(readNBytes("out.bin", len(data)), data))
}

And in main.zig add:

const std = @import("std");

const OUT_FILE = "out.bin";
const BUFFER_SIZE: u64 = 4096;

fn readNBytes(
  allocator: *const std.mem.Allocator,
  filename: []const u8,
  n: usize,
) ![]const u8 {
  const file = try std.fs.cwd().openFile(filename, .{});
  defer file.close();

  var data = try allocator.alloc(u8, n);
  var buf = try allocator.alloc(u8, BUFFER_SIZE);

  var written: usize = 0;
  while (data.len < n) {
    var nwritten = try file.read(buf);
    @memcpy(data[written..], buf[0..nwritten]);
    written += nwritten;
  }

  std.debug.assert(data.len == n);
  return data;
}

const Benchmark = struct {
  t: std.time.Timer,
  file: std.fs.File,
  data: []const u8,
  allocator: *const std.mem.Allocator,

  fn init(
    allocator: *const std.mem.Allocator,
    name: []const u8,
    data: []const u8,
  ) !Benchmark {
    try std.io.getStdOut().writer().print("{s}", .{name});

    var file = try std.fs.cwd().createFile(OUT_FILE, .{
      .truncate = true,
    });

    return Benchmark{
      .t = try std.time.Timer.start(),
      .file = file,
      .data = data,
      .allocator = allocator,
    };
  }

  fn stop(b: *Benchmark) void {
    const s = @as(f64, @floatFromInt(b.t.read())) / std.time.ns_per_s;
    std.io.getStdOut().writer().print(
      ",{d},{d}\n",
      .{ s, @as(f64, @floatFromInt(b.data.len)) / s },
    ) catch unreachable;

    b.file.close();

    var in = readNBytes(b.allocator, OUT_FILE, b.data.len) catch unreachable;
    std.debug.assert(std.mem.eql(u8, in, b.data));
    b.allocator.free(in);
  }
};

Keep it simple: write()

Now let's add the naive version of writing bytes to disk: calling write() repeatedly until all data has been written to disk.

In main.go:

func main() {
  size := 104857600 // 100MiB
  data := readNBytes("/dev/random", size)

  const RUNS = 10
  for i := 0; i < RUNS; i++ {
    benchmark("blocking", data, func(f *os.File) {
      for i := 0; i < len(data); i += BUFFER_SIZE {
        size := min(BUFFER_SIZE, len(data)-i)
        n, err := f.Write(data[i : i+size])
        if err != nil {
          panic(err)
        }

        assert(n == BUFFER_SIZE)
      }
    })
  }
}

And in main.zig:

pub fn main() !void {
  var allocator = &std.heap.page_allocator;

  const SIZE = 104857600; // 100MiB
  var data = try readNBytes(allocator, "/dev/random", SIZE);
  defer allocator.free(data);

  const RUNS = 10;
  var run: usize = 0;
  while (run < RUNS) : (run += 1) {
    {
      var b = try Benchmark.init(allocator, "blocking", data);
      defer b.stop();

      var i: usize = 0;
      while (i < data.len) : (i += BUFFER_SIZE) {
        const size = @min(BUFFER_SIZE, data.len - i);
        const n = try b.file.write(data[i .. i + size]);
        std.debug.assert(n == size);
      }
    }
  }
}

Let's build and run these programs and store the results to CSV we can analyze with DuckDB.

Go first:

$ go build main.go -o gomain
$ ./gomain > go.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'go.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.07251540000000001s	1.4GB/s

And Zig:

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'zig.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.0656907669s	1.5GB/s

Alright, we've got a baseline now and both language implementations are in the same ballpark.

Let's add a simple io_uring version!

io_uring, 1 entry, Go

The iouring-go library has really excellent documentation for getting started.

To keep it simple, we'll use io_uring with only 1 entry. Add the following to func main() after the existing benchmark() call in main.go:

benchmark("io_uring", data, func(f * os.File) {
  iour, err := iouring.New(1)
  if err != nil {
    panic(err)
  }
  defer iour.Close()

  for i := 0; i < len(data); i += BUFFER_SIZE {
    size := min(BUFFER_SIZE, len(data)-i)
    prepRequest := iouring.Pwrite(int(f.Fd()), data[i : i+size], uint64(i))
    res, err := iour.SubmitRequest(prepRequest, nil)
    if err != nil {
      panic(err)
    }

    <-res.Done()
    i, err := res.ReturnInt()
    if err != nil {
      panic(err)
    }
    assert(size == i)
  }
})

Note that benchmark takes care of f.Seek(0) before each run. And it also validates that the file contents are equivalent to the input data. So it validates the benchmark for correctness.

Alright, let's run this new Go implementation with io_uring!

$ go mod init gomain
$ go mod tidy
$ go build main.go -o gomain
$ ./gomain > go.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'go.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.0811486s	1.3GB/s
io_uring	0.5083049999999999s	213.2MB/s

Well that looks terrible.

Let's port it to Zig to see if we notice the same behavior there.

io_uring, 1 entry, Zig

There isn't an official Zig tutorial on io_uring I'm aware of. But io_uring.zig is easy enough to browse through. And there are tests in that file that also show how to use it.

And now that we've explored a bit in Go the basic gist should be similar:

initialize io_uring
submit an entry
wait for it to finish
move on

Add the following to fn main() after the existing benchmark block in main.zig:

{
  var b = try Benchmark.init(allocator, "iouring", data);
  defer b.stop();

  const entries = 1;
  var ring = try std.os.linux.IO_Uring.init(entries, 0);
  defer ring.deinit();

  var i: usize = 0;
  while (i < data.len) : (i += BUFFER_SIZE) {
    const size = @min(BUFFER_SIZE, data.len - i);
    _ = try ring.write(0, b.file.handle, data[i .. i + size], i);

    const submitted = try ring.submit_and_wait(1);
    std.debug.assert(submitted == 1);

    const cqe = try ring.copy_cqe();
    std.debug.assert(cqe.err() == .SUCCESS);
    std.debug.assert(cqe.res >= 0);
    const n = @as(usize, @intCast(cqe.res));
    std.debug.assert(n <= BUFFER_SIZE);
  }
}

Now build and run:

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'zig.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.06650093630000001s	1.5GB/s
io_uring	0.17542890139999998s	597.7MB/s

Well it's similarly pretty bad. But our implementation ignores one major aspect of io_uring: batching requests.

Let's do some refactoring!

io_uring, N entries, Go

To support submitting N entries, we're going to have an inner loop running up to N that fills up N entries to io_uring.

Then we'll wait for the N submissions to complete and check their results.

We'll keep going until we write the entire file.

All of this can stay inside the loop in main, I'm just dropping preceding whitespace for nicer formatting here:

benchmarkIOUringNEntries := func (nEntries int) {
  benchmark(fmt.Sprintf("io_uring_%d_entries", nEntries), data, func(f * os.File) {
    iour, err := iouring.New(uint(nEntries))
    if err != nil {
      panic(err)
    }
    defer iour.Close()

    requests := make([]iouring.PrepRequest, nEntries)

    for i := 0; i < len(data); i += BUFFER_SIZE * nEntries {
      submittedEntries := 0
      for j := 0; j < nEntries; j++ {
        base := i + j * BUFFER_SIZE
        if base >= len(data) {
          break
        }
        submittedEntries++
        size := min(BUFFER_SIZE, len(data)-i)
        requests[j] = iouring.Pwrite(int(f.Fd()), data[base : base+size], uint64(base))
      }

      if submittedEntries == 0 {
        break
      }

      res, err := iour.SubmitRequests(requests[:submittedEntries], nil)
      if err != nil {
        panic(err)
      }

      <-res.Done()

      for _, result := range res.ErrResults() {
        _, err := result.ReturnInt()
        if err != nil {
          panic(err)
        }
      }
    }
  })
}
benchmarkIOUringNEntries(1)
benchmarkIOUringNEntries(128)

There are some specific things in there to notice.

First, toward the end of the file we may not have N entries to submit. We may have 1 or even 0.

If we have 0 to submit, we need to not even submit anything otherwise the Go library hangs. Similarly, if we don't slice requests to requests[:submittedEntries], the Go library will segfault if submittedEntries < N.

Other than that, let's build and run this!

$ go build -o gomain
$ ./gomain > go.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'go.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.0740368s	1.4GB/s
io_uring_128_entries	0.127519s	836.6MB/s
io_uring_1_entries	0.46831579999999995s	226.9MB/s

Now we're getting somewhere! Still half the throughput but a 4x improvement from using only a single entry.

Let's port the N entry code to Zig.

io_uring, N entries, Zig

Unlike Go we can't do closures, so we'll have to make benchmarkIOUringNEntries a top-level function and keep the calls to it in the loop in main:

pub fn main() !void {
    var allocator = &std.heap.page_allocator;

    const SIZE = 104857600; // 100MiB
    var data = try readNBytes(allocator, "/dev/random", SIZE);
    defer allocator.free(data);

    const RUNS = 10;
    var run: usize = 0;
    while (run < RUNS) : (run += 1) {
        {
            var b = try Benchmark.init(allocator, "blocking", data);
            defer b.stop();

            var i: usize = 0;
            while (i < data.len) : (i += BUFFER_SIZE) {
                const size = @min(BUFFER_SIZE, data.len - i);
                const n = try b.file.write(data[i .. i + size]);
                std.debug.assert(n == size);
            }
        }

        try benchmarkIOUringNEntries(allocator, data, 1);
        try benchmarkIOUringNEntries(allocator, data, 128);
    }
}

And for the implementation itself, the only two big differences from the first version are that we'll bulk-read completion events (cqes) and that we'll create and wait for many submissions at once.

fn benchmarkIOUringNEntries(
  allocator: *const std.mem.Allocator,
  data: []const u8,
  nEntries: u13,
) !void {
  const name = try std.fmt.allocPrint(allocator.*, "iouring_{}_entries", .{nEntries});
  defer allocator.free(name);

  var b = try Benchmark.init(allocator, name, data);
  defer b.stop();

  var ring = try std.os.linux.IO_Uring.init(nEntries, 0);
  defer ring.deinit();

  var cqes = try allocator.alloc(std.os.linux.io_uring_cqe, nEntries);
  defer allocator.free(cqes);

  var i: usize = 0;
  while (i < data.len) : (i += BUFFER_SIZE * nEntries) {
    var submittedEntries: u32 = 0;
    var j: usize = 0;
    while (j < nEntries) : (j += 1) {
      const base = i + j * BUFFER_SIZE;
      if (base >= data.len) {
        break;
      }
      submittedEntries += 1;
      const size = @min(BUFFER_SIZE, data.len - base);
      _ = try ring.write(0, b.file.handle, data[base .. base + size], base);
    }

    const submitted = try ring.submit_and_wait(submittedEntries);
    std.debug.assert(submitted == submittedEntries);

    const waited = try ring.copy_cqes(cqes[0..submitted], submitted);
    std.debug.assert(waited == submitted);

    for (cqes[0..submitted]) |*cqe| {
      std.debug.assert(cqe.err() == .SUCCESS);
      std.debug.assert(cqe.res >= 0);
      const n = @as(usize, @intCast(cqe.res));
      std.debug.assert(n <= BUFFER_SIZE);
    }
  }
}

Let's build and run:

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'zig.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
blocking	0.0674331114s	1.5GB/s
iouring_128_entries	0.06773539590000001s	1.5GB/s
iouring_1_entries	0.1855542556s	569.9MB/s

Huh, that's surprising! We caught up to blocking writes with io_uring in Zig, but not in Go, even though we made good progress in Go.

Ring buffers

But we can do a bit better. We're doing batching, but the API is called "io_uring" not "io_batch". We're not even making use of the ring buffer behavior io_uring gives us!

We are waiting for all submitted results complete. But there's no reason to do that. Instead we should submit as much as we can. But we should not block waiting for completions. We should handle completions when they happen. And we should retry submissions until we're done reading. Retrying if there's no space for the moment.

Unfortunately the Go library doesn't seem to expose this ring behavior of io_uring. Or I've missed it.

But we can do it in Zig. Let's go.

io_uring, ring buffer, Zig

We need to change the way we track which offsets we need to submit so far. We also need to keep the loop going until we are sure we have written all data. And we need to stop blocking on the number we submitted; never blocking at all.

fn benchmarkIOUringNEntries(
  allocator: *const std.mem.Allocator,
  data: []const u8,
  nEntries: u13,
) !void {
  const name = try std.fmt.allocPrint(allocator.*, "iouring_{}_entries", .{nEntries});
  defer allocator.free(name);

  var b = try Benchmark.init(allocator, name, data);
  defer b.stop();

  var ring = try std.os.linux.IO_Uring.init(nEntries, 0);
  defer ring.deinit();

  var cqes = try allocator.alloc(std.os.linux.io_uring_cqe, nEntries);
  defer allocator.free(cqes);

  var written: usize = 0;
  var i: usize = 0;
  while (i < data.len or written < data.len) {
    var submittedEntries: u32 = 0;
    var j: usize = 0;
    while (true) {
      const base = i + j * BUFFER_SIZE;
      if (base >= data.len) {
        break;
      }
      const size = @min(BUFFER_SIZE, data.len - base);
      _ = ring.write(0, b.file.handle, data[base .. base + size], base) catch |e| switch (e) {
        error.SubmissionQueueFull => break,
        else => unreachable,
      };
      submittedEntries += 1;
      i += size;
    }

    _ = try ring.submit_and_wait(0);
    const cqesDone = try ring.copy_cqes(cqes, 0);

    for (cqes[0..cqesDone]) |*cqe| {
      std.debug.assert(cqe.err() == .SUCCESS);
      std.debug.assert(cqe.res >= 0);
      const n = @as(usize, @intCast(cqe.res));
      std.debug.assert(n <= BUFFER_SIZE);
      written += n;
    }
  }
}

The code got a bit simpler! Granted, we're omitting error handling.

Build and run:

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'zig.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
iouring_128_entries	0.06035423609999999s	1.7GB/s
iouring_1_entries	0.0610197624s	1.7GB/s
blocking	0.0671628515s	1.5GB/s

Not bad!

Crank it up

We've been inserting 100MiB of data. Let's go up to 1GiB to see how that affects things. Ideally the more data we write the more we reflect realistic long-term results.

In main.zig just change SIZE to 1073741824. Rebuild and run:

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'out.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
iouring_128_entries	0.6063814535s	1.7GB/s
iouring_1_entries	0.6167537295000001s	1.7GB/s
blocking	0.6831747749s	1.5GB/s

No real difference, perfect!

Let's make one more change though. Let's up the BUFFER_SIZE from 4KiB to 1MiB.

$ zig build-exe main.zig
$ ./main > zig.csv
$ duckdb -c "select column0 as method, avg(cast(column1 as double)) || 's' avg_time, format_bytes(avg(column2::double)::bigint) || '/s' as avg_throughput from 'out.csv' group by column0 order by avg(cast(column1 as double)) asc"

method	avg_time	avg_throughput
iouring_128_entries	0.2756831357s	3.8GB/s
iouring_1_entries	0.27575404880000004s	3.8GB/s
blocking	0.2833337046s	3.7GB/s

Hey that's an improvement!

Control

All these numbers are machine-specific obviously. So what does an existing tool like fio say? (Assuming I'm using it correctly. I await your corrections!)

With a 4KiB buffer size:

$ fio --name=fiotest --rw=write --size=1G --bs=4k --group_reporting --ioengine=sync
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1)
fiotest: (groupid=0, jobs=1): err= 0: pid=2437359: Thu Oct 19 23:33:42 2023
  write: IOPS=282k, BW=1102MiB/s (1156MB/s)(1024MiB/929msec); 0 zone resets
    clat (nsec): min=2349, max=54099, avg=2709.48, stdev=1325.83
     lat (nsec): min=2390, max=54139, avg=2752.89, stdev=1334.62
    clat percentiles (nsec):
     |  1.00th=[ 2416],  5.00th=[ 2416], 10.00th=[ 2416], 20.00th=[ 2448],
     | 30.00th=[ 2448], 40.00th=[ 2448], 50.00th=[ 2448], 60.00th=[ 2480],
     | 70.00th=[ 2512], 80.00th=[ 2544], 90.00th=[ 2832], 95.00th=[ 3504],
     | 99.00th=[ 5792], 99.50th=[15296], 99.90th=[19584], 99.95th=[20096],
     | 99.99th=[22656]
   bw (  KiB/s): min=940856, max=940856, per=83.36%, avg=940856.00, stdev= 0.00, samples=1
   iops        : min=235214, max=235214, avg=235214.00, stdev= 0.00, samples=1
  lat (usec)   : 4=97.22%, 10=2.03%, 20=0.71%, 50=0.04%, 100=0.01%
  cpu          : usr=17.35%, sys=82.11%, ctx=26, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1102MiB/s (1156MB/s), 1102MiB/s-1102MiB/s (1156MB/s-1156MB/s), io=1024MiB (1074MB), run=929-929msec

1.2GB/s is about in the ballpark of what we got.

And with a 1MiB buffer size?

$ fio --name=fiotest --rw=write --size=1G --bs=1M --group_reporting --ioengine=sync
fiotest: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1
fio-3.33
Starting 1 process
fiotest: Laying out IO file (1 file / 1024MiB)

fiotest: (groupid=0, jobs=1): err= 0: pid=2437239: Thu Oct 19 23:32:09 2023
  write: IOPS=3953, BW=3954MiB/s (4146MB/s)(1024MiB/259msec); 0 zone resets
    clat (usec): min=221, max=1205, avg=241.83, stdev=43.93
     lat (usec): min=228, max=1250, avg=251.68, stdev=45.80
    clat percentiles (usec):
     |  1.00th=[  225],  5.00th=[  225], 10.00th=[  227], 20.00th=[  227],
     | 30.00th=[  231], 40.00th=[  233], 50.00th=[  235], 60.00th=[  239],
     | 70.00th=[  243], 80.00th=[  249], 90.00th=[  262], 95.00th=[  269],
     | 99.00th=[  302], 99.50th=[  318], 99.90th=[ 1074], 99.95th=[ 1205],
     | 99.99th=[ 1205]
  lat (usec)   : 250=80.96%, 500=18.85%
  lat (msec)   : 2=0.20%
  cpu          : usr=4.26%, sys=94.96%, ctx=3, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3954MiB/s (4146MB/s), 3954MiB/s-3954MiB/s (4146MB/s-4146MB/s), io=1024MiB (1074MB), run=259-259msec

3.9GB/s is also roughly in the same ballpark we got.

Our code seems reasonable!

What's next?

None of this is original. fio is a similar tool, written in C, with many different IO engines including libaio and writev support. And it has many different IO workloads.

But it's been enjoyable to learn more about these APIs. How to program them and how they compare to eachother.

So next steps could include adding additional IO engines or IO workloads.

Also, either I need to understand Iceber's Go library better or its API needs to be loosened up a little bit so we can get that awesome ring buffer behavior we could use from Zig.

Keep an eye out here and on my io-playground repo!

Selected responses after publication

wizeman on lobsters suggests measuring at least 30 seconds worth of writing data and fsync()-ing if you want to test the entire IO subsystem and not just hitting the kernel cache.

Digging into io_uring has been on my list for a long time now! This week I finally made made some progress.

Let's go on a little journey through a few increasingly complex (and useful) implementations of writing a file to disk with io_uring.https://t.co/gR9K2OQs2R pic.twitter.com/TMaC8QYL6k
— Phil Eaton (@eatonphil) October 19, 2023

The List

Weblog on marginalia.nu — Thu, 19 Oct 2023 03:00:00 GMT

As a general observation, I tend to be more productive when I know what to do next at any given moment. There are days when I’ve seemingly gotten a “week” of work done on an afternoon, those are the days when what I needed to do was very clear, and I basically just had a list of items to tick off one by one. There have admittedly also those ignoble weeks weeks when I’ve gotten an afternoon’s work done, mostly they are weeks when it’s not been at all clear what to do next.

crosstable.love

Kartik Agaram — Wed, 18 Oct 2023 10:00:00 GMT

crosstable.love is a little app I whipped up for tracking standings during the Cricket World Cup, just to avoid the drudgery of resorting rows as new results come in.

video; 20 seconds

theft deterrent

Blogity blog blog. Journal — Wed, 18 Oct 2023 00:10:17 GMT

My thought, to help with our problem of theft today (2023) is that Target, et al, copy the Costco model, and require membership for entry into the store.

Have membership cost $0, but is revocable upon video evidence of stealing shit. Swipe to get into the store, pay when you leave.

Stocketa

Paul Stamatiou — Mon, 16 Oct 2023 03:00:00 GMT

Please read this Stocketa article in a browser to see embedded Stocketa videos.

It was March 2020, I was in New England when covid quarantine had just begun and I found myself much more homebound. In these situations I'm not one to just do nothing. I always have some sort of project or hobby to keep me busy, be it taking and editing photos, writing detailed blog posts, or coding something.

I had always wanted to learn iOS development, so this felt like the perfect opportunity. I was particularly keen to learn SwiftUI to build my designs natively. Years back I had dabbled with writing an Objective-C app, but it never quite filled me with joy, to say the least. As a designer, I've spent the last decade getting good with prototyping tools and frameworks to mimic realistic, interactive mobile prototypes. Building it myself was the natural next step to give me even more control over the experience, and get even deeper into the craft.

I started learning Swift and SwiftUI basics with courses like Design+Code, and Hacking with Swift. But I knew myself—I learn best with a project of my own. I knew exactly what I wanted to build: a better stock portfolio tracker. The stock market had taken a plunge with covid and I was doing dozens of trades per week with everything on sale. It would be fun to have a simple app of my own to track the performance of my trades instead of dealing with the subpar sites and apps from my stockbrokers.

Little did I know this project would consume my nights and weekends for over two years, often spending 10-20 hours a week on it. I built a ton of functionality, sweat all the details, wrote a comprehensive node backend for it, created an LLC for it, and managed an active Testflight. I was learning Swift and SwiftUI along the way, so I was often redesigning and rebuilding as I went.

Eventually, I decided to stop pursuing it. I'll get into the why a bit later. This post is for me to document Stocketa. I designed and built a lot that I'm proud of and want to have somewhere to archive it before I pull the plug.

The mostly-finished App Store screenshots that would have been published.

What is was Stocketa

Stocketa was a simple portfolio tracker designed with care. It came out of my frustration of having to check the clunky apps and sites that my brokers had just to get a glimpse at how my investments were doing. I'm a casual investor and don't need much from a tool like this but I had assets in various places, and I just wanted one easy and well-designed place to check them. I wanted to see real per-transaction gain/loss info, based on my own cost basis.

While there are tons of apps, brokers, and neobanks that address this, nothing quite felt like what I wanted. There were several types of solutions in this space:

First off, there are simple stock apps like Apple Stocks that let you see stock charts, news, and more. They have good data and don't require any accounts to use. However, they offer no functionality to log your holdings and track your gains/losses.
There are lots of broker apps (e.g., Robinhood) that let you trade and track your holdings, but they only let you see your holdings from them, not from any other financial services you might use.
Then there's a whole category of what I'd call advanced stock services like TradingView (and several that have since died or pivoted over the years), but they're not mobile-first or as elegant as I wanted.
There are broader investment advisor services like Personal Capital. While they do often have some aggregator functionality to link up your other accounts, they're not that powerful for diving into individual assets, aren't particularly well-designed, and are primarily focused on upselling you on their own investment advising services (seriously, I used to get endless 8am calls from Personal Capital).
And there are lots of smaller apps on the App Store where you can track your holdings but the design and UX leaves a lot to be desired. I've tried most of them.

From a design perspective, I generally found that financial apps bias towards density when displaying your assets. They assume you have tons of stocks and prioritize a list view where you have to tap in to a detail view to see more info. I found myself constantly tapping in, manipulating a chart or looking at some basic stats, then swiping back and repeating all that for several stocks. It was not easy or the kind of at a glance info I wanted.

That left me with my challenge. I wanted a simple portfolio tracker aimed at the casual investor. It didn't have to do it all, but it should aim to do a few things well. An app just for me.

Going custom

I was more than thrilled to be working on a new app from scratch. After years of working at large companies with rigid design systems and gobs of stakeholders for every little design decision, I was yearning for some creative freedom. Freedom from process, freedom from typical design constraints. I would be in full control, designing and building it myself. It was my side project after all.

I had gotten tired of designing for traditional mobile apps. You know the type.. you add a tab bar, a typical navigation stack, page header and the app already feels significantly basic, muted, and limited out of the gate.

This was my place to have fun. I wanted to go custom and toss out whatever components the system gives you for free. And I knew that by having to figure out how to build various UI components from scratch, I'd learn a lot along the way. Almost all of Stocketa's UI is custom.

I will say that going custom is almost always the absolutely wrong thing to do 99% of the time. The operating system you're working in—be it iOS, Android, macOS—has invested a ton in a highly-considered suite of components. They're remarkably accessible, well-tested, and work on a variety of devices.

It's absolutely no small feat to consider creating a custom component for your app. However, when going custom does make sense and you're okay with the added scope and responsibility, the app becomes your canvas to do whatever you want and make it yours.

The first area where custom made sense for me was general app navigation. When I got started, I knew I wanted this app to have minimal app chrome. I didn't want a standard tab bar and header taking up space. I wasn't going to require much navigation: most things would just live inside the stock cards from the home timeline.

The app would be focused around showing more with less. Personally, I only cared about tracking a few stocks, but I hated the work of constantly having to tap in to see a detail view, or interact with the chart. As such, the main scroll view needed to be more immersive and not get occluded by unnecessary app chrome.

It started with app navigation, custom stock cards, sheets, and more. Needless to say, I spent a lot of time on the details with everything being custom. Below are a few examples:

Please read this Stocketa article in a browser to see embedded Stocketa videos.

Designing and building

I started with the basics: getting a scroll view of stock cards working, adding charts, and hooking it up to a database for persistence (initially Core Data before moving to Firebase). Then I added the requisite network calls and polling to update the stock card from a financial data provider. I started with IEX as the data provider as they had a decent API, reasonable pricing, and allowed commercial use, which would be required for a consumer app like this to go on the App Store.

Working on an early build in 2020.

One of the initial key decisions with Stocketa was that I wanted to track stock transactions in a comprehensive way. Simply logging the number of shares you had for each stock was not going to cut it. This ended up being a significant amount of work to do properly, but it was necessary:

For each stock sale or purchase I wanted to be able to display not only the total gain/loss, but per-transaction gain/loss so you could tell if an individual transaction was a good trade.
When it comes to stock sales, I needed to know the cost basis method (FIFO, LIFO, et cetera) so I could correctly calculate and display the realized and unrealized gains on a particular transaction. When selling shares, I also needed to know from which previous transaction or transactions the shares would be sold so I could get the correct cost basis and accurately calculate the gain/loss based on the cost basis method.
And then there was dealing with stock splits, which seems simple at first glance but ended up being quite a bit of work to build in a robust way: not simply altering previous transactions. They should be automatically applied for new splits, but also able to be manually applied if you didn't log split-adjusted transactions.

A glimpse at part of how comprehensive transaction types in Stocketa were shown.

At some point I gave it a name: Stocketa. In Greek you might say "marketa" (Μαρκέτα, though αγορά might be a more proper word) when referring to a market, so this was like coalescing Stock and Market with that in mind.

When I was interviewing for a new job in 2021, I created these slides as part of my presentation after talking about my work at Twitter for the previous 9 years. These weren't all of Stocketa slides, and there was a lot of additional talking points I had for each slide, but you get the idea.

Slides touching on various parts of Stocketa as of late 2021. The design of many of these screens is now outdated, but it's a good look at a point in time.

A note about SwiftUI

I kept designing, building.. and then redesigning and rebuilding. One of the side effects of learning Swift and SwiftUI as I was building this was that I was constantly redoing specific pages and parts of the app as I went. I continually found better ways to build things, and similarly I would continually refine designs. About a year or so into building Stocketa I wrote about my experience with SwiftUI. Much of that still rings true.

SwiftUI has come a long way since iOS 13. It's more performant, there's more parity in available components and functionality compared with UIKit, and you get more control over things. And with iOS 17, you get some goodies like shaders, keyframes, scroll transitions, and more.

Your browser does not support the video tag. Xcode showing a SwiftUI preview.

There's still a gap between the types of things you can do with AppKit and UIKit, but SwiftUI is undeniably a powerful tool that's dramatically more approachable. And not just a coding tool; it's a great design tool. Designers can quickly lay things out, hook up interactions, and get a real feel for a design using a native tool.

Almost all of Stocketa's UI is built with SwiftUI. There were only a few occasions when I found it necessary to use UIKit to achieve some goal.¹ None of this would have been possible, or drawn me to take a closer look into building iOS apps, if it hadn't been for SwiftUI.

What I built

I used the app daily and continued pouring time into every feature and detail to meet my high quality bar. While I initially planned for this to be a simple to build app that would not require a backend, or even any user accounts, that did not last for long. I quickly realized I needed my own backend to unlock a few things.

At first, this was so I didn't have to include the API key for my financial data provider inside the app for security reasons—I wouldn't want someone to find it and start running up my bill by abusing it. I created a backend proxy for the API with rate-limiting and throttling. Then, I wanted to have it be authenticated with user accounts, so I added Sign In With Apple along with a stock data caching system so that I only had to fetch stats for a stock once with a certain time period (depending on the particular stat that was either seconds, 24 hours, or weekly) regardless of however many user accounts needed that stock info. The main goal with that was to also reduce my costs associated with financial data API usage.

By this point I really started enjoying the speed at which I could update endpoints and business logic entirely on the backend. I started moving logic I had in the app to the backend. Stocketa was starting to come together.

Even though this was just a solo project, I used Linear to keep track of tasks, ideas, features, customer feedback, and milestones.

I had prioritized lists of things to build and bugs to fix. I was using—and loving—Linear for this. Here's a list of most of the things I built for Stocketa:

Please read this Stocketa article in a browser to see embedded Stocketa videos.

I ran a small TestFlight alpha with close friends, then increasingly I added more people. At one point I was up to around 1,000 people. That was the most I wanted to allow. I received lots of feedback from overall great first impressions (below), to general feature requests, and of course lots of bug reports.

Already the app is filling a need for me in its personal approach to tracking investments: no pushing for sales, no integrations with brokers and no unnecessary permissions. Brilliant.

Simply amazed at the SwiftUI work you’ve poured into this app. So beautiful. The onboarding alone is a masterclass in attention to detail. Bravo.

love it, this feels so polished and slick already... really great use of SwiftUI

Just got the invite, added my stocks... the animations are beautiful, design is clean and precise... it’s all around beautiful. Congrats!

F**k, man. This app is incredible. You’re building all of this on your own?

Dude. This app is f**king beautiful. Really great job. Just imported shares - super easy.

This NUX is insane. The particles when you add a stock. Gives me the happy brain chemicals..

I had thousands on a waiting list but there were API and hosting costs associated with the increased usage that I didn't want to bear yet, as well as a huge increase in support and customer feedback that took a lot of time to manage. You start to hear the same types of feedback, and it just becomes a chore to manage that many emails. That good news was that something was resonating and people cared enough to go through the work to send in bug reports and feature requests.

The website

At some point early in feature development I took a break to focus on the website. At first I wanted a simple landing page to build interest, and collect emails for a waitlist so I could keep people in the loop. And I didn't want to spend that much time on it.

Your browser does not support the video tag. Video: Initial Stocketa teaser homepage.

There was nothing fancy or particularly special about this initial page. It was just something quick I wanted to get out the door. And now, a while later, there's a lot I hate about this early design. The background texture was too strong, the headline alignment feels odd with the phone.. but I digress.

In that initial design I did start playing with some wave-related elements—a loose reference to stock market charts going up and down. And then to add to that, I built a few mini stock cards gently hovering in place (thanks to offset-path CSS along with 2 keyframe animations) in the background. Each stock card had an SVG line chart—loosely based on that stock's actual performance around the time—that would animate on page load. I ended up keeping a version of these mini stock cards on the latest version of the website, and then on an onboarding screen inside the app.

The original inspiration for these random floating stock cards was from my now out-of-date work page where I highlighted projects with some large cards, each with their own intro animation based on the project. One of them was for my work on Twitter Video, where video clip thumbnails would fan out from the phone on hover (shown below).

Your browser does not support the video tag. The floating mini stock cards on the Stocketa homepage were inspired by the hover state animation on this project card from my old portfolio page.

Homepage design `v1.0`

Your browser does not support the video tag. Video: The first full version of Stocketa.com in early 2021, currently archived here. The App Store QR code was meant to be displayed on scroll but in this version I didn't have that hooked up as the app wasn't live.

In early 2021 I ended up getting around to designing and building out the first complete homepage for Stocketa. I had enough functionality and features built that it felt worthy of investing some time into showcasing them on the site.

Lots of app websites at the time (and still now) had the typical hero section on their homepage: some app screenshot in a phone frame off to one side, with accompanying header and subtitle text adjacent. Nothing wrong with that, but I wanted something a bit more engaging. I wasn't entirely interested in optimizing for conversion, and cramming a lot in above the page fold.

Initial design explorations in figma.

I thought it would be nice to try to do something where the 3D phone frame was angled and then began to change position on scroll while a brief teaser video of the app functionality played. Apple had done something similar in recent years with product pages, such as the first-generation AirPods Pro. They used over 100 frames of the AirPods that would progress as you scrolled:

Your browser does not support the video tag. Example of a technique used by Apple.

There's a lot of ways to achieve an effect like this. One route route is using Lottie and essentially converting each frame of your video or animation to a base64-encoded string in a json file for the Lottie web player to manage. It's not a flawless solution, however, and can incur some large file sizes depending on what you're doing. It's best for vector micro-animations, not large devices with complex visual content. I also didn't want to incur the cost of that large initial json download blocking the very first thing you see on the site.

Instead I opted to have javascript display a sequence of frames composited to canvas along with an offscreenCanvas for performance. I used Rotato to import my short Stocketa screencast, create a 3D phone video, and then create a PNG sequence of frames. I wasn't super pleased with the quality—the phone frame didn't look super realistic and had some rough pixelated edges—but it was good enough compared to doing it all by hand with other design tools.

Your browser does not support the video tag. Scroll-based Stocketa 3D phone, paired with subtle parallax for header elements and the floating stock cards. There were some other minor scroll effects like the gradient glow behind the phone spreading.

I then exported optimized transparent webp, and jpg files (with accompanying alpha masks created in Sketch to save bandwidth) and had the javascript detect which file type to use for the animation. While I was pleased with how this effect turned out, it ended up becoming a rather huge annoyance when I wanted to update the content later—it's a pain to generate, optimize, and update the image sequence each time.

Other bits of the website: cards for various features.

And finally, as for the rest of the content on this site I went with a basic feature card approach, sometimes called a "bento box" layout; particularly when varying card sizes are used. This is a layout that I've grown to like a bit less over the years as it has become very common.

Homepage design `v2.0`

Your browser does not support the video tag. Video: New Stocketa site

A year later, it was time for another redesign. Stocketa had more functionality, and a lot more refinements. I had more to share, and I had grown tired of the 3D phone frame interaction. It was neat but it was just there for show; it wasn't the best way to show off parts of the app as it scrolled off screen. It was also a pain to update the frame sequences. I essentially just wanted to take screenshots of the real app from the iOS simulator and upload them.

Separately, I didn't want a super long page with a lot of repetitive modules and screenshots. I wanted a skimmable feature list. I began to wonder if I could get away with just a single phone frame on the site that showed everything.

I eventually landed on a two pane approach: the device frame was always in view, fixed on the right side, while the scrollable left side contained the feature list. The trade-off with this design was that it would require hovering over each feature item to be able to update the screenshot shown in the device frame. I figured an interesting compromise to that would be having the phones stack up and away on scroll. Almost like the old Apple Time Machine backup UI.

I built several versions of this and found that it didn't feel great. There wasn't much scroll distance so if you scrolled at a normal or fast pace, you'd see a bunch of phones flicker by you as the animations tried to keep up. It was too busy and hectic, not to mention a lot of state management to deal with jumping between your current scroll-based position screenshot to a hover-based screenshot if you hovered over a feature list item. I ended up simplifying a bit and only having the scroll-based UI manage just a few states: the intro phone frame, the first feature screenshot, and then a final animation at the bottom of the page.

Your browser does not support the video tag. The website's mobile layout uses a carousel of screenshots in the header.

After sorting out the core interactions, I had to figure out how to deal with mobile and tablet interactions. The fix two-pane layout made this a bit of a challenge. I ended up turning the header into a carousel of device frames you could scroll through.

And then finally, I added a few small bits of polish ✨:

Hero title text gradient changes based on scroll
The icon accent color and background circle color change based on scroll
When hovering over any feature list item, there is a subtle background color change created by an overlaid gradient. SVG-created noise/grain is also overlaid on top of the background.
When hovering over any feature list item, or when you're reached the end of the page, there's a subtle icon particle emitter that floats up icons from the bottom. Each feature used a different icon for this.

Your browser does not support the video tag. Particle emitter effects: icons float up from the bottom, based on the feature list item being hovered on. There's also another very subtle emitter generating some tiny floating particles.

Overall I liked this design, but looking back at almost a year later, the gradient text and background colors feel a bit too much for my taste now.

API troubles

Eventually it became clear that the quality of financial data Stocketa relied on was not cutting it. The first provider I used, IEX, would sometimes have charts outdated, inaccurate prices (especially for less frequently traded assets on the IEX exchange), and lacked data for OTC markets (they required you sign a pricey license directly with OTC), as well as significantly limited data for Nasdaq-listed stocks (even basic things like pre-market and after-hours data). The last straw was when IEX dropped mutual fund data entirely, all without any advance notice.

I later used another provider that allowed commercial use that didn't charge too much ($100/month) but I had even more problems with their data and reliability. Endpoints were even returning old and incorrect data, and even once alerted to issues, their team would not only ignore the bugs but claim they didn't exist. I couldn't ship Stocketa with unreliable financial data to people that would be paying me for a quality app.

I also could not get certain types of data with paid APIs. I had to build a scraping engine on my backend that would visit a few sites to get the data I needed for less updated data like dividend yield, earnings dates, and even for OEF data (other providers would be more than a day delayed for this at times).

Needless to say, this was all frustrating. It was a hassle, hard to maintain, and unreliable. At the same time, my backend continued getting more involved to deal with more and more of the intricacies of dealing with stocks. Things like dealing with companies merging or stock tickers renaming.

~13k LOC node/express backend (shown in Zed but I now prefer Cursor) for Cloud Run. It manages auth, caching, news, core business logic, cleaning up data, notifications, widgets, working with multiple API, and a bit of scraping. This was my first time doing anything of this size with node and typescript.

Starting out I thought getting access to a stock market data API would be the easy part. I quickly learned about the world of market data feeds, SIPs (Securities Information Processor), and vendor agreements.

Quality market data exists, it's just not made for indie app developers. It's very expensive. Several leading U.S. market data providers have commercial use data pricing starting at $2,000 per month. And that's not for everything, if you need things like index data or options data, there's more charges ahead.

I spoke with one of these providers last year, trying to get them to understand how small app developers need affordable commercial use data. They presented a startup-focused plan that they were pitching for $499/month in addition to a per-MAU fee. Even that I would still consider very expensive for my needs, and it seems like that may have fallen by the wayside and is no longer promoted on their website.

I could spend months developing Stocketa, incurring that monthly API cost until I went live (or risk getting shutdown if I tried to use a cheaper non-commercial plan during the TestFlight phase) then not make enough money to cover costs on the App Store and have to shut it down shortly after anyways.

Overall, it feels like no one in this cares to cater to the developer and startup crowd, or they believe there's not much money for them to pursue there. This was the same case with Plaid when I briefly looked into getting access to their investments API to automatically sync holdings data. They required some sort of minimum length contract.

Why I stopped working on Stocketa

When I started this project, I really began to wonder why where weren't any great indie apps focused on stocks like this. Just simple, well-designed apps to help the casual investor keep track of their investments. Well I think I was starting to get my answer: getting quality financial data, even just for the United States, is a nightmare. You can only get it if you have a lot of money to invest in your project. And good luck if you really want a great API with comprehensive international data.

I can't help but draw parallels to what happened recently with the Twitter and Reddit APIs. The risk of running your project or business solely on top of another company's API, no matter how reliable you think they may be, is extremely high.

Stocketa was always meant to be a side project—never a real business. For it to be a real business, I would need to keep adding functionality and introducing more ways to make money. I couldn't just have a simple and elegant little app tracking your stocks. It would have to do more, like dipping into financial advice services, or offering stock trading. That's a hyper-competitive space lined with massive companies. That would mean a busier, cluttered app. The exact reason why I started Stocketa was because of all those cluttered apps.

I decided to stop working on Stocketa for a few reasons:

Data: As mentioned above, all the expenses and challenges associated with acquiring reliable, affordable, and high-quality financial data that would make it feasible to release this as a reasonably priced subscription-based app.
Support: The associated investment in customer support. Based on how people were using my TestFlight, it was going to be a significant investment to keep up with support, emails, maintenance, and ongoing feature development. However, part of this was likely due to complexity incurred from having to fix or skirt around unreliable data. There were lots of things I wanted to keep improving and refining with the app and that wouldn't end any time soon.

If I were to release and charge for Stocketa, I would feel especially on the hook to resolve issues in a timely manner and that's not something I can accomodate now because of the next point.
Time: And most importantly, I've found something more worthy of my complete focus and attention: Rewind AI. I sunk a lot of nights and weekends into Stocketa over the years. It consumed every spare moment, even taking over my occasional time writing blog posts here when I had something to share. That was something I really missed. Now, I would much rather be focusing my creative thinking time on how I can improve and grow Rewind.

I originally started working on Stocketa with one goal: to learn Swift and SwiftUI my way with a project that was interesting to me. At the time that was scratching my own itch of keeping track of my investments. While it turned into a much larger and unshipped project, I learned a ton about native iOS development and was able to push myself creatively.

These days I'm happiest investing my design, product and development time into Rewind AI, where my SwiftUI knowledge learned with Stocketa has been tremendously helpful.

Footnotes

1 I only had to go down to UIKit a few times. One time for things like custom text fields for more control over interactions, styling, text input, and formatting. Another to use a CollectionView for drag-to-reorder stocks. A third time for creating particle emitters in a few places, like confetti effects, as well as one area where I have some twinkling stars in a night illustration when the stock market is closed. And one last time to get more control over gestures. I needed to get a location when a long press begins and SwiftUI LongPressGesture() didn't provide that, so I used UILongPressGestureRecognizer instead.

2023–10–15: New GIT mirror

megi's PinePhone Development Log — Sun, 15 Oct 2023 01:00:00 GMT

2023–10–14: Orange Pi 5 Plus and QuartzPro64 upstreamed

megi's PinePhone Development Log — Sat, 14 Oct 2023 01:00:00 GMT

Increment only numbers matching regex in Vim

The NeoSmart Files — Fri, 13 Oct 2023 20:47:14 GMT

Long-time vim or neovim users are probably already aware that visually selecting a block of text then pressing CTRL + A in vim will result in any numbers in the selected block of text being incremented by 1. This works … Continue reading →

The post Increment only numbers matching regex in Vim first appeared on The NeoSmart Files.

What I’m up to - October 2023

Philip I. Thomas — Thu, 12 Oct 2023 22:58:04 GMT

This is my monthly newsletter about what I'm up to, which I send in place of social media.

✨ What I was up to in September

I launched Booklet, a forum as an alternative to chat. It's my attempt to make communication more asynchronous for professional groups. Knowledge workers check chat every 6 minutes, fracturing their attention and preventing deep work. Booklet solves this by encouraging long-form, threaded discussions - and summarizing all activity into one single email per day.

I wrote about my motivation for Booklet on the Contraption Company site.

The actual launch of Booklet went better than expected - with thousands of visitors in the first day, and hundreds of communities created. I've been iterating rapidly based on user feedback, including rolling out a variety of OpenAI-powered tools to improve the email summary.

Some of the early use case for Booklet have been:

Investors connecting their portfolio founders in a network
A marketplace building a community of workers to engage them between gigs
A Substack newsletter adding a subscriber community
Companies replacing Google Groups for internal announcements

If you know anybody looking for a solution like these, please send them to Booklet. And, you can try out Booklet by joining its customer community at hq.booklet.group

Besides that, I had a wonderful trip to Stockholm where the Vasa Museum fascinated me, I ate too many cardamom buns, and I stopped by the Nobel Prize Museum. I also attended the New York Coffee Festival, and briefly stopped by Chicago for a wedding.

🤔 Things to share

Articles: The Magical Japanese Art of Luggage Forwarding, The Loneliness Economy, Protocol Daddies, The Jargon File.
Books: Coddling of the American Mind.
Podcasts: Airbnb CEO Brian Chesky on early rejection, customer focus - a look at how Jony Ive is turning Airbnb into a centralized product organization (see "Trend" below). Three Star Coffee Service: Coffee at restaurant Noma.
Apps: Helicone for OpenAI monitoring and caching.
Video: "How to Live an Asymmetric Life"
Trend: With the downturn, more tech CEOs seem to be retaking the "Head of product" role.
Coffees I'm drinking: Man Vs. Machine Supernatural, Heart Guatemala Amate, and notable IRL coffees at Café Pascal and Drop Coffee.

📫 What I'm up to in September

Working on Booklet, and enjoying a first month without travel in a long time.

📍 Where I'll be

NYC all month 🗽

(Let me know if we overlap)

📸 Photo

Caradmom buns for fika in Stockholm

The Vasa - a ~500 year old ship that was salvaged and now is on display in an eponymous museum in Stockholm

Intricate carvings on the Vasa

Football Queries, part 1

infrequent oscillations — Tue, 10 Oct 2023 09:55:39 GMT

In which we note some queries on the football database.

Please to meet you, hope you guessed my name?

Steve Kemp's Blog — Mon, 09 Oct 2023 12:00:00 GMT

"Hello, my name is Steve" - those are words I've said a million times in my life, however they are not true words.

If you want to get all technical about things, my name has always been SteveN.

Mostly this hasn't mattered to me, or anybody else, I introduce myself as Steve, people call me Steve, and Steve is the name that makes me turn my head, when shouted across a bar. However things changed when I moved to Finland.

In Finland I had to open new bank accounts, sign mortgages, hand over IDs, and there were many many pieces of paper I signed, or forms I filled out. Unfortunately I screwed up:

If I were thinking clearly I'd think "Oh, this is something official, I'd best write SteveN".
If I were distracted, or not being careful I'd write my name as "Steve", and then sign it as Steve.

The end result? I've been in Finland for approximately eight years, and I have some official documentation calling me Steve, and some other official documentation calling myself Steven. (For example my "Permanent Residency Permit" calls me Steve, but my Social Security ID knows me as Steven.)

Every now and again somebody queries the mismatch, and there are daily moments of pain where I have to interact with different agencies, so I made the obvious decision: I'm gonna change my name.

A fee of €60 and a simple online form was sufficient to initiate the process. The processing time was given as "one to five months" on the official forename changing page, but happily the process was complete in a month.

I will now need to do a little housekeeping by getting updated bank-cards, etc, and then complete the process by changing my UK passport to match. Hopefully this won't take too long - but I guess if Finland knows me as Steve and the UK knows me as Steven I'll still be in a bit of a screwed up state, albeit one that is consistent in each country!

Not a big change really, but also it feels weird to suddenly say "Hello, my name is Steve" and mean it.

People are weird.

Names are interesting.

The end.

Fin.

tcpproxy 0.4 released

The NeoSmart Files — Sun, 08 Oct 2023 21:53:36 GMT

This blog post was a bit delayed in the pipeline, but a new release of tcproxy, our educational async (tokio) rust command line proxy project, is now available for download (precompiled binaries or install via cargo). I was actually surprised … Continue reading →

The post tcpproxy 0.4 released first appeared on The NeoSmart Files.

Moving Marginalia to a New Server

Weblog on marginalia.nu — Sat, 07 Oct 2023 03:00:00 GMT

So the search engine is moving to a new server soon, thanks to the generous grant mentioned recently. If you visit search.marginalia.nu now, it may or may not use the old or new server. It’ll be like this for a while, since I need them both for testing and maintenance type work. I’ll also apologize if this post is a bit chaotic. It is a reflection of a very chaotic couple of weeks that apart from setting up this migration also involved a very short notice invitation for a presentation at ossym23.

Go database driver overhead on insert-heavy workloads

Notes on software development — Thu, 05 Oct 2023 03:00:00 GMT

The most popular SQLite and PostgreSQL database drivers in Go are (roughly) 20-76% slower than alternative Go drivers on insert-heavy benchmarks of mine. So if you are bulk-inserting data with Go (and potentially also bulk-retrieving data with Go), you may want to consider the driver carefully. And you may want to consider avoiding database/sql.

Some driver authors have noted and benchmarked issues with database/sql.

So it may be the case that database/sql is responsible for some of this overhead. And indeed the variations between drivers in this post will be demonstrated by using database/sql and avoiding it. This post won't specifically prove that the variation is due to the database/sql interface. But that doesn't change the premise.

Not covered in this post but something to consider: JetBrains has suggested that other frontends like sqlc, sqlx, and GORM do worse than database/sql.

This post is built on the workload, environment, libraries, and methodology in my databases-intuition repo on GitHub. See the repo for details that will help you reproduce or correct me.

INSERT workload

In this workload, the data is random and there are no indexes. Neither of these aspects matter for this post though because we're comparing behavior within the same database among different drivers. This was just a workload I already had.

Two different data sizes are tested:

10M rows with 16 columns, each column is 32 bytes
10M rows with 3 columns, each column is 8 bytes

Each test is run 10 times and we record median, standard deviation, min, max and throughput.

SQLite

Both variations presented here load 10M rows using a single prepared statement called for each row within a single transaction.

The most popular driver is mattn/go-sqlite3.

It is roughly 20-40% slower than another driver that avoids database/sql.

10M Rows, 16 columns, each column 32 bytes:

Timing: 56.53 ± 1.26s, Min: 55.05s, Max: 59.62s
Throughput: 176,893.65 ± 3,853.90 rows/s, Min: 167,719.97 rows/s, Max: 181,646.02 rows/s

10M Rows, 3 columns, each column 8 bytes:

Timing: 15.92 ± 0.25s, Min: 15.69s, Max: 16.67s
Throughput: 628,044.37 ± 9,703.92 rows/s, Min: 599,852.91 rows/s, Max: 637,435.60 rows/s

The other driver I tested is my own fork of bvinc/go-sqlite-lite called eatonphil/gosqlite. I forked it because it is unmaintained and I wanted to bring it up-to-date for tests like this.

10M Rows, 16 columns, each column 32 bytes:

Timing: 45.51 ± 0.70s, Min: 43.72s, Max: 45.93s
Throughput: 219,729.65 ± 3,447.56 rows/s, Min: 217,742.98 rows/s, Max: 228,711.51 rows/s

10M Rows, 3 columns, each column 8 bytes:

Timing: 10.44 ± 0.20s, Min: 10.02s, Max: 10.68s
Throughput: 957,939.60 ± 18,879.43 rows/s, Min: 936,114.60 rows/s, Max: 998,426.62 rows/s

PostgreSQL

Both variations presented use PostgreSQL's COPY FROM support. This is significantly faster for PostgreSQL than doing the prepared statement we do in SQLite. (Here are my results for doing prepared statement INSERTs in PostgreSQL if you are curious.)

The most popular PostgreSQL driver is lib/pq. The performance issues with lib/pq are well-known, and the repo itself is marked as no longer developed.

It is roughly 44-76% slower than an alternative driver that avoids database/sql.

10M Rows, 16 columns, each column 32 bytes:

Timing: 104.53 ± 2.40s, Min: 102.57s, Max: 110.08s
Throughput: 95,665.37 ± 2,129.25 rows/s, Min: 90,847.08 rows/s, Max: 97,490.96 rows/s

10M Rows, 3 columns, each column 8 bytes:

Timing: 8.16 ± 0.43s, Min: 7.44s, Max: 8.80s
Throughput: 1,225,986.47 ± 66,631.53 rows/s, Min: 1,136,581.82 rows/s, Max: 1,343,441.37 rows

The other driver I tested is jackc/pgx, without database/sql.

10M Rows, 16 columns, each column 32 bytes:

Timing: 46.54 ± 1.60s, Min: 44.09s, Max: 49.51s
Throughput: 214,869.42 ± 7,265.10 rows/s, Min: 201,991.37 rows/s, Max: 226,801.07 rows/s

10M Rows, 3 columns, each column 8 bytes:

Timing: 5.20 ± 0.44s, Min: 4.71s, Max: 5.96s
Throughput: 1,923,722.79 ± 156,820.46 rows/s, Min: 1,676,894.32 rows/s, Max: 2,124,966.60 rows/

The discrepancies here are even greater than with the different SQLite drivers.

Workloads with small resultset

I won't go into as much detail but if you're doing queries that don't return many rows, the difference between drivers is negligible.

See here for details.

Conclusion

If you are doing INSERT-heavy workloads, or you are processing large number of rows returned from your SQL database, you might want to try benchmarking the same workload with different drivers.

And specifically, there is likely no good reason to use lib/pq anymore for accessing PostgreSQL from Go. Just use jackc/pgx.

For INSERT-heavy workloads in Go, you may want to switch database drivers. For PostgreSQL and SQLite, the popular drivers are 20-76% slower for this workload in my tests.

Some driver developers have reported issues with database/sql as an interface.https://t.co/NLVp0P2uiV pic.twitter.com/RxTbgMZ1MG
— Phil Eaton (@eatonphil) October 6, 2023

Quitting while you're ahead

Tilting at Windmills — Thu, 05 Oct 2023 03:00:00 GMT

I often wonder, how many actors are there who quietly disappeared beacause they made a bunch of money from one lucky success? Hollywood…

Throughput Theater

jodavaho.io — Thu, 05 Oct 2023 03:00:00 GMT

I regret to inform you that member initialization lists will not make your program noticeably faster.

The point of this post is to vent, so it is highly opinionated.

Style-based performance optimizations are a waste of time
Good fundamentals are more important than micro-optimizations
Compile and test times are vastly underrated
For some reason, code / function generalization is valued higher than ergonomics or real performance.

I form this opinion based on experiences with:

The near-constant debate over style, move semantics, constructor initialization lists, and other micro-optimizations
Making small changes and seeing large performance gains, sometimes 1,000-100,000x by:
- Using hardware more efficiently
- Reframing the core problem
- Using the right data structures
- Using the right algorithms

In short, if you’re worried about the performance problems with initialize+copy vs std::move, you’re either doing something very wrong or you’re down to the last 0.1% of performance improvements.

Perl One-Liners Guide book announcement

learnbyexample — Wed, 04 Oct 2023 03:00:00 GMT

Hello!

I am pleased to announce a new version of my Perl One-Liners Guide ebook. Examples, exercises, solutions, descriptions and external links were added/updated/corrected.

When it comes to command line text processing, there are several well known tools like grep for filtering, sed for substitution and awk for field processing. Compared to such tools, Perl has a feature rich regular expression engine, plenty of builtin modules and a thriving ecosystem. Another advantage is that Perl is more portable.

This ebook will show examples for filtering and substitution features, field processing, using standard and third-party modules, multiple file processing, how to construct solutions that depend on multiple records, how to compare records and fields between two or more files, how to identify duplicates while maintaining input order and so on.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of Perl One-Liners Guide for FREE till 07-October-2023. You can still pay if you wish ;)

All Books Bundle is just $12 (normal price $32), includes all my 13 programming ebooks.

What's new?🔗

Command version updated to Perl 5.38.0
- option -g slurps entire file contents
Many more exercises added
Long sections split into smaller ones
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to Perl One-Liners Guide
New cover image

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Testimonials🔗

This is fantastic! 👏 I use Perl one-liners for record and text processing a lot and this will be definitely something I will keep coming back to - I’ve already learned a trick from “Context Matching” (9) 🙂

— feedback on Linux@lemmy.ml

Table of Contents🔗

Preface
One-liner introduction
Line processing
In-place file editing
Field separators
Record separators
Using modules
Multiple file input
Processing multiple records
Two file processing
Dealing with duplicates
Perl rename command

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/learn_perl_oneliners/.

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_perl_oneliners for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tips, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, Gumroad rating, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors.

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_perl_oneliners/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Evolution of value

Blogity blog blog. Journal — Wed, 04 Oct 2023 00:09:00 GMT

The Evolution of Value: From Land, Labor, and Capital to Trust, Time, and Perspective

Introduction

The world has undergone significant transformations in the way value is created, measured, and exchanged. Traditionally, the three pillars of economic value were land, labor, and capital. However, in the modern era, these have been supplanted by less tangible but equally vital resources: trust, time, and perspective (POV).

The Traditional Triad: Land, Labor, and Capital

Land

In agrarian societies, land was the primary source of wealth. It provided the raw materials for food, shelter, and trade. Control over land often equated to power and economic stability.

Labor

Labor transformed raw materials into goods. The skill and efficiency of labor determined the quality and quantity of output, affecting the overall economic health of a society.

Capital

Capital, both financial and physical, acted as the catalyst for growth. It enabled investment in technology and infrastructure, facilitating the expansion of both land and labor productivity.

The Modern Triad: Trust, Time, and Perspective

Trust

In today’s interconnected world, trust has become a cornerstone of value. Whether it’s trust in a brand, a technology, or a system, it enables transactions and relationships that form the basis of modern economies.

Time

Time has always been a finite resource, but its importance has escalated in the age of information. Efficiency, speed, and the ability to manage time effectively are now critical factors in economic success.

Perspective

In a world saturated with information and options, perspective or POV has become a valuable asset. Unique insights, innovative thinking, and the ability to solve complex problems are highly sought after.

The Shift Explained

Technological Advancements

The digital revolution has minimized the importance of land and physical capital. Virtual real estate and digital assets have become new forms of land and capital.

Globalization

The global market has made labor more of a commodity, easily outsourced or automated. Trust, however, cannot be outsourced; it must be built and maintained.

Information Age

The explosion of information has made time and perspective more valuable. The ability to quickly assimilate information and offer a unique POV is invaluable.

Conclusion

The transition from land, labor, and capital to trust, time, and perspective reflects the evolving complexities of the modern world. While the traditional triad laid the foundation for economic systems, the modern triad is shaping the future, emphasizing the intangible yet indispensable resources that drive today’s economies.

Four Pillars of Disagreement

Blogity blog blog. Journal — Tue, 03 Oct 2023 04:00:00 GMT

The Four Pillars of Disagreement: Non-Negotiable Categories

Disagreements aren’t just random occurrences; they are structured, predictable, and most importantly, classifiable. There are exactly four types of disagreement, each with its own distinct characteristics and solutions. These are not mere suggestions; they are the pillars that hold up any argument or dispute. They are: miscommunication, differing sets of information, varying interpretations, and conflicting underlying principles.

Miscommunication: The Unquestionable Culprit

Let’s get this straight: when two colleagues can’t even agree on a meeting time—one thinking it’s at 3 PM and the other at 4 PM—that’s not just an oversight. It’s a glaring example of miscommunication, pure and simple. The solution is equally straightforward: clarify, confirm, and move on.

Differing Sets of Information: The Undeniable Divider

Consider two employees locked in debate over a new company policy. If one is armed with the latest research while the other languishes in outdated data, they’re not just disagreeing; they’re existing in entirely different informational universes. The fix? Update, align, and proceed. Anything less is a disservice to the discussion.

Varying Interpretations: The Inarguable Confounder

Text messages are not just words on a screen; they are a battleground for interpretation. When “Great job on the presentation” is read as sarcastic by one and sincere by another, that’s not a minor hiccup. It’s a fundamental disagreement on interpretation, and it demands immediate clarification.

Conflicting Underlying Principles: The Irrefutable Core

When two friends argue politics from opposing ideological platforms—one valuing individual freedom and the other social equality—there’s no skirting the issue. This is a clash of principles, as foundational as it gets. Don’t expect a quick fix; this type of disagreement calls for deep reflection and, often, an acceptance of ideological diversity.

Conclusion: The Immutable Truth

These are not just types of disagreements; they are the framework within which all disagreements occur. Whether it’s the undeniable role of miscommunication, the clear-cut impact of differing information, the indisputable effect of varying interpretations, or the unyielding influence of conflicting principles, these four categories are the bedrock of all disputes. Recognize them, address them, and only then can you navigate the complex landscape of human disagreement. The goal is not to win, but to understand.

A Tale of Two Pharmacies

Pxtl.ca — Tue, 03 Oct 2023 03:00:00 GMT

What follows is the story of why I no longer go to my old pharmacy. This is recounted from memory, and some of the details may be inaccurate, but the gist is there.

drops in at local pharmacy

"I'm running out of my meds, can you refill them?"

"Sorry, you have to come closer to refill-day, so come back Saturday."

"Okay, I'll come back then."

...

"Okay, it's Saturday, here to refill my meds."

"... yeah, you're out of refills. We'll have to talk to your doctor."

"... but it's the weekend?"

"Oh, we'll sort it out on Monday."

...

Closing the gap on fediverse hashtag visibility with hashtag-importer

Linux Engineer's random thoughts — Mon, 02 Oct 2023 01:00:00 GMT

I have released a small application, hashtag-importer, that users of a Mastodon instance can use to slowly import more content from low-traffic hashtags, into their instance (with their admin's permission).

Why hashtag-importer

In the fediverse, your server might not see all posts made by everyone; it should only see posts …

Intercepting and modifying Linux system calls with ptrace

Notes on software development — Sun, 01 Oct 2023 03:00:00 GMT

How software fails is interesting. But real-world errors can be infrequent to manifest. Fault injection is a formal-sounding term that just means: trying to explicitly trigger errors in the hopes of discovering bad logic, typically during automated tests.

Jepsen and ChaosMonkey are two famous examples that help to trigger process and network failure. But what about disk and filesystem errors?

A few avenues seem worth investigating:

A custom FUSE filesystem
An LD_PRELOAD interception layer
A ptrace system call interception layer
A SECCOMP_RET_TRAP interception layer
Or, symbolic analysis a la Alice from University of Wisconsin-Madison

I would like to try out FUSE sometime. But LD_PRELOAD layer only works if IO goes through libc, which won't be the case for all programs. ptrace is something I've wanted to dig into for years since learning about gvisor.

SECCOMP_RET_TRAP doesn't have the same high-level guides that ptrace does so maybe I'll dig into it later. And symbolic analysis might be able to detect bad workloads but it also isn't fault injection. Maybe it's the better idea but fault injection just sounds more fun.

So this particular post will cover intercepting system calls (syscalls) using ptrace with code written in Zig. Not because readers will likely write their own code in Zig but because hopefully the Zig code will be easier for you to read and adapt to your language compared to if we had to deal with the verbosity and inconvenience of C.

In the end, we'll be able to intercept and force short (incomplete) writes in a Go, Python, and C program. Emulating a disk that is having an issue completing the write. This is a case that isn't common, but should probably be handled with retries in production code.

This post corresponds roughly to this commit on GitHub.

A bad program

First off, let's write some code for a program that would exhibit a short write. Basically, we write to a file and don't check how many bytes we wrote. This is extremely common code; or at least I've written it often.

$ cat test/main.go
package main

import (
        "os"
)

func main() {
        f, err := os.OpenFile("test.txt", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0755)
        if err != nil {
                panic(err)
        }

        text := "some great stuff"
        _, _ = f.Write([]byte(text))

        _ = f.Close()
}

With this code, if the Write() call doesn't actually succeed in writing everything, we won't know that. And the file will contain less than all of some great stuff.

This logical mistake will happen rarely, if ever, on a normal disk. But it is possible.

Now that we've got an example program in mind, let's see if we can trigger the logic error.

ptrace

ptrace is a somewhat cross-platform layer that allows you to intercept syscalls in a process. You can read and modify memory and registers in the process, when the syscalls starts and before it finishes.

gdb and strace both use ptrace for their magic.

Google's gvisor that powers various serverless runtimes in Google Cloud was also historically based on ptrace (PTRACE_SYSEMU specifically, which we won't explore much in this post).

Interestingly though, gvisor switched only this year (2023) to a different default backend for trapping system calls. Based on SECCOMP_RET_TRAP.

You can get similar vibes from this Brendan Gregg post on the dangers of using strace (that is based on ptrace) in production.

Although ptrace is cross-platform, actually writing cross-platform-aware code with ptrace can be complex. So this post assumes amd64/linux.

Protocol

The ptrace protocol is described in the ptrace manpage, but Chris Wellons and a University of Alberta group also wrote nice introductions. I referenced these three pages heavily.

Here's what the UAlberta page has to say:

We fork and have the child call PTRACE_TRACEME. Then we handle each syscall entrance by calling PTRACE_SYSCALL and waiting with wait until the child has entered the syscall. It is in this moment we can mess with things.

Implementation

Let's turn that graphic into Zig code.

const std = @import("std");
const c = @cImport({
    @cInclude("sys/ptrace.h");
    @cInclude("sys/user.h");
    @cInclude("sys/wait.h");
    @cInclude("errno.h");
});

const cNullPtr: ?*anyopaque = null;

// TODO //

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();

    var args = try std.process.argsAlloc(arena.allocator());
    std.debug.assert(args.len >= 2);

    const pid = try std.os.fork();

    if (pid < 0) {
        std.debug.print("Fork failed!\n", .{});
        return;
    } else if (pid == 0) {
        // Child process
        _ = c.ptrace(c.PTRACE_TRACEME, pid, cNullPtr, cNullPtr);
        return std.process.execv(
            arena.allocator(),
            args[1..],
        );
    } else {
        // Parent process
        const childPid = pid;
        _ = c.waitpid(childPid, 0, 0);
        var cm = ChildManager{ .arena = &arena, .childPid = childPid };
        try cm.childInterceptSyscalls();
    }
}

So like the graphic suggested, we fork and start a child process. That means this Zig program should be called like:

$ zig build-exe --library c main.zig
$ ./main /actual/program/to/intercept --and --its args

Presumably, as with strace or gdb, we could instead attach to an already running process with PTRACE_ATTACH or PTRACE_SEIZE (based on the ptrace manpage) rather than going the PTRACE_TRACEME route. But I haven't tried that out yet.

With the child ready to be intercepted, we can implement the ChildManager that actually does the interception.

ChildManager

The core of the ChildManager is an infinite loop (at least as long as the child process lives) that waits for the next syscall and then calls a hook for the sytem call if it exists.

const ChildManager = struct {
    arena: *std.heap.ArenaAllocator,
    childPid: std.os.pid_t,

    // TODO //

    fn childInterceptSyscalls(
        cm: *ChildManager,
    ) !void {
        while (true) {
            // Handle syscall entrance
            const status = cm.childWaitForSyscall();
            if (std.os.W.IFEXITED(status)) {
                break;
            }

            var args: ABIArguments = cm.getABIArguments();
            const syscall = args.syscall();

            for (hooks) |hook| {
                if (syscall == hook.syscall) {
                    try hook.hook(cm.*, &args);
                }
            }
        }
    }
};

Later we'll write a hook for the sys_write syscall that will force an incomplete write.

Back to the protocol, childWaitForSyscall will call PTRACE_SYSCALL to allow the child process to start up again and continue until the next syscall. We'll follow that by wait-ing for the child process to be stopped again so we can handle the syscall entrance.

    fn childWaitForSyscall(cm: ChildManager) u32 {
        var status: i32 = 0;
        _ = c.ptrace(c.PTRACE_SYSCALL, cm.childPid, cNullPtr, cNullPtr);
        _ = c.waitpid(cm.childPid, &status, 0);
        return @bitCast(status);
    }

Now that we've intercepted a syscall (after waitpid finishes blocking), we need to figure out what syscall it was. We do this by calling PTRACE_GETREGS and reading the rax register which according to amd64/linux calling convention is the syscall called.

Registers

PTRACE_GETREGS fills out the following struct:

struct user_regs_struct
{
  unsigned long r15;
  unsigned long r14;
  unsigned long r13;
  unsigned long r12;
  unsigned long rbp;
  unsigned long rbx;
  unsigned long r11;
  unsigned long r10;
  unsigned long r9;
  unsigned long r8;
  unsigned long rax;
  unsigned long rcx;
  unsigned long rdx;
  unsigned long rsi;
  unsigned long rdi;
  unsigned long orig_rax;
  unsigned long rip;
  unsigned long cs;
  unsigned long eflags;
  unsigned long rsp;
  unsigned long ss;
  unsigned long fs_base;
  unsigned long gs_base;
  unsigned long ds;
  unsigned long es;
  unsigned long fs;
  unsigned long gs;
};

Let's write a little amd64/linux-specific wrapper for accessing meaningful fields.

const ABIArguments = struct {
    regs: c.user_regs_struct,

    fn nth(aa: ABIArguments, i: u8) c_ulong {
        std.debug.assert(i < 4);

        return switch (i) {
            0 => aa.regs.rdi,
            1 => aa.regs.rsi,
            2 => aa.regs.rdx,
            else => unreachable,
        };
    }

    fn setNth(aa: *ABIArguments, i: u8, value: c_ulong) void {
        std.debug.assert(i < 4);

        switch (i) {
            0 => { aa.regs.rdi = value; },
            1 => { aa.regs.rsi = value; },
            2 => { aa.regs.rdx = value; },
            else => unreachable,
        }
    }

    fn result(aa: ABIArguments) c_ulong { return aa.regs.rax; }

    fn setResult(aa: *ABIArguments, value: c_ulong) void {
        aa.regs.rax = value;
    }

    fn syscall(aa: ABIArguments) c_ulong { return aa.regs.orig_rax; }
};

One thing to note is that the field we read to get rax is not aa.regs.rax but aa.regs.orig_rax. This is because rax is also the return value and PTRACE_SYSCALL gets called twice for some syscalls on entrance and exit. The orig_rax field preserves the original rax value on syscall entrance. You can read more about this here.

Getting and setting registers

Now let's write the ChildManager code that actually calls PTRACE_GETREGS to fill out one of these structs.

    fn getABIArguments(cm: ChildManager) ABIArguments {
        var args = ABIArguments{ .regs = undefined };
        _ = c.ptrace(c.PTRACE_GETREGS, cm.childPid, cNullPtr, &args.regs);
        return args;
    }

Setting registers is similar, we just pass the struct back and call PTRACE_SETREGS instead:

    fn setABIArguments(cm: ChildManager, args: *ABIArguments) void {
        _ = c.ptrace(c.PTRACE_SETREGS, cm.childPid, cNullPtr, &args.regs);
    }

A hook

Now we finally have enough code to write a hook that can get and set registers; i.e. manipulate a system call!

We'll start by registering a sys_write hook in the hooks field we check in childInterceptSyscalls above.

    const hooks = &[_]struct {
        syscall: c_ulong,
        hook: *const fn (ChildManager, *ABIArguments) anyerror!void,
    }{.{
        .syscall = @intFromEnum(std.os.linux.syscalls.X64.write),
        .hook = writeHandler,
    }};

If we look at the manpage for write we see it takes three arguments

The file descriptor (fd) to write to
The address to start writing data from
And the number of bytes to write

Going back to the calling convention that means the fd will be in rdi, the data address in rsi, and the data length in rdx.

So if we shorten the data length, we should be creating a short (incomplete) write.

    fn writeHandler(cm: ChildManager, entryArgs: *ABIArguments) anyerror!void {
        const fd = entryArgs.nth(0);
        const dataAddress = entryArgs.nth(1);
        var dataLength = entryArgs.nth(2);

        // Truncate some bytes
        if (dataLength > 2) {
            dataLength -= 2;
            entryArgs.setNth(2, dataLength);
            cm.setABIArguments(entryArgs);
        }
    }

In a more sophisticated version of this program, we could randomly decide when to truncate data and randomly decide how much data to truncate. However, for our purposes this is sufficient.

But there are some real problems with this code. When I ran this program against a basic Go program, I saw duplicate requests.

Ah ok, PTRACE_SYSCALL gets hit when you both enter and exit a syscall.

So each time you call PTRACE_SYSCALL and you do stuff, you just call it again afterwards to handle/wait for the exit. pic.twitter.com/PjmNwcMepG
— Phil Eaton (@eatonphil) September 29, 2023

So the deal with PTRACE_SYSCALL is that for (most?) syscalls, you get to modify data before the data actually is handled by the kernel. And you get to modify data after the kernel has finished the syscall too.

This makes sense because PTRACE_SYSCALL (unlike PTRACE_SYSEMU) allows the syscall to actually happen. And if we wanted to, for example, modify the syscall exit code, we'd have to do that after the syscall was done not before it started. We are modifying registers directly after all.

All this means for our Zig code is that when we handle sys_write, we need to call PTRACE_SYSCALL again to process the syscall exit. Otherwise we'd reach this writeHandler for both entries and exits, which would require some additional way of disambiguating entrances from exits.

    fn writeHandler(cm: ChildManager, entryArgs: *ABIArguments) anyerror!void {
        const fd = entryArgs.nth(0);
        const dataAddress = entryArgs.nth(1);
        var dataLength = entryArgs.nth(2);

        // Truncate some bytes
        if (dataLength > 2) {
            dataLength -= 2;
            entryArgs.setNth(2, dataLength);
            cm.setABIArguments(entryArgs);
        }

        const data = try cm.childReadData(dataAddress, dataLength);
        defer data.deinit();
        std.debug.print("Got a write on {}: {s}\n", .{ fd, data.items });

        // Handle syscall exit
        _ = cm.childWaitForSyscall();
    }

We could put the cm.childWaitForSyscall() waiting for the syscall exit in the main loop and I did try that at first. However, not all syscalls seemed to have the same entry and exit hook and this resulted in the hooks sometimes starting with a syscall exit rather than a syscall entry. So rather than making the code more complicated, I decided to only wait for the exit on syscalls I knew had an exit (by observation at least), like sys_write.

Multiple writes? No bad logic?

So I had this code as is, correctly handling syscall entrances and exits, but I was seeing multiple write calls. And the text file I was writing to had the complete text I wanted to write. There was no short write even though I truncated the data length.

Ok so what happens in this Go program if I truncate the amount of data?

I assumed Go would do nothing since all I did was call `f.Write()` once and `f.Write()` returns a number of bytes written.

But actually, it still writes everything! pic.twitter.com/OSalKEbERM
— Phil Eaton (@eatonphil) September 29, 2023

This took some digging into Go source code to understand. If you trace what os.File.Write() does on Linux you eventually get to src/internal/poll/fd_unix.go:

// Write implements io.Writer.
func (fd *FD) Write(p []byte) (int, error) {
        if err := fd.writeLock(); err != nil {
                return 0, err
        }
        defer fd.writeUnlock()
        if err := fd.pd.prepareWrite(fd.isFile); err != nil {
                return 0, err
        }
        var nn int
        for {
                max := len(p)
                if fd.IsStream && max-nn > maxRW {
                        max = nn + maxRW
                }
                n, err := ignoringEINTRIO(syscall.Write, fd.Sysfd, p[nn:max])
                if n > 0 {
                        nn += n
                }
                if nn == len(p) {
                        return nn, err
                }
                if err == syscall.EAGAIN && fd.pd.pollable() {
                        if err = fd.pd.waitWrite(fd.isFile); err == nil {
                                continue
                        }
                }
                if err != nil {
                        return nn, err
                }
                if n == 0 {
                        return nn, io.ErrUnexpectedEOF
                }
        }
}

This might be common knowledge but I didn't realize Go did this. And when I tried out the same basic program in Python and even C, the behavior was the same. The builtin write() behavior on a file (in many languages apparantly) is to retry until all data is written, with some exceptions.

This makes sense since files on disk, unlike file descriptors backed by network sockets, are generally always available. Compared to a network connection, disks are physically close and almost always stay connected. (With some obvious exceptions like network-attached storage and thumb drives.)

So to trigger the short write, the easiest way seems to have the sys_write call return an error that is NOT EAGAIN since the code will retry if that is the error.

After looking through the list of errors that sys_write can return, EIO seems like a nice one.

So let's do our final version of writeHandler and on the syscall exit, we'll modify the return value (rax in amd64/linux) to be EIO.

    fn writeHandler(cm: ChildManager, entryArgs: *ABIArguments) anyerror!void {
        const fd = entryArgs.nth(0);
        const dataAddress = entryArgs.nth(1);
        var dataLength = entryArgs.nth(2);

        // Truncate some bytes
        if (dataLength > 2) {
            dataLength -= 2;
            entryArgs.setNth(2, dataLength);
            cm.setABIArguments(entryArgs);
        }

        // Handle syscall exit
        _ = cm.childWaitForSyscall();

        var exitArgs = cm.getABIArguments();
        dataLength = exitArgs.nth(2);
        if (dataLength > 2) {
            // Force the writes to stop after the first one by returning EIO.
            var result: c_ulong = 0;
            result = result -% c.EIO;
            exitArgs.setResult(result);
            cm.setABIArguments(&exitArgs);
        }
    }

Let's give it a whirl!

All together

Build the Zig fault injector and the Go test code:

$ zig build-exe --library c main.zig
$ ( cd test && go build main.go )

And run:

$ ./main test/main

And check test.txt:

$ cat test.txt
some great stu

Hey, that's a short write! :)

Sidenote: Reading data from the child

We accomplished everything we set out to, but there's one other useful thing we can do: reading the actual data passed to the write syscall.

Just like how we can get the child process registers with PTRACE_GETREGS, we can read child memory with PTRACE_PEEKDATA. PTRACE_PEEKDATA takes the child process id and the memory address in the child to read from. It returns a word of data (which on amd64/linux is 8 bytes).

We can use the syscall arguments (data address and length) to keep calling PTRACE_PEEKDATA on the child until we've read all bytes of the data the child process wanted to write:

    fn childReadData(
        cm: ChildManager,
        address: c_ulong,
        length: c_ulong,
    ) !std.ArrayList(u8) {
        var data = std.ArrayList(u8).init(cm.arena.allocator());
        while (data.items.len < length) {
            var word = c.ptrace(
                c.PTRACE_PEEKDATA,
                cm.childPid,
                address + data.items.len,
                cNullPtr,
            );

            for (std.mem.asBytes(&word)) |byte| {
                if (data.items.len == length) {
                    break;
                }
                try data.append(byte);
            }
        }
        return data;
    }

And we could modify writeHandler to print out the entirety of the write message each time (for debugging):

    fn writeHandler(cm: ChildManager, entryArgs: *ABIArguments) anyerror!void {
        const fd = entryArgs.nth(0);
        const dataAddress = entryArgs.nth(1);
        var dataLength = entryArgs.nth(2);

        // Truncate some bytes
        if (dataLength > 2) {
            dataLength -= 2;
            entryArgs.setNth(2, dataLength);
            cm.setABIArguments(entryArgs);
        }

        const data = try cm.childReadData(dataAddress, dataLength);
        defer data.deinit();
        std.debug.print("Got a write on {}: {s}\n", .{ fd, data.items });

        // Handle syscall exit
        _ = cm.childWaitForSyscall();

        var exitArgs = cm.getABIArguments();
        dataLength = exitArgs.nth(2);
        if (dataLength > 2) {
            // Force the writes to stop after the first one by returning EIO.
            var result: c_ulong = 0;
            result = result -% c.EIO;
            exitArgs.setResult(result);
            cm.setABIArguments(&exitArgs);
        }
    }

That's pretty neat!

Next steps

Short writes are just one of many bad IO interactions. Another fun one would be to completely buffer all writes on a file descriptor (not allowing anything to be written to disk at all) until fsync is called on the file descriptor. Or forcing fsyncs to fail.

An interesting optimization would be to apply seccomp filters so that rather than paying a penalty for watching every syscall, I only get notified about the ones I have hooks for like sys_write. Here's another post that explores ptrace with seccomp filters.

Credits: Thank you Charlie Cummings and Paul Khuong for reviewing a draft of this post!

Selected responses after publication

oscooter on Reddit gave some tips on using ptrace, including using process_vm_readv instead of PTRACE_PEEKDATA to read memory from the tracee process.

Fault injection is a scary-sounding term. Intercepting and modifying Linux system calls sounds scary too.

But it's a neat way to trigger logical errors in programs, to build confidence we wrote code correctly.

Let's trigger short writes to disk in Zig!https://t.co/0C3tWt3vtT pic.twitter.com/OS7auDe8jR
— Phil Eaton (@eatonphil) October 1, 2023

Wordcount III: Beating the system `wc`

Bytepawn - Marton Trencseni — Fri, 29 Sep 2023 01:00:00 GMT

In this follow-up to the previous article about writing a C++ version of the Unix command-line utility wc, I make some modifications to beat my system wc in performance tests.

Publishing my first game using pico-8

Ben E. C. Boyter — Thu, 28 Sep 2023 03:00:00 GMT

TL/DR; Play/Download a copy from itch.io

I had always wanted to create games. In fact its one of the reasons I went down the development path. For various reasons I never invested enough time (probably too much time playing them) and went into writing code for businesses.

In hindsight that was probably the right move from a financial perspective, since it seems most game developers tend to be fairly underpaid and the drama in the game development space makes high school teen drama look tame.

Vim tip 33: editing with text objects

learnbyexample — Mon, 25 Sep 2023 03:00:00 GMT

Combining motions such as w, % and f with editing commands like d, c and y require precise positioning to be effective.

Vim also provides a list of handy context based options to make certain editing use cases easier using the i and a text object selections. You can easily remember the difference between these two options by thinking i as inner and a as around.

diw delete a word regardless of where the cursor is on that word
- equivalent to using de when the cursor is on the first character of the word
diW delete a WORD regardless of where the cursor is on that WORD
daw delete a word regardless of where the cursor is on that word as well as a space character to the left/right of the word depending on its position in the current sentence
dis delete a sentence regardless of where the cursor is on that sentence
yas copy a sentence regardless of where the cursor is on that sentence as well as a space character to the left/right
cip delete a paragraph regardless of where the cursor is on that paragraph and change to Insert mode
dit delete all characters within HTML/XML tags, nesting is taken care as well
- see :h tag-blocks for details about corner cases
di" delete all characters within a pair of double quotes, regardless of where the cursor is within the quotes
da' delete all characters within a pair of single quotes along with the quote characters
ci( delete all characters within () and change to Insert mode
- works even if the parenthesis are spread over multiple lines, nesting is taken care as well
ya} copy all characters within {} including the {} characters
- works even if the braces are spread over multiple lines, nesting is taken care as well

You can use a count prefix for nested cases. For example, c2i{ will clear the inner braces (including the braces, and this could be nested too) and then only the text between braces for the next level.

See :h text-objects for more details.

Video demo:

Old-School CGI Scripts!

Steve Kemp's Blog — Sun, 24 Sep 2023 22:00:00 GMT

I'm not sure if I've talked about my job here, but I recently celebrated my one year anniversary - whilst on a company offsite trip to Sweden. When I joined the company there were approximately 100 people employed by it. Nowadays the numbers are much higher.

Having more people around is pretty awesome, but I realized that there were a lot of people wandering around the office who I didn't recognize so it occurred to me to make a game of it.

I had the idea I could write a slack bot to quiz me on my colleagues:

Show a random face, using the Slack profile picture.
Give a list of 5 names.
Ask me which was correct.

I spent an hour messing around with various Slack APIs, and decided the whole thing was too much of a hassle. Instead I wrote a simple script to download the details of all members of the workspace:

Name.
Email address.
Profile picture URL.

Then using that data, users.json, I hacked up a simple web application in Python, using the flask API. There only needed to be two pages:

A page ("/") to show five random images, each with five random names beneath them.
A page ("/quiz") to receive the HTTP POST, and score.

All in all this took only two hours or so. Old-school CGI is pretty awesome like that - Hidden values meant the whole thing could be stateless:

 <input type="hidden" name="1answer" value="Bob Smith" ..
 <input type="hidden" name="1profile" value="Sales" ..
 <input type="hidden" name="1url" value="https://.." ..

 <input type="hidden" name="2answer" value="Sally Smith" ..
 <input type="hidden" name="2profile" value="Sales" ..
 <input type="hidden" name="2url" value="https://.." ..

The only downside is that I don't have any authentication, so there is no ability to have a leaderboard. I've looked at the Okta samples and I think it would be easy to add, but I guess that would make it more complex and less portable. That said I'm not sharing the code this time, so who cares if it is tied to the company?

Anyway sometimes I forget how fast and easy it is to spinup a random virtual machine and present a HTTP(S) service for interactive use. This is one of those times when I remembered.

How I would've designed Overwatch 2

Pxtl.ca — Sun, 24 Sep 2023 03:00:00 GMT

Okay I’ve been playing the Overwatch 2 anniversary events and considering how many of them are small-group team-vs-team things (Catch the Patchmari, Starwatch). I slept on these the first time through and playing them now I’m surprised at how much I’m enjoying them, particularly StarWatch despite some of its dumber ideas (like the AI-controlled Doomfist). This has cemented my previous opinion:

OW2 should’ve been 4v4 (2 DPS, 1 tank, 1 healer) with some kind of fast-respawning mechanic for the healer, and a standard double-team mode (8v8) either in quickplay or the arcade or something

Then the number of heroes and the popularity of the roles reflects the number in play - there are twice as many DPS heroes to choose from, twice as many DPS players, and twice as many on the team.

Quantified Kilimanjaro (or Procrastination)

PREPEND — Sat, 23 Sep 2023 03:00:00 GMT

I climbed Kilimanjaro via the Rongai route in October of 2012, capturing runkeeper, and other metrics all along the way. I’ve also been meaning to write this post since then and it’s taken over 10 years to finally overcome procrastination and write up how I logged my data up to and down from the summit.

Planning - what route, when, and what coordinator

I think I first wanted to climb Kilimanjaro when I heard about it in Toto’s Africa song. It seemed pretty cool as the tallest single standing mountain in Africa and just mysterious as this giant mountain sticking up out of the wilds. And then as I aged, it seemed part of culture with Hemingway’s Snows of Kilimanjaro and just every time I heard about it, it seemed like an amazing thing to do, like visit Shangri-La.

I love hiking, but am pretty weak and not willing to train for real mountain climbing. I learned tht Kilimanjaro is over 19,000 feet but doesn’t require any technical training or special equipment so it seemed like something that might be achievable. And one night, while hiking the Inca Trail with my uncle John and friend Calros, we started talking about “wouldn’t it be cool to do this with Kilimanjaro.” That was in 2006, and in 2012, John and I went on another trip and summited Kilimanjaro. So it’s definitely doable for a tourist.

Researching Kilimanjaro, I read quite a bit. I relied mostly on some guidebooks, mostly Kilimanjaro: A Trekker’s Guide by Alex Steward (2004, Ciccerone) and Kilimanjaro - The Trekking Guide to Africa’s Highest Mountain by Henry Stedman, and I also remember reading Michael Moushabeck’s Kilimanjaro: A Photographic Journey to the Roof of Africa that wasn’t very useful but had many nice photos. I think I like books over web sites because of the editing and density of information vs filtering through tons of web pages.

Route

Kilimanjaro is really popular so there are many different routes to take:

Rongai - less busy, and has more time at each altitude for acclimatization - ended up selecting this one for 7 days because of acclimatization, this is usually 6 days but we added an extra day to acclimatize with our tour operator
Marangu - most popular, sleeping in huts, steeper climb - ended up not picking this because it had a lower success rate and seemed like it would be more crowded
Macheme - second most popular, I think people call this the “whiskey” route as it’s supposed to be hard
Lemosho - considered this because it’s also longest at 8 days, but ended up not taking it because the extra day didn’t work out with my work’s vacation schedule, and it had some driving and drop off rather than all hiking
Umbwe - shortest and quickest route

There’s a new route now called “Northern Circuit” that I don’t remember that is 9 days so I didn’t consider that in our planning.

I think we picked Rongai because it was kind of in a goldilocks zone of not being too long and not too short. My biggest concern was acclimatization because I read that’s the biggest reason for failure so this seemed like a good route to gain altitude slowly.

Joseph Mosha, my guide, handed me a copy of this hand drawn map and it was like my bible during the trek. I love it so much and have it framed on my wall.

Coordinator

It’s possible to climb Kilimanjaro on your own. While I wanted to be cost conscious, I still had a pretty open window and budget of when to go, so I wanted to find a tour operator. The trip is offered by lots of travel groups but I think most resell programs or have remote people scheduling local operators.

I wanted to make sure the operator was locally based to both support Tanzania people as well as to make sure porters and guides were treated ethically and paid. I wanted to use the African Walking Company but didn’t find a way to book directly through them so ended up booking through PeakPlanet to get to them. Back in 2012, I paid $2450 for the 7 day rongai. This was the price to add an extra day to their advertised 6-day trip. It looks like the now offer an 9-day rongai for about $3k.

This is for a small group trip where I think there was a max size of 8 people or something and they guaranteed the departure dates. There were cheaper operators if you had more flexible dates or could handle cancellations.

I also made sure this included airport transfers and shoulder lodging and wanted to pay a single price to handle all the “incidentals.” So this price included pickup from Arusha (JRO) airport, transport to Moshi and staying in their lodge before and after the trek.

This ended up being really important because there were only three people on our trek and other operators would have delayed or cancelled. PeakPlanet/AWS said that they would definitely depart and return on our selected dates and they stuck to that.

Using local staff was really important to us because I thought that Tanzanians would be more knowledgeable and all guides and porters must be local, so if non-Tanzanians are involved those are extra people beyond the actual people carrying out the trek. And PeakPlanet/AWS was really great in that we only met with and worked with Tanzanians and the two guides we had - Joseph Mosha and Elija Joel - were very experienced (dozens of summits), knowledgeable about climbing and mountain sickness, and very friendly and caring. I think Elija is the single most important reason I was able to complete and without his help, I don’t think I would make it.

PeakPlanet did have a lot of guidance on equipment, training, and general information that made preparing for the trip easier.

Here’s the route map they shared for Rongai…

When to go

I wanted to visit Kilimanjaro before the snows were gone and but wanted to find the best weather possible. I ended up picking late September 28th through October 6th to hit the end of the rainy season and also near a full moon on summit. Weather ended up being great and we didn’t have any rain at all. And I saw almost no snow as the trail was dry even up to the summit.

Equipment

The Stedman book was super helpful for all equipment purchases. I think I booked my trip and flight in February, so had about six months to shop for deals on equipment.

My basic plan was to have multiple layers, travel light, and have waterproof stuff just in case. Porters carried my bad so I had a 30 pound limit on that and just a day bag that I carried.

I rented my sleeping bag and tent from the operator, but ended up bringing the following gear:

Zamberlan boots - this is probably the most important item and I spent the most time. I considered Saloman, Asolo, Lowa, and every boot sold in Atlanta and online. I went with Zamberlan because of the fit, reputation, and features. I went with the 996 Vioz GTX as they were waterproof with goretex, leather, vibram soles, and a toe box that fit well with me. The waterproof ended up being unnecessary as I never experience snow, but they have been really handy since then going through streams and puddles and never worrying. I still wear them on week-end hikes and they are almost done. I’d guess I have over 1,000 miles over 12 years and have definitely got my money’s worth. I think I spent $300. Also, I have the original laces and they’ve held up really well. I spent about six months breaking them in on local trails and hikes.
Icebreaker merino wool t-shirts x 2 - these were amazing for both warmth as well as durability. I alternated them each day and they were still not funky by the end of the trip. They are itchy the first day but afterwards very comfortable. I still wear them 11 years later.
Icebreaker 260 thermal top and leggings. My top has a hood though that fits snugly with just the face exposed. Icebreaker didn’t have as many options back then and I was able to get them at REI on sale. I wore these on the last few days when it got cold and they were really helpful.
Smartwool socks (x3) and underwear (x3). More merino and I alternated these each day. I think my base layer was all merino and it worked out pretty well.
Patagonia Ultralight Down Hoody - this was very light at 900 grams and packed into its own little bag. I wore it at camp at night and on the summit when I pretty much wore everything
1-2 fleece mid layers that I don’t remember them. Comically I think they were lululemon or REI.
Arc’teryx Zenta AR Glove - these looked cool and also lasted, but my fingers were so cold on summit, even wearing inner wool liner gloves. I would buy something else instead.
Arc’teryx Alpha LT Hard Shell - I think this was the most expensive item I bought, but I liked it. It was extremely light so I carried it in my day bag even from the beginning. It provided no warmth but completely stopped wind when worn over my layers and had vents under the arms I opened up to help regulate heat. I still have it and wear it frequently so I think it’s a decent buy. I think I bought an XL and wish I had bought an XXL because I end up wearing it over everything else. I also bought a soft shell and ended up not wearing it because I just put this over my layers.
Camelbak Tycoon - 3 liter water capacity with 20 liters dry storage. This had insulation for the water bladder and drinking tube. Camelbak doesn’t make this any more but it’s great and I still use it for all my hikes. Camelbak support has been amazing too, the clip across my chest that holds the straps in place broke and camelbak sent me a new one free.
I brought waterproof outer pants that I don’t remember the brand. I never had to use them.
Nomad GoalZero solar kit and battery - this was really useful for charging my iphone every day. I was able to log every day’s hike and this solar kit for $100 from REI worked out well enough that I just hung the panels on my backpack each day, and charged my phone at night. I had an old iphone with gps.
RunKeeper
Some gps altitude app that shows me altitude based on GPS but didn’t need signal. I can’t remember the name, but used it and screenshot each day. I can’t find the screenshots, but recorded the altitudes into my diary.
Moleskine Squared Reporter Hardcover (that the don’t make any more).
Some astronaut pen that wrote in subzero temperatures, upside down, and under pressure.

Medications and Vaccinations

People debate whether to take Diamox or not. I read about the side effects and they were minimal so I went to a travel clinic for my vaccinations and had them give me a prescription for Diamox. I tested it out for a few days to see if I had any negative effects, and didn’t. So I took it throughout my trip. It did make me urinate a bit more than usual, but that’s not so bad as you want to drink lots of water.

I also took Malarone to protect against Malaria and took that before leaving the US and after returning. Got the same prescription from my travel clinic. I’ve taken it a few times for other trips since then and haven’t had any negative effects.

I had a prescription for xanax and took that some nights to help sleep through the screaming animals (monkeys maybe) and cold. Some people advise against this, but I think it was a life saver as it can be hard to sleep when it’s 0 degrees and you have altitude adjustment and are on the side of a mountain.

My travel clinic gave me the following vaccinations:

Typhoid
Hepatitis B
Polio
Meningococcal meningitis

I already had Hepatitis A and Yellow Fever from a previous trip and they were still good. I also had DPT and MMR.

CDC recommends Rabies but I didn’t get it as it’s expensive, has multiple courses, and it’s extremely rare to encounter wildlife on Kilimanjaro. If I had gone on a safari, I would have considered it more.

Daily log

I was really into quantified self back in 2011 and 2012 so I wanted to make sure to keep a log each day. I used a fitbit to track my exercise and food and lost 120+ pounds. And I used RunKeeper to log each hike throughout the day. And kept a paper record in my notebook.

I didn’t find any detailed logs of the actual hikes, so I thought I would capture it all here in case people wanted to see specific routes, distances, times, altitude, etc. Of course, it’s 11 years later, but then the best time to plant a tree is 20 years ago and the second best is today.

Runkeeper doesn’t really have a nice way to share or embed these hikes any more, so I’m just screenshotting each one. I’ve used runkeeper forever and it’s nice they keep the data, but I wish they had better exports.

I flew ATL->JFK->AMS->JRO and arrived on 9/27 and stayed in Kilimanjaro Mountain Resort in Marangu.

September 28, Friday

9/28 - stayed in hotel in Marangu with a trip into Moshi for shopping. Prep day to get over jet lag. $80 for the taxi from hotel to Moshi and back. Paid for everything in the trip in US dollars.

Comically, this is the only day I got a photo of Kilimanjaro looming in the distance. You could see it through the trees from the hotel. I spent the entire trek with Kilimanjaro slowing getting closer and then farther away. I must have looked at it 2,000 times. But never too a picture. Each time looking at it, I was tired and looking at where I was going and never took a photo.

September 29 (Day1), Saturday

5.54 miles, 641 meters gained. Camped at 2,641 meters.

Started at 1035 at the base of Rongai route. Made camp at 3pm. Started at 2,000 meters and set camp at 2,641. After camp, took a 45 minute walk up and back for acclimatization.

4 specific segments for today’s hikes.

September 30 (Day2), Sunday

6.92 miles, 1018 meters gained. Camped at 3,659 meters.

4 segments today as well.

Day2-Hike1, 1.73 miles, 1:05:29, finished 9:42am

| gpx | kml

Day2-Hike2, 1.38 miles, 1:43:46, finished 11:24am

| gpx | kml

Day2-Hike3, 2.25 miles, 1:37:46, finished 2:57pm

| gpx | kml

Day2-Hike4, 1.56 miles, 1:21:00, finished 4:37pm

| gpx | kml

October 1 (Day3), Monday

2.20 miles, 677 meters gained. Camped at 4,336 meters.

Just a short distance today, but a lot of altitude gained. Camped at Mawenzi Tarn at the base of Mawenzi Peak. This is the first day I really started to notice Acute Mountain Sickness (AMS). I had “thick thinking” where it would just take a long time to think through things, it took me 3 minutes to put batteries in my headlamp and that would normally take 30 seconds.

Day3-Hike1, 2.20 miles, 2:50:17, finished 11:52am

| gpx | kml

October 2 (Day4), Tuesday

1.3 miles, 164 meters gained. Camped at 4,500 meters.

Stayed at Mawenzi Tarn again and spent the day just hanging out and acclimatizing. This is the only day I touched snow as I went hiked around the base to some standing snow.

We were above the clouds and it was really beautiful just relaxing after a few days and using lots of sunblock. Also spent time with clouds rolling over us and it was really neat to be inside the cloud as it passed over.

Day4-Hike1, 1.31 miles, 1:39:39, finished 10:45am

| gpx | kml

October 3 (Day5), Wednesday

5.32 miles, 215 meters gained. Camped at 4,715 meters.

Arrived at Kibo Hut camp “Base camp”. So it took about 21 miles to get here. We arrived pretty early at 1pm. We napped until dinner at 5, then sleep until 11, eat a light meal and start the summit. The it should have taken 6 hours to Gillman’s point, then 2 hours to Uhuru, then 2-4 hours back down. We didn’t actually sleep in the hut, we stayed in our standard tents.

Day5-Hike1, 5.32 miles, 3:30:54, finished 11:16am

| gpx | kml

October 4 (Day6), Thursday

So technically, I started summit at 11:56pm but include it as day 6.

3 segments today, but only the first with runkeeper.

3.50 miles, 1195 meters gained to summit.

Phone battery died when I pulled it out at summit to end runkeeper and take a picture. Luckily I had it inside my layers and it tracked everything from Kibo hut to Uhuru.

So what’s interesting is my diary has the max altitude as 5910 meters even though the actual hight is 5895 meters. So this is wrong, but I kept it in my records as I measured altitudes consistently with the same app so kept this as the altitude for peak assuming that my app would measure wrong, consistently so at least this lets me calculate the altitude gained or lost.

3.50 miles back down from summit and it took 2 hours. Then napped for an hour and got up to walk about six miles to camp at Horombu Hut, 3,720 meters, so 995 meters lost from last camp.

Day6-Hike1, 3.50 miles, 7:16:58, finished 7:12am

| gpx | kml

Day6-Hike2, 3.50 miles, ~2:00, finished 9:16am

Not sure exactly how long this took, my diary says 2 hours.

Day6-Hike3, ~6 miles, ???, finished ~4:05pm

Not sure how long this took, but my diary entry was at 4:05 so I guess I finished a little before then.

October 5 (Day7), Friday

13.02 miles, 1,830 meters lost. Camped at 1890 in hotel.

2 segments.

Day7-Hike1, 7.74 miles, 3:11:35, finished 11:27am

| gpx | kml

Day7-Hike2, 5.28 miles, 2:15:43, finished 2:29pm

| gpx | kml

Summary Table

Date	Start	End	Time Walking (hours:minutes)	Distance (miles)	Altitude Gain (meters)	End Altitude (meters)	Cumulative Distance (miles)
9/29	1101	1618	3:28	5.54	641	2,641	5.54
9/30	0837	1637	5:47	6.92	1,018	3,659	12.46
10/1	0902	1152	2:50	2.20	677	4,336	14.66
10/2	0906	1018	1:39	1.31	164	4,500	15.97
10/3	0746	1126	3:31	5.32	215	4,715	21.29
10/4 summit	2356	0712	7:16	3.50	1,195	5910	24.79
10/4 descent	0721	1605	???	9.50	-995	3720	34.29
10/5	0816	1429	5:27	13.02	-1,830	1,890	37.31

47.31 miles total, 24.79 miles up, 22.52 down; 6.75 average per day.

3910 meters of elevation gain; average of 652 across six days climbing.

Advice

I hope this is helpful. I wanted to know about the specifics of the hikes when I was researching and I wasn’t able to see the kind of pace that people kept.

What’s interesting to me is that the time actually hiking was rather short each day. Summit day was really long with about 16 hours, but other than that it was usually just a few hours of hiking and the rest of the time hanging around at camp.

I would bring less equipment next time as I think I probably had 3 or 4 backup layers and fleeces. I just wore the same thing. Maybe two copies of each to alternate them. I definitely wouldn’t bring a soft shell.

I didn’t bring lightweight but warm camp shoes so I had to wear my boots around camp.

Altitude was rough so I would try to train more. I live on the east coast and climbed our highest mountains, but that’s only like 5,000 feet. To do it again, I think I’d spend more time in Arusha/Marangu as it’s about 2,000 meters high. And I’d take more time than 7 days. The longer rongai sounds nice or maybe that new northern circuit trek.

I’m glad that I didn’t skimp on my operator and it was worth everything. The food was nice and the campsite was always great and clean and ready to go when we got there.

I’d bring more cash to tip. I think I tipped $300 total and that was all the cash I brought. But since our group was smaller than usual with only 3 people, I would have liked to tip more to make up for it.

Procrastination

So I’ve had all this data and all these notes since the trip. I’ve been thinking it over in my head for all these years. I never sat down to write and collect until today. Why did I do that? I’m interested in the trek. I like seeing all the numbers line up. But what’s been pushing this off? I’m not sure. It was hard as I think it took about 8 hours to write everything up and figure out what I wrote in my notes.

I find it interesting when I know what to do and why, but just don’t do it. I hope to figure out why I do that.

Things I read / References

A Disk Usage Mystery

Weblog on marginalia.nu — Sat, 23 Sep 2023 03:00:00 GMT

I ran into a bit of a puzzling situation yesterday, testing some of the new index construction changes before they’re going live in a few days. The process crashed with a pretty non-descript stack trace complaining about illegal instructions, so first glance it looked more like it was within the realm of freak JVM bug, cosmic ray, hardware error maybe. I was doing this on my developer workstation, which also spawned a popup complaining that the hard drive it was working on had nearly run out of space, and inferred that the error probably was due to memory mapping more space onto a disk than what was possible.

Wordcount II: Introducing a cleaner C++ class hierarchy to `wc`

Bytepawn - Marton Trencseni — Sat, 23 Sep 2023 01:00:00 GMT

In this follow-up to the previous article about writing a C++ version of the Unix command-line utility wc, I make the class structure more complicated to keep separate concerns and functionality in different C++ classes. The result ends up being significantly more complex than the original, but does not make the overall program easier to understand or modify.

How do databases execute expressions?

Notes on software development — Thu, 21 Sep 2023 03:00:00 GMT

Databases are fun. They sit at the confluence of Computer Science topics that might otherwise not seem practical in life as a developer. For example, every database with a query language is also a programming language implementation of some caliber. That doesn't include all databases though of course; see: RocksDB, FoundationDB, TigerBeetle, etc.

This post looks at how various databases execute expressions in their query language.

tldr; Most surveyed databases use a tree-walking interpreter. A few use stack- or register-based virtual machines. A couple have just-in-time compilers. And, tangentially, a few do vectorized interpretation.

Throughout this post I'll use "virtual machine" as a shorthand for stack- or register-based loops that process a linearized set of instructions. I say this since it is sometimes fair to call a tree-walking interpreter a virtual machine. But that is not what I mean when I say virtual machine in this post.

Stepping back

Programming languages are typically implemented by turning an Abstract Syntax Tree (AST) into a linear set of instructions for a virtual machine (e.g. CPython, Java, C#) or native code (e.g. GCC's C compiler, Go, Rust). Some of the former implementations also generate and run Just-In-Time (JIT) compiled native code (e.g. Java and C#).

Less commonly these days in programming languages does the implementation interpret off the AST or some other tree-like intermediate representation. This style is often called tree-walking.

Shell languages sometimes do tree-walking. Otherwise, implementations that interpret directly off of a tree normally do so as a short-term measure before switching to compiled virtual machine code or JIT-ed native code (e.g. some JavaScript implementations, GraalVM, RPython, etc.)

That is, while some major programming language implementations started out with tree-walking interpreters, they mostly moved away from solely tree-walking over a decade ago. See JSC in 2008, Ruby in 2007, etc.

My intuition is that tree-walking takes up more memory and is less cache-friendly than the linear instructions you give to a virtual machine or to your CPU. There are some folks who disagree, but they mostly talk about tree-walking when you've also got a JIT compiler hooked up. Which isn't quite the same thing. There has also been some early exploration and improvements reported when tree-walking with a tree organized as an array.

And databases?

Databases often interpret directly off a tree. (It isn't, generally speaking, fair to say they are AST-walking interpreters because databases typically transform and optimize beyond just an AST as parsed from user code.)

But not all databases interpret a tree. Some have a virtual machine. And some generate and run JIT-ed native code.

Methodology

If a core function (in the query execution path that does something like arithmetic or comparison) returns a value, that's a sign it's a tree-walking interpreter. Or, if you see code that is evaluating its arguments during execution, that's also a sign of a tree-walking interpreter.

On the other hand, if the function mutates internal state such as by assigning a value to a context or pushing to a stack, that's a sign it's a stack- or register-based virtual machine. If a function pulls its arguments from memory and doesn't evaluate the arguments, that's also an indication it's a stack- or register-based virtual machine.

This approach can result in false-positives depending on the architecture of the interpreter. User-defined functions (UDFs) would probably accept evaluated arguments and return a value regardless of how the interpreter is implemented. So it's important to find not just functions that could be implemented like UDFs, but core builtin behavior. Control flow implementations of functions like if or case can be great places to look.

Note: In talking about a broad swath of projects, maybe I've misunderstood one or some. If you've got a correction, let me know! If there's a proprietary database you work on where you can link to the (publicly described) execution strategy, feel free to pass it along! Or if I'm missing your public-source database in this list, send me a message!

Survey

Cockroach (Ruling: Tree Walker)

Judging by functions like func (e *evaluator) EvalBinaryExpr that evaluates the left-hand side and then evaluates the right-hand side and returns a value, Cockroach looks like a tree walking interpreter.

It gets a little more interesting though, since Cockroach also supports vectorized expression execution. Vectorizing is a fancy term for acting on many pieces of data at once rather than one at a time. It doesn't necessarily imply SIMD. Here is an example of a vectorized addition of two int64 columns.

ClickHouse (Ruling: Tree Walker + JIT)

The ClickHouse architecture is a little unique and difficult for me to read through – likely due to it being fairly mature, with serious optimization. But they tend to document their header files well. So files like src/Functions/IFunction.h and src/Interpreters/ExpressionActions.h were helpful.

They have also spoken publicly about their pipeline execution model; e.g. this presentation and this roadmap issue. But it isn't completely clear how much pipeline execution (which is broader than just expression evaluation) connects to expression evaluation.

Moreover, they have publicly spoken about their support for JIT compilation for query execution. But let's look at how execution works when the JIT is not enabled. For example, If we take a look at how if is implemented, we know that the then and else rows must be conditionally evaluated.

In the runtime entrypoint, executeImpl, we see the function call executeShortCircuitArguments which in turn calls ColumnFunction::reduce() which evaluates each column vector that is an argument to the function and then calls execute on the function.

So from this we can tell the non-JIT execution is a tree walker and that it is over chunks of columns, i.e. vectorized data, similar to Cockroach. However in ClickHouse execution is always over column vectors.

In the original version of this post, I had some confusion about the ClickHouse execution strategy. Robert Schulze from ClickHouse helped clarify things for me. Thanks Robert!

DuckDB (Ruling: Tree Walker)

If we take a look at how function expressions are executed, we can see each argument in the function being evaluated before being passed to the actual function. So that looks like a tree walking interpreter.

Like ClickHouse, DuckDB expression execution is always over column vectors. You can read more about this architecture here and here.

Influx (Ruling: Tree Walker)

Influx originally had a SQL-like query language called InfluxQL. If we look at how it evaluates a binary expression, it first evaluates the left-hand side and then the right-hand side before operating on the sides and returning a value. That's a tree-walking interpreter.

Flux was the new query language for Influx. While the Flux docs suggest they transform to an intermediate representation that is executed on a virtual machine, there's nothing I'm seeing that looks like a stack- or register-based virtual machine. All the evaluation functions evaluate their arguments and return a value. That looks like a tree-walking interpreter to me.

Today Influx announced that Flux is in maintenance mode and they are focusing on InfluxQL again.

MariaDB / MySQL (Ruling: Tree Walker)

Control flow methods are normally a good way to see how an interpreter is implemented. The implementation of COALESCE looks pretty simple. We see it call val_str() for each argument to COALESCE. But I can only seem to find implementations of val_str() on raw values and not expressions. Item_func_coalesce itself does not implement val_str() for example, which would be a strong indication of a tree walker. Maybe it does implement val_str() through inheritance.

It becomes a little clearer if we look at non-control flow methods like acos. In this method we see Item_func_acos itself implement val_real() and also call val_real() on all its arguments. In this case it's obvious how the control flow of acos(acos(.5)) would work. So that seems to indicate expressions are executed with a tree walking interpreter.

I also noticed sql/sp_instr.cc. That is scary (in terms of invalidating my analysis) since it looks like a virtual machine. But after looking through it, I think this virtual machine only corresponds to how stored procedures are executed, hence the sp_ prefix for Stored Programs. MySQL docs also explain that stored procedures are executed with a bytecode virtual machine.

I'm curious why they don't use that virtual machine for query execution.

As far as I can tell MySQL and MariaDB do not differ in this regard.

MongoDB (Ruling: Virtual Machine)

Mongo recently introduced a virtual machine for executing queries, called Slot Based Execution (SBE). We can find the SBE code in src/mongo/db/exec/sbe/vm/vm.cpp and the main virtual machine entrypoint here. Looks like a classic stack-based virtual machine!

It isn't completely clear to me if the SBE path is always used or if there are still cases where it falls back to their old execution model. You can read more about Mongo execution here and here.

PostgreSQL (Ruling: Virtual Machine + JIT)

The top of PostgreSQL's src/backend/executor/execExprInterp.c clearly explains that expression execution uses a virtual machine. You see all the hallmarks: opcodes, a loop over a giant switch, etc. And if we look at how function expressions are executed, we see another hallmark which is that the function expression code doesn't evaluate its arguments. They've already been evaluated. And function expression code just acts on the results of its arguments.

PostgreSQL also supports JIT-ing expression execution. And we can find the switch between interpreting and JIT-compiling an expression here.

QuestDB (Ruling: Tree Walker + JIT)

QuestDB wrote about their execution engine recently. When the conditions are right, they'll switch over to a JIT compiler and run native code.

But let's look at the default path. For example, how AND is implemented. AndBooleanFunction implements BooleanFunction which implements Function. An expression can be evaluated by calling a getX() method on the expression type that implements Function. AndBooleanFunction calls getBool() on its left and right hand sides. And if we look at the partial implementation of BooleanFunction we'll also see it doing getX() specific conversions during the call of getX(). So that's a tree-walking interpreter.

Scylla (Ruling: Tree Walker)

If we take a look at how functions are evaluated in Scylla, we see function evaluation first evaluating all of its arguments. And the function evaluation function itself returns a cql3::raw_value. So that's a tree-walking interpreter.

SQLite (Ruling: Virtual Machine)

SQLite's virtual machine is comprehensive and well-documented. It encompasses more than just expression evaluation but the entirety of query execution.

We can find the massive virtual machine switch in src/vdbe.c.

And if we look, for example, at how AND is implemented, we see it pulling its arguments out of memory (already evaluated) and assigning the result back to a designated point in memory.

SingleStore (Ruling: Virtual Machine + JIT)

While there's no source code to link to, SingleStore gave a talk at CMU that broke down their query execution pipeline. Their docs also cover the topic.

TiDB (Ruling: Tree Walker)

Similar to DuckDB and ClickHouse, TiDB implements vectorized interpretation. They've written publicly about their switch to this method.

Let's take a look at how if is implemented in TiDB. There is a vectorized and non-vectorized version of if (in expression/control_builtin.go and expression/control_builtin_generated.go respectively). So maybe they haven't completely switched over to vectorized execution or maybe it can only be used in some conditions.

If we look at the non-vectorized version of if, we see the condition evaluated. And then the then or else is evaluated depending on the result of the condition. That's a tree-walking interpreter.

Conclusion

As the DuckDB team points out, vectorized interpretation or JIT compilation seem like the future for database expression execution. These strategies seem particularly important for analytics or time-series workloads. But vectorized interpretation seems to make the most sense for column-wise storage engines. And column-wise storage normally only makes sense for analytics workloads. Still, TiDB and Cockroach are transactional databases that also vectorize execution.

And while SQLite and PostgreSQL use the virtual machine model, it's possible databases with tree-walking interpreters like Scylla and MySQL/MariaDB have decided there is not significant enough gains to be had (for transactional workloads) to justify the complexity of moving to a compiler + virtual machine architecture.

Tree-walking interpreters and virtual machines are also independent from whether or not execution is vectorized. So that will be another interesting dimension to watch: if more databases move toward vectorized execution even if they don't adapt JIT compilation.

Yet another alternative is that maybe as databases mature we'll see compilation tiers similar to what browsers do with JavaScript.

Credits: Thanks Max Bernstein, Alex Miller, and Justin Jaffray for reviewing a draft version of this! And thanks to the #dbs channel on Discord for instigating this post!

I spent some time looking into how various databases execute expressions in their query language.

Most of them have a tree-walking interpreter, some have a virtual machine, and some do just-in-time compilation.

Let's dig into some database code to see!https://t.co/BIGtHKh1X4 pic.twitter.com/nmhe9HmYw7
— Phil Eaton (@eatonphil) September 21, 2023

F-Zero 99

Pxtl.ca — Tue, 19 Sep 2023 03:00:00 GMT

i got a little carried away with this nostalgia game. I loved the old F-Zero and this remake lets you play the classic against 98 other players at once.

Vim tip 32: text and indent settings

learnbyexample — Tue, 19 Sep 2023 03:00:00 GMT

Here are some text and indent Vim settings that you can put in the vimrc file to customize your editor. See :h options.txt for complete reference.

filetype plugin indent on enables loading of plugin and indent files
- these files become active based on the type of the file to influence syntax highlighting, indentation, etc
- :echo $VIMRUNTIME gives your installation directory (indent and plugin directories would be present in this path)
- see :h vimrc-filetype, :h :filetype-overview and :h filetype.txt for more details
set autoindent copy indent from the current line when starting a new line
- useful for files not affected by indent setting
- see also :h smartindent
set textwidth=80 guideline for Vim to automatically move to a new line with 80 characters as the limit
- white space is used to break lines, so a line can still be greater than the limit if there's no white space
- default is 0 which disables this setting
set colorcolumn=80 create a highlighted vertical bar at column number 80
- use highlight ColorColumn setting to customize the color for this vertical bar
- see vi.stackexchange: Keeping lines to less than 80 characters for more details
set shiftwidth=4 number of spaces to use for indentation (default is 8)
set tabstop=4 width for the tab character (default is 8)
set expandtab use spaces for tab expansion
set cursorline highlight the line containing the cursor

Video demo:

Software 3.0 and the AI Engineer Landscape (talk notes & slides)

swyx's site RSS Feed — Mon, 18 Sep 2023 22:50:48 GMT

2024 update

Thank you

Letters To A New Developer — Mon, 18 Sep 2023 19:12:21 GMT

Dear reader, Welcome to the last letter. I couldn’t have imagined what would happen when I penned my first letter 5 years ago. (I still think you should learn version control, for the record.) This blog outlasted two jobs. 280+ posts and 220,000+ visitors later, this project has changed my life in so many ways. … Continue reading →

Steam engine or bridge?

Blogity blog blog. Journal — Fri, 15 Sep 2023 23:23:00 GMT

When looking at the bigger picture of writing and maintaining complex software systems, I look at it and ask: have I built the software-equivalent of a bridge, or a steam engine. While a bridge needs mainteance and inspection, after it’s built, it largely just sits there. You spend a lot of time up-front, designing and engineering it before even a single brick is laid.

A steam engine on the other hand, only just sits there if you don’t want it to do what it was designed to do - be an engine. It needs a team of people working tirelessly round the clock babysitting it to cajole it into performing.

‘ls’ is a bridge piece of software. It basically sits there. But SaaS companies’ backends are a complex steam engine piece of software, which is why you need SREs.

Marginalia Search receives FUTO Grant

Weblog on marginalia.nu — Fri, 15 Sep 2023 03:00:00 GMT

I’m happy to announce that the generous people at FUTO have granted the project $15,000 with no strings attached to help the search engine out with some more server power. FUTO is a young Austin, TX-based organization “dedicated to developing, both through in-house engineering and investment, technologies that frustrate centralization and industry consolidation”. It’s one to keep an eye on, I believe their heart is in the right place and they have every possibility of making a real difference.

What I'm up to - September 2023

Philip I. Thomas — Tue, 12 Sep 2023 15:54:53 GMT

I'm Philip, and this is my monthly newsletter about what I'm up to, which I send in place of social media.

✨ What I was up to in August

Launched FRCTNL, a community of fractional tech workers.
Continued building Booklet, which is the software powering FRCTNL.
Returned to Mexico City, where Quintonil reaffirmed its rank as my #2 favorite restaurant in the world.
Ate a lot of tsukemen.
Graded at a crossword puzzle tournament.

🤔 Things to share

Articles: Do you avoid the news? You’re in growing company. (See "News Minimalist" below). Walk for the Boredom of it All. Scans of coffee equipment.
Books: Steve Jobs by Walter Isaacson (re-reading after a decade). Cold Start Problem (again), Hacker and Painters.
Podcast: Frank Slootman on The Knowledge Project (a refreshing take on high-performance cultures).
Apps: News Minimalist (now the only news I'm reading).
Video: Tunnel Vision: An Unauthorized BART Ride
Other: Noma Projects finally ships to the USA. Once, a new project from 37Signals.Thinking about Eichler homes. "Where founders live matters again."
Trend: PE for startups (such as Third South Capital and Tiny).
Coffees I'm drinking: La Cabra Elida.

📫 What I'm up to in September

Making a couple of brief trips for weddings
Hoping to open Booklet access more broadly

📍 Where I'll be

Sept 21-25: Stockholm 🇸🇪
Sept 29-Oct 1: Chicago 🏢

(Let me know if we overlap)

📸 Photo

Bugs were a (tasty) theme in Mexico

More rigid

Pxtl.ca — Tue, 12 Sep 2023 03:00:00 GMT

If you watch "Well There's Your Problem" podcast (about engineering disasters), Alice has an iconoclastic opinion on things like expansion plates and swivel joints and the general flexibility of good engineering: "No, make it more rigid!" I have taken that to heart with some recent home improvement projects.

CallerArgumentExpression and extension methods don’t mix

The NeoSmart Files — Mon, 11 Sep 2023 20:17:55 GMT

This post is for the C# developers out there and takes a look at the interesting conjunction of [CallerArgumentExpression] and static extension methods – a mix that at first seems too convenient to pass up. A quick recap: [CallerArgumentExpression] landed … Continue reading →

The post CallerArgumentExpression and extension methods don’t mix first appeared on The NeoSmart Files.

CLI tip 33: manipulating string case with GNU sed

learnbyexample — Mon, 11 Sep 2023 03:00:00 GMT

sed provides escape sequences to change the case of replacement strings, which might include backreferences, shell variables, etc.

Sequence	Description
`\E`	indicates the end of case conversion
`\l`	convert the next character to lowercase
`\u`	convert the next character to uppercase
`\L`	convert the following characters to lowercase (overridden by `\U` or `\E`)
`\U`	convert the following characters to uppercase (overridden by `\L` or `\E`)

First up, changing case of only the immediate next character after the escape sequence.

# match only the first character of a word
# use & to backreference the matched character
# \u would then change it to uppercase
$ echo 'hello there. how are you?' | sed 's/\b\w/\u&/g'
Hello There. How Are You?

# change the first character of a word to lowercase
$ echo 'HELLO THERE. HOW ARE YOU?' | sed 's/\b\w/\l&/g'
hELLO tHERE. hOW aRE yOU?

# match lowercase followed by underscore followed by lowercase
# delete the underscore and convert the 2nd lowercase to uppercase
$ echo '_fig aug_price next_line' | sed -E 's/([a-z])_([a-z])/\1\u\2/g'
_fig augPrice nextLine

Next, changing case of multiple characters at a time.

# change all alphabets to lowercase
$ echo 'HaVE a nICe dAy' | sed 's/.*/\L&/'
have a nice day
# change all alphabets to uppercase
$ echo 'HaVE a nICe dAy' | sed 's/.*/\U&/'
HAVE A NICE DAY

# \E will stop further conversion
$ echo 'fig_ aug_price next_line' | sed -E 's/([a-z]+)(_[a-z]+)/\U\1\E\2/g'
fig_ AUG_price NEXT_line
# \L or \U will override any existing conversion
$ echo 'HeLLo:bYe gOoD:beTTEr' | sed -E 's/([a-z]+)(:[a-z]+)/\L\1\U\2/Ig'
hello:BYE good:BETTER

Finally, examples where escapes are used next to each other.

# uppercase first character of a word
# and lowercase rest of the word characters
# note the order of escapes used, \u\L won't work
$ echo 'HeLLo:bYe gOoD:beTTEr' | sed -E 's/[a-z]+/\L\u&/Ig'
Hello:Bye Good:Better

# lowercase first character of a word
# and uppercase rest of the word characters
$ echo 'HeLLo:bYe gOoD:beTTEr' | sed -E 's/[a-z]+/\U\l&/Ig'
hELLO:bYE gOOD:bETTER

Video demo:

See also my CLI text processing with GNU sed ebook.

Wordcount I: Implementing the Unix command-line tool `wc` in modern C++

Bytepawn - Marton Trencseni — Sun, 10 Sep 2023 01:00:00 GMT

After reading the excellent book Beautiful C++ about the language's latest features, I implement the Unix command-line tool wc in modern C++.

Swyx's Simple Guide to Singapore

swyx's site RSS Feed — Thu, 07 Sep 2023 19:59:56 GMT

A personal guide to Singapore for foreign friends visiting.

Python tip 33: sorting iterables based on multiple conditions

learnbyexample — Tue, 05 Sep 2023 03:00:00 GMT

In an earlier tip, you learned how to sort iterables based on a key. You can use a sequence like list or tuple to specify a tie-breaker condition when two or more items are deemed equal under the primary sorting rule.

>>> books = ('Mage Errant', 'Piranesi', 'Cradle', 'The Weirkey Chronicles', 'Mistborn')

# sorts based on the number of words
# retains original order for items with the same number of words
>>> sorted(books, key=lambda b: b.count(' '))
['Piranesi', 'Cradle', 'Mistborn', 'Mage Errant', 'The Weirkey Chronicles']

# items with the same number of words are further sorted in alphabetic order
>>> sorted(books, key=lambda b: (b.count(' '), b))
['Cradle', 'Mistborn', 'Piranesi', 'Mage Errant', 'The Weirkey Chronicles']

To sort in descending order, usually the reverse=True keyword argument is used. But what if the primary and secondary rules are opposites? If one of the rule is numerical in nature, you can simply negate the number to reverse the order.

# descending order based on the number of words
# ascending alphabetic order for items with the same number of words
>>> sorted(books, key=lambda b: (-b.count(' '), b))
['The Weirkey Chronicles', 'Mage Errant', 'Cradle', 'Mistborn', 'Piranesi']

# reverse the above result
>>> sorted(books, key=lambda b: (-b.count(' '), b), reverse=True)
['Piranesi', 'Mistborn', 'Cradle', 'Mage Errant', 'The Weirkey Chronicles']

Brag post: Hacker News Front Page entries

learnbyexample — Tue, 05 Sep 2023 03:00:00 GMT

In case you haven't yet read this nice post "Get your work recognized: write a brag document" by Julia Evans, please do that first.

I definitely found it nice to collect which of my content have reached Hacker News front page over the past 4 years. As I wrote in my book writing experience post, the responses I got for my GNU awk one-liners collection was one of the stepping stones towards my career as a technical author.

Here's the list so far, ordered by oldest first:

Learn to use Awk with hundreds of examples — 478 points, Oct 2017, 116 comments
Show HN: I wrote a book on GNU grep and ripgrep — 182 points, June 2019, 53 comments
Show HN: I wrote a book on Python regular expressions — 193 points, Aug 2019, 50 comments
Show HN: An eBook with hundreds of GNU Awk one-liners — 539 points, April 2020, 48 comments
Show HN: Ruby One-Liners Cookbook — 191 points, Sept 2020, 36 comments
Perl One-Liners Cookbook — 126 points, Nov 2020, 47 comments
Show HN: "100 Page Python Intro" eBook — 107 points, Feb 2021, 26 comments
Paying my bills with 'free' ebooks — 85 points, Mar 2021, 22 comments
Show HN: "Command line text processing with GNU Coreutils" eBook — 117 points, Oct 2021, 20 comments
Show HN: Improve your Python regex skills with 75 interactive exercises — 175 points, Nov 2021, 12 comments
Vim prank: alias vim='vim -y' — 341 points, Jan 2022, 259 comments
Vim Reference Guide — 244 points, Mar 2022, 110 comments
Show HN: Interactive exercises for Linux CLI text processing commands — 69 points, Dec 2022, 7 comments
CLI text processing with GNU awk — 419 points, Aug 2023, 129 comments

As is the case with other social media platforms, being an active participant on Hacker News definitely helps. Apart from commenting on other topics, I also post links to projects and resources that I felt were useful. The number of such links reaching front page outnumbers my own content links.

It’s not magic (part 2)

Letters To A New Developer — Mon, 04 Sep 2023 15:16:47 GMT

Dear new developer, When you see a senior engineer who knows the code base like the back of her hand maneuver around it at the speed of thought, it sure looks like magic. When a mysterious bug pops up and you spend a couple of hours trying to replicate it, but then an old hand … Continue reading →

AI: The Somnium Files review

ブラック — Mon, 04 Sep 2023 03:00:00 GMT

Eye spy with my little AI.

AI: The Somnium Files is an interactive murder mystery story. You play as Date Kaname, an amnesiac special agent trying to track down a mysterious serial killer using an AI-powered eyeball and a machine that allows invading the dreams of investigation targets to unlock the secrets of their minds by solving puzzles.

But despite being a murder mystery, it kinda doesn't feel like one during a good chunk of it. The mood of it is kinda all over the place and quite a vast chunk of the game is spent engaging in various kinds of comedy routines. I imagine the humour won't land on everyone – it is quite Japanese and trends towards lower than higher end of the brow. I did generally like the comedic bits but it does induce a level of mood whiplash as you go from fawning over a porn magazine to gruesome murder in the span of 20 minutes. Thankfully it does manage to stay serious during the more serious and somber moments of the story, so it's not incredibly jarring.

The Somnium Files features a relatively unique way of branching the storyline. Taking certain actions in the game puts you in a different branch in the story, and some of the branches lock you out in middle of it, telling you to go finish other branches to unlock it. It even unlocks a route, lets you read through it for a bit and then locks it up again in order to throw you onto another, now-unlocked route. At first I wasn't really sure what to think of it, but now in hindsight, I do like the implementation. It does a good job at keeping you guessing for a good chunk of the story, preventing you from reaching the juiciest bits first while simultaneously dropping hints across the different branches.

As for the story itself, I really liked it. It's interesting, intriguing and engaging. It was fun to gather the clues throughout the story and work together the connections. And boy, are there connections. This thing is more interconnected than the World Wide Web. I'm proud that I was able to sleuth some of the plot points but I still didn't figure out the true culprit until the big reveal. As a novice to mysteries, the solution felt a bit out of left field, but not in a way that felt unsatisfying or unfair. And once you had the solution, a lot of the story prior to that just started clicking into places. Even some offhand remarks, made seemingly in jest, fit into the puzzle.

Beyond just the murder mystery, I also enjoyed overall character stories. Most of the early endings don't end you with solving the crime but still had satisfying and emotional finales to them. I may have even shed a tear or two for this perverted detective adventure. That being said, the story isn't all that realistic. You can probably figure that out within minutes when they tell you that you have an artificial eyeball that contains a highly sophisticated artificial intelligence. So there's definitely a degree of sci-fi going on here. However, it also has other very unrealistic elements too for you to suspend your disbelief. You might have a little girl with the power to take down a bear with her bare hands, or have a squadron of gunmen shooting worse than the freaking Stormtroopers. But if this kind of a thing doesn't faze you, there's a fun and interesting story waiting for you.

Most of the game is spent in interactive point-and-click segments where you explore the world and talk to the various characters. I don't think there's really any way to fail these, so it plays more like a visual novel with some leeway as to how many things you want to interact with. You also have some action bits with quick-time events and interrogation events where you need to pick evidence to show to a suspect to reel them in. The core gameplay is quite simple without really anything to particularly like or dislike, although I would've liked to be able to see what objects had new dialogue and which didn't. Could've saved me from clicking on Boss' computer for the ninth time to see if there was dialogue hidden behind it.

The other part of the game, and arguably the most video game part of the whole thing, are the titular Somnium segments. These are puzzle segments where you need to figure out how to unlock the secrets of the target's mind before you run out of time. But counter to this description, these are relatively slow affairs. You have 360 seconds to spare and each action you can take inside the dream takes up an arbitrary amount of time, and idling in place happens in super-slow motion. Running towards a door and kicking it open might be just a second, or it might be closer to a minute, depending on the stage and conditions. And if you run out of time, you'll fail and have to replay through most if not all of it again. These dream sequences are also where the actual branching happens, so you won't need a walkthrough for routing.

I'm a bit of two minds about the Somnium segments. On one hand, they were an interesting and unique aspect of the game, with the occasional moment of hilarity. And while some of the puzzles I could determine by remembering what clues I'd gathered during the normal gameplay and applying them in the dream, some of the aspects required just utterly pure guesswork. How am I supposed to know whether to stick my head or my hand inside a hole? And if you guess wrong and run out of time, you're gonna waste a bunch of time running through the same already-solved puzzles in order to progress through. These puzzles do definitely add to the game, but I just wish they didn't waste as much of my time if my guesses didn't work out. After a fail, it just becomes an exercise of mechanical repetition and holding down the skip button.

The graphics are so-so. The world detail leaves a lot of room for improvement but the character models are mostly good with excellent designs. Some of the facial animations look a bit iffy at times, but I did quite like the overall 3D character look. There's a sense of charm about it. I wasn't necessarily expecting that since I've usually preferred 2D sprites over 3D models in other games. Iris and Mizuki were both very cute in the game. And of course, with everyone being a 3D model, you could also have properly animated cutscenes in the game. Well, cutscene animation was also a bit strange at times too, but it's still a more visual affair than most visual novels.

No complaints on the sound side. Opting for the original Japanese voices, the voice acting was pretty good. Not its greatest asset but still good. The soundtrack was nothing special but it did its job providing the mood.

The technical side however is a bit sketchy. This is quite clearly a console port and has some basic graphics options inside a separate launcher that don't all even work properly. It also forces you to choose between keyboard/mouse and controller input in the launcher, even though you can still use a controller with keyboard/mouse controls. You just don't get the proper button prompts for a controller. Annoyingly it also ignored my choice for Japanese voice acting for the ending movie for reasons unknown. Granted, it never crashed on me or anything, but a bit more polish on the technical front would've been nice.

I would definitely recommend reading through AI: The Somnium Files. Its gameplay won't win it any recognition but story is interesting, rather fun, occasionally touching and most likely will keep you guessing until the end. I'm definitely going to be checking out the sequel, AI: The Somnium Files - Nirvana Initiative, at some point. I imagine I also need to check out other Uchikoshi production too.

Practice and Experience

bayindirh — Mon, 04 Sep 2023 03:00:00 GMT

During my lifetime, I have met with people who had certain skills, like reading the weather, or knowing a technical subject inside out, or being able to learn pretty quickly while being thorough. I have admired these people, and wanted to learn what made them tick. The main question was "How is this possible, and what can I learn from these people", and the feeling accompanying this question was always a healthy wonder and awe, not envy or anger.

Moreover, I noticed that these people didn't boast their skills, in any way. A neighbor may make a passing remark about tomorrow's weather, or a friend will casually note a pitfall I may encounter during implementing something. Moreover, things progress exactly the way they noted earlier. This like being among a band of oracles, and both feels awesome and makes you wonder about how this is possible.

As life progressed, and I tried to quench my thirst for learning, a pattern started to emerge. I noticed that I carry a big body of knowledge about a couple of subjects, without trying for it. Interestingly, the dialogues would progress the same. A casual remark about a potential problem followed by a "I didn't think of this" look. These occurrences surprised me, generating more questions about the process, converting main question to "How I did this?"

The question prompted me to observe myself more intently and closely.

This observation lead me to two revelations about the process. i.e. How practice and experience works, and how adding time to these changes things.

The first revelation was how practice and experience work in tandem. It's always said that "practice makes perfect", and to perfect ourselves, we practice. Practice is irreplaceable and invaluable all in itself. It sharpens the skill and the mind, reducing the mental cost of the skill while increasing the quality of the result. However, it's the experience you gain adds the subtle and impeccable parts to your knowledge and how you handle things. The edge cases, possible problems, the smell of how things are going can't be obtained without experience. A textbook, a mentor, or a course may give you a direction or a simple sense, but sharpening that is up to you and it's only possible with the experience you get over time. Because, even if you practice deliberately, you sharpen what you know. What you don't know can't be practiced.

However, experience does not make you better at that instant. I t needs time. Time is required to let the experience sink in, get introduced into your practice, and get considered during the next real-world scenario requires that knowledge. When you take that step, and get experience on top of the existing one, because you need to go step by step, the knowledge will start to accumulate.

This accumulation is slow and not recognizable, but when you need to tap into that knowledge, it'll show itself.

Until next time,

Be kind.

Eight years of organizing tech meetups

Notes on software development — Mon, 04 Sep 2023 03:00:00 GMT

This is a collection of random personal experiences. So if you don't want to read everything, feel free to skip to the end for takeaways.

I write because I'd like to see more high-quality meetups. And maybe my little bit of experience will help someone out.

2015: Philadelphia

I first tried to organize a meetup in Philly in 2015. I was contracting at the time and I figured a meetup might be a good way to source contracts or just meet interesting people. I created the "Philadelphia Software in Business" (or some other similarly vaguely named) group on Meetup.com.

I didn't have any network; the first companies I worked for were not in Philly. But Meetup.com got me a few tens of people joining the group.

My first challenge was finding a place to meet. I didn't know what I was doing so I looked at restaurants, bars, and cafes for dedicated event space. Needless to say, renting space was expensive on its own. And there was always an additional required minimum dollar spent per attendee.

I ultimately found a place near the Schuylkill River. Maybe it was a community event space. Maybe I paid for it. I can't remember.

The first and only time I hosted an event for the group, I got a surprising number of people for such a vague topic. There were maybe 6 of us. I was the youngest by far (I was 20), they were middle age. Excel users and one visionary type.

There was no real point to the meetup and I didn't continue doing it.

2016 - 2017: Linode

While I was at Linode, I organized "hack nights". I didn't ask for anyone's approval before starting it. I just said I'd be ordering pizza for anyone interested in staying after work to hack on Linode-related projects. I was willing to pay for the pizza, in part because I didn't want to risk being shut down by asking. But caker paid for it each time.

I was nervous because people would show up and ask for pizza and not want to hack. It was company-provided under the aspiration of doing Linode-related work. Maybe I mentioned this or not. I can't remember. I'm pretty sure they got their pizza.

Aside from myself, developers at Linode didn't really attend. The folks who attended were support staff or folks from the technical writing team who wanted more experience coding.

I ran this for maybe 3 to 5 Wednesdays before not continuing. It was pretty fun! But staying after work for a few hours each Wednesday lost its charm.

Book Club

Another time at Linode I started a book club. I was very torn about attempting to make the book club open to anyone in the area or just to Linode employees.

I knew I'd probably get more people to attend if I made it public. But I wasn't sure if Linode would be cool with having external folks in the office. Before they moved to the Old City office, visitors weren't really a thing.

So I made it private to Linode. And I started with the most obvious book for your average developer: Practical Common Lisp.

I am pretty sure I learned one big trick by this time though. When I announced I'd be starting the book club I said something like this:

Hey folks! I'm thinking of starting a book club. A book I have in mind to start with is Practical Common Lisp. If I get at least one other person to join in then I'll move forward!

I ended up getting two folks: one developer and one support staff member. We held the book club for 30 minutes once a week, covering one chapter each week. I was the only one who read anything I think, but the other two guys faithfully showed up for discussion.

I didn't ask for permission to do this either. And this time we met during company time. I think it was 2-2:30PM.

It was fun. We finished the book. But Practical Common Lisp probably wasn't a good choice. And I don't think I started a second book.

2017 - 2020: False starts

I moved to NYC and joined a small startup (~20 employees). Linode was 100+ employees.

We were in a WeWork so I considered starting a book club that was public to the WeWork. I had learned by then the law of numbers: I probably wouldn't get anyone from my company to join.

I considered putting up posters around the WeWork to advertise. But in the end, I didn't end up going through with anything.

I did present at a few meetups in NYC during this time. But I didn't organize anything.

And then the pandemic hit and everything disappeared.

2021 - 2022: Virtual

In 2021 I started contracting again, thinking about starting a company. I wanted a community to be at the center.

So I started a Discord focused on software internals.

I had a bit more of a network at this point so I posted about the Discord on Twitter and got 100 likes or something and slowly started gaining folks in the Discord.

I knew it was going to do better if I was pretty active in it so I made sure to post interesting blog posts at some regular interval. About compilers or databases or something.

The Discord didn't turn out to help me out much in the starting-a-company front. Or I didn't use it effectively for that.

I wanted more of an independent Discord of cool people who like to learn about systems internals. And that's what I got.

This turned out to be ok though because I stopped working on that company and the Discord is still around and I still get to hang out with cool people.

This Discord is still around and hit 1,700 members recently. Among other things, it has developers from many different database companies in it these days. They hang out and help out the noobs like me learn about database internals.

I culled inactive members recently, so today the total is around 1,100.

Hacker Nights

During the pandemic I became frustrated that all the good meetups disappeared so I decided to start an online one that would be somewhat tied to the Discord and be about software internals.

I would find 2 or 3 people to present for 10-20 minutes each on anything to do with software internals. We'd meet once a month at 8PM NY time I think.

To get speakers I'd mostly DM people who I saw do interesting things on Twitter or Hacker News. I was lucky to have Philip O'Toole (author of rqlite), Simon Eskildsen (author of the Napkin Math blog), Rasmus Andersson, and many other excellent folks speak.

You can find videos of these talks on YouTube.

The events were organized on Meetup.com. The group grew quickly and I'd have about 100 people RSVP to each event. 10-20 normally showed up.

I'd post a Zoom link on Meetup.com. Sometimes Meetup.com crashed right as the meetup started, so no one could get a link. That was fun.

On two different nights I had Zoom bombers show up and play crazy music or impersonate other members of the call and act weirdly (Zoom lets you change your name after you've joined the call).

I learned a little bit about how to administrate a Zoom meeting.

I ran Hacker Nights for 5 months. It was tiring to find speakers, tiring to deal with Zoom bombers. It was thankless and I wasn't really enjoying it.

I was proud though that I was offering a channel for developers to learn about software internals of compilers, databases, etc. And it was great to meet many interesting speakers and attendees.

2023: Designing Data Intensive Applications

A month ago I put out a call on Twitter for folks in NYC interested in reading through the book Designing Data Intensive Applications.

I'd read the book before and while it was challenging, I knew it was immensely useful to any developer who works with data or an API.

By this time I'd learned my second trick: not asking for public responses.

I said something like:

Hey folks! I'm thinking of starting a book club meeting in Midtown NYC reading through Designing Data Intensive Applications. DM me if you'd be interested! If I get 2 other interested folks this will be on!

I got maybe 40 DMs and 20 of them were based in NYC. Attendence thus would have been higher if I made the book club virtual. But virtual events take about as much effort as in-person events and somehow feel less rewarding. So I went through with the NYC group.

I'm sure I could have gotten some company to provide us space, but this would just mean more negotation for me and tedium for everyone involved (bring your ID to be checked in, make sure you're registered, etc.).

The group would meet every 2 weeks and cover 2 chapters at a time. We'd meet for 30 minutes. To avoid needing to find a place to meet, we'd meet in public at Bryant Park. (There turns out to be plenty of available seating on Fridays at 9AM in Bryant Park. When it rains we meet online.)

I wanted to keep the overhead minimal and the timeline slightly aggressive. We'd be through the book in only 3 months. No crazy commitment.

We've meet twice now and are 25% done the book. Attendance has been around 7 to 9 people each time so far, or a little less than 50%.

They're almost all software developers, with one manager I think, who work for a variety of large and small tech companies.

I'm loving it so far. And if it continues to go well, I'll probably continue running in-person book clubs.

But it would only meet a few months a year, giving me a few month breaks from running it.

Takeaways: The meh

Organizing any event takes effort. Meetups are especially hard because you need to find a place to run the meetup, you probably want to provide food, and you need to find speakers.

Often you can find a single place to host the meetup, but you have to constantly search for new speakers. Even one of the greatest meetups in NYC, Papers We Love, seems to be struggling to find speakers.

The CMU Database Group and the Distributed Systems Reading Group seem to have the right idea though. They only run sessions part of the year, and they plan out all sessions in advance (including speakers).

However, they are both virtual. And I'm not so interested in running virtual events anymore.

Takeaways: The good

For one, meetups are an awesome way to meet random people and expand your network.

Two, they're educational. Even beyond the content you are meeting about, there's the discussion alongside it you wouldn't get by yourself. And you, as organizer, get to pick the topic.

These work out great for me. I love to meet people, and I love to learn.

Tricks

Starting something new is embarassing because you're putting yourself out there. Maybe no one in your network shares your interests (to the degree or in the direction you do).

My tricks are:

Most importantly: keep things low key! Don't stress people out. Before learning and networking, the point of meetups is (or should be) fun.
Saying you are "thinking about X" is a lightweight way to gauge interest. As compared to just saying you're "starting X", which gives you less room to back out if there turns out not to be interest.
Asking people to DM you with interest is less embarrassing than asking for people to respond in public. Not everyone would want to respond in public. If there's interest in private, you can share the interest in public later on. But if you only ask for responses in public and there are no responses, that can feel embarassing.
Indicating success criteria can help people understand how big you're thinking of. I'm normally fine with doing something as small as only two other people, so I say that. It's kind of like how Kickstarters work with minimum funding levels.

These ideas apply to corporate planning too. I think about them when I'm sharing some new idea in company Slack as much as when I share on Twitter.

A note on attendance rates: 10-20% actual attendance versus RSVP seems normal. If you get a higher percentage of people actually attending versus RSVP-ing you're doing pretty well!

Finding sponsors

One final idea is about paying for space or paying for food. Companies with space and money for food are often willing to partner with folks willing to do the work to run an event.

Running your own event in a company's space is advertising for them. They get to be associated with cool tech. It's a chance for them to pitch their open positions.

Obviously this happens often when you start a meetup hosted by your own company. But you can also find other companies to host space.

The kind of people to find to make this happen are senior developers or engineering managers, often on Twitter and sometimes on LinkedIn.

I haven't done this myself yet because I'm not ready to commit to running a meetup. But I see it happen. And it's the approach I'd take if I were to run a real meetup again.

Though now that I've got some time off there are a few talks I'd like to do myself.

Wrote a post on my experience organizing tech meetups of various stripes over the years. And a few things I've learned.

"meetups" taken pretty broadly to include online communities, book clubs, and actual speaker events.https://t.co/xnd0LTneup pic.twitter.com/w1oEaSNDHb
— Phil Eaton (@eatonphil) September 4, 2023

CLI tip 21: inplace file editing with GNU awk

learnbyexample — Sat, 02 Sep 2023 03:00:00 GMT

You can use the -i option with GNU awk to load libraries. The inplace library comes by default with the GNU awk installation. Thus, you can use -i inplace to modify the original input itself. Make sure to test that the code is working as intended before using this option.

$ cat table.txt
brown bread mat cake 42
blue cake mug shirt -7
yellow banana window shoes 3.14

# retain only the first and third fields
$ awk -i inplace '{print $1, $3}' table.txt
$ cat table.txt
brown mat
blue mug
yellow window

You can provide a backup extension by setting the inplace::suffix special variable. For example, if the input file is ip.txt and inplace::suffix='.orig' is used, the backup file will be named as ip.txt.orig.

$ cat marks.txt
  Name    Physics  Maths
 Moe  76  82
Raj  56  64

$ awk -i inplace -v inplace::suffix='.bkp' -v OFS=, '{$1=$1} 1' marks.txt
$ cat marks.txt
Name,Physics,Maths
Moe,76,82
Raj,56,64

# original file is preserved in 'marks.txt.bkp'
$ cat marks.txt.bkp
  Name    Physics  Maths
 Moe  76  82
Raj  56  64

Earlier versions of GNU awk used INPLACE_SUFFIX variable instead of inplace::suffix. Also, you can use inplace::enable variable to dynamically control whether files should be inplaced or not. See gawk manual: Enabling In-Place File Editing for more details.

See this unix.stackexchange thread for details about security implications of using the -i option and workarounds.

Video demo:

See my CLI text processing with GNU awk ebook if you are interested in learning about the GNU awk command in more detail.

2023–09–02: Linux repository changes

megi's PinePhone Development Log — Sat, 02 Sep 2023 01:00:00 GMT

On Grind - from Investor to Creator to Founder

swyx's site RSS Feed — Thu, 31 Aug 2023 09:21:01 GMT

Sarah asks a provoking question that has been on my mind a lot as I transition from part time creator to founder:

Absurd Success

Weblog on marginalia.nu — Wed, 30 Aug 2023 03:00:00 GMT

So… I’ve had the most unreal week of coding. Zero exaggeration, I’ve halved the RAM requirements of the search engine, removed the need to take the system offline during an upgrade, removed hard limits on how many documents can be indexed, and quadrupled soft limits on how many keywords can be in the corpus. It’s been a long term goal to keep it possible to run and operate the system on low-powered hardware, and so far improvements have been made, to the point where my 32 Gb RAM developer machine feels spacey rather than cramped, but this set of changes takes it several notches further.

Zürich's new 74 bus fast charging station

Stefan Schüller — Tue, 29 Aug 2023 19:00:16 GMT

VBZs new 74 Electric Bus charging setup.

Vim tip 31: mark frequently used locations

learnbyexample — Tue, 29 Aug 2023 03:00:00 GMT

You can save frequently visited locations using marks for quicker navigation to those positions in the file. You can also pair marks with motion commands for tasks like copying, deleting, etc.

ma mark location in the file using the alphabet a
- you can use any of the 26 alphabets
- use lowercase alphabets to work within the current file
- use uppercase alphabets to work from any file
- :marks will show a list of the existing marks
`a move to the exact location marked by a
'a move to the first non-blank character of the line marked by a
'A move to the first non-blank character of the line marked by A (this will work for any file where the mark was set)
d`a delete from the current character to the character marked by a
- marks can be paired with any command that accept motions like d, y, >, etc

Motion commands that take you across lines (for example, 10G) will automatically save the location you jumped from in the default ` mark. You can move back to that exact location using `` or the first non-blank character using '`. Note that the arrow and word motions aren't considered for the default mark even if they move across lines.

See :h mark-motions for more ways to use marks.

Video demo:

DataEngBytes 2023

Ben E. C. Boyter — Sat, 26 Aug 2023 03:00:00 GMT

I was recently selected to present at DataEngBytes 2023 after submitting an idea for a talk based on my blog post Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100.

I think it went well, other than not having the slides able to present for the first 10 minutes. After 5 mins or so I decided to just plough on since it was less of a visual talk anyway. I did have a few people come to me afterwards and thank me for it which was nice.

Better be first 99% of the time than second 100% of the time

Bytepawn - Marton Trencseni — Fri, 25 Aug 2023 01:00:00 GMT

A review of the Donald MacKenzie's book Trading at the Speed of Light, which gives an excellent history and inside-peek of the world of High Frequency Trading, or HFT.

Quickly make any LÖVE app programmable from within the app

Kartik Agaram — Tue, 22 Aug 2023 21:51:16 GMT

It's a very common workflow. Type out a LÖVE app. Try running it. Get an error, go back to the source code.

How can we do this from within the LÖVE app? So there's nothing to install?

This is a story about a hundred lines of code that do it. I'm probably not the first to discover the trick, but I hadn't seen it before and it feels a bit magical.

Sleeping at Night

Weblog on marginalia.nu — Tue, 22 Aug 2023 03:00:00 GMT

I’ve started going on a long walk each morning immediately as I wake up, and it’s had the unexpected side-effect of fixing my broken circadian rhythm. For decades, as long as I can remember, I’ve been what you might consider a serious night owl. Regardless of how long I slept or when I woke up, I would get nothing requiring any sort of thought done until sometime after lunch, and it wasn’t really until late at night that my brain really kicked into gear.

It’s not magic (part 1)

Letters To A New Developer — Mon, 21 Aug 2023 15:05:12 GMT

Dear new developer, I remember the first time I pulled up MapQuest in the 2000s. I looked for directions to a place I’d never been. A custom map popped up, which was cool. But even more impressive was the turn by turn directions. I printed them both out and followed the instructions. That felt like … Continue reading →

CLI tip 32: text processing between two files with GNU awk

learnbyexample — Mon, 21 Aug 2023 03:00:00 GMT

awk is handy to compare records and fields between two or more files. The key features used in the solution below:

For two files as input, NR==FNR will be true only when the first file is being processed
next will skip rest of the script and fetch the next record
a[$0] by itself is a valid statement. It will create an uninitialized element in array a with $0 as the key (assuming the key doesn't exist yet)
$0 in a checks if the given string ($0 here) exists as a key in the array a

$ cat colors_1.txt
teal
light blue
green
yellow
$ cat colors_2.txt
light blue
black
dark green
yellow

# common lines
$ awk 'NR==FNR{a[$0]; next} $0 in a' colors_1.txt colors_2.txt
light blue
yellow

# lines from colors_2.txt not present in colors_1.txt
$ awk 'NR==FNR{a[$0]; next} !($0 in a)' colors_1.txt colors_2.txt
black
dark green

Note that the NR==FNR logic will fail if the first file is empty, since NR wouldn't get a chance to increment. You can set a flag after the first file has been processed to avoid this issue. See this unix.stackexchange thread for more workarounds.
# no output
$ awk 'NR==FNR{a[$0]; next} !($0 in a)' /dev/null <(seq 2)

# gives the expected output
$ awk '!f{a[$0]; next} !($0 in a)' /dev/null f=1 <(seq 2)
1
2

Here's an example of comparing specific fields instead of whole lines. When you use a , separator between strings to construct the array key, the value of SUBSEP is inserted. This special variable has a default value of the non-printing character \034 which is usually not used as part of text files.

$ cat marks.txt
Dept    Name    Marks
ECE     Raj     53
ECE     Joel    72
EEE     Moi     68
CSE     Surya   81
EEE     Tia     59
ECE     Om      92
CSE     Amy     67

$ cat dept_name.txt
EEE Moi
CSE Amy
ECE Raj

$ awk 'NR==FNR{a[$1,$2]; next} ($1,$2) in a' dept_name.txt marks.txt
ECE     Raj     53
EEE     Moi     68
CSE     Amy     67

Video demo:

See also my CLI text processing with GNU awk ebook.

Cyberpunk 2077 review

ブラック — Sun, 20 Aug 2023 03:00:00 GMT

My favourite game of the 2020s (so far).

I've poured about 400 hours towards Cyberpunk 2077 across all the platforms but I'm still finding it quite hard to write a review of the game. Cyberpunk 2077 has famously been kind of a mess, so why have I dedicated so much of my time towards it? I'm not sure if I know the answer.

I think one of the reasons why I like Cyberpunk 2077 is just the world. Night City is dense and grandiose stage that is essentially one of the main characters of the game with its overwhelming presence. There might be some issues when it comes to AI to make it appear like the city was full of real people going about their real lives akin to recent Grand Theft Auto games, it is still a sublime place to be in. I at least do not remember having as much fun exploring Los Santos in GTA V as I did when exploring the various alleys and roofs of Kabuki. Night City is called The City of Dreams and its design truly embodies it.

What didn't feel that strong was traversing the world though. You can fast travel between points you have already discovered, but I used it sparingly because I actually wanted to spend time in Night City. Sprinting around the city was actually a really valid option for moving short distances and I probably used it more than you'd think. However, for longer distances, you really need some kind of a motorised device. And the car physics in Cyberpunk 2077 are not the greatest. I've definitely encountered worse driving games but this isn't Grand Theft Cyberpunk.

However, I did discover the ultimate way to travel in Night City: the Yaiba Kusanagi CT-3X. While cars have a very funky handling model, zipping around on a motorcycle is actually super fun and the Kusanagi is the best of them. And in addition to being fun to drive, you can use the Kusanagi to go basically anywhere and cut through the Night City congestion. Really the only downside of the Kusanagi is that it's so much fun that it overshadows the rest of the vehicles. Kind of a shame, since the game is full of fun and cool vehicle designs. One of the cars even talks to you!

While exploring that world, it also becomes pretty evident that Cyberpunk 2077 is a part of a larger franchise. There's a shocking amount of lore stuffed into the game. So much so that I imagine that only few players are actually going to go through all of it. But even if you're not going to be engaging with all of it, it still has the effect of making Cyberpunk 2077 feel like it's an actual world and not just shallow set dressing. I've definitely taken an interest in the Cyberpunk franchise after playing through 2077. If only I had some friends to play the TTRPG with.

Another strength of the game is the writing. I feel like a lot of games these days have quite unnatural writing to them. My guess would be that they're trying to emulate the kind of "witty" writing that is present in current Hollywood blockbusters. Thankfully Cyberpunk 2077 doesn't feel like that even though they hired a Hollywood actor to appear in it. The dialogue seems a bit weird or otherworldly when compared to standard English though but I feel like that might be a desired effect. And I'm not referring to the specific 2077 slang words that are used. It would be kinda boring if people living in an alternative future spoke English exactly like people in our regular 2020 did.

I quite enjoyed the cast of characters too. You cannot have a huge influence over the kind of a person V is but I liked her (female V forever). She also meshes well with your main sidekick that's with you for most of the game and I enjoyed the interactions between the two. There's also quite a large array of interesting secondary characters and even tertiary characters have some characterisation. The only characters that I hated, I hated because they were awful people that could only be fixed with a bullet to the skull, not because they were badly written characters. I guess the only issue I had with the characters was that the voice actors were often voicing several characters, so you might encounter a shopkeeper that sounds exactly like your girlfriend.

The main story of the game is good although not necessarily amazing. I also feel like it's relatively short for a single-player roleplaying game. I think a main story only playthrough will only eat about 20 hours of your time. Perhaps it was kept short to give players a chance to finish it since relatively few people ever actually complete games. Personally I think I could've done with additional fleshing of the main story. I'm not sure if the Cyberpunk 2077 story is the most interesting story possible in the Cyberpunk universe, since there is quite a lot of material available, but it was interesting enough to make me want more of it.

However, to complement the relative brevity of the main story, the game has quite a lot of optional content. There's questlines for the main secondary characters, various side missions, gigs by the local fixers, cyberpsychos to subjugate and requests to "keep the peace" (neutralise targets) from the police. And the optional content is much higher quality than many other open world games where the additional content is mainly just collectibles. Some of my favourite missions in the game are in fact optional. Even the police requests, which are the lowest effort side content in the game, still have some amount of setting and variation to them. None of optional content felt like a chore or a lazy collectible like in some other games. In fact, I might take a detour while traveling from one story mission to another to just clear out a couple of NCPD murder requests.

The talent system allows quite personalised ways of playing as V. For my first playthrough, I went for a pretty generic jack of all trades gunner build that didn't really strike me in any way. After that I went for a more stealthy builds, which were much more to my liking. And on my fourth playthrough, I opted for a katana-wielding Sandevistan ninja build, which I really enjoyed. Never really tried out a serious gunner build nor have I tried a netrunner build, as those didn't seem as interesting. Maybe when I decide that it's time to complete Cyberpunk 2077 for a fifth time that I might experiment with those.

The way that you specialise your V alters the core gameplay quite a bit, and since I've never tried out all of the possible ways to play the game, I can't comment on all of them. But I do generally like the gunplay, the melee combat and the stealth mechanics. Even though I never committed properly to be a gunner, I always found it to be satisfying enough. And I quite liked the combination of stealth and hacking abilities to be able to circumvent cameras and people. And the Sandevistan katana build was just visceral catharsis with the way you could overpower your opponents.

I do find that the talent system does contribute to quite an uneven difficulty curve though. I always found the game to start off way harder than it ended, since at the start, you have so much worse gear, so much worse cybernetics and lacked all talents. When playing on the hard difficulty, it was quite easy to get yourself killed if you made any mistakes. And after you'd obtained your equipment and talents towards the end of the game, it was so much harder to die to a point of being a rare event. This is not exactly a rare effect with games built around equipment and talents, but for the sequel, I hope that CDPR flattens out the difficulty curve a bit.

Beyond just trying out different sorts of builds, Cyberpunk 2077 offers pretty good replayability in general. I should know, I've played it through four times already! I'm pretty sure I've discovered something new every time I've played through the game, and not just because they've kept patching the game ever since it launched in 2020. There's always been something new to discover in the environments, some kind of a way to complete a quest that I hadn't tried before, or something completely else. And judging by the stuff I've encountered on YouTube and various websites, there's still stuff that I've never seen before. There's usually a couple of ways to complete any given quest, so you might opt to go for a guns blazing approach on your first try and for a stealthy approach on the second. The various builds also unlock new ways to approach some of the quests.

The game is gorgeous too. I think this is still the apex of gaming graphics after two years of its release, and the system requirements really match that. Cranking the settings to their highest (ignoring path tracing because it's just a technology preview), I was able to get around 80 FPS at 1440p with an RTX 4090. Thankfully you can tweak the settings for a more optimised result on weaker systems, but this is really a game that benefits from beefy hardware. The beautiful graphics really just further enhance the beauty of Night City. Soundtrack of the game is also rather good, and I find myself listening to many of the tracks outside of the game too. Never Fade Away in particular is currently topping the charts on my Last.fm profile. I also loved the voice acting for female V, and really none of the cast really disappointed me with their performance.

The game isn't perfect though, as was quite widely covered when the game launched. Far from it. There were so many different glitches and bugs when the game came out that it was hard to go ten minutes without encountering something. Thankfully there's been over two and a half years of effort put towards the game to polish it, but it's still not quite enough. Luckily the effort has managed to fix the worst game-breaking issues that were present in the beginning. The kind of bugs that would prevent progress or otherwise really make you miserable. But you will definitely still encounter people continuing to walk around after being decapitated or see a car plow through a wall. Bugs that break your immersion but not your progression. I imagine that Cyberpunk 2077 will never truly be jank-free.

It also feels like they've managed to fix worst of the crashing too, since I think I only encountered a single crash on my new gaming PC. With my previous rig, I remember encountering several crashes in a single session. Not sure if that was an issue with the PC/installation I had at the time or if it was just the older versions. Hopefully newcomers to the game can enjoy it with only an occasional crash – since really no AAA game on the PC is truly 100% stable.

I'm not really sure how relevant my review is going to be though. I'm writing this review just a few weeks before the new expansion, Phantom Liberty, comes out and overhauls the game. I haven't actually looked at a detailed list on all of the changes because I want to experience them with fresh eyes once they come out, but the little I've seen online suggests that they're actually doing quite a lot of changes. Hopefully the changes are just for the better and don't ruin anything that I currently love in the game. But since Cyberpunk 2077 has been quite the enjoyable experience for me, I do have a level of trust in CD Projekt Red to not muck it up at this point.

Cyberpunk 2077 is kind of a rough product even after its polish, but one that has always captivated me and kept me playing, even though I still have a hard time explaining why. It's a fun game with a good story and writing, and it'd be hard to not recommend such a game.

2023–08–20: QuartzPro 64

megi's PinePhone Development Log — Sun, 20 Aug 2023 01:00:00 GMT

Bomb Rush Cyberfunk review

ブラック — Sat, 19 Aug 2023 03:00:00 GMT

Underwhelming spiritual successor to Jet Set Radio.

While I may not be the biggest Jet Set Radio fan out there, I still find it to be a pretty fun action/platform game. And when I saw that someone was going to create a Jet Set Radio style game for the modern age, I did get quite excited. However, having dipped my toes into the game now, I feel rather disappointed.

The movement in Bomb Rush Cyberfunk feels noticeably worse than what Jet Set Radio offered. And I think that's a two-pronged issue, one part being controls and one part being level design. On the control side, I had trouble getting my character to control as I wish. Feels like the characters can cut far too sharp in a direction when coming from a jump, and like it happens in an unpredictable fashion. And the control issues are not helped by the fact that the controls are more intricate than Jet Set Radio's, with many more movement possibilities. I could've taken less if the counterbalance was that it felt more intuitive to move.

And the early levels feel very sparse in comparison to Jet Set Radio with large open areas without any parts for grinding or jumping. In Jet Set Radio, if you saw a ramp going upwards, you could most likely grind against it. But in Bomb Rush Cyberfunk, it feels like I was constantly trying to grind against level elements that just didn't support it. And having to second-guess what parts of the level work for movement and which don't really kills the flow. The levels really didn't feel like they were designed to allow for stylishly combining various tricks to your heart's contents as long as you had the skills. Maybe I'd learn the movement controls in due time but I'm not sure if the level design would ever allow the same kind of free-flowing movement that Jet Set Radio had.

The approach to levels seems very different between the two. Jet Set Radio would throw you into a level with a map of all tag locations and big honking arrows pointing to the tags inside the game. Bomb Rush Cyberfunk meanwhile throws you into a map without any tag locations, no arrows and tells you to go spray enough of them to unlock the next objective. Jet Set Radio was more of a time attack whereas Bomb Rush Cyberfunk seems to be more of an exploration. And I don't think exploration really meshes well with the kind of movement-based gameplay associated with Jet Set Radio.

After I managed to clear the first couple of reputation gates by spraying tags, the gameplay really slowed down as I was now just trying to hunt down any tags I'd missed instead of doing stylish grinds and jumps towards my next objective. The only way to tell apart done and undone tags is with this small shimmering effect on top of the sprays, which is much harder to spot in movement. The minimap would only show you the location of the reputation gate, which would angrily send you back because you hadn't done enough spraying yet. You can't even use the spray button prompt to figure out if you've done a tag, since for some reason you can spray the same tag multiple times. As far as I could tell, there was no benefit in doing so, but it was for some reason still possible.

The spraying itself also makes little sense. Jet Set Radio showed you exactly what kind of joystick movements you should do, and if you screwed up the sequence of moves, you'd lose valuable spray paint and would need to go collect more. Here, you seem to be able to just wiggle the joystick around for about a second and you're done. No sense of accomplishment for a job well done, no punishment for spraying with your eyes closed. At least I was never able to screw up a single tag by just wiggling my joystick around without paying attention. You might as well just have the spraying happen automatically.

Bomb Rush Cyberfunk also differs from its spiritual forefather by having a combat system. A combat system that doesn't really feel fun. You might be thrown into a stage with a handful of mob cops and one flying cop, who will fly around and then deliver you heaven's punishment from above. And the game doesn't really feel like giving you any helping hand to figuring out where this flying cop is relative to your position, making it much harder to dodge the incoming attacks. And trying to take down the flying cop is also painful due to lacking any kind of a lock-on system, which seems like something a game throwing many opponents at you at once should have. And even when you are successfully taking down your opponents, it doesn't feel satisfying. More like just some flailing around. If the combat was completely optional, maybe in the style of Mirror's Edge, I might give it a pass, but as far as I can tell, some combat sections are not optional.

The combat sections also seem to suffer from the same lack of direction as the spraying part of the game. I got thrown into a combat section, and I didn't really know what was my objective. Should I just outrun and escape the cops? Should I beat up every single cop coming my way? Or should I just focus on the flying cop that was in the cutscene? Once again, the only direction that the game seemed to be telling me was "go earn more reputation". Guidance that was not really helpful when I was being hunted by the jetpack police.

The game does very much pull off the Jet Set Radio style. It does very much look like it was just another Jet Set Radio game released on the Dreamcast, for better and for worse. But while the graphics do look old, it is a very functional design. That was probably to be expected considering how well Jet Set Radio has aged. I might not like the levels from a gameplay perspective, I did enjoy them from an aesthetics perspective. Just maybe if they didn't feel so empty. Soundtrack seemed fine too, although it doesn't rival Jet Set Radio in the sound department. Then again, really no game can. Jet Set Radio's soundtrack was a just succession of bangers, whereas Bomb Rush Cyberfunk has good songs and then it has less good songs.

I really would've liked to love this game, but I don't think I can as it is. For the 40-euro price tag, it feels pretty underwhelming and undercooked. Some of the issues I had with the game could probably be improved, but I'm not sure if the core gameplay could be improved to a degree that made this feel like worth its price tag. Perhaps once it hits the bargain bin, I might give it another chance. But right now Bomb Rush Cyberfunk is unable to meet my expectations and I opted to refund my copy of the game instead. Not sure if I've ever opted to do that on Steam before. A shame.

Portal with RTX review

ブラック — Fri, 18 Aug 2023 03:00:00 GMT

This thing is horrendous.

I'd heard that Portal with RTX didn't run great, but I thought that when people said that, it meant that their RTX 3060 couldn't run the game. Thankfully, I have an RTX 4090, so I can just brute force through any bad optimisations. What I didn't expect was that the game runs like dogshit on whatever piece of technology you throw at it.

I launched the game and alt-tabbed before I even loaded in the first stage. The game crashed.

Relaunched the game and cranked the graphical settings. The game crashed.

Started the game for the third time, played through enough to get the half-powered portal gun and a bit more, and then alt-tabbed. The game crashed.

This was all within 10 minutes of playing. I cannot remember if I have ever had a game behave this badly on me before.

My 1% FPS during the start of the game was also around ten. With a 7800X3D and a 4090. I could run maxed out Cyberpunk 2077 without DLSS smoother than this game. If this kind of hardware cannot run this game without massive stuttering, there is surely no gaming PC on this planet that can run this game smoothly.

It's also not decisively better graphically, even though that's the only reason for this thing to exist. The environment lighting does look more natural than the 2007 version, but it still very much looks like a PlayStation 3 era game. They did change some assets but not everything, so you are still seeing 2007 quality, just with improved lighting. I guess the improvements are nice enough that were this a working product that I would do my Portal replay in this version. But considering how busted this version is, it's so not worth it to even try.

Thankfully this is just a free tech demo. Had they charged me even 50 cents for this experience, I would've asked for a refund. The only valid use for this game is to generate screenshots and video clips for Nvidia marketing material. Otherwise it's just an embarrassing and damaging showcase for their RTX Remix.

250k downloads of Latent Space Pod

swyx's site RSS Feed — Thu, 17 Aug 2023 10:24:44 GMT

I last talked about my Latent Space adventures in April and last December. Even as a well regarded developer-part-time-creator, the Latent Space Newsletter + Pod has done much better than I usually do. Here are the stats as of today:

DEATHLOOP review

ブラック — Wed, 16 Aug 2023 03:00:00 GMT

A first-person game about trying to break out of a time loop.

When DEATHLOOP was first shown to public audiences, I figured that it'd be a shooter game focused around on a time loop. And while it does involve both of those things, it's not really that. It's more of a repeating exploration game, where you go around in circles, collecting clues so that you can solve the temporal puzzle and bust your way out of the titular loop.

After laying down the premise and teaching you your basic abilities in a linear sequence of gameplay, the game becomes very non-linear by giving you some clues and telling you go to figure out how to solve things. It feels a bit daunting at first, and I waffled around exploring unnecessarily a bit before I understood that I could really just ignore everything but the visionary clues and I'd be golden. Once you understand to just follow those, the game doesn't feel like you're just supposed to explore around and take a stab in the dark. Trying to just blindly explore your way out of it is just gonna bury you in information. I still have absolutely no idea what I was supposed to do with some of the non-plot discoveries I made.

The story isn't really much anything. There's some exposition about what happened previously to begin the loop, but it's not really important and the game doesn't put much effort into it. Well, that or I just couldn't find the scraps of paper and tape recorders that explain the backstory. The few real revelations of the story also took place right before the end for me, since I guess I just played the non-linear story in that order. The meat of the game's story is really just the clues themselves, and not much anything. Granted, the overall puzzle is an enjoyable mystery but it's not a deep and involved plot. The ending is kinda weird and abrupt too.

The cast of characters is mainly you and the villains that you need to kill, with one villain being particularly chatty for plot reasons. The less chatty villains are basically all just massive unlikable narcissists for you to hate. This is probably for the best, since if you actually plan on finishing the game, you'll probably have a hard time counting how many times you've killed them. I don't want to claim that they're completely shallow or undeveloped, but quite a lot of them never really registered to me as actual people. More like objectives with legs. And the only non-villain character left over is your player character, who is actually a pretty distinct character and all around fun fella with a penchant for talking to himself.

For exploration, the main activity of the game, there's four different maps and four different times of day when you can go into them. And while the maps themselves don't really change, the intel you can gather out of them changes with the time of day. I don't think a fresh player can beat this game without coming and going to every single map so much that they should just install a revolving door. And I can't say that I'm really a fan of this. It just kinda feels like lazy level design to make us pour through the same four, relatively small maps with a fine comb. The maps really just aren't that big so you'll definitely get intimate with them in 5–10 hours of gameplay.

The gunplay in DEATHLOOP doesn't feel amazing. It's fine but not really anything more of that. I think that's because the game actually wants you to stealth through everything. The game does actually suggest to you that you're free to go through the maps as you please, including the possibility of going guns blazing, but the way things blow up in your face if you use firearms suggests that they would really prefer you to sneak around instead. And with the gunplay being what it is, it never really felt like "time for action, baby" as much as it felt like "oh, I fucked up, can I get out of this with these guns and the couple of bullets left in my pocket?" Also felt kinda weird that the magic powers you can get in the game seem very combat-focused when that's really second fiddle to stealth.

Much like the gunplay, stealth gameplay is "fine". It's not the best nor the worst stealth experience I've had in a game. My chance to go by undetected did get significantly better when I got access to the silenced SMG and I actually had the possibility of deleting enemies without groping their butt. Actually, the silenced SMG is so good that a more pro-stealth player might consider it overpowered, since it allows for insta-kills via precise headshots, and if you fuck up, silent rapid fire onto an enemy until it stops moving. Supposedly there were other silenced weapons to also find, but I didn't really need anything more than the SMG.

What actually felt a lot more satisfying than sneaking around was taking the high ground with a sniper rifle and taking potshots at the enemies from the distance. Position yourself far away enough and they won't be able to hear the gunshots, leaving them unable to hunt you down and helpless to protect their tender little heads. It also feels more in character as a big burly black ex-military dude to snipe down your enemies than try to sneak your way around them.

Occasionally the chatty villain also decides to invade the map you're in. The first time this happened, I flew into panic since I had no idea where I was getting shot at and had a hard time fighting her off. Pretty quickly after that I started to figure out her patterns and it became more of an annoyance than a massive threat. I even managed to sneak around her with invisibility, turn off my invisibility for a bit to lure her in, then go invisible again to knife her in the back for some sweet insta-kill revenge. You also need to go do some busywork to unlock your escape route after this invasion, so that really just doubled the annoyance of it all. Not really a big fan of this particular feature.

There's also a multiplayer aspect where you can play as the invader, but I never touched it. I want to play a single-player campaign that I can put on pause if I needed to go pee. Thankfully opting out of the multiplayer aspect is very much possible, although I believe it did automatically start off in online mode.

Possibly my favourite part of the game is just the presentation. DEATHLOOP is dripping with this cartoony 1960s vibe and it's fantastic. The sound design is also great. The stealth music is very good at building a tense atmosphere, and the shift in music when you go from stealth to combat, or from combat to stealth, also works well to indicate your status, so it also works from a gameplay perspective. Colt's voice acting is also good, although I'm not so sure about the rest of the voice acting. It at least didn't feel as strong. But the overall presentation of the game is still to its benefit.

Technically the game seems sound. I get pretty low system utilisation when running max settings at 1440p with a 240 FPS cap, so I assume that this game runs pretty fine on low-end hardware as well. Not a lot of bugs either, although I did manage to softlock several times in the inventory screen, and a couple of times an enemy body ragdolled wildly into space after I've killed them. I also read somewhere that the loading times were tied to your FPS. Can't really confirm that claim, but if that's true, then it sounds kinda bad. Thankfully my loading times were pretty decent with a 7800X3D, an RTX 4090 and a fast SSD. Overall, I don't have a lot of things to gripe about the technical side – a lot of games are put out in far worse state.

DEATHLOOP is kind of a mixed bag for me. The story didn't feel that special and the gameplay didn't feel that special either. The time loop mechanic, even though it was an interesting concept, didn't really make the game feel that special either. But despite none of the aspects of the game really grabbing me, I still felt compelled to pour about 23 hours into the game to solve the puzzle and reach the ending, so I guess DEATHLOOP was doing something right. I'd give the game a cautious recommendation if you enjoy exploration and stealth, but I wouldn't say that it's a must play title.

Python tip 32: positive lookarounds

learnbyexample — Wed, 16 Aug 2023 03:00:00 GMT

Lookarounds help to create custom anchors and add conditions within a regex definition. These assertions are also known as zero-width patterns because they add restrictions similar to anchors and are not part of the matched portions. Negative lookarounds were discussed in this post. The syntax for positive lookarounds is shown below:

(?=pat) positive lookahead assertion
(?<=pat) positive lookbehind assertion

Here are some examples:

>>> s = '42 apple-5, fig3; x-83, y-20: f12'

# extract digits only if it is followed by ,
# note that end of string doesn't qualify as this is a positive assertion
>>> re.findall(r'\d+(?=,)', s)
['5', '83']

# extract digits only if it is preceded by - and followed by ; or :
>>> re.findall(r'(?<=-)\d+(?=[:;])', s)
['20']

# replace 'par' as long as 'part' occurs as a whole word later in the line
>>> re.sub(r'par(?=.*\bpart\b)', '[\g<0>]', 'par spare part party')
'[par] s[par]e part party'

With lookbehind assertion (both positive and negative), the pattern used for the assertion cannot imply matching variable length of text. Fixed length quantifier is allowed. Different length alternations are not allowed, even if the individual alternations are of fixed length.

>>> s = 'pore42 tar3 dare7 care5'

# not allowed
>>> re.findall(r'(?<=tar|dare)\d+', s)
re.error: look-behind requires fixed-width pattern

# workaround for r'(?<!tar|dare)\d+'
>>> re.findall(r'(?<!tar)(?<!dare)\d+', s)
['42', '5']

# workaround for r'(?<=tar|dare)\d+'
>>> re.findall(r'(?:(?<=tar)|(?<=dare))\d+', s)
['3', '7']

The third-party regex module (https://pypi.org/project/regex/) offers advanced features like variable-length lookbehinds, subexpression calls, etc.

Video demo:

See also my 100 Page Python Intro and Understanding Python re(gex)? ebooks.

Working Backwards Book Summary

Svedic.org — Tue, 15 Aug 2023 14:39:15 GMT

Why read the Working Backwards book? Amazon is not very popular right now, with many media criticizing its treatment of workers and its cut-throat culture. Still, love them or hate them, there are some undeniable achievements. Amazon is a tech … Continue reading →

Thinking about functional programming

Notes on software development — Tue, 15 Aug 2023 03:00:00 GMT

Someone on Discord asked about how to learn functional programming.

The question and my initial tweet on the subject prompted an interesting discussion with Shriram Krishnamurthi and other folks.

So here's a slightly more thought out exploration.

And just for backstory sake: I spent a few years a while ago programming in Standard ML and I wrote a chunk of a Scheme implementation. I'm not an expert, but I have a bit of background.

Hey, this is a free opinion.

Concepts from functional programming

When people talk about functional programming, I think of a few key choices you can make while programming:

Immutability by default
(Tail) recursion by default
First-class functions (and the suite of tools that go along with it. e.g. map, reduce/fold)

And if you have experience as a programmer, you either get the basic gist of these tenets or you can easily read about the basics.

That is, while most programmers I've met understood the basics, most programmers I've met were not particularly comfortable or fluent expressing programs with these ideas.

For myself, the only way I got comfortable expressing code with these ideas was lots of practice (as I mentioned above). And yet, even after I did a bunch of programming in Standard ML and Scheme, I really didn't see a particular benefit to practicing in a language other than one with which I wa already generally comfortable.

You have to learn a lot of other random things when you pick up Scheme or Standard ML that aren't just: practice immutability by default, recursion, and first-class functions.

So I think it's kind of misguided when person A asks how to learn functional programming and person B responds that they should learn Haskell or OCaml or whatever. I see this happen pretty often online.

Beyond any "language for functional programming" as a recommendation in general, Haskell is a particularly egregious suggestion to make in my opinion because not only are you trying to practice functional programming tenets but you're also dealing with a complex type system and lazy evaluation.

Instead, practice immutability, recursion, map/reduce in whatever language you like.

Programming languages

If you want to study programming languages, that's awesome. However, functional programming doesn't really have any direct connection to studying programming languages.

Languages are all over the place. Scheme, Standard ML, and Haskell are worlds apart, even within the functional programming family.

And modern languages have mostly adapted the aspects of functional programming that used to be unique 20 years ago.

Moreover, there are many other worthwhile families of languages to learn about:

Imperative/C-like (ok, you probably already know these)
Stack-based (JVM, x86 assembly sort of, Forth)
Array-oriented (APL, J)
Declarative (CSS, SQL, TLA+, Prolog)
Data (HTML, JSON, YAML)
Proof assistants (Isabelle/HOL, Coq)

The list isn't exhaustive, and the variations within families can be massive. But the point is that functional programming doesn't mean crazy programming languages or crazy programming ideas. Functional programming is a subset of crazy programming languages and crazy programming ideas.

If you want to learn about crazy programming languages and crazy programming ideas, you should! Go for it!

Introduction to Computer Science

SICP is famous as the (former) introductory textbook for computer science at MIT, and for its use of Scheme and the Metacircular Evaluator.

I don't have any experience teaching beginners how to program so I don't have thoughts on if this made sense. That's for folks like Shriram to think about.

However, I'm a half-decent programmer and I can't make it through this book. If you liked the book or want to read it, that's great! But I don't recommend it to anyone.

And many introductory Computer Science textbooks just don't make much sense to give to experienced programmers. For an experienced programmer, they can be quite slow!

Most of the folks I see asking about how to learn functional programming are experienced programmers.

Do whatever you feel like doing

I don't mean to overanalyze things, or get you overanalyzing things. If you want to learn functional programming by writing Haskell, that's awesome, you should go for it.

Wanting to do something is basically the best motivation there is.

The only reason I write this sort of post is so that folks who think that using Haskell or Standard ML or Scheme or reading SICP is the only way to learn functional programming see those ideas aren't necessarily true.

Write a Scheme!

Finally, for folks with time and motivation wanting to seriously work out their functional programming muscles, writing a Scheme implementation with a decent chunk of the standard library can be an immensely enjoyable project.

You'll learn a lot about languages and compilers and algorithms and data structures. It's leetcode with meaning.

Wrote a short post on this idea about different things to think about when talking about learning functional programming

1. Core concepts (immutability, first-class functions, recursion)

2. Exploring programming languages

3. Teaching CS to studentshttps://t.co/k4LzvnHbNs
— Phil Eaton (@eatonphil) August 16, 2023

Free Interior Design With Sweet Home 3D and SketchUp

Svedic.org — Sat, 12 Aug 2023 19:39:26 GMT

If you need a do-it-yourself interior design for your flat, house, basement, or dictator mansion—with completely free software—you are at the right place. Below is my experience with choosing software, finding 3D furniture models, finding textures, rendering, and furnishing the … Continue reading →

Life in 1080p

Weblog on marginalia.nu — Sat, 12 Aug 2023 03:00:00 GMT

I recently got a smaller computer screen. It’s actually not that small, it’s 27", but the resolution is modest compared to what is available. And in short, it’s fantastic. It’s not an expensive screen, it’s not a fancy screen; but it’s comparatively a small screen. For a few years I was using a 34" ultra-wide monitor, which has been causing me nothing but grief. It’s sort of crept up on me that so many small annoyances in my computer-use all originated from using this screen.

Message Queues, State Machines, Actors, UI

Weblog on marginalia.nu — Sat, 12 Aug 2023 03:00:00 GMT

This is a bit of an what I’ve been working on style of post. It’s also a bit of a complement for the release notes of the upcoming release which should be dropping in a week or so. There’s some spit and polish still missing from these things, but if I don’t write about them now too much will have been ejected from the cache to make a well written post about it.

Releasing my tools under the MIT License was probably a mistake

Donat Studios — Fri, 11 Aug 2023 08:15:30 GMT

For basically as long as I can remember, I have been a staunch defender of permissive licenses over the more copyleft variety. I want you to use things I've written. On top of that I don't believe it's my place to force you to then open source things you have written that expand upon my source code. Hence my extensive use of the MIT License.

The MIT License is very simple. It basically says you can do whatever you want with the code, but you have to include the license and copyright when redistributing the code itself. It does not however require you to open source any modifications you make to the code, nor provide attribution to the non-code products of the work.

With my libraries, this has served me very well. It's great, people use and contribute fixes back, and often donate back to me. I love it. I want you to use my code in your proprietary software. I open sourced it so you could do that. I specifically build open source libraries because I solved a problem I thought other people might face and wanted to save them time. Lifting the sea for all boats and all that. I have zero problem continuing to use the MIT License on my libraries.

I also open sourced the tools I host on my website, including but not limited to Pixel Circle / Oval Generator and Batch RewriteRule Generator. The MIT License seemed like the natural choice because I've been using it for everything I've released since the early 2000's.

I'd considered these as just part of my website far more so than something I thought people would reuse. My intentions open sourcing these tools, unlike my libraries, was to promote the community helping me to improve them - and they certainly have done so. I have gotten so much great feedback and contributions to these tools, and I am very happy with that. Massive shout out to Tyson453 for his insanely awesome work improving the speed of the Pixel Circle / Oval Generator.

Against my well meaning intentions however, websites re-hosting my tools have been popping up like weeds. In some cases, they are even beating me in search results for my own tools. With noted exception, they don't credit me as the author or provide any sort of link back. Many of them have made minor or major modifications to the tools, and next to none provide the source to those modifications. A number of them even have the gall to post links advertising them in the comments of my own tools.

Most irksome of all, in a fair number of cases they sit centrally on pages covered in ads and SEO keywords. My tools are being associated with a genuinely bad user experience.

This is all perfectly legal and allowed under the MIT License. That's on me. It just is not in the spirit of what I intended. I didn't have the foresight to see this coming. I didn't think people so lacked in the spirit of open source. I wanted to promote community contributions, not to have them monetized by other people who don't even provide the source to their modifications. I wanted to grow the tools as a community, not have closed source forks of them overtake my own open source versions.

There's not a lot I can do. I could try to contact the owners of these sites and ask them to provide attribution, but there's no obligation for them to do so. I could try to contact the owners of these sites and ask them to provide the source to their modifications, but again there is no obligation for them to do so. I really wish to see the changes shared alike, and the vain part of me would like attribution as the original author. At the end of the day though I chose the MIT License and it is what it is. It simply does not require any of that.

On top of everything else, until very recently and for completely unclear reasons, my website Donat Studios was completely blocked from Bing search results. This in turn meant I didn't show up in DuckDuckGo or Yahoo results either. This meant that for a time the only results when searching these sites for my tools were these re-hosted versions.

I actually managed to reach a human at Bing and get this resolved, but it took a lot of work and I'm still not sure why it happened in the first place.

I am considering relicensing my tools under some sort of Attribution-ShareAlike license similar to the BY-SA the content on this site is licensed under. It would not apply to current and past versions of the tools as that code is licensed MIT and you can't un-ring that bell. It would however apply to future versions. This would still promote community contributions, but would also require that any modifications be shared alike meaning that the modified source would have to be provided and I would have to be credited as the original author.

At the very least I will definitely think twice about the license I choose for future tools. I don't want to be a jerk about it, but I also don't want to see them monetized by others without even a link back to my website.

Update on minimalism

bayindirh — Fri, 11 Aug 2023 03:00:00 GMT

As a part of my minimalism journey, I'm trying to reduce my digital footprint. This means I'm using less services, less devices, less applications. In other words, trying to do more, with less.

This concerted effort has created a set of dynamics and side-effects which I didn't anticipate. While it feels strange, the resulting outcome is nice.

Before embarking on this journey, I used to have a desktop computer with lots of disk capacity, a nice sound system, and a nice monitor. My laptop computer was a mobile replacement of it, carrying the most essential things, plus the whole development toolchain I used to have on my desktop computer. As a result, I used my desktop computer almost all the time, while using the laptop while on the go.

Now, I have a single laptop computer with a much smaller disk. My office provides me a desktop and another laptop. This means, I am using my laptop much more, and need to adjust to its capabilities and limitations.

Talking of adjusting, it turned out that my desktop computer was doing much more than I anticipated. Since it had more than enough storage and I used it almost everyday, I never felt the need for the cloud. Secondly, I unknowingly created and perfected workflows around that machine. Removing it created a strange void and disorganization on its wake. I'm still working on adapting these workflows to what I have. This part is not going fast.

Another part of this adjusting came from the applications I used to use on the desktop computer, especially for my photography and music collections. Over the years, I have chosen a set of applications and used them to their limits, so that I know their behavior like my name. Losing these applications would be bad, however the most important ones are working on macOS. This saved me a ton of time both in the short and long time.

The last part of this adjustment process is handling and organizing massive amount of files coming from decades of computing history. These files are currently being moved to a cloud provider. After they are consolidated there, they will be organized slowly. Some of these files will be live on offline storage (i.e. portable external hard drives), some will stay in the cloud, and some will be deleted. Remaining ones will be backed up regardless of their location.

It becomes clearer that I have deployed myself an "invisible infrastructure" over the years, without even noticing, as I change the way I accomplish things. Currently, almost none of this infrastructure is in place, but a better, smaller yet more efficient version is being built step by step. Let's see how it goes and where it arrives.

Until next time,

Be kind.

How to allow popups in dynamically created webviews in Electron.js

swyx's site RSS Feed — Wed, 09 Aug 2023 10:41:27 GMT

My smol menubar project utilizes Electron's special webview tag to dynamically generate a list of sub browser windows for chat. For the last couple months I've had an issue with the SSO popups in this, namely that they just don't work at all, presumably because Electron blocks popups for you by default.

Vim tip 30: some general Vim settings

learnbyexample — Tue, 08 Aug 2023 03:00:00 GMT

Here are some general Vim settings that you can put in the vimrc file to customize your editor. See :h options.txt for complete reference.

set history=200 increase default history from 50 to 200
- there are separate history lists for : commands, search patterns, etc
set nobackup disable backup files
set noswapfile disable swap files
colorscheme murphy a dark theme
- you can use :colorscheme followed by a space and then press Tab or Ctrl+d to get a list of the available color schemes
set showcmd show partial Normal mode command on Command-line and character/line/block-selection for Visual mode
set wildmode=longest,list,full use Bash-like tab completion
- first tab will complete as much as possible
- second tab will provide a list
- third and subsequent tabs will cycle through the completion options

:h 'history' will give you the documentation for the given option (note the use of single quotes).

You can use these settings from the Command-line mode as well, but will be active for the current Vim session only. Settings specified in the vimrc file will be loaded automatically at startup.

Video demo:

The Benefits of Sticking Around

Letters To A New Developer — Mon, 07 Aug 2023 17:33:00 GMT

Dear new developer, Over the last twenty years, I’ve had two work modes. I’ve either been at a company for a number of years (3, 8, 2, and 3) or been a contractor/consultant working for clients for weeks or months. Both modes have upsides; I wrote before about the benefits of a small consulting company. … Continue reading →

The Legacy of Bram Moolenaar

Jan van den Berg — Mon, 07 Aug 2023 09:29:16 GMT

This weekend we learned that Bram Moolenaar had passed away at the age of 62. And this news affected me more than I expected. Like so many: I did not know Bram personally. But I’ve been using a tool made by Bram for more than half my life — at least weekly, sometimes daily. That […]

Walk code repositories respecting .gitignore files in Go

Ben E. C. Boyter — Mon, 07 Aug 2023 03:00:00 GMT

Since I maintain a few projects that deal with source code one of the things I needed badly was a way to parse and understand gits .gitignore and .ignore files in order to get as much accuracy as possible.

I had previously tried using code I lifted from The Platinum Searcher albeit with some fixes to avoid crashes. Annoyingly however it never implemented glob’s correctly. I tried searching around for another implementation but none appeared to work as expected.

Control vs Acceptance

The Healthy Dev — Sat, 05 Aug 2023 11:35:24 GMT

An axis for judging your actions and reactions

Colo vs Cloud

Blogity blog blog. Journal — Fri, 04 Aug 2023 04:33:00 GMT

what is the cheapest real actual server you can buy, new, with a support contract from a reputable vendor like HP/Dell/etc? (Assume we’re running both the db and web-server on one server.) And what’s the cost of sitting it in a reputable colo? And then multiply that by, let’s say two, to get redundancy in a us-west and us-east region. Ideally, we’d want more regions to serve the entire world, but let’s just go with two for now.

Let’s say $10k for a server w/ support contract, x2, plus $400/mo for colo from Hurricane Electric (https://he.net/colocation.html) in us-west. It’s a promo deal, and only in us-west, but let’s just go with that price. $400*60 for 5 years (standard lifetime of computer equipment) = $24k. So $34k per server * 2 servers / 60 months ~= $1,130/month amortized. Thankfully, we’re not in cloud, so that’s the cost whether it’s 1 request/second, or 1,000 requests/second.

A Raspberry Pi or old laptop in your basement with one consumer-grade ISP, and not factoring in the cost of electricity does not count here. Those are obviously going to be massively cheaper, but they’re not remotely suitable for enterprise.

Meanwhile, Lambda gives you 1M requests and 3.2 million compute-seconds a month for free. This works out to be 0.4 rps (requests/second) or 22 requests / minute if spread out evenly. Which isn’t, like, a lot, but you haven’t paid a single cent to AWS for this yet. Let’s say you want to get to 40 rps (to keep the math easy) or 100M requests over the month. According to the AWS calculator (https://calculator.aws/#/addService/Lambda), this’ll run you $120/month, or $120*24 ~= $3k for 2 years.

But that doesn’t count hosting for your static files with CloudFront, or does it count the RDS instance backing it, and we’d use AWS Cognito for user-auth since we’ve accepted AWS vendor lock-in. (Which isn’t great, but that’s the state of the industry.)

CloudFront: Free Tier 1TB, 10M requests/month. 10 TB/month will run you $1k/month https://calculator.aws/#/addService/CloudFront

RDS: This will vary greatly. A teeny tiny db.m6g.large instance w/ 2 CPUs and 8 GiB of RAM is going to run you $250/month, but a big one fat juicy db.r5d.24xlarge will run you closer to $20k/month. https://calculator.aws/#/addService/RDSMySQL

Cognito: With 1,000 active users, this’ll be $50/month; 10,000 users is $500/mo, and 100,000 users is $5k/month.

Between Lambda, CloudFront, RDS, and Cognito you could easily blow past that $1,130/month estimate for a proper colo’d pair of servers, if your service gets at all popular. But if you stick within the free tier for Lambda and Cloudfront, then you’re only looking at RDS and Cognito costs, which can vary greatly. Anywhere from $500/month to $26k/month, or more.

Except to get the same service in a colo as a $26k/month AWS bill, you’d need to spend way more and have much more than the single server in a rack that I started with. Thankfully, with something like Equinix Smart Hands, no one has to drive to the colo to futz with a server when it develops a bad hard drive.

Graph it out for your particular use case which one is better, but cloud infrastructure has the benefit of opex vs capex, as well as the opportunity cost of time. AWS Amplify will let you stand up a whole platform hacking on a Saturday before Dell can even quote you a price on a server or Equinix can answer the phone.

I also did reject a raspberry pi or old laptop server early on, but that’s not to be underestimated. If I’ve already got some server running at home for my house, running a service on that, fronted by Cloudflare can take you quite far.

It depends on your use case. The big thing is that AWS/Cloud charges for egress, so if you’re making a media distribution compnay, aka the Netflix model, you’re better off building your own, but the cloud is actuallly cost competitive.

What I’m up to - August 2023

Philip I. Thomas — Thu, 03 Aug 2023 18:17:07 GMT

I'm Philip, and this is my monthly newsletter about what I'm up to, which I send in place of social media.

✨ What I was up to in July

Booklet is a product I've been building to help communities communicate in a more structured and readable way. This month I finally had some people besides me start to use it. It's been fun to get the product into the wild, and now I'm iterating rapidly based on their feedback ahead of a more public release.

I finally got to reading the Noma Guide to Fermentation after visiting there last summer, and I've been doing a lot of experiments with lacto-fermentation. (See some photos below). It's been fun to go to the farmer's market every week and to cook with what's local and seasonal.

I recently celebrated one full year of working full-time on Contraption Company. Building new ways to help people work online has been a fun, rewarding, and creatively fulfilling journey. I'm proud of our commitment to keeping the business intentionally small and independent, and I'm excited about what the next year holds.

🤔 Things to share

Articles: The Carlota Perez Framework for thinking about technology cycles. How to Do Great Work.
Books: Designing Coffee looks at interior design of cafes around the world. A Work in Progress: A Journal documents a fascinating creative process. Algebra of Happiness has some good advice.
Podcast: Robin Sloan on Social Media After Twitter.
Apps: Collective automates and optimizes the back office for one-person LLCs.
Trend: Location sharing between friends - with granularity of either "City" or "Exact."
Coffees I'm drinking: Prolog Aquiares Anaerobic Natural was yummy. Prodigal Finca La Soledad Geisha was clean. Process Nestor Lasso Ombligon was funky.

📫 What I'm up to in August

Traveling back to Mexico City, where I'm excited to try some new restaurants and cafes.
Welcoming more Booklet users.
Launching a couple of my own communities with Booklet.

📍 Where I'll be

Aug 3-7: Mexico City 🇲🇽

(Let me know if we overlap)

📸 Photos

A fermentation in three photos . . .

Beginning - a haul from the farmer's market

Middle - setting up the fermentation, which for berries takes about a week

End - fermented cherry gastrique with roasted chicken and purple potatoes

--

Thanks for reading! If we haven't chatted in a while, I'd love to hear from you. Send an email to say hi, share your thoughts, or just catch up.

How to Blow Up a Category - Netlify's New Era and The JAMstack Endgame

swyx's site RSS Feed — Wed, 02 Aug 2023 08:12:50 GMT

note - this is a hasty written braindump of feelings as emotions as I don't have the time to polish this essay up to my usual standards, but still wanted to capture this important moment and end in my life. pardon any poorly phrased and organized thoughts. The JSJam guys did a roundup of reactions here and Ricmac did one on The New Stack.

2023–08–02: Update on the Pinephone Pro U-Boot build

megi's PinePhone Development Log — Wed, 02 Aug 2023 01:00:00 GMT

Muv-Luv Alternative: Total Eclipse review

ブラック — Tue, 01 Aug 2023 03:00:00 GMT

The Total Eclipse anime was actually my first experience with the entire Muv-Luv universe – which I imagine was the case for a lot of current anime-watching Muv-Luv fans. I actually don't remember anything about it other than the first two episodes (which are not even covered in the VN). And while it has been eleven years since the anime premiered, having a forgettable story is not exactly a good sign for the VN. Then again, this meant that I came to the visual novel version of Total Eclipse with pretty fresh eyes.

The visual novel story is entirely kinetic and feels a bit meandering, with the focus switching between political intrigue, life or death action, or high school romantic comedy, with a good chunk of the female cast competing who can have the most obnoxious teenage crush on the protagonist. Political intrigue across the major powers of the Alternative timeline does take the driver's seat for most of the story though. Even the action scenes are drenched in geopolitics.

Ostensibly the main objective of the story is bringing the XFJ Program to completion but it doesn't really feel like it. Rather, it feels like the story is about whatever is the current crisis that is currently unfolding and which may or may not relate to the XFJ program. I'm also not sold on the ending as it felt like the villain had better motives than the protagonist. Of course, the ending converges to work out somehow at the end because the writers didn't want to write a story where the protagonist does dumb things and meets a dumb end.

There's also some questions raised during the story that were not really ever answered. Perhaps the idea was to leave room to further explore some of the factions in the future. But since Total Eclipse as a franchise is already over 15 years old, I doubt we'll ever get some kind of a satisfactory explanation to some stuff. At the moment it just feels like they're introducing unknown characters of unknown factions with unknown motives and you just have to deal with it.

In general though, I felt at least engaged with the story overall and even actually captivated during certain parts of the story. And despite the ending involving some questionable developments, it does thankfully hit some good emotional tones and is ultimately something I felt somewhat satisfied with.

To anyone interested in Total Eclipse, I'd say that the requirement for actually enjoying the experience is a general interest in the Muv-Luv Alternative universe – the story takes place just before the events of Muv-Luv Alternative and only expands on a tiny part of the Alternative story. Standing on its own two feet, it's not really a strong story, making the decision of making Total Eclipse the first animated Muv-Luv story questionable. But as an additional story in the Muv-Luv Alternative universe for Muv-Luv fans, it's decent enough to recommend.

I did notice that the character arc for a lot of the characters followed the same pattern. They introduce a character as supremely unlikeable, throw them into the ringer for a bit and then have them calm down and become somewhere between tolerable and likeable. I noticed that they did it for the protagonist, Takamura, Cryska, the Chinese girl, and the other Japanese-American test pilot. Thankfully the protagonist and Takamura calm down relatively fast, otherwise the experience might've been quite a slog. I think they might've even tried doing it with Tarisa but she's far too adorable to be unlikeable. But despite the repeating character development pattern, the characters of Total Eclipse are ultimately likeable and interesting.

Presentation in Total Eclipse is the best that Muv-Luv has yet to offer. I did initially feel a bit weirded out by the colour palette, as it's very muted in comparison to many other entries in the series. However, after getting used to it, I can say that the art is very good. Both the sprites and CGs are high quality, and there's a good amount of CGs to boot with 195 entries in the album. Only very occasionally I saw something that looked weird or low quality.

They're also utilising a good amount of animation for the sprites in order to wring out a bit more life out of static images. And for the first time ever, Total Eclipse also utilises a decent amount of actual 3D animated cutscenes during the story. While they managed to give a good kinetic feeling to the mech action in Alternative, it's completely different to see TSF manoeuvre when it's given a fully animated short cutscene. The only downside of the 3D cutscenes is that there's not that many of them and it feels like they mostly happen during the first half. Hopefully the level of animation shown in Total Eclipse is the new standard for new Muv-Luv stories, as I am quite a fan of it.

On the sound side, there's not that much innovation. Yuuya is kind of a weird protagonist in that he's mostly not voiced but still is given a voice from time to time. At first I thought they only voiced him during flashbacks and when the story is not from his point-of-view, but later in the story, his lines are sometimes voiced and sometimes not. Not really sure the rationale for this, as THE DAY AFTER had Hibiki fully voiced. At least it's not super distracting. Otherwise the cast and voices are solid even if I can't really find anything special to say about it. What I did particularly enjoy however was the usage of insert songs during some bigger action events. I'm a huge sucker for insert songs for action scenes.

No issues with general ergonomics and display. A much better reading experience than with the old rUGP titles. I read through Total Eclipse exclusively on my Steam Deck and the performance was what I've come to expect from titles using the Ages Mark 2 engine, which was mostly solid. Like with the other games, my Steam Deck exhibited the same kind of audio crackling after resuming the game from sleep. And even though this is a Steam Deck verified game, there's still no official controller layout.

I finally however decide to test the Pause Games plugin and it seemed to make the whole Steam Deck audio experience a lot more solid. After installing the plugin and enabling the "pause before suspend" feature, it felt like I could resume Total Eclipse from sleep a lot better than previously without having massive and continuous audio crackling. Highly recommend this tweak to anyone experiencing audio crackling in Ages Mark 2 games. Wish I'd tried it back when I read through THE DAY AFTER series.

All in all, Total Eclipse is a decent 25-hour delve into the geopolitical and weapon development politics of the Muv-Luv Alternative world, with occasional hot-blooded action to balance things out. Just don't make the same mistake âge did and try to make it your introduction to the Muv-Luv world.

CLI tip 31: concatenate files column wise

learnbyexample — Tue, 01 Aug 2023 03:00:00 GMT

The paste command is typically used to merge two or more files column wise. By default, paste adds a tab character between corresponding lines of input files.

$ cat colors_1.txt
Blue
Brown
Orange
Purple
$ cat colors_2.txt
Black
Blue
Green
Orange

$ paste colors_1.txt colors_2.txt
Blue    Black
Brown   Blue
Orange  Green
Purple  Orange

You can use the -d option to change the delimiter between the columns. The separator is added even if the data has been exhausted for some of the input files.

$ paste -d'|' <(seq 3) <(seq 4 5) <(seq 6 8)
1|4|6
2|5|7
3||8

# note that the space between -d and empty string is necessary here
$ paste -d '' <(seq 3) <(seq 6 8)
16
27
38

# use newline separator to interleave file contents
$ paste -d'\n' <(seq 11 12) <(seq 101 102)
11
101
12
102

You can use empty files to get multicharacter separation between the columns. The pr command is better suited for this task.

$ paste -d' : ' <(seq 3) /dev/null /dev/null <(seq 4 6)
1 : 4
2 : 5
3 : 6

$ pr -mts' : ' <(seq 3) <(seq 4 6)
1 : 4
2 : 5
3 : 6

Video demo:

See paste command chapter from my Command line text processing with GNU Coreutils ebook for more details.

2023–07–31: My new U-Boot builds

megi's PinePhone Development Log — Mon, 31 Jul 2023 01:00:00 GMT

2023–07–31: rk2aw released along with new U-Boot builds

megi's PinePhone Development Log — Mon, 31 Jul 2023 01:00:00 GMT

Inside your head.

Blogity blog blog. Journal — Sun, 30 Jul 2023 10:00:00 GMT

People sometimes say a problem is “inside your head”, but the idiotic thing of it is, of course everyone lives “inside their head”. It’s where consciousness lies (unless you’re an ancient Egyptian and believe it comes from the heart, which is “just” the muscle that pumps blood around, and isn’t packed full of neurons). So it’s a stupid statement to begin with. I live, as do you, and everyone else, “inside my head”. I literally can’t leave it. 24/7, I’m in my head even when I’m sleeping, I can’t escape it. And neither can you. All of my perceptions, everything I see, touch, or smell is filtered through my brain which is in my head. What’s odious then, isn’t the statement itself, but the implication that I could just “shake it off” with just the right set of magical thinking. The problem then, is everybody who claims “it’s all in your head”, proclaims the problem solved, and walks off with their fingers in their ears, not listening to people suffering, not helping them “get out of their heads”, not even trying to help get to that place of magical thinking that would fix everything.

They say the solution is “in your head”. If it is, they’re a right shite bunch of bastards for keeping this magical thinking to themselves and letting the rest of us suffer. If instead we focused on listening to the people suffering, and have identified that it’s in their heads, then how do we get them out of it? How do we expand their consciousness to be “outside” their heads? How do we modify their brain so the head they’re in is different? We have all sorts of drugs for that, both legal and illegal, as well as a host of new therapies, some approved by the FDA, as in “has a scientifically proven effect”, using magnetic fields applied to the brain, eg TMS (Transcranial Magnetic Stimulation), to help people get their heads, which they live 24/7 in, to be different ones than the ones that are giving them so much trouble!

Thanks to doctors actually listening to patients (which is a revolutionary concept, I know. Not all doctors are super geniuses, but the ones who are able to listen to people who’s problems are “in their heads” are all smarter than those who dismiss people’s problems for being “in their heads”. Especially neurologists, who have dedicated their career to the study of what’s inside heads.), people are getting better, healing the brain, and once again living fulfilling lives, with more than 3 spoons a day of energy. At the worst of it, watching Netflix was too exhausting. Concentrating enough to write code was out of the question. After TMS, powerful psychiatric medication, a course of psilocybin and other off label courses to help me “get out of my head”, and a huge amount of talk therapy on top, I’d now consider myself cured of whatever lingering after effects of Covid and the pandemic I suffered. It took a lot of work, I still live “in my head” but I’m able to leave my bed enough day of the week, and even leave my house most of them as well.

Feel free to tell me my problems are in my head so long as you’re going to help me get out of it; help me pay for doctors, help me get to the doctor, help me pay for therapies and therapists and actually get to them, help me navigate insurance and get on disability while I need it, help me with housework while it’s too much for me to manage.

“It’s in your head” is the beginning of helping to figure out how to solve the problem, not the solution it and of itself.

via HN

What are the types of happiness?

jodavaho.io — Sun, 30 Jul 2023 03:00:00 GMT

As I sit in a hotel lobby at 4:00 AM drinking awful coffee (story for another time), I’m reflecting on:

If happiness from a boon or gain or accomplishment is just the loss of the pain of not having that gain, then maybe we can just short circuit that process and stop wanting the thing in the first place

Of course, this is a well-studied idea¹. But what gets me is: Is that what happiness is? Can I really be happy by removing desire (which, presumptively, prevents happiness by showing me a future I don’t have). Certainly, some class of happiness can be “solved” by this anti-FOMO strategy. But it seems so zero-sum.

This idea assumes that I believe I will be happier with that thing in my life, and therefore I am less happy for not having it. I don’t think this is the case at all. I want to look forward to things. I want to feel closer every day to a goal, for example. This “happiness of progress” seems to be totally different.

example: Kids

Nobody could ever have told me I’d be happy as a father. It was something I wanted, but my happiness was not realized when the kids were born. They themselves bring the happiness with their delightful little goblin selves every day. This sort of “lost the sadness of not having kids” does not adequately explain the joy of being a father²:

But what kind of happiness is that? It seems to be the happiness of relationship, a totally different kind.

Types of happiness

Where this train of thought leads me to is: What are the types of happiness, and what are the viable strategies for preserving or achieving them. I know there are types of love but without looking for it, I’m motivated to try to try and derive this list myself first.³

Here’s a rough list:

The loss of persistent discomfort
The end of a journey
The joy of laziness, or The contentment of having no obligations
Discovery and creation
Personal connection
The joy of being

And of course, there are many that are derived or from combinations of these:

the joy of collaboration is roughly “personal connection” plus “creation”

I think this is an incomplete list, but it allows one (me only?) to not feel too guilty about desire, while at the same time feeling happy in the moment for various reasons. This feels less absolutist and more adult.

. See any old philosophy. Or, I suggest read Happiness Hypothesis for a decent summary. ↩︎
Side note: The stoics would say I should relish this feeling, but face the prospect of losing it every day, to keep the feeling fresh and sweet. Yes, thinking of the kids being sully parent-hating teenagers that love speeding and hate school … well that does make these days all the sweeter. ↩︎
Where even might we find this list? Perhaps an evolutionary biologist would say that “Status” is the most important thing, and all other derive from this. ↩︎

Your Code is not Yours

The Healthy Dev — Thu, 27 Jul 2023 23:38:44 GMT

But our tools make it easy to think it is

Things I hate about my Tesla

jodavaho.io — Thu, 27 Jul 2023 03:00:00 GMT

I own a 2022 Tesla Model Y. Or rather, my powerhouse wife does. Regardless, I’ve driven approximately 10,000 miles in one, and have formed a long list of opinions.

The Good

The performance
The cold / icy driving experience is fantastic. It’s the easiest to drive in slippery conditions that I’ve owned except my old toyota 4x4 SUV, but that’s a high bar.
The adaptive cruise control eliminates 90% of the annoyances from highway driving (though the industry has caught up)
The 360 cameras and 360 range sensors make driving in tight confines very easy.
The incident reports. This is the video replay of any person or vehicle approaching or touching your car. I’ve already seen it save an insurance report and aid in accident reporting.
The appearance
The superchargers are awesome. It’s absolutely never been a problem to visit one the 3 to 6 times per year we have to use them. We charge at home overnight, and have not set foot in a gas station except to pick up Diet Coke since we bought the Tesla. It seems crazy we don’t all do this, honestly. I can’t wait for gasoline to be a speciality product.

The Bad

Entering the car

Someday there will be a zombie movie where a minor character is running from zombies, turns the corner to see their pristine Tesla, thank god, and will sadly be eaten while they fumble with the app, key cards, and door locking mechanisms (see next section).

When approaching the car to drive:

10% of the time it does nothing. I stand confused as the dash reads “Cameras on”. Then I take out my keycard, wipe it over the window frame trying to find the magic, unlabelled “unlock” spot, and eventually enter.
30% of the time it unlocks just fine, I enter, sit down, and drive away - except my legs are cramped and some techno is blaring because it selected my wife’s seat and radio settings
10% of the time it unlocks just fine, I enter, sit down, and it won’t let me drive. It insists I place a keycard under the armrest to drive.
50% of the time it unlocks just fine, I enter, sit down, and drive away fine.

Obviously I created those percentages, but it seems about right.

Special case: Using the trunk

Using the trunk is a special case. The “right” way is to go through the above process to enter the car, but instead of driving, you use the tablet (see next section) to unlock the trunk.

But, idiot that I am, I assume it will auto-unlock, and I’ll be able to open the trunk by pressing the open button on the trunk itself. According to above, there’s a 90% chance it’ll unlock. But this is wrong because the Tesla doesn’t unlock when you approach from behind, only when you approach the cab it seems.

20% of the time, the car unlocks so I can open the trunk fine
40% of the time I have to go up to the cab to get the car to unlock, but the trunk won’t unlock. I have to open the driver’s side door to, I guess, fully unlock the trunk, at which time I can go back and open it.
40% of the time I just open the driver’s side door (see prev section!) and then open trunk using console.

Walk-away lock

There’s no easy way to temporarily disable walk-away lock. This means that when you’re loading kids, luggage, or anything, and it’ll take multiple trips, and you don’t want the car to lock, turn off climate control, and shutdown, you have to leave a door open. You don’t even have to walk away, just shutting the door while you’re not in the driver’s seat will shut the car down.

And you can enable climate control remotely, but as soon as you open then close the door (e.g., you put a kid in), the whole car will shut down regardless of if you walked away or not. Imagine the Tesla locking your kids in the car, and oops, you drop your phone.

The app

All of this is fine if I can open the app to do any of this. But the app is useless. No matter where I am or where the car is, there’s a 90% chance I cannot connect my app to the car. I suspect my wife disabled my app access at this point.

That fucking tablet.

OK here we go. This is the big one.

In the tesla the UX is awful. You can do exactly one thing at a time, and transitioning between things to do takes at least a click and swipe. Think about all the things you might want to do in a car:

Open the glove box
Turn on/off the windshield wipers
Change the radio station
Stream via spotify
Pair a phone
Open the trunk
Adjust the speakers
Enable air conditioning / heat
Lock the windows
Turn on/off seat warmers
Enable rear fans
Check the range to your destination
View the upcoming turns in navigation
Input a new destination for your navigation
anything else …

Yes, you need to use the tablet to do that. Nope, you cannot do any of those at the same time. Yes, you cannot adjust the winshield wipers without taking your eyes off the road and looking at the bottom corner of the tablet.

There are almost no buttons in the car - just the tablet. Each thing you want to do requires a full-screen interaction, meaning if the passenger wants to do any of this they block the entire in-car display with a fancy popup modal. And don’t get me started on how annoying it is when the FM tuner is left up and blocks my navigation instructions.

This is far and away my biggest complaint about the car. They were arrogant enough to just believe they could make a functional UI from a tablet bolted onto their dashboard. It is a major distraction to safe driving to do anything in that car.

The cold weather performance is not good.

While in general the range is good, the cold weather range is awful. We can drive about half as far as normal, and any statements about pre-conditioning the battery alleviating that are false. You can precondition to gain back some range, but still, a trip to our parents’ place in the country is doable in summer (if you dont run AC), but requires a stop at a supercharger in winter.

The WTF

When you are using cruise control you cannot disable windshield wipers.

I’m not kidding. Why? Because Sir Elon insisted that this car is made using cameras only, and the camera is behind the windshield, so to do adaptive speed control it needs to have a pristine windshield. Most other manufacturers use radar. Going forward, I expect more lidar.

The adaptive cruise control will slam on the brakes for no apparent reason.

About once every 50 miles, it will execute a pretty dangerous-feeling braking. Why? No idea, but most likely the cameras detected a phantom car or obstacle. It once got real nervous about a bridge that was paved a lighter color than the road we were driving on. It will also slow down if a car looks like it’s in front of it when we’re all turning on a highway. I find these things to be dangrous enough that I question the purchase regularly.

Why I am starting a hardcore tech company in my 50s

I Am Not a Robot — Wed, 26 Jul 2023 20:48:02 GMT

Or at least this is the story I tell myself about it

Using the SonicWall Connect Tunnel with Firefox on a Chromebook

Jan van den Berg — Wed, 26 Jul 2023 20:21:49 GMT

Yes, you read that correctly. Firefox on a Chromebook! Without tricks. Or at least, not many tricks. Why? When you want to use the SonicWall Connect Tunnel software (from the SMA 1000 Series) on your Chromebook the suggested SonicWall Mobile Connect app does not work properly. I don’t know why, but there is a solution. […]

Python tip 31: next() function

learnbyexample — Tue, 25 Jul 2023 03:00:00 GMT

The next() builtin function can be used on an iterator (but not iterables) to retrieve the next item. Once you have exhausted an iterator, trying to get another item will result in a StopIteration exception. Here's an example:

>>> names = (m for m in dir(tuple) if '__' not in m)

>>> next(names)
'count'
>>> next(names)
'index'
>>> next(names)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Here's a practical example to get a random item from a list without repetition:

>>> import random 

>>> names = ['Jo', 'Ravi', 'Joe', 'Raj', 'Jon']
>>> random.shuffle(names)

>>> random_name = iter(names)
>>> next(random_name)
'Jon'
>>> next(random_name)
'Ravi'

You can set a default value to be returned instead of the StopIteration exception. Here's an example:

>>> letters = iter('fig')

>>> next(letters, 'a')
'f'
>>> next(letters, 'a')
'i'
>>> next(letters, 'a')
'g'
>>> next(letters, 'a')
'a'
>>> next(letters, 'a')
'a'

Video demo:

See also my 100 Page Python Intro ebook.

Correctly configuring incoming SPF in Exim on Debian

Jan van den Berg — Tue, 25 Jul 2023 00:17:31 GMT

The Debian documentation is sparse on how to correctly configure incoming SPF checks in the Debian Exim package. It is sparse in the sense that it tells you what to install (spf-tools-perl) but it is not clear WHERE to put the very important macro. It only says: This is provided via the macro CHECK_RCPT_SPF, set it […]

How to deal with fatal: bad object HEAD in git

swyx's site RSS Feed — Mon, 24 Jul 2023 16:57:34 GMT

as a cheapo who uses Apple iCloud (the world's worst sync service) as a sync service for my Obsidian Second Brain, I have recently run into this issue a lot:

What We Mean When We Say “Software Engineer”

Letters To A New Developer — Mon, 24 Jul 2023 15:15:00 GMT

This is a guest post from Doug Durham. Enjoy. Dear new developer, It’s very easy to throw terms around in our (or any) industry. In our hurried culture, we latch onto words or phrases that may not fully encapsulate their original intent. I’m afraid that is what has happened to the label of “software engineer”. … Continue reading →

Introduction to Marketing Mix Modeling

Bytepawn - Marton Trencseni — Sun, 23 Jul 2023 01:00:00 GMT

I describe the concept of Marketing Mix Modeling using Google's LightweightMMM library.

Creation is our higher purpose

jodavaho.io — Fri, 21 Jul 2023 03:00:00 GMT

Building things is our higher purpose

Ergo, the most shameful thing is not not finish a creation

I feel deep in my bones that making things is the higher purpose of humanity. It’s our secret weapon against devolving into tribes. The simplest creations improve ones life, and that’s what makes us human: Having agency to improve our lot. The true strength of our gift is making for others. Making something desireable from something less desireable underpins every part of cities, is the foundation of medicine, and a basic requirement of our economy. It will make us an interstellar species, because we can make it so if we want. We can make anything so, if we really want it. The wanting is the hardest part.

credit: Midjourney

Everything made is in service of this divine. So, I’ve realized that the most shameful thing I do is not finish. I haven’t created until it’s made available for myself or others.

I’ve made a life decision: No more projects until my current top 3 are done. When one is done, I sub it out and can sub one in. My current top 3 are:

Highfleet Ship Optimizer: Which has taken so long I doubt anyone will be playing the game by the time I finish it. Should be done soon. Using SCIP, Python on AWS Lambda, and a frontend hosted for free on Vercel. Nice and cheap at a bout a buck a month.
Personal finance automation. Not quite automated trading, but at least real-time monitoring and some basic savings deposits. Gonna be a CLI integration with Plaid to fetch balances. Might be a nice open-source project. I’d love a nice 80s terminal aesthetic to it for funzies.
Gunplay-heavy, cyberpunk-like tabletop rpg system, to include better statistics from rolling, a more systematic skill tree, increased lethality over typical games, and better system for tracking influence, connections, and wealth. A friend and I are play-testing it.

The backlog is long, but I’ll feel better if I can finish a few here and there rather than letting them all wait around and catching the latest idea.

Note: I copied this from a letter I wrote to a friend. I hope he won’t mind.

Vim tip 29: greedy quantifiers

learnbyexample — Wed, 19 Jul 2023 03:00:00 GMT

Quantifiers can be applied to literal characters, dot metacharacter, groups, backreferences and character classes.

* match zero or more times
- abc* matches ab or abc or abccc or abcccccc but not bc
- Error.*valid matches Error: invalid input but not valid Error
- s/a.*b/X/ replaces table bottle bus with tXus since a.*b matches from the first a to the last b
\+ match one or more times
- abc\+ matches abc or abccc but not ab or bc
\? match zero or one times
- \= can also be used, helpful if you are searching backwards with the ? command
- abc\? matches ab or abc. This will match abccc or abcccccc as well, but only the abc portion
- s/abc\?/X/ replaces abcc with Xc
\{m,n} match m to n times (inclusive)
- ab\{1,4}c matches abc or abbc or xabbbcz but not ac or abbbbbc
\{m,} match at least m times
- ab\{3,}c matches xabbbcz or abbbbbc but not ac or abc or abbc
\{,n} match up to n times (including 0 times)
- ab\{,2}c matches abc or ac or abbc but not xabbbcz or abbbbbc
\{n} match exactly n times
- ab\{3}c matches xabbbcz but not abbc or abbbbbc

Greedy quantifiers will consume as much as possible, provided the overall pattern is also matched. That's how the Error.*valid example worked. If .* had consumed everything after Error, there wouldn't be any more characters to try to match valid. How the regexp engine handles matching varying amount of characters depends on the implementation details (backtracking, NFA, etc).

See :h pattern-overview for more details.

If you are familiar with other regular expression flavors like Perl, Python, etc, you'd be surprised by the use of \ in the above examples. If you use \v very magic modifier, the \ won't be needed.

Video demo:

Six months with my Steam Deck

ブラック — Sun, 16 Jul 2023 03:00:00 GMT

I've had my Steam Deck for six months now and I am surprised at just how transformative of a device it has been for me.

When the Steam Deck was first announced, I was very sceptical of it. Back in August 2021, I remarked in a conversation how I only saw the use case for the Steam Deck in emulators and visual novels. And in January 2022, I commented how I wasn't interested in buying one. To be fair, my reasoning for not buying one was pretty sound: I was almost never away from home where I keep my gaming PC, and I already had a Nintendo Switch and two PlayStation Vita. But in December 2022, I changed my mind: I thought it seemed neat, especially for visual novels. And about a month later, I picked up a second-hand 256 GB model for 400€.

Now six months later, I absolutely adore my little Steam Deck. It's a really cool gaming device. Unfortunately due to Steam breaking the hours per platform tracking that SteamDB had, I can no longer see how many hours I've played on Linux (Steam Deck being my only Linux gaming PC), but my current estimate is that I've spent over 120 hours playing on the Steam Deck. Around 20 hours per month is not nothing for me, especially since I'm rarely more than five kilometres away from home / my gaming PC.

UPDATE: Turns out that if you register your own Steam API key, you can fetch your own Linux-specific Steam playtime. My total Steam Deck (or rather Linux) playtime is currently 131 hours!

It's actually extremely easy to forget that the Steam Deck is a weird, specialised gaming PC that is running on top of modified Arch Linux even if almost all of my Steam library is just Windows games. You can see that a lot of work has gone into Proton to get it work as flawlessly as it does. It's of course not perfect, but it is downright sublime all things considered.

So far on my Deck, I've read through six Muv-Luv visual novels (with the seventh currently underway), played through Bastion and Jet Set Radio, tried out a bit of Forza Horizon 5, Grand Theft Auto V and San Andreas, and put many hours towards Grand Theft Auto IV, Tokyo Xanadu eX+ and Vampire Survivors. Not everything was a perfect fit on the Steam Deck, and not everything worked completely right, but I still feel extremely satisfied with what I've managed to get done with a Linux-based handheld gaming PC. Haven't managed to try emulation yet though since I've never been huge into emulation, but maybe I'll take a stab at it at some point as well. I've at least heard the emulation experience is quite good.

It's also quite easy to juggle a lot of games on the Steam Deck. Even though I had the 256 GB model, I immediately spruced it up with a microSD card. Flash storage has gotten so cheap that slapping an extra 512 GB of storage to my Steam Deck only cost me 38.10€. I actually had to check my receipts and discovered that's about the same that I paid for my 16 GB Vita memory card nine years ago. And now, thanks to my 685.7 GB of usable storage, I have 43 games installed on my Steam Deck with 115.7 GB left over. That's a lot of game library for such a small device.

I've also been pleasantly surprised at how manageable of a device the Steam Deck is. On paper, it seemed absolutely massive compared to my Nintendo Switch and PlayStation Vitas. But when I held it up for the first time, it didn't actually feel that massive or heavy. I was expecting it to feel like a brick but I think the weight is balanced well with its size, so it doesn't feel particularly dense. Navigation on the Steam Deck is also well-implemented, with its touchscreen and trackpads, the latter of which is particularly nice for the occasional text entry. While I haven't done any massive marathon sessions, I don't really have any issues with the ergonomics.

The Steam Deck has been so excellent that I think it has managed to make PC my primary gaming platform for the first time in a decade.

Back when the PlayStation 4 and the PlayStation Vita were new products, Sony touted the benefits of their ecosystem by promising that you could buy a digital copy of a game for both your PS4 and your Vita just once with Cross-Buy, and then sync your progress between the home console and the handheld console with Cross-Save. Unfortunately third-party support for these features, and the PlayStation Vita as a whole, never really took off, and Sony never had the fortitude to carry on with the experiment. Well, unless they manage to surprise with the upcoming Project Q.

However, it feels like Valve has managed to do what Sony couldn't. Since the Steam Deck is just a small PC and has direct access to every single game in my Steam Library, I can now buy a PC game just once and have it available both at home and abroad. A lot of games these days also support Steam Cloud, so I can realistically carry on where I left off on the other device – at least if the Steam Cloud support is properly implemented (I've seen it not be).

I think the biggest proof of my transformation is the fact that ever since January, I have not picked up a single game for any of my consoles. Contrast this to my estimate of having spent approximately $3800 on console games between January 2021 and January 2023. Ever since the Steam Deck, my interest in my gaming consoles has dropped off significantly, and my interest in PC gaming has increased dramatically. My Nintendo Switch might offer the portability of my Steam Deck and my PlayStation 5 might match my gaming PC in delivering high-fidelity experiences, but neither can offer me both. But PC gaming / Steam definitely now can.

I'm in fact currently waiting for a bunch of (very expensive) gaming computer parts to arrive, hoping to have a brand-new rig ready for my summer holiday. If I'm lucky, I'll have my hands on those tomorrow. Sort of hard to argue that I'm not a PC gamer at this point and it's at least partially my Steam Deck's fault.

I have to give it to Valve's business sense – releasing the Steam Deck, a good and affordable portable gaming PC, is probably a massive boon for their bottom line. I've kept track of how much money I've spent on PC games this year and the total comes to 572.88€. 451.59€ of that is directly within Steam and 121.29€ on third-party sites like Humble and Fanatical. Not sure what cut Valve gets from third-party sites that deliver games as Steam activation keys, but with a 30% cut inside their own store, they've made at least 130€ off my 2023 Steam purchases alone.

I'm actually pretty satisfied for having spent less than 600€ in about a half a year though. Compared to my previous console software spend, this is much cheaper. Possibly because I always opted for a physical console copy and that plastic ain't cheap. Maybe at some point my software savings will offset the huge amount of money that I'm throwing at the hardware.

And since Steam is the first-party store on this device, I'm definitely avoiding all stores that are not Steam or don't give out Steam keys. You can probably get GOG and Epic Games Store working on this device but I really can't be bothered to. Not when the first-party experience is so good. So far I've picked up Cyberpunk 2077 for -50% on Steam after having bought it in 2021 from GOG, and despite Epic Games Store giving out free copies of Control and Death Stranding, I bought a 14.12€ eight-game bundle mainly for Control and paid 9.99€ for a Humble Choice for Death Stranding Director's Cut. I'm spending money that I didn't have to, just to have my gaming life consolidated on this one platform as much as possible. This is some truly next-level evil genius financial brilliance on Valve's part.

But even though I've showered praise on Valve for the Steam Deck, there are still a couple of things about the Steam Deck that I don't like.

What might be the most annoying part of using the Steam Deck portably for me is the fact that the display polariser is 90 degrees wrong. This means that it's completely impossible to use the Steam Deck while wearing sunglasses and I like to protect my eyes when I drag myself outdoors. If I'm out and about, for example sitting in the commuter train, and I want to play my Steam Deck, I'd need to find some place where to put my sunglasses for the duration. That sounds like a great way to either lose or destroy my sunglasses, which is why I opted to read a book instead during my last train ride. Apple has managed to make my iPhone work while wearing sunglasses, so I'm sure Valve could fix it too.

The battery calibration also feels off on my device. At times it has sat at 99% charge for the longest time, and then started rapidly discharging afterwards. I've also had the device pop a low battery warning for 10% remaining, followed by the Steam Deck shutting itself off within 10 seconds. Afterwards it never charged the battery when I connected it to a power cord. It just stayed at 0% for over an hour and didn't start charging until I reconnected the power cable, forcibly shutting off the device in the process. Thankfully my Deck is never too far away from a power source in my usage so I've never suffered too badly, but this sort of a thing is still an annoying paper cut in the ownership experience.

And whilst this isn't really a problem caused by the Steam Deck or Valve, I've also been annoyed when it has come to patching games. Some visual novels really need translation patches for a proper experience, but the Wine-based game install environment doesn't really facilitate that. Attempting to patch my copy of CHAOS;HEAD NOAH on the Steam Deck was such a pain in the neck that I think I'm just going to try copying over the patched game installation from my gaming PC to my Deck. Maybe in the future these patches will be made with considerations for the Steam Deck, but right now it's an annoyance.

But still, for a 400-euro weird little first-generation gaming thingy, the Steam Deck is a hoot. I don't have any regrets about buying mine and I'd recommend it to anyone even remotely curious about getting one. During the latest Steam seasonal sale, the barrier of entry was even at just 377.10€, so the financial investment isn't too bad. Most likely you won't regret it and even if you do, you can probably load it off on the used market. Some weirdos do in fact buy used Steam Decks.

Bots, the second

infrequent oscillations — Wed, 12 Jul 2023 22:43:43 GMT

In which we look at our second bot.

We put a distributed database in the browser – and made a game of it

Notes on software development — Tue, 11 Jul 2023 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

CLI tip 30: extract only the matching portions

learnbyexample — Tue, 11 Jul 2023 03:00:00 GMT

The grep command provides the -o option to extract only the matching portions. Here are some examples using the BRE/ERE regexp flavors:

# whole words made up of lowercase alphabets and digits only
$ s='coat Bin food Apple (tar12) best fig_42'
$ echo "$s" | grep -owE '[a-z0-9]+'
coat
food
tar12
best

# extract characters from the start of string based on a delimiter
$ echo 'apple:123:banana:cherry' | grep -o '^[^:]*'
apple

# sequence of characters surrounded by double quotes
$ echo 'I like "mango" and "guava"' | grep -oE '"[^"]+"'
"mango"
"guava"

# whole words that have at least one consecutive repeated character
$ s='effort flee facade oddball rat tool'
$ echo "$s" | grep -owE '\w*(\w)\1\w*'
effort
flee
oddball
tool

And here are some examples with the PCRE flavor:

# numbers >= 100 if there are leading zeros
# same as: grep -owE '0*[1-9][0-9]{2,}'
$ echo '0501 035 154 12 26 98234' | grep -woP '0*+\d{3,}'
0501
154
98234

# extract digits only if it is preceded by - and not followed by ,
$ s='42 apple-5, fig3; x-83, y-20: f12'
$ echo "$s" | grep -oP '(?<=-)\d++(?!,)'
20

# extract digits that follow =
$ echo 'apple=42, fig=314' | grep -oP '=\K\d+'
42
314

# all digits and optional hyphen combo from the start of string
$ echo '123-87-593 42 apple-12-345' | grep -oP '\G\d+-?'
123-
87-
593

# all words except those surrounded by double quotes
$ s='I like2 "mango" and "guava"'
$ echo "$s" | grep -oP '"[^"]+"(*SKIP)(*F)|\w+'
I
like2
and

Use ripgrep if you want to add some more text to the matching portions, or perhaps you need to handle multiple capture groups. Here's an example:

$ echo 'apple=42, fig=314' | rg -o '(\w+)=(\d+)' -r '$2:$1'
42:apple
314:fig

Video demo:

See my CLI text processing with GNU grep and ripgrep ebook if you are interested in learning about the GNU grep and ripgrep commands in more detail.

Sending a Difficult Email

Letters To A New Developer — Mon, 10 Jul 2023 17:03:22 GMT

Dear new developer, Sometimes you have to send a difficult or awkward email. But, before you send it, think about whether it should be an email or if an in-person conversation, phone call, or video call is best. If it is an awkward topic, it’ll be an awkward conversation too, but the additional information bandwidth … Continue reading →

Bots, part 1

infrequent oscillations — Mon, 10 Jul 2023 13:54:21 GMT

In which we begin to enumerate some bots.

Investor Coffee Story

jodavaho.io — Mon, 10 Jul 2023 03:00:00 GMT

Oh sweet, coffee

One time, $employer was having an all-hands event. During these events, everyone is in town from all over. I walked in the first day and saw a table of treats and gourmet coffee, and was like “oh sweet, coffee”. I walked up and was intercepted by someone who told me “Those are for the investors”. Of course the first thing out of my mouth was “I’m investing my life” like that was supposed to mean anything to him.

Life is a funny little sequence of random occurrences that leads to me being escorted back to the rows of computer desks with an empty coffee cup in my hand while the VP of Engineering says “We’ll bring some over if there’s any left”.

To the best of my knowledge, $employer does supply coffee to employees, and no, it’s not gourmet, but it is just fine. This is a story about me being dumb and missing the obvious signals that this was special coffee for special people. There it is. If I didn’t write that down I’d forget it sometday, and now it will live well past me (as long as someone pays my AWS bill). And yes, this was edited after $employer.HR contacted me.

Fixing AMD CPU Scaling on Fedora

smcleod.net — Sun, 09 Jul 2023 09:41:37 GMT

Enabling the new AMD P-State Driver on Fedora with Kernel 6.4

Brain & Background Processes

bayindirh — Sat, 08 Jul 2023 03:00:00 GMT

During the pandemic, I have found a nice application called Pagico, which allowed me to dump everything in my mind, divide them into projects and actionable items. This process enabled me to plan my short term and medium term workload effortlessly, and see what I'm doing with my life.

One of the things I done with Pagico was to create deadlines for almost all the tasks I added into it. Combined with the built-in Gantt chart, this method created a nice way to plan the week ahead. I was able to use this method for almost three years without any big problems.

However, life being life, some tasks slipped, some went into back burner, but I never removed the dates from them, converting my tasks into a big rolling ball of mud, keeping it approximately the same size via disciplined effort.

Recently, I started noticing problems about concentration, deep work and completing the same amount of tasks, causing this ball of mud to grow. I am not a workaholic, but my baseline performance slipped for this long for the first time and, this was bothering me a lot.

After a week-long work travel and week-long vacation back to back, I returned to office and decided to write most important tasks into a list, on paper, returning to my old ways. I didn't open Pagico once, and used that list to finish my tasks. At the end of the week, not only I was able to finish them all, but I squeezed in what happened during that week, too. This made me remember a study I read, and gave me an idea for trying.

The study has found a simple yet impactful fact: "Even if you don't see/use your open browser tabs, your mind is aware that they are there. In turn, you leak your concentration power with this awareness". Maybe, the same was true for my Pagico tasks, I concluded. They were not urgent, yet I attached dates to them, creating a sense of urgency. Because of this, my brain was trying to schedule these tasks at the background. In other words, I was making myself miserable by overloading myself for no reason.

After realizing this, I removed dates from all tasks in Pagico, except a couple of hard deadlines I have about work, and I feel liberated. I am aware that these tasks are there, yet I have a much clearer vision about urgency of things at hand. I can visit the lists once a week, order them by urgency, add deadlines to things I want to tackle in that particular week, and leave the rest alone, inside Pagico. On the other hand, I'm not afraid to cancel things which are not progressing, because I don't have infinite time and selecting the battles I want to fight is an important part of resource planning. Because at the end life is a resource management and strategy game.

Next, I have to prune my 500+ open tabs at the office and bookmark what's important and close the rest.

This small experiment and its profound effect made me remember that productivity is a journey, not a target. The ways, methods and strategies we use have to evolve after they fulfill their roles and carry us to the next level, to allow us to grow further and hopefully live a more fulfilling and enjoyable life.

Until next time,

Be kind.

Up but how high?

infrequent oscillations — Thu, 06 Jul 2023 22:12:04 GMT

In which we spoil the Ted Lasso S3 finale.

2023–07–06: Sneak peek of rk2aw – what's comming up soon

megi's PinePhone Development Log — Thu, 06 Jul 2023 01:00:00 GMT

Python tip 30: zip() function

learnbyexample — Tue, 04 Jul 2023 03:00:00 GMT

You can use the zip() builtin function to iterate over two or more iterables simultaneously. In every iteration, you'll get a tuple with an item from each of the iterables. Here's an example:

>>> names = ['Joe', 'Mei', 'Rose', 'Ram']
>>> physics = [86, 91, 76, 80]
>>> maths = [77, 92, 81, 83]

>>> for n, p, m in zip(names, physics, maths):
...     print(f'{n:5}: {p},{m}')
... 
Joe  : 86,77
Mei  : 91,92
Rose : 76,81
Ram  : 80,83

Here are some examples using list comprehensions and generator expressions:

>>> p = [1, 3, 5]
>>> q = [3, 214, 53]

>>> [i + j for i, j in zip(p, q)]
[4, 217, 58]

# inner product
>>> sum(i * j for i, j in zip(p, q))
910

By default, zip() will silently stop when the shortest iterable is exhausted:

>>> fruits = ('apple', 'banana', 'fig', 'guava')
>>> qty = (100, 25, 42)

>>> for f, q in zip(fruits, qty):
...     print(f'{f:6}: {q}')
... 
apple : 100
banana: 25
fig   : 42

The strict keyword argument was added in the Python 3.10 version. When set to True, this will raise an exception if the iterables are not of the same length:

>>> for f, q in zip(fruits, qty, strict=True):
...     print(f'{f:6}: {q}')
... 
apple : 100
banana: 25
fig   : 42
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: zip() argument 2 is shorter than argument 1

Video demo:

See also my 100 Page Python Intro ebook.

[Newbie Tip] What to do when you 403 during pypi twine upload

swyx's site RSS Feed — Mon, 03 Jul 2023 19:40:24 GMT

I'm pretty new to the pypi packaging ecosystem so recently ran into some trouble pushing a python package:

What I'm up to - July 2023

Philip I. Thomas — Mon, 03 Jul 2023 18:51:08 GMT

✨ What I've been up to

We finished moving back to NYC last month. I made a quick stopover to Canada for Tailwind Connect, where I met some interesting people and learned some new CSS techniques. I also visited Puerto Rico for a few days.

The book Bowling Alone: The Collapse and Revival of American Community was a fantastic read. One thing I'm thinking about is telephones versus televisions - telephones are bidirectional while TVs are one-directional. As a society, I think we under-appreciate how shift from telephones to TVs has driven a move toward more parasocial relationships. Social media followed this trend - it started more telephone-like, promoting social relationships, but moved over time became more TV-like (epitomized by Tiktok) where people consume rather than create.

🤔 Things to share

Articles: Cities and Ambition. How Parking Ruined Everything. Stop Talking to Each Other and Start Buying Things: Three Decades of Survival in the Desert of Social Media. How to Do Great Work - "... the right strategy is to not plan too much".
Books: Watching the English.
Apps: Anki, Preply.
Coffees I'm drinking: Heart Honduras Encarnación Enamorado.

📫 What I'm up to this month

Booklet, a new Contraption Co. product, is welcoming its first users this month. Stay tuned for more details.

📍 Where I'll be

July 10-14: San Juan, PR 🇵🇷

(Let me know if we overlap!)

📸 Photos

Last dinner in Chicago

Puerto Rico

A simple app for drawing Wardley Maps

Kartik Agaram — Sun, 02 Jul 2023 10:00:00 GMT

wardley.love is a reskin of snap.love for drawing Wardley Maps. I've been using it a lot; here's one example:

Using Whisper to Transcribe Podcasts

swyx's site RSS Feed — Sat, 01 Jul 2023 22:31:05 GMT

Prerequisites

I don't think the $70 game is that expensive

ブラック — Sat, 01 Jul 2023 03:00:00 GMT

Time for some hot bourgeois takes.

I recently ordered some manga from Book Depository right before they shut down all operations (RIP). In fact, I got five total volumes of My Dress-Up Darling in English, as the impending closure of the site prompted some good discounts: on average I paid 13.37 € per volume. On other online book retailers, each volume was closer to 17.50 €, and my previous Book Depository orders for the same series last year were also closer to 17 € than they were to 13 €.

My Dress-Up Darling is great. It's funny, it's cute, it's romantic, it's well drawn. It's just all around great. Whenever I start reading a volume, I just gobble it up like a greedy pig. So much so that I decided to take out a timer and see how long it took me to finish the fifth volume.

It took me just 35 minutes to finish the volume.

If I use this amount of time with my favourite way of evaluating entertainment, euro per hours of enjoyment, My Dress-Up Darling is about 23 €/h even with the cheap pricing I got with Book Depository. With the more expensive prices on other sites, it'd be closer to 30 €/h. I love the manga but this still seems like a pretty awful money-to-time ratio.

With this in mind, I decided to go through the list of games I've completed lately to get a comparison for the amount of entertainment I've sucked out of those games compared to the amount of money I paid for them.

Game	Price	Hours played	Price/hour
Bastion	0.00 €¹	17.7 h	0.00 €/h
Ghostwire: Tokyo	0.00 €¹	39.4 h	0.00 €/h
Jet Set Radio	1.19 €	4.5 h	0.26 €/h
Muv-Luv photonflowers*	10.49 €	10.1 h	1.04 €/h
Muv-Luv photonmelodies♮	12.49 €	29.1 h	0.43 €/h
NEEDY GIRL OVERDOSE	9.30 €	11.2 h	0.83 €/h
Synergia	~2.23 €²	3.7 h	0.60 €/h
THE DAY AFTER 00	8.99 €	4.5 h	2.00 €/h
THE DAY AFTER 01	15.11 €	8.2 h	1.84 €/h
THE DAY AFTER 02	15.11 €	6.7 h	2.26 €/h
THE DAY AFTER 03	15.11 €	7.7 h	1.96 €/h

Gift
Bought in a bundle

All of my recently finished games absolutely obliterate My Dress-Up Darling when it comes to the amount of entertainment for the price. However, they're all quite clearly old and cheap games. New games are expensive! Often going for $70 at the time as they're released!

But what if they were too? Well…

Game	Price/hour ($70)
Bastion	$3.96/h
Ghostwire: Tokyo	$1.78/h
Jet Set Radio	$15.56/h
Muv-Luv photonflowers*	$6.93/h
Muv-Luv photonmelodies♮	$2.41/h
NEEDY GIRL OVERDOSE	$6.25/h
Synergia	$18.92/h
THE DAY AFTER 00	$15.56/h
THE DAY AFTER 01	$8.54/h
THE DAY AFTER 02	$10.45/h
THE DAY AFTER 03	$9.09/h

At $70 per game, absolutely every single one of them becomes much worse value than what I actually paid for them. But even if we assume naive $1 = 1€ conversion, all of them are still cheaper for the amount of time I got out of them than that single volume of My Dress-Up Darling. Even Synergia, which I think is not worth the money or the time.

I also have a bunch of unfinished games that I've dabbled with during the year. Let's see how they hold up in reality and against the $70 benchmark:

Game	Price	Hours played	Price/hour	Price/hour ($70)
CHAOS;HEAD NOAH	6.29 €	2.8 h	2.25 €/h	$25.00/h
Cyberpunk 2077 (Steam)	29.99 €	30.8 h	0.97 €/h	$2.27/h
Mass Effect Legendary Edition	49.79 €	31.3 h	1.59 €/h	$2.24/h
Mirror's Edge Catalyst	4.99 €	6.1 h	0.82 €/h	$11.48/h
Muv-Luv Alternative Total Eclipse	23.79 €	5.3 h	4.49 €/h	$13.21/h
Tokyo Xanadu eX+	~2.23 €¹	7.9 h	0.28 €/h	$8.86/h
Vampire Survivors	3.99 €	14.8 h	0.27 €/h	$4.73/h

Bought in a bundle

Those again feel pretty reasonable when it comes to value. CHAOS;HEAD NOAH, a visual novel that I've only read for a couple hours so far, is the worst performer on the list and fares quite badly against the $70 benchmark. And yet, they still all fare better than my manga. And according to HowLongToBeat, I can expect 20+ hours from CHAOS;HEAD NOAH even with my speedreading, bringing the benchmark value in the $3.50/h range (0.31 €/h real value). So unless I drop any of these titles, the cost-per-hour is only going to become more and more reasonable.

Granted, books do come with one big advantage: they don't require any additional purchases. It's a fully self-contained product. However, I feel like my manga collection still fares quite poorly value-wise even when accounting for the hardware. The last number I have for my Steam Deck playtime was around 110 hours. Having paid 400 € for mine, that translates to roughly 3.64 €/h. And that's with under half a year of ownership with a recluse that spends around 23 hours of their day at home. I imagine by the time that I replace the device, especially if I can still recoup some of its purchase price on the used market, that the final cost of ownership will become a rounding error. As long as I just don't break it.

I also recently calculated that were I to throw my gaming PC into the lake right now, it would have had a price of 2.60 €/hour of gaming. And if I were able to sell it for half what I paid for it (which seems like a reasonable expectation looking at prices for used gaming PCs) then it'd drop to 1.30 €/h. I'm also one of those weirdos that has a completely dedicated PC for gaming, as my primary computer of choice is a Mac. I imagine most people gaming on PC do actually use their gaming PC as their primary computer anyways, so the value of the hardware over time becomes negligible.

One additional piece of media that we can benchmark games against is movies. Finnkino movie tickets seem to range somewhere between 14.90 € and 22.00 €, and a movie is probably somewhere between 90 and 150 minutes in runtime. That gives us a range of 5.96 to 14.67 €/h. So had I paid $70 for my copy of NEEDY GIRL OVERDOSE, I'd still be looking at the cheaper end of the value spectrum when it comes to movie tickets. And Ghostwire: Tokyo pricing might not even cover the parking.

And the thing is: it's very easy to not pay $70 for a game. Not a single game that I've listed here was bought for anywhere near $70. In my opinion, $70 is mostly just a tax that is levied against the impatient who must get the latest AAA game right at launch. Games go frequently on discount, and if you're a PC gamer, you have a pretty decent selection of storefronts to shop in.

To quantify this in data, I arbitrarily chose some bigger releases from the past year and checked out what kind of savings are/were possible for them if you just waited for a bit using gg.deals (great website btw).

Game	Launch price	Current price	Historical low	Price drop
Atomic Heart	59.99 €	39.59 €	39.59 €	34%
Call of Duty: Modern Warfare II	69.99 €	38.49 €	38.49 €	45%
Dead Island 2	59.99 €	43.79 €	35.99 €	27% (40%)
Dead Space remake	59.99 €	41.99 €	35.99 €	30% (40%)
Diablo IV	69.99 €	69.99 €	69.99 €	0%
Forspoken	79.99 €	39.99 €	39.99 €	50%
Hogwarts Legacy	59.99 €	40.65 €	40.65 €	32%
Marvel's Spider-Man Remastered	59.99 €	30.43 €	29.73 €	49% (50%)
Redfall	69.99 €	55.74 €	52.49 €	20% (25%)
Resident Evil 4 remake	59.99 €	44.39 €	38.69 €	26% (36%)
Returnal	59.99 €	37.37 €	35.99 €	38% (40%)
Star Wars Jedi: Survivor	69.99 €	52.49 €	52.49 €	25%
The Callisto Protocol	59.99 €	29.99 €	26.99 €	50% (55%)
The Last of Us Part I	59.99 €	46.21 €	44.99 €	23% (25%)

Looking at this data, I'd say it becomes quite clear that games do get pretty decent discounts within just a year of coming out. The only game on my list of recent major titles that wasn't discounted was Diablo IV and that came out 25 days ago. Of course, that could also just a sign that the last year has had quite a lot of bad games (hello Forspoken and Redfall) and botched releases (looking at you, Star Wars Jedi: Survivor and The Last of Us Part I). But even the universally acclaimed Elden Ring (released 16 months ago) can be bought for 40% off its launch price right now. Just chip away at your backlog for some months and you can probably grab the next AAA release for a good discount.

As far as I'm concerned, gaming is still an incredible value for money, even with the new $70 price tag for AAA releases. Just as long as you just play the damn games.

Collecting the dues

There is however something that I find to be incredibly bad value: collector's editions.

For example, a copy of Tears of the Kingdom is currently available on a Finnish retailer's site for around 63 € but if you want the collector's edition, you need to pay 160 €. That is 97 € more for the collector's edition compared to the base game. Sure, Tears of the Kingdom might be a blockbuster that offers somewhere between 50 and 200 hours for a playthrough, but you can get that experience for just 63 €. So what does the 154% collector's edition markup give you?

A 204-page hardcover artbook, a steelbook, a poster and four pins.

Personally, I'm not really a fan of steelbooks, posters and pins. I think steelbooks look awkward among non-steelbooks on the shelf and they're not really even that valuable. You got a free steelbook upgrade if you pre-ordered Persona and got the launch day version. Not sure how many people like to hang posters but I'm definitely not one of them. And sure, this might be a premium steel poster, but how much can that be worth? It's not even that big. And do people outside of school children decorating their bags have a use for pins?

The only thing that strikes me as desirable is the artbook, but even then this collector's edition seems like bad value. I actually bought the Persona 5 artbook when it came out and it's very thick at 432 pages. And how much did I pay for it back in 2017? Around 24 €. And the Zelda one is half the size for an almost 100 € upsell. It's closer to what I got with Persona 4 Official Design Works, which was 192 pages for 34 €, and that's probably with a reasonable markup since I bought it six years after release. (These days the English version of the Persona 4 art book seems to be fetching over 100€.)

To me it seems like collector's editions are a way for game publishers to increase profit margins on games as base games have not kept up with inflation. The standard game price jumped to $60 during the PlayStation 3 era, and $60 in 2007 is worth about the same as $90 in 2023. Of course, they do need to make a couple more trinkets to stuff in those boxes for them to sell, but I've rarely seen a worthwhile collector's edition, at least one that doesn't cost an arm and a leg.

Now I have to admit that I have actually bought a collector's edition for a game in the recent years. It was the Cyberpunk 2077 Collector's Edition for the PS4. It does come with quite a lot of stuff, although the value of some of them can be debated. Inside the massive box, you get the game, a steelbook, a world compendium, some postcards, a map, a collection of stickers, a metal keychain, some metal pins, a guide leaflet, embroidered patches, a 216-page hardcover artbook and a large figurine of male V on a motorcycle.

I'm no expert on collector's editions but I reckon the Cyberpunk 2077 one includes a lot more stuff than your average collector's edition. It's somewhat debatable whether or not that stuff is good but it certainly has it. However, it's also quite a lot more expensive than your average collector's edition at 249 €. So why on earth did I get it? Well, because I got it for 79.95 € during a Black Friday sale. I guess the launch reception of the game really had an effect on its retail price. And since the game collector in me wanted to pick up a disc version of the atrocious 1.00 version of Cyberpunk 2077 anyways, I figured I might as well get the artbook, a cool keychain and a figurine at the same time. The base game hovered around 20 € at the time so it was really only 60 € more for those. What a bargain! Or so I tell myself at least.

Still, I have limited love for these expensive collections of trinkets. I think I'm going to keep avoiding collector's editions and instead put my money towards standard versions of games – which I'll continue to mostly grab from various sales.

The Traction vs Trust Equation of Cofounder Dating

swyx's site RSS Feed — Thu, 29 Jun 2023 04:20:24 GMT

One of the most common problems in the beginning stage of a founder journey is establishing the cofounder group. I thought I would jot down some notes from my own experience + that of friends in similar situations.

CLI text processing with GNU sed book announcement

learnbyexample — Thu, 29 Jun 2023 03:00:00 GMT

Hello!

I am pleased to announce a new version of my CLI text processing with GNU sed ebook. Examples, exercises, solutions, descriptions and external links were added/updated/corrected.

This book will help you learn the GNU sed command step-by-step from beginner to advanced levels with hundreds of examples and exercises. In addition to command options, regular expressions will also be discussed in detail.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of CLI text processing with GNU sed for FREE till 10-July-2023. You can still pay if you wish ;)

Other offers:

CLI text processing with GNU grep and ripgrep is FREE
All Books Bundle is $12 (normal price $32) — all my 13 programming ebooks

What's new?🔗

Command version updated to GNU sed 4.9
Many more exercises added, and you can practice some of them using this interactive TUI app
Long sections split into smaller ones
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to CLI text processing with GNU sed
New cover image
Images centered for EPUB format

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Interactive TUI app🔗

I also wrote an interactive TUI app based on some of the exercises from the ebook. Reference solutions are also provided.

Table of Contents🔗

Preface
Introduction
In-place file editing
Selective editing
BRE/ERE Regular Expressions
Flags
Shell substitutions
z, s and f command line options
append, change, insert
Adding content from file
Control structures
Processing lines bounded by distinct markers
Gotchas and Tricks
Further Reading

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/learn_gnused/.

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_gnused for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tips, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_gnused/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

Premier League 2023

infrequent oscillations — Thu, 29 Jun 2023 00:19:43 GMT

In which we analyse, loosely, the 2023 EPL.

Audacity Tips

Jan van den Berg — Wed, 28 Jun 2023 23:29:59 GMT

This is a public note to myself for working with Audacity; which I don’t do too often, and I want to make sure I don’t forget it. I recently created a 5 hour music project: a DJ radio show. What I need from Audacity is: That’s mostly it. Audacity is a very powerful tool, but […]

On "Software Engineering" circa 2023

--Mike-- — Tue, 27 Jun 2023 22:29:21 GMT

I've been a programmer since the 1980s. I feel that the peak of the field was somewhere between 1995 and 2000. We had Windows 95/98, the Internet, and all programs were local applications run on a desktop, that people had gotten very productive on.

The existence of Visual Basic and VBA support in the Microsoft Office Suite made it possible, and even practical, for most domain experts to build usable applications that allowed everyone to get their jobs done. If there were performance problems, or it needed to be made more reliable, professional programmers would be brought in to rebuild things in a more properly designed manner.... it was at this point that we almost shifted to being actual Software Engineers, and professionalized.

Since then VB was cast into the pyre as a sacrifice to the very unnecessary migration to .Net, and the bloat that ensued as desktop programming lost a decade of productivity, people decided to just shove everything onto the web.

It was only as this was starting to happen that Steve Jobs further crippled programming by introducing the iPhone, and suddenly GUI applications were expected to work on tiny screens (in either orientation) without proper input hardware like 3 button mice and keyboards, connected across a slow and unreliable network connection.

Needless to say, the last 2 decades have been a total loss as far as programmer productivity goes, with one shining exception.... GIT. Git has its flaws, mostly arising when people don't realize it's a set of snapshots that fake storing deltas, and not the other way around.

GIT/GitHub, et al... are fantastic. The ability to just keep multiple machines up to sync without hassle in seconds is sooooo good. I used to keep stacks of floppy disks with ZIP files of source code, all manually managed.

In the future, we need to recover to the point where you can drag/drop GUI elements and have them work anywhere, like we were with VB/Delphi/Hypercard.

When we get there, we'll let users build basic applications, and we can finally professionalize and apply actual engineering practices to the art of programming.

Until then, please stop calling it engineering. We don't put in anywhere near the effort that Margaret Hamilton (the first actual Software Engineer) and crew did, in safely getting men to the moon. We're programmers, not Engineers. As Uncle Bob said, we don't profess anything. We certainly don't use engineering practices as described by The Engineer Guy.

---

Example: You can plug a lamp into an outlet, and in the US, it can draw up to 15 amperes, and under almost all circumstances, you can't damage the wiring in the house via a fault in the load.

We have no equivalent in software. Chroot, sandboxes, etc... are far too unsafe. We have no standard way of letting the user choose resources to give to applications at run time.

The worst part is, most people don't even see the deficiency. Imagine the current power grid with no fuses or circuit breakers.... the first wiring mistake would crash civilization.

We can do better, we must do better.

Growth as a Software Engineer

The Healthy Dev — Mon, 26 Jun 2023 23:29:55 GMT

Key skills to go beyond senior

Pick up the dog poop

Letters To A New Developer — Mon, 26 Jun 2023 14:51:50 GMT

Dear new developer, I was out for a run the other day. People walk dogs along my running route and I had noticed a few days before a bag of dog poop. I had run on by, assuming someone was planning to grab it on the way back from their walk. But then I saw … Continue reading →

Muv-Luv photonmelodies♮ review

ブラック — Mon, 26 Jun 2023 03:00:00 GMT

Time for the second Muv-Luv side story collection. This time with three stories: Adoration, Resurrection and Altered Fable.

Adoration

Adoration is described as "a rare glimpse into what the world of Muv-Luv Alternative is like outside of Japan", taking place in Dover, England and focusing around the Cerberus battalion. However, it is not a hardboiled European military action as you might believe from the summary. Rather, it's a lighthearted comedy starring Makabe Seijuro, a man with a hundred elder brothers and an off-brand Lelouch.

That being said, Adoration is a fun read. I found myself grinning quite a bit when the serious Royal Guard cadet faces absurd situations with the German troops. Big-time Alternative fans looking for a serious character drama might not enjoy their stay, but if you enjoyed Extra, you might enjoy the adventures of Cadet Makabe. Budget Lelouch also encounters lucky pervert situations at a pace that might make Shirogane jealous, so you have some ecchi action to entertain you as well.

Nonetheless, there are actually serious moments of character development as well. It might be a fairly brief read, clocking at around four hours to completion, but Makabe does in fact grow as part of the story. And the rest of the cast, while not experiencing the same kind of a character arc, are extremely likable. Even the one "asshole" of the story is still a likable dude.

The writer also seems to have exercise every opportunity to cram in references to other in vogue series. I'm not calling Makabe an off-brand Lelouch for nothing.

Adoration is approximately four hours long, which is fairly good for a side story.

Resurrection

From the beginning of Adoration, I figured that it'd be a fairly serious and somber story, much akin to Rain Dancers in photonflowers*.

It's definitely not that. It's in fact probably the most moronic that Muv-Luv has ever gotten. It's exceedingly absurd and silly.

That being said, Resurrection is not a bad read. As long as you are not expecting a story as serious as the beginning would suggest, you'll probably find some amusement in it. However, it is very hard to take seriously – and the story tries to have some serious parts to it, which are pretty hard to digest given how stupid everything that became before it was.

Resurrection also features very good art and animation. Kozuki and the Valkyries play a pretty big part in the story, and they all get new sprites that I found vastly superior to their old ones. And animating the sprites when it came to stuff like throwing punches is definitely competing for the top places when it comes to Muv-Luv stories. I think the only other Muv-Luv story that might rival it is Total Eclipse (which I've sampled for a couple of hours in between reading photonmelodies♮).

Resurrection was approximately seven and a half hours long, so a good chunk longer than Adoration.

Altered Fable

Altered Fable is the meat and potatoes of the entire package, taking up over 12 hours of my time to finish. But while it might be the main dish of photonmelodies♮, it feels more like a dessert for Muv-Luv fans. And while it's the direct sequel to Muv-Luv Alternative (at least until Muv-Luv INTEGRATE comes out to muddy the waters), there's not really a central plot or discernible character development. It's more like a bunch of random events strung together. Definitely not required reading, but I wouldn't expect that from a side story collection anyways.

That is not to say that there's zero plot. There is an underlying story but it really only surfaces every once in a while with a conclusion at the end. Otherwise it's just random Extraverse romps, but now with a much larger cast than you had in Extra. At least that underlying story does explain a lot of events that happen during Altered Fable that otherwise seem like really convenient coincidences to tie it into Alternative.

Altered Fable also has pretty intricate routing for a side story, even though there's not really routes per se. The main story is fixed, and you can really only affect what small events play during it and which of the girls you get to spend time with. Personally, I would've loved a full on Kashiwagi route but I'll take the little scraps I get here.

Going for 100% completion was kind of a pain though, and took me over nine hours of additional reading/skipping, as there's quite a bit of additional content beyond a single playhrough and it's not really clear what choices unlock what. And while I couldn't find a route flowchart, there is thankfully a walkthrough. Trying to get the full gallery unlocked without it was a fool's errand.

But ignoring the pain of being a completionist, Altered Fable was definitely a treat. A moment of fan service to sunset the adventures of Takeru. Just don't take it too seriously.

photonmelodies♮

While photonmelodies♮ features less stories than the previous photonflowers*, it feels like a tighter package. Many of the stories in photonflowers* were very short, and you couldn't really get into the stories in such a time. In contrast the shortest story in photonmelodies♮ is a respectable four hours, and the total package has at least 20-30 hours of content.

Technically photonmelodies♮ is safe and sound. While I prefer the Ages Mark 2 engine (used in THE DAY AFTER and Total Eclipse remasters) for its text rendering and some of its options, the venerable rUGP performs fine and runs basically perfect on the Steam Deck. In fact, it doesn't even suffer from the audio crackling bug that I've experienced with the Ages Mark 2 titles.

Muv-Luv photonmelodies♮ is a pretty easy recommendation to fans of Muv-Luv, and especially for those that enjoyed photonflowers*.

Vim tip 28: miscellaneous motion and reposition commands

learnbyexample — Mon, 26 Jun 2023 03:00:00 GMT

Moving within the visible window:

H move to the first non-blank character of the top (home) line of the visible window
M move to the first non-blank character of the middle line of the visible window
L move to the first non-blank character of the bottom (low) line of the visible window

Reposition the current line:

Ctrl+e scroll up by a line
Ctrl+y scroll down by a line
zz reposition the current line to the middle of the visible window
- useful to see context around lines that are nearer to the top/bottom of the visible window
zt reposition the current line to the top of the visible window
zb reposition the current line to the bottom of the visible window

See :h 'scrolloff' option if you want to always show context around the current line.

Video demo:

Generating SQL with LLMs for fun and profit

I Am Not a Robot — Fri, 23 Jun 2023 02:39:14 GMT

Do you trust a language model to generate and execute code on the fly? We need to talk.

Leadership models V: The Hero's Journey

Bytepawn - Marton Trencseni — Fri, 23 Jun 2023 01:00:00 GMT

The Hero's Journey, or Monomyth, is a narrative pattern identified by scholar Joseph Campbell that appears across a wide range of cultures and eras, and is also a useful mental model in Leadership.

CLI tip 29: define fields using FPAT in GNU awk

learnbyexample — Tue, 20 Jun 2023 03:00:00 GMT

In awk, the FS variable allows you to define the input field separator. In contrast, FPAT (field pattern) allows you to define what should the fields be made up of.

$ s='Sample123string42with777numbers'
# one or more consecutive digits
$ echo "$s" | awk -v FPAT='[0-9]+' '{print $2}'
42

$ s='coat Bin food tar12 best Apple fig_42'
# whole words made up of lowercase alphabets and digits only
$ echo "$s" | awk -v FPAT='\\<[a-z0-9]+\\>' -v OFS=, '{$1=$1} 1'
coat,food,tar12,best

$ s='items: "apple" and "mango"'
# get the first double quoted item
$ echo "$s" | awk -v FPAT='"[^"]+"' '{print $1}'
"apple"

FPAT is often used for CSV input where fields can contain embedded delimiter characters. For example, a field content "fox,42" when , is the delimiter.

$ s='eagle,"fox,42",bee,frog'

# simply using , as separator isn't sufficient
$ echo "$s" | awk -F, '{print $2}'
"fox

For such simpler CSV input, FPAT helps to define fields as starting and ending with double quotes or containing non-comma characters.

# * is used instead of + to allow empty fields
$ echo "$s" | awk -v FPAT='"[^"]*"|[^,]*' '{print $2}'
"fox,42"

The above will not work for all kinds of CSV files, for example if fields contain escaped double quotes, newline characters, etc. See stackoverflow: What's the most robust way to efficiently parse CSV using awk? for such cases. You could also use other programming languages such as Perl, Python, Ruby, etc which come with standard CSV parsing libraries or have easy access to third party solutions. There are also specialized command line tools such as xsv.

Video demo:

See also my CLI text processing with GNU awk ebook.

Metaprogramming in Zig and parsing CSS

Notes on software development — Mon, 19 Jun 2023 03:00:00 GMT

I knew Zig supported some sort of reflection on types. But I had been confused about how to use it. What's the difference between @typeInfo and @TypeOf? I ignored this aspect of Zig until a problem came up at work where reflection made sense.

The situation was parsing and storing parsed fields in a struct. Each field name that is parsed should match up to a struct field.

This is a fairly common problem. So this post walks through how to use Zig's metaprogramming features in a simpler but related domain: parsing CSS into typed objects, and pretty-printing these typed CSS objects.

I live-streamed the implementation of this project yesterday on Twitch. The video is available on YouTube. And the source is available on GitHub.

If you want to skip the parsing steps and just see the metaprogramming, jump to the implementation of match_property.

Parsing CSS

Let's imagine a CSS that only has alphabetical selectors, property names and values.

The following would be valid:

div {
  background: black;
  color: white;
}

a {
  color: blue;
}

Thinking about the structure of this stripped down CSS we've got:

CSS properties that consist of property names and values (in our case the property names are limited to background and color)
CSS rules that have a selector and a list of rules
CSS sheets that have a list of rules

Turning that into Zig in main.zig:

const std = @import("std");

const CSSProperty = union(enum) {
    unknown: void,
    color: []const u8,
    background: []const u8,
};

const CSSRule = struct {
    selector: []const u8,
    properties: []CSSProperty,
};

const CSSSheet = struct {
    rules: []CSSRule,
};

The parser is going to look for CSS rules which contain a selector and a list of CSS rules. The entrypoint is that simple:

fn parse(
    arena: *std.heap.ArenaAllocator,
    css: []const u8,
) !CSSSheet {
    var index: usize = 0;
    var rules = std.ArrayList(CSSRule).init(arena.allocator());

    // Parse rules until EOF.
    while (index < css.len) {
        var res = try parse_rule(arena, css, index);
        index = res.index;
        try rules.append(res.rule);

        // In case there is trailing whitespace before the EOF,
        // eating whitespace here makes sure we exit the loop
        // immediately before trying to parse more rules.
        index = eat_whitespace(css, index);
    }

    return CSSSheet{
        .rules = rules.items,
    };
}

Let's implement the eat_whitespace helper we've referenced. It increments a cursor into the css file while it sees whitespace.

fn eat_whitespace(
    css: []const u8,
    initial_index: usize,
) usize {
    var index = initial_index;
    while (index < css.len and std.ascii.isWhitespace(css[index])) {
        index += 1;
    }

    return index;
}

In our stripped-down version of CSS all we have to think about is ASCII. So the builtin std.ascii.isWhitespace() function is perfect.

Next, parsing CSS rules.

`parse_rule()`

A rule consists of a selector, opening curly braces, any number of properties, and closing curly braces. We need to remember to eat whitespace between each piece of syntax.

And we'll reference a few more parsing helpers we'll talk about next for the selector, braces, and properties.

const ParseRuleResult = struct {
    rule: CSSRule,
    index: usize,
};
fn parse_rule(
    arena: *std.heap.ArenaAllocator,
    css: []const u8,
    initial_index: usize,
) !ParseRuleResult {
    var index = eat_whitespace(css, initial_index);

    // First parse selector(s).
    var selector_res = try parse_identifier(css, index);
    index = selector_res.index;

    index = eat_whitespace(css, index);

    // Then parse opening curly brace: {.
    index = try parse_syntax(css, index, '{');

    index = eat_whitespace(css, index);

    var properties = std.ArrayList(CSSProperty).init(arena.allocator());
    // Then parse any number of properties.
    while (index < css.len) {
        index = eat_whitespace(css, index);
        if (index < css.len and css[index] == '}') {
            break;
        }

        var attr_res = try parse_property(css, index);
        index = attr_res.index;

        try properties.append(attr_res.property);
    }

    index = eat_whitespace(css, index);

    // Then parse closing curly brace: }.
    index = try parse_syntax(css, index, '}');

    return ParseRuleResult{
        .rule = CSSRule{
            .selector = selector_res.identifier,
            .properties = properties.items,
        },
        .index = index,
    };
}

The parse_syntax helper is pretty simple, it does a bounds check and increments the cursor if the current character matches the one you pass in.

fn parse_syntax(
    css: []const u8,
    initial_index: usize,
    syntax: u8,
) !usize {
    if (initial_index < css.len and css[initial_index] == syntax) {
        return initial_index + 1;
    }

    debug_at(css, initial_index, "Expected syntax: '{c}'.", .{syntax});
    return error.NoSuchSyntax;
}

This calls attention to debugging messages on failure. When we fail to parse a syntax, we want to give a useful error message and point at the exact line and column of code where the error happens.

So let's implement debug_at.

`debug_at`

First, we iterate over the entire CSS source code until we find the entire line that contains the index where the parser failed. We also want to identify the exact line and column corresponding to that index.

fn debug_at(
    css: []const u8,
    index: usize,
    comptime msg: []const u8,
    args: anytype,
) void {
    var line_no: usize = 1;
    var col_no: usize = 0;

    var i: usize = 0;
    var line_beginning: usize = 0;
    var found_line = false;
    while (i < css.len) : (i += 1) {
        if (css[i] == '\n') {
            if (!found_line) {
                col_no = 0;
                line_beginning = i;
                line_no += 1;
                continue;
            } else {
                break;
            }
        }

        if (i == index) {
            found_line = true;
        }

        if (!found_line) {
            col_no += 1;
        }
    }

Then we print it all out in a nice format for users (which will likely just be ourselves).

    std.debug.print("Error at line {}, column {}. ", .{ line_no, col_no });
    std.debug.print(msg ++ "\n\n", args);
    std.debug.print("{s}\n", .{css[line_beginning..i]});
    while (col_no > 0) : (col_no -= 1) {
        std.debug.print(" ", .{});
    }
    std.debug.print("^ Near here.\n", .{});
}

Ok, popping our mental stack, if we look back at parse_rule we still need to implement parse_identifier and parse_property.

`parse_identifier`

An "identifier" for us here is just going to be an ASCII alphabetical string (i.e. [a-zA-Z]+). We're going to really simplify CSS because we're going to use this method for parsing not just selectors but property names and even property values.

Zig again has a nice builtin std.ascii.isAlphabetical we can use.

const ParseIdentifierResult = struct {
    identifier: []const u8,
    index: usize,
};
fn parse_identifier(
    css: []const u8,
    initial_index: usize,
) !ParseIdentifierResult {
    var index = initial_index;
    while (index < css.len and std.ascii.isAlphabetic(css[index])) {
        index += 1;
    }

    if (index == initial_index) {
        debug_at(css, initial_index, "Expected valid identifier.", .{});
        return error.InvalidIdentifier;
    }

    return ParseIdentifierResult{
        .identifier = css[initial_index..index],
        .index = index,
    };
}

In reality, CSS properties are highly complex. Parsing CSS correctly isn't the main aim of this post though. :)

`parse_property`

The final piece of CSS we need to parse is properties. These consist of a property name, then a colon, then a property value, and finally a semicolon. And within each piece we eat whitespace.

const ParsePropertyResult = struct {
    property: CSSProperty,
    index: usize,
};
fn parse_property(
    css: []const u8,
    initial_index: usize,
) !ParsePropertyResult {
    var index = eat_whitespace(css, initial_index);

    // First parse property name.
    var name_res = parse_identifier(css, index) catch |e| {
        std.debug.print("Could not parse property name.\n", .{});
        return e;
    };
    index = name_res.index;

    index = eat_whitespace(css, index);

    // Then parse colon: :.
    index = try parse_syntax(css, index, ':');

    index = eat_whitespace(css, index);

    // Then parse property value.
    var value_res = parse_identifier(css, index) catch |e| {
        std.debug.print("Could not parse property value.\n", .{});
        return e;
    };
    index = value_res.index;

    // Finally parse semi-colon: ;.
    index = try parse_syntax(css, index, ';');

    var property = match_property(name_res.identifier, value_res.identifier) catch |e| {
        debug_at(css, initial_index, "Unknown property: '{s}'.", .{name_res.identifier});
        return e;
    };

    return ParsePropertyResult{
        .property = property,
        .index = index,
    };
}

Finally we get to the first bit of metaprogramming. Once we have a property name and value, we need to turn that into a Zig union.

That's what match_property() is going to be responsible for doing.

`match_property`

This function needs to take a property name and value and return a CSSProperty with the correct field (matching up to the property name passed in) and assigned to the value passed in.

If we didn't have metaprogramming or reflection, the implementation might look like this:

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    if (std.mem.eql(u8, name, "color")) {
        return CSSProperty{.color = value};
    } else if (std.mem.eql(u8, name, "background")) {
        return CSSProperty{.background = value};
    }

    return error.UnknownProperty;
}

And that is not necessarily bad. In fact it may be how a lot of production code looks over time as product needs evolve. You can keep the internal field name unrelated to the external field name.

However for the sake of learning, we'll try to implement the same thing with Zig metaprogramming.

And specifically, we can take a look at lib/std/json/static.zig to understand the reflection APIs.

Specifically, if we look at line 210-226 of that file, we can see them iterating over fields of a Union:

        .Union => |unionInfo| {
            if (comptime std.meta.trait.hasFn("jsonParse")(T)) {
                return T.jsonParse(allocator, source, options);
            }

            if (unionInfo.tag_type == null) @compileError("Unable to parse into untagged union '" ++ @typeName(T) ++ "'");

            if (.object_begin != try source.next()) return error.UnexpectedToken;

            var result: ?T = null;
            var name_token: ?Token = try source.nextAllocMax(allocator, .alloc_if_needed, options.max_value_len.?);
            const field_name = switch (name_token.?) {
                inline .string, .allocated_string => |slice| slice,
                else => return error.UnexpectedToken,
            };

            inline for (unionInfo.fields) |u_field| {

Then right after that (lines 226-243) we see them conditionally modifying the result object:

            inline for (unionInfo.fields) |u_field| {
                if (std.mem.eql(u8, u_field.name, field_name)) {
                    // Free the name token now in case we're using an allocator that optimizes freeing the last allocated object.
                    // (Recursing into parseInternal() might trigger more allocations.)
                    freeAllocated(allocator, name_token.?);
                    name_token = null;

                    if (u_field.type == void) {
                        // void isn't really a json type, but we can support void payload union tags with {} as a value.
                        if (.object_begin != try source.next()) return error.UnexpectedToken;
                        if (.object_end != try source.next()) return error.UnexpectedToken;
                        result = @unionInit(T, u_field.name, {});
                    } else {
                        // Recurse.
                        result = @unionInit(T, u_field.name, try parseInternal(u_field.type, allocator, source, options));
                    }
                    break;
                }

We can see that the .Union => |unionInfo| condition is entered by switching on @typeInfo(T) (line 149) and that T is a type (line 144).

We don't have a generic type though. We know we are working with a CSSProperty. And we know CSSProperty is a union so we don't need the switch either.

So let's apply that to our match_property implementation.

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    const cssPropertyInfo = @typeInfo(CSSProperty);

    for (cssPropertyInfo.Union.fields) |u_field| {
        if (std.mem.eql(u8, u_field.name, name)) {
            return @unionInit(CSSProperty, u_field.name, value);
        }
    }

    return error.UnknownProperty;
}

And if we try to build that we'll get an error like this:

main.zig:15:31: error: values of type '[]const builtin.Type.UnionField' must be comptime-known, but index value is runtime-known
    for (cssPropertyInfo.Union.fields) |u_field| {

Zig's "reflection" abilities here are comptime only. So we can't use a runtime for loop, we must use a comptime inline for loop.

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    const cssPropertyInfo = @typeInfo(CSSProperty);

    inline for (cssPropertyInfo.Union.fields) |u_field| {
        if (std.mem.eql(u8, u_field.name, name)) {
            return @unionInit(CSSProperty, u_field.name, value);
        }
    }

    return error.UnknownProperty;
}

As far as I understand it, this loop is basically unrolled and the generated code would look a lot like our hard-coded initial version.

i.e. it would probably look like this:

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    const cssPropertyInfo = @typeInfo(CSSProperty);

    if (std.mem.eql(u8, "background", name)) {
        return @unionInit(CSSProperty, "background", value);
    }

    if (std.mem.eql(u8, "color", name)) {
        return @unionInit(CSSProperty, "color", value);
    }

    if (std.mem.eql(u8, "unknown", name)) {
        return @unionInit(CSSProperty, "unknown", value);
    }

    return error.UnknownProperty;
}

Again that's just how I imagine the compiler to generate code from the Union field reflection and inline for over the fields.

Try compiling that code. I get this:

main.zig:17:58: error: expected type 'void', found '[]const u8'
            return @unionInit(CSSProperty, u_field.name, value);

Thinking about the generated code makes it especially clear what's happening. We have an unknown field in there that has a void type. You can't assign a string to void.

We know at runtime that the condition where that happens should be impossible because the user shouldn't enter unknown as a property name. (Though now that I write this, I see they actually could. But let's pretend they wouldn't.)

So the problem isn't a runtime failure but a comptime type-checking failure.

Thankfully we can work around this with comptime conditionals.

If we wrap our current condition in an additional conditional that is evaluated at comptime and filters out the unknown pass of the inline for loop, the compiler shouldn't generate any code trying to assign to the unknown field.

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    const cssPropertyInfo = @typeInfo(CSSProperty);

    inline for (cssPropertyInfo.Union.fields) |u_field| {
        if (comptime !std.mem.eql(u8, u_field.name, "unknown")) {
            if (std.mem.eql(u8, u_field.name, name)) {
                return @unionInit(CSSProperty, u_field.name, value);
            }
        }
    }

    return error.UnknownProperty;
}

And indeed, if you try to compile it, this works. Since the conditional is evaluated at compile time, we can imagine the code the compiler generates is this:

fn match_property(
    name: []const u8,
    value: []const u8,
) !CSSProperty {
    const cssPropertyInfo = @typeInfo(CSSProperty);

    if (std.mem.eql(u8, "background", name)) {
        return @unionInit(CSSProperty, "background", value);
    }

    if (std.mem.eql(u8, "color", name)) {
        return @unionInit(CSSProperty, "color", value);
    }

    return error.UnknownProperty;
}

The unknown field has been skipped.

In retrospect, I realize that the unknown field probably isn't even needed. We could eliminate it from the CSSProperty union and get rid of that comptime conditional. However, sometimes there are in fact private fields you want to skip. And I wanted to show how to deal with that case.

For the last bit of metaprogramming, let's talk about displaying the resulting CSSSheet we'd get after parsing.

`sheet.display()`

If we didn't have metaprogramming and wanted to display the sheet, we'd have to switch on every possible union field.

Like so (I've modified the CSSSheet struct definition so it includes this method):

    fn display(sheet: *CSSSheet) void {
        for (sheet.rules) |rule| {
            std.debug.print("selector: {s}\n", .{rule.selector});
            for (rule.properties) |property| {
                switch (property) {
                    .unknown => unreachable,
                    .color => |color_value| std.debug.print("  color: {s}\n", .{color_value}),
                    .background => |background_value| std.debug.print("  background: {s}\n", .{background_value}),
                };
            }
            std.debug.print("\n", .{});
        }
    }

This is already a little annoying and could get unwieldy as we add fields to the CSSProperty union.

Instead we can use the inline for (@typeInfo(CSSProperty).Union.fields) |u_field| method to iterate over all fields, skip the unknown field at comptime, and print out the field name and value by matching on the current value of the property enum by using the @tagName builtin.

    fn display(sheet: *CSSSheet) void {
        for (sheet.rules) |rule| {
            std.debug.print("selector: {s}\n", .{rule.selector});
            for (rule.properties) |property| {
                inline for (@typeInfo(CSSProperty).Union.fields) |u_field| {
                    if (comptime !std.mem.eql(u8, u_field.name, "unknown")) {
                        if (std.mem.eql(u8, u_field.name, @tagName(property))) {
                            std.debug.print("  {s}: {s}\n", .{
                                @tagName(property),
                                @field(property, u_field.name),
                            });
                        }
                    }
                }
            }
            std.debug.print("\n", .{});
        }
    }

`main`

Finally, we pull it all together with a little main function.

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();

    const allocator = arena.allocator();

    // Let's read in a CSS file.
    var args = std.process.args();

    // Skips the program name.
    _ = args.next();

    var file_name: []const u8 = "";
    if (args.next()) |f| {
        file_name = f;
    }

    const file = try std.fs.cwd().openFile(file_name, .{});
    defer file.close();

    const file_size = try file.getEndPos();
    var css_file = try allocator.alloc(u8, file_size);
    _ = try file.read(css_file);

    var sheet = parse(&arena, css_file) catch return;
    sheet.display();
}

And try it against some tests.

$ zig build-exe main.zig
$ cat tests/basic.css
div {
    background: white;
}
$ ./main tests/basic.css
selector: div
  background: white

Nice! Let's try it against a more complex test.

$ cat tests/multiple-blocks.css
div {
    background: black;
    color: white;
}

a {
    color: blue;
}
$ ./main tests/multiple-blocks.css
selector: div
  background: black
  color: white

selector: a
  color: blue

Awesome. And against a bad CSS sheet:

$ cat tests/bad-property.css
a {
    big: pink;
}
$ ./main cat tests/bad-property.css
Error at line 2, column 4. Unknown property: 'big'.


    big: pink;
    ^ Near here.

We've got it!

Addendum: `@field`

The docs were quite clear about using @field(object, fieldName) to access the value of an object of type @TypeOf(object) at field fieldName.

And the docs do mention @field() can be used as LHS but that only really struct me when I was browsing the Zig JSON code like at line 307.

I didn't use that in this little project but I've used it elsewhere, so it I wanted to call this LHS behavior out.

Wrote a short post on parsing CSS as a way to motivate some basic exploration of metaprogramming in Zig.

I heavily referenced Zig's builtin JSON parser when learning this. And it is referenced multiple times in the post as well.https://t.co/CX6jXSLGiR pic.twitter.com/jAJJZ0pONQ
— Phil Eaton (@eatonphil) June 19, 2023

Real-world experiments: 5 Lessons from Google, Bing, Netflix and Alibaba

Bytepawn - Marton Trencseni — Sun, 18 Jun 2023 01:00:00 GMT

I discuss five lessons from large-scale experiments conducted by Google, Bing, Netflix and Alibaba: Kohavi's 1 out of 3 rule, Google's 41 shades of blue, Bing's unexpected big win, Alibaba's personalization experiment and Netflix' movie image personalization.

Full Time

Weblog on marginalia.nu — Fri, 16 Jun 2023 03:00:00 GMT

I’m working on Marginalia Search full time. I left the office for the last time today, and it’s the strangest feeling. I’ve quit jobs, taken time off work, been laid off, but this is different from any of those things. This is deliberate. There’s a note of relief. I’ve essentially been working two pretty demanding jobs; one for pay and one for passion and the joy of making a difference.

Python tip 29: negative lookarounds

learnbyexample — Tue, 13 Jun 2023 03:00:00 GMT

Lookarounds help to create custom anchors and add conditions within a regex definition. These assertions are also known as zero-width patterns because they add restrictions similar to anchors and are not part of the matched portions. The syntax for negative lookarounds is shown below:

(?!pat) negative lookahead assertion
(?<!pat) negative lookbehind assertion

Here are some examples:

# change 'cat' only if it is not followed by a digit character
# note that the end of string satisfies the given assertion
# 'catcat' has two matches as the assertion doesn't consume characters
>>> re.sub(r'cat(?!\d)', 'dog', 'hey cats! cat42 cat_5 catcat')
'hey dogs! cat42 dog_5 dogdog'

# change 'cat' only if it is not preceded by _
# note how 'cat' at the start of string is matched as well
>>> re.sub(r'(?<!_)cat', 'dog', 'cat _cat 42catcat')
'dog _cat 42dogdog'

# change whole word only if it is not preceded by : or -
>>> re.sub(r'(?<![:-])\b\w+', 'X', ':cart <apple: -rest ;tea')
':cart <X: -rest ;X'

Lookarounds can be placed anywhere and multiple lookarounds can be combined in any order. They do not consume characters nor do they play a role in matched portions. They just let you know whether the condition you want to test is satisfied from the current location in the input string.

# extract all whole words that do not start with a/n
>>> ip = 'a_t row on Urn e note Dust n end a2-e|u'
>>> re.findall(r'(?![an])\b\w+', ip)
['row', 'on', 'Urn', 'e', 'Dust', 'end', 'e', 'u']

# since the three assertions used here are all zero-width,
# all of the 6 possible combinations will be equivalent
>>> re.sub(r'(?!\Z)\b(?<!\A)', ' ', 'output=num1+35*42/num2')
'output = num1 + 35 * 42 / num2'

Video demo:

See also my 100 Page Python Intro and Understanding Python re(gex)? ebooks.

Top Traits to Look for When Hiring Exceptional Engineers

Letters To A New Developer — Mon, 12 Jun 2023 18:22:10 GMT

This is a guest post from Manish. Enjoy. Dear new developer, As leaders, we often need to hire new talent to expand our resources, replace team members, or fill vacant positions. While technical skills are important, there are also several non-technical traits that can be equally or even more valuable. Consider the following factors when … Continue reading →

AMD RX 7900 XTX review

ブラック — Mon, 12 Jun 2023 03:00:00 GMT

The worst graphics card I've had the displeasure of owning.

Now, I'm not going to lie: I'm no graphics card expert. Before the RX 7900 XTX, the three graphics card that I've owned were the Nvidia RTX 3060, the Nvidia GTX 760 and the Nvidia GeForce 9800 GT. If I don't misremember, those were bought in 2021, 2014 and 2008 respectively. Having only had experience with four different GPUs over a span of 15 years definitely doesn't give me the credentials to review a graphics card properly.

However, I can still say that the AMD RX 7900 XTX is definitely the worst card I have ever owned. Why? Because it just doesn't work.

May

I bought my 7900 XTX at the beginning of May. Thankfully AMD.com uses DHL, my absolute favourite courier service in Finland, and I managed to get my grubby little hands on my GPU in mere five days after placing the order. After I got done with work, I painstakingly replaced the RTX 3060 inside my gaming PC (sandwich-style PC cases are not easy to work in), booted up and installed drivers. And then, I decided to boot up the most graphically intensive game I have: Cyberpunk 2077.

Turns out that the 7900 XTX could really only muster around 40 frames per second with the ray tracing settings I had. I would have expected a little more for a 1100-euro investment. But opening up the AMD performance overlay showed a culprit: the GPU junction temperature was at 110°C.

Turns out that the 7900 XTX hot spot temperature problem that was in the news four months ago was still a thing.

Well, nothing to do about it but contact AMD support. To their credit, they did immediately acknowledge that these products have a fault, although they seemed to be downplaying it to a degree ("may impact the performance but the card can still be used"). Nevertheless, I asked them to generate me an RMA so I could get my card switched for a working one. Then about a week passed before I got my shipping label because AMD support was kinda slow. Maybe they are dealing with a big influx of unhappy 7900 XTX owners?

Since my assumption was that it'd only take a couple of weeks to get a card back, and since working inside the mini-ITX case gets on my nerves, I decided that I wouldn't reinstall my old graphics card back. It'll be back in a jiffy and I can just sate my gaming desires with the Steam Deck in the meanwhile.

Unfortunately due to the poor pick-up schedules at my nearest DHL location, it took around a week from leaving the return package to it actually arriving at AMD's return location. At this point, it had already been two weeks since I got the initial card and that I submitted the RMA request. And then, AMD's partner kept sitting on the package for around week and a half, and so May turned to June.

June

Finally in June, I got my second DHL shipment notification and DHL then hurried over the package to my front door in less than 24 hours. Again, immediately after work I painstakingly stuffed the 7900 XTX into my computer case and launched Cyberpunk 2077 to finally see what kind of performance the GPU can dish when it's not throttling.

It overheated and throttled immediately.

Unfortunately the second card came just against the weekend, so I had no choice but to continue the already-long email chain with AMD customer support and try to get another RMA open. Although since I'd just spent the past three weeks looking at my half-finished gaming PC gathering dust, I was leaning more towards a refund than having to do the same process again.

Then Monday comes and AMD replies. They couldn't ascertain that the graphics card was overheating if I didn't send them measurements from the AMD Radeon software. This felt a bit odd considering the first RMA they didn't bother even asking for evidence but I still went ahead and gathered them their precious Radeon screenshots. At least it's not difficult to document when the card hits 110°C as soon as you load a Cyberpunk 2077 save.

Having proven to AMD that the card was in fact overheating using their own tools, AMD customer support had another quest for me: send system diagnostics reports of my gaming PC and prove to them that the GPU power cables (two 8-pin connectors) are connected. Again, I have a hard time understanding how this is necessary since a month ago the same customer support agent had told me that these cards have a known flaw.

Unfortunately that was not enough. What was now required of me was to update my BIOS, do a clean reinstall of all AMD drivers, send details of the power supply that I was using for the card, and install the graphics card without the riser card my mini-ITX case requires. There is no clearance to install any kind of a graphics card with the motherboard in the case, so doing this would have required completely disassembling my entire PC.

I'd had enough.

I told AMD customer support that I was done playing their games. No BIOS update was going to fix this issue and AMD's drivers cannot be so bad that they turn the GPU into a hot plate while it's throttling. But thankfully after being told by a friend and having been proven by der8auer, these cards actually have a single orientation in which they won't overheat. Unfortunately, that was not an orientation that was achievable mounting inside my case.

So I did the only sensible thing imaginable: I grabbed two rolls of toilet paper and very carefully perched my gaming PC on its back to get the card in a vertical orientation. And while the PC might have been very unstable when it came to sitting on those bog rolls, the card did stay at a stable 80°C while playing Cyberpunk 2077. Clocks were also staying up and the fans were running a whopping 1100 RPM slower.

And those stats stayed stable for over 20 minutes. Then I turned the computer back up again. Temperatures immediately skyrocketed. Not even a minute and the junction temperature was 110°C, forcing the card to throttle itself. So the cooler was most definitely busted for the second time in a row.

Thankfully, telling AMD customer support how fed up I was with their diagnosis for an issue they are already aware of and sending them the graphs of the vertical orientation test worked. After about a week of constant contact with AMD and I got my second RMA.

And so, today, I dispatched the AMD RX 7900 XTX towards AMD's fulfillment center. Hopefully in a couple of weeks time, I will have my money back and I can leave this episode behind me.

Future

I think there are two lessons that I've learned from this experience.

First of all, I really need to do the research before putting down this kind of money towards a product. On some level, I was aware that AMD had a overheating problem with some of its products. But that was in the news months ago. And it only affected some units (or so AMD claimed). That's surely not something that I'd encounter. Twice.

And secondly, maybe just avoid buying reference graphics cards. They're clearly not that good and at least when it comes to AMD, neither is the customer support. And based on the discussions I've had with people, it seems like Nvidia support is not world-class either.

But I'm still on the market for a new graphics card. The reason why I picked the RTX 3060 was because I hadn't owned a gaming PC for over seven years at the time, and I wanted to try out PC gaming again with a careful approach. But with the amount of PC gaming I've done lately, and the inherent low performance of the card, I want to upgrade. But to what?

One of the reasons why I went with the 7900 XTX was because I didn't want to give my money to Nvidia and support the insane prices that they charge for their cards. The AMD RX 7900 XTX and Nvidia RTX 4080 have comparable performance but the 7900 XTX is over 200€ cheaper. But now that I've put my money where my mouth is, and fought against the Nvidia hegemony, the only things I have left to show for it are a long email thread with AMD customer support and additional scratches in my GPU bracket.

So now I need to choose: do I go for a custom 7900 XTX design like the Sapphire PULSE 7900 XTX and pay a premium over the reference cooler design just to get a working card, or do go back to Team Green with my tail between my legs and pay hundreds of euro more, to a point where it makes sense to just go for the RTX 4090 instead? I feel like there are no good options here, but Nvidia has yet to burn me.

Searching for The Early Founder Cadence

swyx's site RSS Feed — Mon, 12 Jun 2023 02:51:01 GMT

I've been quite inspired by David Sacks' The Cadence ever since I read it. It prescribes an operational process and ideal team structure for a 50-500 person startup - running sales, finance, product, and marketing in sync in quarterly cycles with effective communication between functions.

Killing Community

Weblog on marginalia.nu — Sun, 11 Jun 2023 03:00:00 GMT

This is a theory that’s previously been stated in log/39-normie-hypothesis.gmi, but I think it’s worth expanding on as it’s become very relevant with the recent Reddit shit-show actualizing just how bad that website has gotten along with social media in general. I think the model demonstrates how the ’enshittification’ process is an inevitability with any social media that is run on a venture capital model. An online community can be like a village, where you have familiar faces, collective experiences, shared values and so forth.

Conditional Probabilities and Simpson's Paradox

Bytepawn - Marton Trencseni — Sun, 11 Jun 2023 01:00:00 GMT

I give examples of "unintuitive" conditional probabilities and discuss Simpson's paradox.

Minecraft Containers

infrequent oscillations — Sat, 10 Jun 2023 09:55:27 GMT

In which we poke around in some chests.

2023–06–10: Orange Pi 5 Plus

megi's PinePhone Development Log — Sat, 10 Jun 2023 01:00:00 GMT

2023–06–09: Multi-distro image supporting both Pinephone and Pinephone Pro

megi's PinePhone Development Log — Fri, 09 Jun 2023 01:00:00 GMT

Mojo is a much better "Objective-C without the C" than Swift ever was

metablog — Fri, 09 Jun 2023 00:54:07 GMT

One of the primary things that people don't understand about Objective-C is that it is a solution of the two language problem, or more precisely a generalisation of the two language problem to the scripted component pattern.

The scripted component pattern itself is a (common) solution to the problem, first identified in the 70s that programming-in-the-large is not the same as programming-in-the-small, that module implementation languages are not necessarily suitable as module interconnection languages.

And so we have all sorts of flexible connection languages, often interpreted (aka glue, scripting, and orchestration languages), starting with the Unix shell, in addition to fast, compiled component languages such as C, C++ and Rust, and a system will usually incorporate at least one of each kind.

But then you run into the two language problem: you have to deal with these two distinct languages, with how they integrate, and with the boundaries of the integration often not matching up very well with the boundaries of the problem you're trying to solve.

Objective-C solved the two language problem by just jamming the two languages into one: Smalltalk for the scripting/integration and C for the component language. Interoperability is smooth and at the statement level, thougha there is some friction due to overlaps caused by integrating two existing languages that were not designed to be integrated.

Mojo essentially uses the Objective-C approach of jamming the two languages into one. Except it doesn't repeat Objective-C's mistake of using the component language as the base (which, inexplicably, Swift didn't just repeat, but actually doubled down on by largely deprecating objects). The reason this is a mistake is that it turns out that the connection language is actually the more general one, the component language is a specialisation of the connection language.

With this realisation, Mojo's approach of making the connection language the base language make sense. In addition, the fact that the component language is a specialisation also means that you don't actually need to jam a full second language into your base, a few syntactic markers to to indicate the specialisations are sufficient.

This is pretty much exactly stage 2 of the 4 stages of Objective-S, so I think they are using exactly the right approach for this. Except of course for the use of Python as the base instead of Smalltalk, which is a pragmatic choice given what they are trying to accomplish, but means your connection language is unduly limited.

Objective-S has the same basic structure, but with a much more capable connection language as the base.

Code Spelunker how it works

Ben E. C. Boyter — Tue, 06 Jun 2023 03:00:00 GMT

I released code spelunker a few days ago https://github.com/boyter/cs and literally one person asked for details on how it worked. Fitting in with my habit of putting any content I produce into my blog what follows is a built out version of it.

So code spelunker started when I noticed someone using Visual Studio to search files recursively in a directory and my own use of ripgrep with fzf. It’s development has been on and off over the last 4 years, roughly broken down into the following categories.

What I'm up to - June 2023

Philip I. Thomas — Mon, 05 Jun 2023 18:23:23 GMT

✨ What I've been up to

It's been a busy month at Contraption Company. I'm amid building a new product called Booklet, which I plan to launch this Summer. It's a community platform designed to be a calmer alternative to Discord and Slack. I've also been helping some startups launch new generative AI products, which has been rewarding.

My 2015 article How Operations Research and Artificial Intelligence Overlap has been picking up traction as AI is in the zeitgeist.

I've been trying to use my Airpods less and be bored more often. It's so easy to plug in and tune out the world.

I got a shout-out in Emma's Indiehackers podcast.

🤔 Things to share

Articles: The Tyranny of Convenience. Yes - "In sum, get out of the house." Ryan Holiday's content marketing strategy, which I'm taking to heart. We stand with the Underdogs. The luxury of building without metrics. It's cheap to build suburbs but expensive to maintain them, creating a "ticking time bomb".
Books: The Pathless Path - highly recommended for capitalist creatives. The Carbon Almanac - data-focused approach to understanding carbon emissions. Watching the English: The Hidden Rules of English Behavior - who doesn't love an entire chapter dedicated to how to talk about the weather, or a section about the "proper" way to eat peas?
Trends: Fractional labor. Everybody in my network seems to be talking about it.
Video: Sonobuoy deploying.
Podcasts: The Infrastructure of Community - "Efficiency is the enemy of social life.”
Gadgets: Flipper Zero is fun.
Coffees I'm drinking: April Pillcocaja - Ecuador - Anaerobic Pacamara (noteworthy!). Kahiwa El Obraje.
Quote: "Startups mostly don't compete against each other, they compete against no one giving a shit" -Justin Kan

📫 What I'm up to this month

With Emma graduating, we're moving back to NYC this month, and will be living in the Lower East Side.

📍 Where I'll be

(Let me know if we overlap!)

June 20-21: Cambridge, ON for Tailwind Connect 🇨🇦

Black truffle pilsner with german chocolate cake was the best thing I ate this month.

Code Spelunker a Code Search Command Line Tool

Ben E. C. Boyter — Mon, 05 Jun 2023 03:00:00 GMT

Code spelunker (cs) or code search is a new command line tool I have been working on and off over the last few years. It allows you to search over code or text files in the current directory either on the console, via a TUI or HTTP server, using some boolean queries or regular expressions. I just recently pushed out a stable first version release and thought I would write a quick post about it.

Getting back into quantifying self for Q3 '23

jodavaho.io — Mon, 05 Jun 2023 03:00:00 GMT

This is the first in a series of posts describing how I’m keeping data about my self, environment, work, and goals. You can read the whole series here

As part of my journalling habit, I used to keep excellent metrics about myself. From the year 2014 to about 2019, I recorded almost everything that I felt was important. This included workouts, spending, and time spent in a variety of areas, such as grad study. I logged each “good” awake hour. This allowed me to generate reports like the following:

The tracked time towards major goals from 2014. Guess when I met my wife? April 2014. And yes, I was trainig to be a fighter at the time, too.

Quick sidebar: You can imagine two ways to get into this. First, you set some goals, then you set up a way to track progress. Complicated goals implies Complicated tracking / reporting. Second way: You just start tracking and make tweaks along the way when you notice too much time or money spent on something. I took the second track back then, and want to again, as otherwise it’s analysis paralysis all the way down.

I’d rather measure and maximize effort (delta-progress / delta-t) than attempt to reach an arbitrarily designed goal.

Let’s break down what I want to track:

Finances
Time spent with Family, improving health, or on engineering-type tasks
Accomplishments vs goals

Let’s just go over finances first, the rest deserve their own posts.

Finances, current system

For the last 1.5 years, I’ve been using ledger to run financial reports. Long story short, it’s a pain to keep the input files synchronized with our actual expenses. It requires manual duplication removal, manual classification, and manual report running. It is nice at quickly querying multiple reports or data sets, but I found that we really only needed an answer for one question: Where did our money go this month?

To get data from financial instutions, I log in, download a custom-date-range csv file, and then run a bash script to convert the csv file to one readable by ledger, then I go transaction-by-transaction:

Is this transaction already in the ledger?
Is the classification for this transaction correct? If not, type “Expenses:Blah” to classify it.
Is the source/ destination account sync’d? If it’s a transfer, I need to go over to the other account’s ledger, and make sure it’s not double counted.

I found:

This is error prone. The double-entry accounting method made it great to know that I had captured all transactions (sum=0), but that wasn’t my primary concern. Instead it created a lot of double-counting, since account A would credit account B with a transfer, but account B would have an incoming transaction from A, and debit A for that amount.
This is tedious. By my current count, we have around 1800 transactions in the ledger for the last 4 months, 100% of which I had to input, classify, and check by hand.
It’s incomprehensible to my wife. The ASCII-column output is wonderful in a hacker sense, but makes no sense to my wife and harms her strong sense of supervisory role. (insert laugh track)

Finances, proposed system

Step 1: Simplify the report generation

Let’s simplify. Ledger is used to parse incoming data and generate interactive reports. The input format is remarkably human readable (look at the example at ledger-cli.org, which tells me the target audience is a somewhat-non-programmer bunch. I am not that audience.

Given that I’m already downloading csv files, let’s just write a jupyter notebook, and use something like viola to render a website that we can view.

Step 2: Simplify the classification of transactions

We’re habitual, most transactions are recurring (Utilities, subscriptions, other bills), or from a few major outlets (Target, Amazon, etc). So, we can get an 80/20 solution by automatically adding a classification column with a few simple rules. The remaining 20% can be tracked as an “other” expense group, and the largest or most frequent charges in that group can easily be plotted with, say, above reporting.

Later, I’d like to teach a neural network to do this form me.

Step 3: Automate the retrieval of data from financial institutions

This is the most complicated step. I’d like a weekly-fetched set of transactions, either for the whole year so I can run the whole batch at once, or to fetch the week’s transactions and append to an ongoing data store.

I’ve looked into ofxtools (readthedocs.org/ofxtools), but some of my financial institutions are either blocking bot access, or it was too difficult for me to configure. This is a solved problem: Plaid will allow developers to fetch transactions, for a modest per-user fee.

Step 4: Preserve ledger output

This is simple, since we import the transaction data, we can just print it out in a ledger-ready format for later use.

Put together

If successful, we’ll have some cron jobs on a tiny computer somewhere in the house:

Fetch year’s financial data from our handful of accounts
Go through and classify most of them using hand-crafted rules, and group the rest for manual inspection
Render a report with a ton of fancy interactive graphs, and a month-to-month and year-to-year status report.
Push that to an internally-accessible website.
Archive the data in an additional ledger-compatible format

I’ll update with how well we’re progressing soon.

You can read the whole series here

‘A person meticulously tracking all the details of their rich life.’ Credit: Midjourney

ChatGPT Summary:

The author discusses their intention to track various aspects of their life, including finances, family time, health improvement, and engineering tasks. They highlight their current system of using ledger-cli for financial reports but find it cumbersome. They propose a simplified system involving Jupyter notebooks, automated classification of transactions, data retrieval from financial institutions, and generating interactive reports. The goal is to create a user-friendly and efficient method for tracking and analyzing personal metrics.

Vim tip 27: regexp anchors

learnbyexample — Mon, 05 Jun 2023 03:00:00 GMT

By default, regexp matches anywhere in the text. You can use line and word anchors to specify additional restrictions regarding the position of matches. These restrictions are made possible by assigning special meaning to certain characters (metacharacters) and escape sequences.

^ restricts the match to the start-of-line
- ^This matches This is a sample but not Do This
$ restricts the match to the end-of-line
- )$ matches apple (5) but not def greeting():
^$ match empty line
\<pattern restricts the match to the start of a word
- word characters include alphabets, digits and underscore
- \<his matches his or to-his or history but not this or _hist
pattern\> restricts the match to the end of a word
- his\> matches his or to-his or this but not history or _hist
\<pattern\> restricts the match between start of a word and end of a word
- \<his\> matches his or to-his but not this or history or _hist

End-of-line can be \r (carriage return), \n (newline) or \r\n depending on your system and fileformat setting.

See :h pattern-atoms for more details.

Video demo:

June Retro

jodavaho.io — Sat, 03 Jun 2023 16:22:56 GMT

Midjourney's rendering of a painting of someone doing a retrospective journal by a lake.

Half way through the year, time for a brief retrospective. Each of these could be a post on their own.

Job

It’s been a little over a year since I left JPL. After a brief hiatus at Amazon, I’m settling into my new job at Outrider.ai. Last year I was laid off from Amazon Robotics, AI, which I don’t think exists any more as an organization. They cited “Marcoeconomic factors” of course. So, the year started with me unemployed, but with some offers on the table. It came down to JPL and Outrider, and I wasn’t very pleased with how I managed that decision making process or juggled multiple interview tracks. I will have to write more about this, because I feel like I burned a few bridges along the way. However, those are two near-perfect options, and it came down to timing: Outrider could move faster with the offer, which helped shore up my dwindling cash reserves.

Outrider has been really fun, in a chaotic, semi-supervised kind of way. I feel like I have a lot of autonomy to work on what I feel is important, room to screw up and fix my own screwups, and plenty of stellar colleagues to learn from. Being a “principal” has been an exercise in restraint, mostly self-imposed, as I tend to go off on technical tangents / crusades that aren’t exactly what leadership wants. For example, screaming and crying about poor meterics. Another time for that story as well…

Hobbies

Most of my hobby time has been taken up with Hunt Showdown. I play regularly with 4 others, and the game has undergone regular content releases that I really appreciate. I’d love to play more Sea of Theives with my nephews!

On Hunt: in addition to the amazingly fun PvE action, Hunt has a match-making-ranking (MMR) system that groups into “Stars”, 1 through 6. You might imagine, there’s a pretty narrow bell curve, roughly corrsponding to mean 4 stars, and +/- 1 standard deviation per star from there. The MMR is based on KDA, or (Kills + Assists)/Deaths, it is weighted by how many stars your opponent has. It’s a fairly involved statistics / estimation problem, which I’d love to write more about, but essentially MMR is increased by kills, decreased by deaths, and not affected by assists. Whereas the KDA rank is improved linearly by kills or assists, but decreased exponentially by deaths. So, my goal of keeping 4 star rank while improving KDA means I have to spend a lot of time assisting (playing on teams and getting assists), or killing low-ranked players (playing solo and going for team wipes). I enjoy both those roles a lot, so it’s a good fit.

On the development side, I made good progress on the highfleet cogitator, thanks as always to my friend Alan Thomas. In my mind, it’s a perfect portfolio project for both of us, since it meshes nicely with my optimization and cloud integration core skills, and Alan’s deep frontend knowledge. Because I read hackernews too much, I had thought of adding a payment option and making a little side money from it, but I doubt that will happen. The Highfleet community is probably a little too niche to target for any kind of monetization, and the project is moving slowly enough that I doubt it’d attract long term payment anyway. I really owe myself a post on the internals and methods, but there’s a few items here.

I peruse hackaday.com and always get a pang of sadness that I’m not doing more with homebrew robotics. Especially since, while living in LA, I had direct access to Supply Frame Design lab. I have had in mind a long term goal of building a submarine, starting with a small pvc pontoon boat. I may have to start that this summer. I’d also love to build more potato guns. These are simple projects with lots of opportunity for nonsensical optimizations like autonomous navigation, underwater mapping, or even beer delivery for subs, and auto-loaders, software controlled firing, aiming, etc for the cannons.

Home / Personal

Life as a homeowner has been challenging. Our back yard is a wasteland of horseweed and dead soil. The hill our house is on is eroding, and needs terracing. I suspect we’ll see a $40,000 quote to get it all sorted out though. But man, do I love having a riding mower and walking out during the weekdays to look for dandelions to kill. I’m really looking forward to getting a functional garage put together, with clean storage, accessible tools, and lots of shelving. One of the most prevalent “discussions” my wife and I have is: Just what to do first.

Side note for another post: I thought marriage would be division of responsibilities, where we’re both CEOs of different parts of the household responsibilities (in a flexible way), but it’s more like a committee of two for every decision, and we both have arbitrarily different priorities. It feels like we’re both micromanagers and “micromanagees” at the same time. I’m reading the book Us, which is interesting in its social commentary, taxing in its narrative-based format, but helping me understand how to deal with this.

Related to tinkering and building, I joined twin cities maker recently. They have a stellar shop, though their electronics workshop is a bit disorganized. Their machine shop, woodshop, and laser cutter selections are very nice. They even have a forge! In addition to working on those boats / guns, I think a good woodworking project will be a playhouse for the girls, which we can keep out back.

Life as a parent to two toddlers couldn’t be more amusing. It’s like living with two goblins/hobbits. Their favorite word by number of uses is “no” by a long shot. Followed by “more”, “cheese”, “please”, not quite in a sentance, but definitely all used for the same goal. Ada has a love of solo play / deep focus, while Noa is a party animal who seems to love chattering and people.

Finally, no human-written retrospective is complete without a “lose weight” goal. In 2014, I was in the best shape of my life. In LA, everyone is looks-oriented to a fault, so when I moved there in 2015, it brought some special treatment when I went out, etc. In the 8 years since then, I’ve tapered off my boxing and working out. Now, as a middle-aged / old man, I’m trying to get back into it. I bought a 20lb weight vest with the goal of accomplishing the murph challenge in a couple years. I went boxing two weeks ago, and really enjoyed it, but have not maganged to return. A rough plan will be 18/6 eating schedules, occasional fasting, some heavy lifting once or twice a week, lighter lifting and running daily, and boxing a few times a month. I’ll see what really hooks me, and drop one or two in favor of that. Of course, I hope it’s boxing that hooks me, but we’ll see.

Conclusion

This has been a rambling, stream of thought post. There’s really no conclusion here except that time flies. If I had written this post in January, it would have been nearly identical. That stagnation is what makes life feel short. I look back and it’s hard to say what I’ve really done, only what I wish I had done. And the wish list gets longer as life goes on.

That’s the real retrospective, I think. I’ve wanted and wanted, and with few exceptions (Highfleet, Job) I’ve not really done much. That needs to change. I think there’s a mindset that if I can just find a day to work on something that I’ll be able to make substantial progress, but over and over that’s proven false - both because I can never find a whole day and because disconnected days are not nearly as productive as they could be. It’s probably much better to work every couple days for an hour or so, so that’s the goal for the rest of the year.

Secrets 4 - Update rugpull

ブラック — Wed, 31 May 2023 03:00:00 GMT

Promise updates, take the money, and then ask for more.

Secrets by Outer Corner is a password manager for macOS and iOS. They just released the latest version of the software, Secrets 4, today and rarely do I feel as outright scammed by software as I do now.

I decided to start using Secrets a couple of months ago after looking for a successor to my previous password manager. It had many good things going for it: it was a native macOS/iOS app, it didn't require a subscription, the data was hosted in iCloud, it had an integration for Windows, and so on.

However, there was one large concern that I had: the last major update was in March 2021, two years prior.

No one wants to buy a product that is unsupported, especially when it comes to software. This is especially important on macOS and iOS side, as Apple is famous for breaking old software in yearly updates. Windows users do not suffer nearly as much from this, as it's not unthinkable to be able to run a Windows 95 application in Windows 11. No wonder Windows developers have been so slow to adopt 64-bit and ARM support.

So I did what any good consumer should do: I dug around some. More specifically, I dug around some on Twitter.

Okay, so Secrets has not seen a big update in over two years, but the developer has made multiple tweets promising an update, and a big one at that. So I would in fact not be buying an abandoned piece of software. So I decided to pull the trigger, and bought both the macOS and iOS versions of Secrets for about 40€ in total.

And then, the developer abandoned the software.

On May 18th, 2023, Outer Corner announced that Secrets 4, the major update being promised since last year, would be a paid upgrade. But to soothe old users, existing users would be promised a "considerable discount". "Okay", I figured, "I guess I need to spend a couple of euro more for this big update."

Today, Secrets 4 was released. The full and complete version of the universal app was offered to existing users as a launch day special for the low, low price of… 50€. That's after the considerable 50% discount. If you wanted to upgrade to the basic version of Secrets 4, you could get it for the low, low price of… 50€ again.

So, having paid 40€ for Secrets just two months prior with a promise of a big update, I am now being asked for somewhere between 50€ and 100€ to get that major update that was promised and advertised multiple times.

Ever since I bought Secrets, I have received a single update. The macOS version of Secrets was updated to version 3.8.5 about a month ago, which "fixed a couple of compatibility issues with macOS Ventura". The iOS version has not been updated in 10 months as of writing.

The developer Paulo Andrade has been deflecting criticism of the paid upgrade by telling people that Secrets users have been enjoying free updates for seven years. Good for those users, I guess. I sure didn't! The update that was coming my way, has been turned into a paid upgrade. I just got that single-sentence update in the two months.

You could argue that "update" does not mean that it's free of charge. However, I feel like "update" is an update to the existing software, and not a completely new application on the App Store and a separate purchase. And I feel like Outer Corner agrees with me, considering the announcement post for Secrets 4 only talks about an "upgrade", not an "update".

It's not like Secrets 3 is a perfect password manager as it is. I'm currently suffering from random freezes when invoking Secrets from the Firefox extension, the image cropper in the application does absolutely nothing, and the image rendering for the thumbnails on macOS is garbage. I always just figured that since the update was coming, I'd just bear with it and hope that the update fixes everything. Had I been told that the version of Secrets would be the end of the road, I'd probably have refunded my purchase within the 14-day cooling off period. Or just not upgraded from the trial to begin with.

Technically the version of Secrets that I have is not abandoned. Completely abandoned, that is. The developer has that they'll support Secrets 3 for "the near future". What is the definition of "near future"? Absolutely no idea, the developer refused to elaborate. My expectations of the future after all of this are very low. I imagine I will be relying on the good graces of Apple to not kill any of the API that Secrets 3 uses in order to use it on future versions of macOS and iOS, and not Outer Corner's support.

I don't like how almost every paid application these days is shifting to a subscription system. That was one of the reasons why I chose Secrets after all. However, the situation with Secrets is even worse than subscriptions – I'm currently doing like 20€ per month of active support, considering that Secrets 3 is now unceremoniously on life support. Secrets has been awful value for money. Even the Photoshop + Lightroom bundle is less than 20€/month, and Adobe software aren't known for being a bargain (at least for non-professionals).

I do understand that Secrets is made by a small indie development company whose developer needs to eat. However, I find the practice of selling an already-axed version of a software without any kind of a warning to be anti-consumer. I feel like companies have two very good alternatives to the way Outer Corner did things: offer a free upgrade to anyone who bought in the last [appropriate amount of time], or make it very clear that you are buying an unsupported product with no future.

If it's not clear already, I think you should not buy Secrets. But if you do, don't expect updates.

CLI tip 28: substitute specific occurrence with GNU sed

learnbyexample — Tue, 30 May 2023 03:00:00 GMT

By using the g flag with the s command (substitute), you can search and replace all occurrences of a pattern. Without the g flag, only the first matching portion will be replaced.

Did you know that you can use a number as a flag to replace only that particular occurrence of matching parts?

# replace only the third occurrence of ':' with '---'
$ echo 'apple:banana:cherry:fig:mango' | sed 's/:/---/3'
apple:banana:cherry---fig:mango

# replace only the second occurrence of a word starting with 't'
$ echo 'book table bus car banana tap camp' | sed 's/\bt\w*/"&"/2'
book table bus car banana "tap" camp

To replace a specific occurrence from the end of the line, you'll have use regular expression tricks:

$ s='apple:banana:cherry:fig:mango'

# replace the last occurrence
$ echo "$s" | sed -E 's/(.*):/\1---/'
apple:banana:cherry:fig---mango

# replace the last but one occurrence
$ echo "$s" | sed -E 's/(.*):(.*:)/\1---\2/'
apple:banana:cherry---fig:mango

# generic formula, where {N} refers to the last but Nth occurrence
$ echo "$s" | sed -E 's/(.*):((.*:){2})/\1---\2/'
apple:banana---cherry:fig:mango

If you combine a number flag with the g flag, all matches from that particular occurrence will be replaced.

# replace except the first occurrence of a word starting with 'b'
$ echo 'book table bus car banana tap camp' | sed 's/\bb\w*/"&"/2g'
book table "bus" car "banana" tap camp

Video demo:

See also my CLI text processing with GNU sed ebook.

Learn how to build tools

Letters To A New Developer — Mon, 29 May 2023 16:57:24 GMT

This is a guest post from Jamie Tanna. Enjoy. Dear new developer, Our jobs and lives are full of repetition, and one of the beauties of being developers is that we can take steps to automate away some of the repetition. Learning to automate, or at least minimise, repetition optimises your work. You can get … Continue reading →

Defaulting git clone to shallow (depth=1)

smcleod.net — Mon, 29 May 2023 01:53:24 GMT

Before adding this to my shell config, I would manually add –depth=1 to all my git clones.

2023–05–29: Touch screen and touch menu support for Pinephone Pro in U-Boot

megi's PinePhone Development Log — Mon, 29 May 2023 01:00:00 GMT

2023–05–28: Some regressions in power use in system sleep on original Pinephone

megi's PinePhone Development Log — Sun, 28 May 2023 01:00:00 GMT

2023–05–27: Implementing display support in U-Boot for Pinephone Pro

megi's PinePhone Development Log — Sat, 27 May 2023 01:00:00 GMT

How likely would you be to block a company from asking you to rate everything they do?

smcleod.net — Fri, 26 May 2023 09:41:37 GMT

NPS Surveys are corporate spam

Bastion review

ブラック — Fri, 26 May 2023 03:00:00 GMT

Time for another blast from the past, this time Bastion from 2011.

I actually played through Bastion for the first time way back in 2012, when it took me about 12 hours to beat the game and do whatever side tasks I did back then. A decade older and wiser, I got through my first playthrough in under eight hours and mopped the rest of the achievements in another ten.

Turns out that Bastion is still a nice game.

You play as The Kid, a silent protagonist who has to smash and fight through a bunch of isometric stages that build around you as you move about the place. The core gameplay loop of Bastion revolves around combat, which is handled using melee and ranged attacks, a shield and special attacks. There is a handful of weapons to choose from, which unlock as you get through the story, and you get a choice of upgrades to further custoize your combat experience.

The combat itself is not particularly challenging, although you definitely can make things harder for yourself using the Idol system. About the Author, which requires you to defeat 30 waves of enemies in the Stranger's Dream while using all of the hardenings available, definitely gives you a run for your money. Even though I managed to beat it in two attempts, the first attempt failing at the last wave, it was not without its share of panic, screaming and aching arms. But if you don't reach for the Idols, combat's more on the relaxing side of the difficulty spectrum.

Oh, and if you decide to go for About the Author: Galleon Mortar and dodge roll, dodge roll, dodge roll.

There is a fairly short and basic story in the game, although it's not particularly gripping. Most of the interesting stuff in the Bastion universe happened before you even came along, and you're mostly told about it from the narrator. If you were hoping for a story-heavy game, you might be disappointed, but it's still a decent, somber narrative.

Bastion's art is highly stylized and gorgeous, although not perfect; due to the isometric perspective and the way the levels are designed, it's hard to tell which parts of the level you can actually access and which you can't. It's also not the easiest to avoid falling entirely off the level, which will snip some off your health bar as a punishment. Still, the overall quality of the art definitely offsets those minor annoyances.

The sound design in the game is absolutely top notch. I'd like to meet the person who doesn't enjoy Logan Cunningham's voice-over narration, which there is a lot. And Bastion's soundtrack ranks in some of the best video game soundtracks that I know of. It's the kind of stuff that you'll want to listen outside the game as well. I know I definitely have.

For my 2023 playthrough of Bastion, I did it entirely on my Steam Deck, as my gaming PC was actually out of commission for the entirety of it. And there is good news on that front: Bastion has a native Linux version and works absolutely flawlessly on SteamOS. Since this game is designed for a controller and is not graphically demanding, it makes for a perfect Steam Deck title.

Considering for how little money you can pick up Bastion these days, and the abundance of good things about it, avoiding it would be foolish if you feel like it would appeal to you in any way.

My 2023 New Mac Setup

swyx's site RSS Feed — Thu, 25 May 2023 21:51:04 GMT

I set up a new Mac for work today. Here's everything I use on a Mac for fullstack web development.

Implementing the Raft distributed consensus protocol in Go

Notes on software development — Thu, 25 May 2023 03:00:00 GMT

As part of bringing myself up-to-speed after joining TigerBeetle, I wanted some background on how distributed consensus and replicated state machines protocols work. TigerBeetle uses Viewstamped Replication. But I wanted to understand all popular protocols and I decided to start with Raft.

We'll implement two key components of Raft in this post (leader election and log replication). Around 1k lines of Go. It took me around 7 months of sporadic studying to come to (what I hope is) an understanding of the basics.

Disclaimer: I'm not an expert. My implementation isn't yet hooked up to Jepsen. I've run it through a mix of manual and automated tests and it seems generally correct. This is not intended to be used in production. It's just for my education.

All code for this project is available on GitHub.

Let's dig in!

The algorithm

The Raft paper itself is quite readable. Give it a read and you'll get the basic idea.

The gist is that nodes in a cluster conduct elections to pick a leader. Users of the Raft cluster send messages to the leader. The leader passes the message to followers and waits for a majority to store the message. Once the message is committed (majority consensus has been reached), the message is applied to a state machine the user supplies. Followers learn about the latest committed message from the leader and apply each new committed message to their local user-supplied state machine.

There's more to it including reconfiguration and snapshotting, which I won't get into in this post. But you can get the gist of Raft by thinking about 1) leader election and 2) replicated logs powering replicated state machines.

Modeling with state machines and key-value stores

I've written before about how you can build a key-value store on top of Raft. How you can build a SQL database on top of a key-value store. And how you can build a distributed SQL database on top of Raft.

This post will start quite similarly to that first post except for that we won't stop at the Raft layer.

A distributed key-value store

To build on top of the Raft library we'll build, we need to create a state machine and commands that are sent to the state machine.

Our state machine will have two operations: get a value from a key, and set a key to a value.

This will go in cmd/kvapi/main.go.

package main

import (
    "bytes"
    crypto "crypto/rand"
    "encoding/binary"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"

    "github.com/eatonphil/goraft"
)

type statemachine struct {
    db     *sync.Map
    server int
}

type commandKind uint8

const (
    setCommand commandKind = iota
    getCommand
)

type command struct {
    kind  commandKind
    key   string
    value string
}

func (s *statemachine) Apply(cmd []byte) ([]byte, error) {
    c := decodeCommand(cmd)

    switch c.kind {
    case setCommand:
        s.db.Store(c.key, c.value)
    case getCommand:
        value, ok := s.db.Load(c.key)
        if !ok {
            return nil, fmt.Errorf("Key not found")
        }
        return []byte(value.(string)), nil
    default:
        return nil, fmt.Errorf("Unknown command: %x", cmd)
    }

    return nil, nil
}

But the Raft library we'll build needs to deal with various state machines. So commands passed from the user into the Raft cluster must be serialized to bytes.

func encodeCommand(c command) []byte {
    msg := bytes.NewBuffer(nil)
    err := msg.WriteByte(uint8(c.kind))
    if err != nil {
        panic(err)
    }

    err = binary.Write(msg, binary.LittleEndian, uint64(len(c.key)))
    if err != nil {
        panic(err)
    }

    msg.WriteString(c.key)

    err = binary.Write(msg, binary.LittleEndian, uint64(len(c.value)))
    if err != nil {
        panic(err)
    }

    msg.WriteString(c.value)

    return msg.Bytes()
}

And the Apply() function from above needs to be able to decode the bytes:

func decodeCommand(msg []byte) command {
    var c command
    c.kind = commandKind(msg[0])

    keyLen := binary.LittleEndian.Uint64(msg[1:9])
    c.key = string(msg[9 : 9+keyLen])

    if c.kind == setCommand {
        valLen := binary.LittleEndian.Uint64(msg[9+keyLen : 9+keyLen+8])
        c.value = string(msg[9+keyLen+8 : 9+keyLen+8+valLen])
    }

    return c
}

HTTP API

Now that we've modeled the key-value store as a state machine. Let's build the HTTP endpoints that allow the user to operate the state machine through the Raft cluster.

First, let's implement the set operation. We need to grab the key and value the user passes in and call Apply() on the Raft cluster. Calling Apply() on the Raft cluster will eventually call the Apply() function we just wrote, but not until the message sent to the Raft cluster is actually replicated.

type httpServer struct {
    raft *goraft.Server
    db   *sync.Map
}

// Example:
//
//  curl http://localhost:2020/set?key=x&value=1
func (hs httpServer) setHandler(w http.ResponseWriter, r *http.Request) {
    var c command
    c.kind = setCommand
    c.key = r.URL.Query().Get("key")
    c.value = r.URL.Query().Get("value")

    _, err := hs.raft.Apply([][]byte{encodeCommand(c)})
    if err != nil {
        log.Printf("Could not write key-value: %s", err)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }
}

To reiterate, we tell the Raft cluster we want this message replicated. The message contains the operation type (set) and the operation details (key and value). These messages are custom to the state machine we wrote. And they will be interpreted by the state machine we wrote, on each node in the cluster.

Next we handle get-ing values from the cluster. There are two ways to do this. We already embed a local copy of the distributed key-value map. We could just read from that map in the current process. But it might not be up-to-date or correct. It would be fast to read though. And convenient for debugging.

But the only correct way to read from a Raft cluster is to pass the read through the log replication too.

So we'll support both.

// Example:
//
//  curl http://localhost:2020/get?key=x
//  1
//  curl http://localhost:2020/get?key=x&relaxed=true # Skips consensus for the read.
//  1
func (hs httpServer) getHandler(w http.ResponseWriter, r *http.Request) {
    var c command
    c.kind = getCommand
    c.key = r.URL.Query().Get("key")

    var value []byte
    var err error
    if r.URL.Query().Get("relaxed") == "true" {
        v, ok := hs.db.Load(c.key)
        if !ok {
            err = fmt.Errorf("Key not found")
        } else {
            value = []byte(v.(string))
        }
    } else {
        var results []goraft.ApplyResult
        results, err = hs.raft.Apply([][]byte{encodeCommand(c)})
        if err == nil {
            if len(results) != 1 {
                err = fmt.Errorf("Expected single response from Raft, got: %d.", len(results))
            } else if results[0].Error != nil {
                err = results[0].Error
            } else {
                value = results[0].Result
            }

        }
    }

    if err != nil {
        log.Printf("Could not encode key-value in http response: %s", err)
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return
    }

    written := 0
    for written < len(value) {
        n, err := w.Write(value[written:])
        if err != nil {
            log.Printf("Could not encode key-value in http response: %s", err)
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            return
        }

        written += n
    }
}

Main

Now that we've set up our custom state machine and our HTTP API for interacting with the Raft cluster, we'll tie it together with reading configuration from the command-line and actually starting the Raft node and the HTTP API.

type config struct {
    cluster []goraft.ClusterMember
    index   int
    id      string
    address string
    http    string
}

func getConfig() config {
    cfg := config{}
    var node string
    for i, arg := range os.Args[1:] {
        if arg == "--node" {
            var err error
            node = os.Args[i+2]
            cfg.index, err = strconv.Atoi(node)
            if err != nil {
                log.Fatal("Expected $value to be a valid integer in `--node $value`, got: %s", node)
            }
            i++
            continue
        }

        if arg == "--http" {
            cfg.http = os.Args[i+2]
            i++
            continue
        }

        if arg == "--cluster" {
            cluster := os.Args[i+2]
            var clusterEntry goraft.ClusterMember
            for _, part := range strings.Split(cluster, ";") {
                idAddress := strings.Split(part, ",")
                var err error
                clusterEntry.Id, err = strconv.ParseUint(idAddress[0], 10, 64)
                if err != nil {
                    log.Fatal("Expected $id to be a valid integer in `--cluster $id,$ip`, got: %s", idAddress[0])
                }

                clusterEntry.Address = idAddress[1]
                cfg.cluster = append(cfg.cluster, clusterEntry)
            }

            i++
            continue
        }
    }

    if node == "" {
        log.Fatal("Missing required parameter: --node $index")
    }

    if cfg.http == "" {
        log.Fatal("Missing required parameter: --http $address")
    }

    if len(cfg.cluster) == 0 {
        log.Fatal("Missing required parameter: --cluster $node1Id,$node1Address;...;$nodeNId,$nodeNAddress")
    }

    return cfg
}

func main() {
    var b [8]byte
    _, err := crypto.Read(b[:])
    if err != nil {
        panic("cannot seed math/rand package with cryptographically secure random number generator")
    }
    rand.Seed(int64(binary.LittleEndian.Uint64(b[:])))

    cfg := getConfig()

    var db sync.Map

    var sm statemachine
    sm.db = &db
    sm.server = cfg.index

    s := goraft.NewServer(cfg.cluster, &sm, ".", cfg.index)
    go s.Start()

    hs := httpServer{s, &db}

    http.HandleFunc("/set", hs.setHandler)
    http.HandleFunc("/get", hs.getHandler)
    err = http.ListenAndServe(cfg.http, nil)
    if err != nil {
        panic(err)
    }
}

And that's it for the easy part: a distributed key-value store on top of a Raft cluster.

Next we need to implement Raft.

A Raft server

If we take a look at Figure 2 in the Raft paper, we get an idea for all the state we need to model.

We'll dig into the details as we go. But for now let's turn that model into a few Go types. This goes in raft.go in the base directory, not cmd/kvapi.

package goraft

import (
    "bufio"
        "context"
    "encoding/binary"
    "errors"
    "fmt"
    "io"
    "math/rand"
    "net"
    "net/http"
    "net/rpc"
    "os"
    "path"
    "sync"
    "time"
)

type StateMachine interface {
    Apply(cmd []byte) ([]byte, error)
}

type ApplyResult struct {
    Result []byte
    Error  error
}

type Entry struct {
    Command []byte
    Term    uint64

    // Set by the primary so it can learn about the result of
    // applying this command to the state machine
    result  chan ApplyResult
}

type ClusterMember struct {
    Id      uint64
    Address string

    // Index of the next log entry to send
    nextIndex uint64
    // Highest log entry known to be replicated
    matchIndex uint64

    // Who was voted for in the most recent term
    votedFor uint64

    // TCP connection
    rpcClient *rpc.Client
}

type ServerState string

const (
    leaderState    ServerState = "leader"
    followerState              = "follower"
    candidateState             = "candidate"
)

type Server struct {
    // These variables for shutting down.
    done bool
        server *http.Server

    Debug bool

    mu sync.Mutex
    // ----------- PERSISTENT STATE -----------

    // The current term
    currentTerm uint64

    log []Entry

    // votedFor is stored in `cluster []ClusterMember` below,
    // mapped by `clusterIndex` below

    // ----------- READONLY STATE -----------

    // Unique identifier for this Server
    id uint64

    // The TCP address for RPC
    address string

    // When to start elections after no append entry messages
    electionTimeout time.Time

    // How often to send empty messages
    heartbeatMs int

    // When to next send empty message
    heartbeatTimeout time.Time

    // User-provided state machine
    statemachine StateMachine

    // Metadata directory
    metadataDir string

    // Metadata store
    fd *os.File

    // ----------- VOLATILE STATE -----------

    // Index of highest log entry known to be committed
    commitIndex uint64

    // Index of highest log entry applied to state machine
    lastApplied uint64

    // Candidate, follower, or leader
    state ServerState

    // Servers in the cluster, including this one
    cluster []ClusterMember

    // Index of this server
    clusterIndex int
}

And let's build a constructor to initialize the state for all servers in the cluster, as well as local server state.

func NewServer(
    clusterConfig []ClusterMember,
    statemachine StateMachine,
    metadataDir string,
    clusterIndex int,
) *Server {
    // Explicitly make a copy of the cluster because we'll be
    // modifying it in this server.
    var cluster []ClusterMember
    for _, c := range clusterConfig {
        if c.Id == 0 {
            panic("Id must not be 0.")
        }
        cluster = append(cluster, c)
    }

    return &Server{
        id:           cluster[clusterIndex].Id,
        address:      cluster[clusterIndex].Address,
        cluster:      cluster,
        statemachine: statemachine,
        metadataDir:  metadataDir,
        clusterIndex: clusterIndex,
        heartbeatMs:  300,
        mu:           sync.Mutex{},
    }
}

And add a few debugging and assertion helpers.

func (s *Server) debugmsg(msg string) string {
    return fmt.Sprintf("%s [Id: %d, Term: %d] %s", time.Now().Format(time.RFC3339Nano), s.id, s.currentTerm, msg)
}

func (s *Server) debug(msg string) {
    if !s.Debug {
        return
    }
    fmt.Println(s.debugmsg(msg))
}

func (s *Server) debugf(msg string, args ...any) {
    if !s.Debug {
        return
    }

    s.debug(fmt.Sprintf(msg, args...))
}

func (s *Server) warn(msg string) {
    fmt.Println("[WARN] " + s.debugmsg(msg))
}

func (s *Server) warnf(msg string, args ...any) {
    fmt.Println(fmt.Sprintf(msg, args...))
}

func Assert[T comparable](msg string, a, b T) {
    if a != b {
        panic(fmt.Sprintf("%s. Got a = %#v, b = %#v", msg, a, b))
    }
}

func Server_assert[T comparable](s *Server, msg string, a, b T) {
    Assert(s.debugmsg(msg), a, b)
}

Persistent state

As Figure 2 says, currentTerm, log, and votedFor must be persisted to disk as they're edited.

I like to initially doing the stupidest thing possible. So in the first version of this project I used encoding/gob to write these three fields to disk every time s.persist() was called.

Here is what this first version looked like:

func (s *Server) persist() {
    s.mu.Lock()
    defer s.mu.Unlock()

    s.fd.Truncate(0)
    s.fd.Seek(0, 0)
    enc := gob.NewEncoder(s.fd)
    err := enc.Encode(PersistentState{
        CurrentTerm: s.currentTerm,
        Log:         s.log,
        VotedFor:    s.votedFor,
    })
    if err != nil {
        panic(err)
    }
    if err = s.fd.Sync(); err != nil {
        panic(err)
    }
    s.debug(fmt.Sprintf("Persisted. Term: %d. Log Len: %d. Voted For: %s.", s.currentTerm, len(s.log), s.votedFor))
}

But doing so means this implementation is a function of the size of the log. And that was horrible for throughput.

I also noticed that encoding/gob is pretty inefficient.

For a simple struct like:

type X struct {
    A uint64
    B []uint64
    C bool
}

encoding/gob uses 68 bytes to store that data for when B has two entries. If we wrote the encoder/decoder ourselves we could store that struct in 33 bytes (8 (sizeof(A)) + 8 (sizeof(len(B))) + 16 (len(B) * sizeof(B)) + 1 (sizeof(C))).

It's not that encoding/gob is bad. It just likely has different constraints than we are party to.

So I decided to swap out encoding/gob for simply binary encoding the fields and also, importantly, keeping track of exactly how many entries in the log must be written and only writing that many.

`s.persist()`

Here's what that looks like.

const PAGE_SIZE = 4096
const ENTRY_HEADER = 16
const ENTRY_SIZE = 128

// Must be called within s.mu.Lock()
func (s *Server) persist(writeLog bool, nNewEntries int) {
    t := time.Now()

    if nNewEntries == 0 && writeLog {
        nNewEntries = len(s.log)
    }

    s.fd.Seek(0, 0)

    var page [PAGE_SIZE]byte
    // Bytes 0  - 8:   Current term
    // Bytes 8  - 16:  Voted for
    // Bytes 16 - 24:  Log length
    // Bytes 4096 - N: Log

    binary.LittleEndian.PutUint64(page[:8], s.currentTerm)
    binary.LittleEndian.PutUint64(page[8:16], s.getVotedFor())
    binary.LittleEndian.PutUint64(page[16:24], uint64(len(s.log)))
    n, err := s.fd.Write(page[:])
    if err != nil {
        panic(err)
    }
    Server_assert(s, "Wrote full page", n, PAGE_SIZE)

    if writeLog && nNewEntries > 0 {
        newLogOffset := max(len(s.log)-nNewEntries, 0)

        s.fd.Seek(int64(PAGE_SIZE+ENTRY_SIZE*newLogOffset), 0)
        bw := bufio.NewWriter(s.fd)

        var entryBytes [ENTRY_SIZE]byte
        for i := newLogOffset; i < len(s.log); i++ {
            // Bytes 0 - 8:    Entry term
            // Bytes 8 - 16:   Entry command length
            // Bytes 16 - ENTRY_SIZE: Entry command

            if len(s.log[i].Command) > ENTRY_SIZE-ENTRY_HEADER {
                panic(fmt.Sprintf("Command is too large (%d). Must be at most %d bytes.", len(s.log[i].Command), ENTRY_SIZE-ENTRY_HEADER))
            }

            binary.LittleEndian.PutUint64(entryBytes[:8], s.log[i].Term)
            binary.LittleEndian.PutUint64(entryBytes[8:16], uint64(len(s.log[i].Command)))
            copy(entryBytes[16:], []byte(s.log[i].Command))

            n, err := bw.Write(entryBytes[:])
            if err != nil {
                panic(err)
            }
            Server_assert(s, "Wrote full page", n, ENTRY_SIZE)
        }

        err = bw.Flush()
        if err != nil {
            panic(err)
        }
    }

    if err = s.fd.Sync(); err != nil {
        panic(err)
    }
    s.debugf("Persisted in %s. Term: %d. Log Len: %d (%d new). Voted For: %d.", time.Now().Sub(t), s.currentTerm, len(s.log), nNewEntries, s.getVotedFor())
}

Again the important thing is that only the entries that need to be written are written. We do that by seek-ing to the offset of the first entry that needs to be written.

And we collect writes of entries in a bufio.Writer so we don't waste write syscalls. Don't forget to flush the buffered writer!

And don't forget to flush all writes to disk with fd.Sync().

ENTRY_SIZE is something that I could see being configurable based on the workload. Some workloads truly need only 128 bytes. But a key-value store probably wants much more than that. This implementation doesn't try to handle the case of completely arbitrary sized keys and values.

Lastly, a few helpers used in there:

func min[T ~int | ~uint64](a, b T) T {
    if a < b {
        return a
    }

    return b
}

func max[T ~int | ~uint64](a, b T) T {
    if a > b {
        return a
    }

    return b
}

// Must be called within s.mu.Lock()
func (s *Server) getVotedFor() uint64 {
    for i := range s.cluster {
        if i == s.clusterIndex {
            return s.cluster[i].votedFor
        }
    }

    Server_assert(s, "Invalid cluster", true, false)
    return 0
}

`s.restore()`

Now let's do the reverse operation, restoring from disk. This will only be called once on startup.

func (s *Server) restore() {
    s.mu.Lock()
    defer s.mu.Unlock()

    if s.fd == nil {
        var err error
        s.fd, err = os.OpenFile(
            path.Join(s.metadataDir, fmt.Sprintf("md_%d.dat", s.id)),
            os.O_SYNC|os.O_CREATE|os.O_RDWR,
            0755)
        if err != nil {
            panic(err)
        }
    }

    s.fd.Seek(0, 0)

    // Bytes 0  - 8:   Current term
    // Bytes 8  - 16:  Voted for
    // Bytes 16 - 24:  Log length
    // Bytes 4096 - N: Log
    var page [PAGE_SIZE]byte
    n, err := s.fd.Read(page[:])
    if err == io.EOF {
        s.ensureLog()
        return
    } else if err != nil {
        panic(err)
    }
    Server_assert(s, "Read full page", n, PAGE_SIZE)

    s.currentTerm = binary.LittleEndian.Uint64(page[:8])
    s.setVotedFor(binary.LittleEndian.Uint64(page[8:16]))
    lenLog := binary.LittleEndian.Uint64(page[16:24])
    s.log = nil

    if lenLog > 0 {
        s.fd.Seek(int64(PAGE_SIZE), 0)

        var e Entry
        for i := 0; uint64(i) < lenLog; i++ {
            var entryBytes [ENTRY_SIZE]byte
            n, err := s.fd.Read(entryBytes[:])
            if err != nil {
                panic(err)
            }
            Server_assert(s, "Read full entry", n, ENTRY_SIZE)

            // Bytes 0 - 8:    Entry term
            // Bytes 8 - 16:   Entry command length
            // Bytes 16 - ENTRY_SIZE: Entry command
            e.Term = binary.LittleEndian.Uint64(entryBytes[:8])
            lenValue := binary.LittleEndian.Uint64(entryBytes[8:16])
            e.Command = entryBytes[16 : 16+lenValue]
            s.log = append(s.log, e)
        }
    }

    s.ensureLog()
}

And a few helpers it calls:

func (s *Server) ensureLog() {
    if len(s.log) == 0 {
        // Always has at least one log entry.
        s.log = append(s.log, Entry{})
    }
}

// Must be called within s.mu.Lock()
func (s *Server) setVotedFor(id uint64) {
    for i := range s.cluster {
        if i == s.clusterIndex {
            s.cluster[i].votedFor = id
            return
        }
    }

    Server_assert(s, "Invalid cluster", true, false)
}

The main loop

Now let's think about the main loop. Before starting the loop we need to 1) restore persistent state from disk and 2) kick off an RPC server so servers in the cluster can send and receive messages to and from eachother.

// Make sure rand is seeded
func (s *Server) Start() {
    s.mu.Lock()
    s.state = followerState
    s.done = false
    s.mu.Unlock()

    s.restore()

    rpcServer := rpc.NewServer()
    rpcServer.Register(s)
    l, err := net.Listen("tcp", s.address)
    if err != nil {
        panic(err)
    }
    mux := http.NewServeMux()
    mux.Handle(rpc.DefaultRPCPath, rpcServer)

    s.server = &http.Server{Handler: mux}
    go s.server.Serve(l)

    go func() {
        s.mu.Lock()
        s.resetElectionTimeout()
        s.mu.Unlock()

        for {
            s.mu.Lock()
            if s.done {
                s.mu.Unlock()
                return
            }
            state := s.state
            s.mu.Unlock()

In the main loop we are either in the leader state, follower state or candidate state.

All states will potentially receive RPC messages from other servers in the cluster but that won't be modeled in this main loop.

The only thing going on in the main loop is that:

We send heartbeat RPCs (leader state)
We try to advance the commit index (leader state only) and apply commands to the state machine (leader and follower states)
We trigger a new election if we haven't received a message in some time (candidate and follower states)
Or we become the leader (candidate state)

            switch state {
            case leaderState:
                s.heartbeat()
                s.advanceCommitIndex()
            case followerState:
                s.timeout()
                s.advanceCommitIndex()
            case candidateState:
                s.timeout()
                s.becomeLeader()
            }
        }
    }()
}

Let's deal with leader election first.

Leader election

Leader election happens every time nodes haven't received a message from a valid leader in some time.

I'll break this up into four major pieces:

Timing out and becoming a candidate after a random (but bounded) period of time of not hearing a message from a valid leader: s.timeout().
The candidate requests votes from all other servers: s.requestVote().
All servers handle vote requests: s.HandleRequestVoteRequest().
A candidate with a quorum of vote requests becomes the leader: s.becomeLeader().

You increment currentTerm, vote for yourself and send RPC vote requests to other nodes in the server.

func (s *Server) resetElectionTimeout() {
    interval := time.Duration(rand.Intn(s.heartbeatMs*2) + s.heartbeatMs*2)
    s.debugf("New interval: %s.", interval*time.Millisecond)
    s.electionTimeout = time.Now().Add(interval * time.Millisecond)
}

func (s *Server) timeout() {
    s.mu.Lock()
    defer s.mu.Unlock()

    hasTimedOut := time.Now().After(s.electionTimeout)
    if hasTimedOut {
        s.debug("Timed out, starting new election.")
        s.state = candidateState
        s.currentTerm++
        for i := range s.cluster {
            if i == s.clusterIndex {
                s.cluster[i].votedFor = s.id
            } else {
                s.cluster[i].votedFor = 0
            }
        }

        s.resetElectionTimeout()
        s.persist(false, 0)
        s.requestVote()
    }
}

Everything in there is implemented already except for s.requestVote(). Let's dig into that.

`s.requestVote()`

By referring back to Figure 2 from the Raft paper we can see how to model the request vote request and response. Let's turn that into some Go types.

type RPCMessage struct {
    Term uint64
}

type RequestVoteRequest struct {
    RPCMessage

    // Candidate requesting vote
    CandidateId uint64

    // Index of candidate's last log entry
    LastLogIndex uint64

    // Term of candidate's last log entry
    LastLogTerm uint64
}

type RequestVoteResponse struct {
    RPCMessage

    // True means candidate received vote
    VoteGranted bool
}

Now we just need to fill the RequestVoteRequest struct out and send it to each other node in the cluster in parallel. As we iterate through nodes in the cluster, we skip ourselves (we always immediately vote for ourselves).

func (s *Server) requestVote() {
    for i := range s.cluster {
        if i == s.clusterIndex {
            continue
        }

        go func(i int) {
            s.mu.Lock()

            s.debugf("Requesting vote from %d.", s.cluster[i].Id)

            lastLogIndex := uint64(len(s.log) - 1)
            lastLogTerm := s.log[len(s.log)-1].Term

            req := RequestVoteRequest{
                RPCMessage: RPCMessage{
                    Term: s.currentTerm,
                },
                CandidateId:  s.id,
                LastLogIndex: lastLogIndex,
                LastLogTerm:  lastLogTerm,
            }
            s.mu.Unlock()

            var rsp RequestVoteResponse
            ok := s.rpcCall(i, "Server.HandleRequestVoteRequest", req, &rsp)
            if !ok {
                // Will retry later
                return
            }

Now remember from Figure 2 in the Raft paper that we must always check that the RPC request and response is still valid. If the term of the response is greater than our own term, we must immediately stop processing and revert to follower state.

Otherwise only if the response is still relevant to us at the moment (the response term is the same as the request term) and the request has succeeded do we count the vote.

            s.mu.Lock()
            defer s.mu.Unlock()

            if s.updateTerm(rsp.RPCMessage) {
                return
            }

            dropStaleResponse := rsp.Term != req.Term
            if dropStaleResponse {
                return
            }

            if rsp.VoteGranted {
                s.debugf("Vote granted by %d.", s.cluster[i].Id)
                s.cluster[i].votedFor = s.id
            }
        }(i)
    }
}

And that's it for the candidate side of requesting a vote.

The implementation of s.updateTerm() is simple. It just takes care of transitioning to follower state if the term of an RPC message is greater than the node's current term.

// Must be called within a s.mu.Lock()
func (s *Server) updateTerm(msg RPCMessage) bool {
    transitioned := false
    if msg.Term > s.currentTerm {
        s.currentTerm = msg.Term
        s.state = followerState
        s.setVotedFor(0)
        transitioned = true
        s.debug("Transitioned to follower")
        s.resetElectionTimeout()
        s.persist(false, 0)
    }
    return transitioned
}

And the implementation of s.rpcCall() is a wrapper around net/rpc to lazily connect.

func (s *Server) rpcCall(i int, name string, req, rsp any) bool {
    s.mu.Lock()
    c := s.cluster[i]
    var err error
    var rpcClient *rpc.Client = c.rpcClient
    if c.rpcClient == nil {
        c.rpcClient, err = rpc.DialHTTP("tcp", c.Address)
        rpcClient = c.rpcClient
    }
    s.mu.Unlock()

    // TODO: where/how to reconnect if the connection must be reestablished?

    if err == nil {
        err = rpcClient.Call(name, req, rsp)
    }

    if err != nil {
        s.warnf("Error calling %s on %d: %s.", name, c.Id, err)
    }

    return err == nil
}

Let's dig into the other side of request vote, what happens when a node receives a vote request?

`s.HandleVoteRequest()`

First off, as discussed above, we must always check the RPC term versus our own and revert to follower if the term is greater than our own. (Remember that since this is an RPC request it could come to a server in any state: leader, candidate, or follower.)

func (s *Server) HandleRequestVoteRequest(req RequestVoteRequest, rsp *RequestVoteResponse) error {
    s.mu.Lock()
    defer s.mu.Unlock()

    s.updateTerm(req.RPCMessage)

    s.debugf("Received vote request from %d.", req.CandidateId)

Then we can return immediately if the request term is lower than our own (that means it's an old request).

    rsp.VoteGranted = false
    rsp.Term = s.currentTerm

    if req.Term < s.currentTerm {
        s.debugf("Not granting vote request from %d.", req.CandidateId)
        Server_assert(s, "VoteGranted = false", rsp.VoteGranted, false)
        return nil
    }

And finally, we check to make sure the requester's log is at least as up-to-date as our own and that we haven't already voted for ourselves.

The first condition (up-to-date log) was not described in the Raft paper that I could find. But the author of the paper published a Raft TLA+ spec that does have it defined.

And the second condition you might think could never happen since we already wrote the code that said when we trigger an election we vote for ourselves. But since each server has a random election timeout, the one who starts the election will differ in timing sufficiently enough to catch other servers and allow them to vote for it.

    lastLogTerm := s.log[len(s.log)-1].Term
    logLen := uint64(len(s.log) - 1)
    logOk := req.LastLogTerm > lastLogTerm ||
        (req.LastLogTerm == lastLogTerm && req.LastLogIndex >= logLen)
    grant := req.Term == s.currentTerm &&
        logOk &&
        (s.getVotedFor() == 0 || s.getVotedFor() == req.CandidateId)
    if grant {
        s.debugf("Voted for %d.", req.CandidateId)
        s.setVotedFor(req.CandidateId)
        rsp.VoteGranted = true
        s.resetElectionTimeout()
        s.persist(false, 0)
    } else {
        s.debugf("Not granting vote request from %d.", +req.CandidateId)
    }

    return nil
}

Lastly, we need to address how the candidate who sent out vote requests actually becomes the leader.

`s.becomeLeader()`

This is a relatively simple method. If we have a quorum of votes, we become the leader!

func (s *Server) becomeLeader() {
    s.mu.Lock()
    defer s.mu.Unlock()

    quorum := len(s.cluster)/2 + 1
    for i := range s.cluster {
        if s.cluster[i].votedFor == s.id && quorum > 0 {
            quorum--
        }
    }

There is a bit of bookkeeping we need to do like resetting nextIndex and matchIndex for each server (noted in Figure 2). And we also need to append a blank entry for the new term.

Despite the section quoted below in code, I still don't understand why this blank entry is necessary.

    if quorum == 0 {
        // Reset all cluster state
        for i := range s.cluster {
            s.cluster[i].nextIndex = uint64(len(s.log) + 1)
            // Yes, even matchIndex is reset. Figure 2
            // from Raft shows both nextIndex and
            // matchIndex are reset after every election.
            s.cluster[i].matchIndex = 0
        }

        s.debug("New leader.")
        s.state = leaderState

        // From Section 8 Client Interaction:
        // > First, a leader must have the latest information on
        // > which entries are committed. The Leader
        // > Completeness Property guarantees that a leader has
        // > all committed entries, but at the start of its
        // > term, it may not know which those are. To find out,
        // > it needs to commit an entry from its term. Raft
        // > handles this by having each leader commit a blank
        // > no-op entry into the log at the start of its term.
        s.log = append(s.log, Entry{Term: s.currentTerm, Command: nil})
        s.persist(true, 1)

        // Triggers s.appendEntries() in the next tick of the
        // main state loop.
        s.heartbeatTimeout = time.Now()
    }
}

And we're done with elections!

When I was working on this for the first time, I just stopped here and made sure I could get to a stable leader quickly. If it takes more than 1 term to establish a leader when you run three servers in the cluster on localhost, you've probably got a bug.

In an ideal environment (which three processes on one machine most likely is), leadership should be established quite quickly and without many term changes. As the environment gets more adversarial (e.g. processes crash frequently or network latency is high and variable), leadership (and log replication) will take longer.

But just because we have leader election working when there are no logs does not mean we'll have it working when we introduce log replication since parts of voting depend on log analysis.
I had leader election working at one time but then it broke when I got log replication working until I found some more bugs in leader election and fixed them. Of course, there may still be bugs even now.

Log replication

I'll break up log replication into four major pieces:

User submits a message to the leader to be replicated: s.Apply().
The leader sends uncommitted messages (messages from nextIndex) to all followers: s.appendEntries().
A follower receives a AppendEntriesRequest and stores new messages if appropriate, letting the leader know when it does store the messages: s.HandleAppendEntriesRequest().
The leader tries to update commitIndex for the last uncommitted message by seeing if it's been replicated on a quorum of servers: s.advanceCommitIndex().

Let's dig in in that order.

`s.Apply()`

This is the entry point for a user of the cluster to attempt to get messages replicated into the cluster.

It must be called on the current leader of the cluster. In the future the failure response might include the current leader. Or the user could submit messages in parallel to all nodes in the cluster and ignore ErrApplyToLeader. In the meantime we just assume the user can figure out which server in the cluster is the leader.

var ErrApplyToLeader = errors.New("Cannot apply message to follower, apply to leader.")

func (s *Server) Apply(commands [][]byte) ([]ApplyResult, error) {
    s.mu.Lock()
    if s.state != leaderState {
        s.mu.Unlock()
        return nil, ErrApplyToLeader
    }
    s.debugf("Processing %d new entry!", len(commands))

Next we'll store the message in the leader's log along with a Go channel that we must block on for the result of applying the message in the state machine after the message has been committed to the cluster.

    resultChans := make([]chan ApplyResult, len(commands))
    for i, command := range commands {
        resultChans[i] = make(chan ApplyResult)
        s.log = append(s.log, Entry{
            Term:    s.currentTerm,
            Command: command,
            result:  resultChans[i],
        })
    }

    s.persist(true, len(commands))

Then we kick off the replication process (this will not block).

    s.debug("Waiting to be applied!")
    s.mu.Unlock()

    s.appendEntries()

And then we block until we receive results from each of the channels we created.

    // TODO: What happens if this takes too long?
    results := make([]ApplyResult, len(commands))
    var wg sync.WaitGroup
    wg.Add(len(commands))
    for i, ch := range resultChans {
        go func(i int, c chan ApplyResult) {
            results[i] = <-c
            wg.Done()
        }(i, ch)
    }

    wg.Wait()

    return results, nil
}

The interesting thing here is that appending entries is detached from the messages we just received. s.appendEntries() will probably include at least the messages we just appended to our log, but it might include more too if some servers are not very up-to-date. It may even include less than the messages we append to our log since we'll restrict the number of entries to send at one time so we keep latency down.

`s.appendEntries()`

This is the meat of log replication on the leader side. We send unreplicated messages to each other server in the cluster.

By again referring back to Figure 2 from the Raft paper we can see how to model the request vote request and response. Let's turn that into some Go types too.

type AppendEntriesRequest struct {
    RPCMessage

    // So follower can redirect clients
    LeaderId uint64

    // Index of log entry immediately preceding new ones
    PrevLogIndex uint64

    // Term of prevLogIndex entry
    PrevLogTerm uint64

    // Log entries to store. Empty for heartbeat.
    Entries []Entry

    // Leader's commitIndex
    LeaderCommit uint64
}

type AppendEntriesResponse struct {
    RPCMessage

    // true if follower contained entry matching prevLogIndex and
    // prevLogTerm
    Success bool
}

For the method itself, we start optimistically sending no entries and decrement nextIndex for each server as the server fails to replicate messages. This means that we might eventually end up sending the entire log to one or all servers.

We'll set a max number of entries to send per request so we avoid unbounded latency as followers store entries to disk. But we still want to send a large batch so that we amortize the cost of fsync.

const MAX_APPEND_ENTRIES_BATCH = 8_000

func (s *Server) appendEntries() {
    for i := range s.cluster {
        // Don't need to send message to self
        if i == s.clusterIndex {
            continue
        }

        go func(i int) {
            s.mu.Lock()

            next := s.cluster[i].nextIndex
            prevLogIndex := next - 1
            prevLogTerm := s.log[prevLogIndex].Term

            var entries []Entry
            if uint64(len(s.log)-1) >= s.cluster[i].nextIndex {
                s.debugf("len: %d, next: %d, server: %d", len(s.log), next, s.cluster[i].Id)
                entries = s.log[next:]
            }

            // Keep latency down by only applying N at a time.
            if len(entries) > MAX_APPEND_ENTRIES_BATCH {
                entries = entries[:MAX_APPEND_ENTRIES_BATCH]
            }

            lenEntries := uint64(len(entries))
            req := AppendEntriesRequest{
                RPCMessage: RPCMessage{
                    Term: s.currentTerm,
                },
                LeaderId:     s.cluster[s.clusterIndex].Id,
                PrevLogIndex: prevLogIndex,
                PrevLogTerm:  prevLogTerm,
                Entries:      entries,
                LeaderCommit: s.commitIndex,
            }

            s.mu.Unlock()

            var rsp AppendEntriesResponse
            s.debugf("Sending %d entries to %d for term %d.", len(entries), s.cluster[i].Id, req.Term)
            ok := s.rpcCall(i, "Server.HandleAppendEntriesRequest", req, &rsp)
            if !ok {
                // Will retry next tick
                return
            }

Now, as with every RPC request and response, we must check terms and potentially drop the message if it's outdated.

            s.mu.Lock()
            defer s.mu.Unlock()
            if s.updateTerm(rsp.RPCMessage) {
                return
            }

            dropStaleResponse := rsp.Term != req.Term && s.state == leaderState
            if dropStaleResponse {
                return
            }

Otherwise, if the message was successful, we'll update matchIndex (the last confirmed message stored on the follower) and nextIndex (the next likely message to send to the follower).

If the message was not successful, we decrement nextIndex. Next time s.appendEntries() is called it will include one more previous message for this replica.

            if rsp.Success {
                prev := s.cluster[i].nextIndex
                s.cluster[i].nextIndex = max(req.PrevLogIndex+lenEntries+1, 1)
                s.cluster[i].matchIndex = s.cluster[i].nextIndex - 1
                s.debugf("Message accepted for %d. Prev Index: %d, Next Index: %d, Match Index: %d.", s.cluster[i].Id, prev, s.cluster[i].nextIndex, s.cluster[i].matchIndex)
            } else {
                s.cluster[i].nextIndex = max(s.cluster[i].nextIndex-1, 1)
                s.debugf("Forced to go back to %d for: %d.", s.cluster[i].nextIndex, s.cluster[i].Id)
            }
        }(i)
    }
}

And we're done the leader side of append entries!

`s.HandleAppendEntriesRequest()`

Now for the follower side of log replication. This is, again, an RPC handler that could be called at any moment. So we need to potentially update the term (and transition to follower).

func (s *Server) HandleAppendEntriesRequest(req AppendEntriesRequest, rsp *AppendEntriesResponse) error {
    s.mu.Lock()
    defer s.mu.Unlock()

    s.updateTerm(req.RPCMessage)

"Hidden" in the "Candidates (§5.2):" section of Figure 2 is an additional rule about:

If AppendEntries RPC received from new leader: convert to follower

So we also need to handle that here. And if we're still not a follower, we'll return immediately.

    // From Candidates (§5.2) in Figure 2
    // If AppendEntries RPC received from new leader: convert to follower
    if req.Term == s.currentTerm && s.state == candidateState {
        s.state = followerState
    }

    rsp.Term = s.currentTerm
    rsp.Success = false

    if s.state != followerState {
        s.debugf("Non-follower cannot append entries.")
        return nil
    }

Next, we also return early if the request term is less than our own. This would represent an old request.

    if req.Term < s.currentTerm {
        s.debugf("Dropping request from old leader %d: term %d.", req.LeaderId, req.Term)
        // Not a valid leader.
        return nil
    }

Now, finally, we know we're receiving a request from a valid leader. So we need to immediately bump the election timeout.

    // Valid leader so reset election.
    s.resetElectionTimeout()

Then we do the log comparison to see if we can add the entries sent from this request. Specifically, we make sure that our log at req.PrevLogIndex exists and has the same term as req.PrevLogTerm.

    logLen := uint64(len(s.log))
    validPreviousLog := req.PrevLogIndex == 0 /* This is the induction step */ ||
        (req.PrevLogIndex < logLen &&
            s.log[req.PrevLogIndex].Term == req.PrevLogTerm)
    if !validPreviousLog {
        s.debug("Not a valid log.")
        return nil
    }

Next, we've got valid entries that we need to add to our log. This implementation is a little more complex because we'll make use of Go slice capacity so that append() never allocates.

Importantly, we must truncate the log if a new entry ever conflicts with an existing one:

If an existing entry conflicts with a new one (same index but different terms), delete the existing entry and all that follow it (§5.3)

    next := req.PrevLogIndex + 1
    nNewEntries := 0

    for i := next; i < next+uint64(len(req.Entries)); i++ {
        e := req.Entries[i-next]
        if i >= uint64(cap(s.log)) {
            newTotal := next + uint64(len(req.Entries))
            // Second argument must actually be `i`
            // not `0` otherwise the copy after this
            // doesn't work.
            // Only copy until `i`, not `newTotal` since
            // we'll continue appending after this.
            newLog := make([]Entry, i, newTotal*2)
            copy(newLog, s.log)
            s.log = newLog
        }

        if i < uint64(len(s.log)) && s.log[i].Term != e.Term {
            prevCap := cap(s.log)
            // If an existing entry conflicts with a new
            // one (same index but different terms),
            // delete the existing entry and all that
            // follow it (§5.3)
            s.log = s.log[:i]
            Server_assert(s, "Capacity remains the same while we truncated.", cap(s.log), prevCap)
        }

        s.debugf("Appending entry: %s. At index: %d.", string(e.Command), len(s.log))

        if i < uint64(len(s.log)) {
            Server_assert(s, "Existing log is the same as new log", s.log[i].Term, e.Term)
        } else {
            s.log = append(s.log, e)
            Server_assert(s, "Length is directly related to the index.", uint64(len(s.log)), i+1)
            nNewEntries++
        }
    }

Finally, we update the server's local commitIndex to the min of req.LeaderCommit and our own log length.

And finally we persist all these changes and mark the response as successful.

    if req.LeaderCommit > s.commitIndex {
        s.commitIndex = min(req.LeaderCommit, uint64(len(s.log)-1))
    }

    s.persist(nNewEntries != 0, nNewEntries)

    rsp.Success = true
    return nil
}

So the combined behavior of the leader and follower when replicating is that a follower not in sync with the leader may eventually go down to the beginning of the log so the leader and follower have some first N messages of the log that match.

`s.advanceCommitIndex()`

Now when not just one follower but a quorum of followers all have a matching first N messages, the leader can advance the cluster's commitIndex.

func (s *Server) advanceCommitIndex() {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Leader can update commitIndex on quorum.
    if s.state == leaderState {
        lastLogIndex := uint64(len(s.log) - 1)

        for i := lastLogIndex; i > s.commitIndex; i-- {
            quorum := len(s.cluster) / 2 + 1

            for j := range s.cluster {
                if quorum == 0 {
                    break
                }

                isLeader := j == s.clusterIndex
                if s.cluster[j].matchIndex >= i || isLeader {
                    quorum--
                }
            }

            if quorum == 0 {
                s.commitIndex = i
                s.debugf("New commit index: %d.", i)
                break
            }
        }
    }

And for every state a server might be in, if there are messages committed but not applied, we'll apply one here. And importantly, we'll pass the result back to the message's result channel if it exists, so that s.Apply() can learn about the result.

    if s.lastApplied <= s.commitIndex {
        log := s.log[s.lastApplied]

        // len(log.Command) == 0 is a noop committed by the leader.
        if len(log.Command) != 0 {
            s.debugf("Entry applied: %d.", s.lastApplied)
            // TODO: what if Apply() takes too long?
            res, err := s.statemachine.Apply(log.Command)

            // Will be nil for follower entries and for no-op entries.
            // Not nil for all user submitted messages.
            if log.result != nil {
                log.result <- ApplyResult{
                    Result: res,
                    Error:  err,
                }
            }
        }

        s.lastApplied++
    }
}

Heartbeats

Heartbeats combine log replication and leader election. Heartbeats stave off leader election (follower timeouts). And heartbeats also bring followers up-to-date if they are behind.

And it's a simple method. If it's time to heartbeat, we call s.appendEntries(). That's it.

func (s *Server) heartbeat() {
    s.mu.Lock()
    defer s.mu.Unlock()

    timeForHeartbeat := time.Now().After(s.heartbeatTimeout)
    if timeForHeartbeat {
        s.heartbeatTimeout = time.Now().Add(time.Duration(s.heartbeatMs) * time.Millisecond)
        s.debug("Sending heartbeat")
        s.appendEntries()
    }
}

The reason this staves off leader election is because any number of entries (0 or N) will come from a valid leader and will thus cause the followers to reset their election timeout.

And that's the entirety of (the basics of) Raft.

There are probably bugs.

Running kvapi

Now let's run the key-value API.

$ cd cmd/kvapi && go build
$ rm *.dat

Terminal 1

$ ./kvapi --node 0 --http :2020 --cluster "0,:3030;1,:3031;2,:3032"

Terminal 2

$ ./kvapi --node 1 --http :2021 --cluster "0,:3030;1,:3031;2,:3032"

Terminal 3

$ ./kvapi --node 2 --http :2022 --cluster "0,:3030;1,:3031;2,:3032"

Terminal 4

Remember that requests will go through the leader (except for if we turn that off in the /get request). So you'll have to try sending a message to each server until you find the leader.

To set a key:

$ curl http://localhost:2020/set?key=y&value=hello

To get a key:

$ curl http://localhost:2020/get\?key\=y

And that's that! Try killing a server and restarting it. A new leader will be elected so you'll need to find the right one to send requests to again. But all existing entries should still be there.

A test rig

I won't cover the implementation of my test rig in this post but I will describe it.

It's nowhere near Jepsen but it does have a specific focus:

Can the cluster elect a leader?
Can the cluster store logs correctly?
Can the cluster of three nodes tolerate one node down?
How fast can it store N messages?
Are messages recovered correctly when the nodes shut down and start back up?
If a node's logs are deleted, is the log for that node recovered after it is restarted?

This implementation passes these tests and handles around 20k-40k entries/second.

Considerations

This was quite a challenging project. Normally when I hack on stuff like this I have TV (The Simpsons) on in the background. It's sort of dumb but this was the first project where I absolutely could not focus with that background noise.

There are a tedious number of conditions and I am not sure I got them all (right). Numerous ways for subtle bugs.

Race conditions and deadlocks

It's very easy to program in race conditions. Thankfully Go has the -race flag that detects this. This makes sure that you are locking read and write access to shared variables when necessary.

On the other side of race conditions, Go does not help you out with: deadlocks. Once you've got locks in place for shared variables, you need to make sure you're releasing the locks appropriately too.

Thankfully someone wrote a swap-in replacement for the Go sync package called go-deadlock. When you import this package instead of the default sync package, it will panic and give you a stacktrace when it thinks you hit a deadlock.

Sometimes it thinks you hit a deadlock because a method that needs a lock takes too long. Sometimes that time it takes is legitimate (or something you haven't optimized yet). But actually its default of 30s is not really aggressive at all.

So I normally set the deadlock timeout to 2s and eventually would like to make that more like 100ms:

sync.Opts.DeadlockTimeout = 2000 * time.Millisecond

It's mostly the persist() function that causes go-deadlock to think there's a deadlock because it tries to synchronously write a bunch of data to disk.

`go-deadlock` is slow

The go-deadlock package is incredibly useful. But don't forget to turn it off for benchmarks. With it on I get around 4-8k entries/second. With it off I get around 20k-40k entries/second.

Unbounded memory

Another issue in this implementation is that the log keeps growing indefinitely and the entire log is duplicated in memory.

There are two ways to deal with that:

Implement Raft snapshotting so the log can be truncated safely.
Keep only some number of entries in memory (say, 1 million) and read from disk as needed when logs need to be verified. In ideal operation this would never happen since ideally all servers are always on, never miss entries, and just keep appending. But "ideal" won't always happen.

Similarly, there is unbounded and unreused channel creation for notifying s.Apply() when the user-submitted message(s) finish.

net/rpc and encoding/gob

In the persist() section above I already mentioned how I prototyped this using Go's builtin gob encoding. And I mentioned how inefficient it was. It's also pretty slow and I learned that because net/rpc uses it and after everything I did net/rpc started to be the bottleneck in my benchmarks. This isn't incredibly surprising.

So a future version of this code might implement its own protocol and own encoding (like we did for disk) on top of TCP rather than use net/rpc.

Jepsen

Everyone wants to know how a distributed algorithm does against Jepsen, which tests linearizability of distributed systems in the face of network and process faults.

But the setup is not trivial so I haven't hooked it up to this project yet. This would be a good area for future work.

Election timeout and the environment

One thing I noticed as I was trying out alternatives to net/rpc (alternatives that injected latency to simulate a bad environment) is that election timeouts should probably be tuned with latency of the cluster in mind.

If the election timeout is every 300ms but the latency of the cluster is near 1s, you're going to have non-stop leader election.

When I adjusted the election timeout to be every 2s when the latency of the cluster is near 1s, everything was fine. Maybe this means there's a bug in my code but I don't think so.

Client request serial identifier

One major part of the Raft protocol I did not cover is that the client is supposed to send a serial identifier for each message sent to the cluster. This is to ensure that messages are not accidentally duplicated at any level of the entire software stack.

Diego Ongaro's thesis goes into more detail about this than the Raft paper. Search in that PDF for "session".

Again I just completely ignored the possibility of duplicate messages in this implementation so far.

References

Finally, I could not have done this without a bunch of internet help. This project took me about 7 months in total. The first 5 months I was trying to figure it out mostly on my own, just looking at the Raft paper.

The biggest breakthrough came from discovering the author of Raft's TLA+ spec for Raft. Formal methods sound scary but it was truly not too bad! This was the first "implementation" of Raft that was in a single file of code. And under 500 lines.

Jack Vanlightly's guide to reading TLA+ helped a bunch.

Finally, I had to peer at other implementations, especially to figure out locking and avoiding deadlocks.

Here's everything that helped me out.

In Search of an Understandable Consensus Algorithm: The Raft paper.
raft.tla: Diego Ongaro's TLA+ spec for Raft.
Jon Gjengset's Students' Guide to Raft
Jack Vanlightly's Detecting Bugs in Data Infrastructure using Formal Methods (TLA+ Series Part 1): An intro to TLA+.

And useful implementations I looked at for inspiration and clarity.

Hashicorp's Raft implementation in Go: Although it's often quite complicated to learn from since it actually is intended for production.
Eli Bendersky's Raft implementation in Go: Although I got confused following it since it used signed integers and -1 to represent base cases. Signed integers is a fair choice as far as I can tell, I just wanted to only use unsigned integers.
Jing Yang's Raft implementation in Rust: Although I find Rust hard to read.

And I haven't tried these but they look cool:

Cheers!

I wrote about implementing Raft in Go. By far the most challenging project I've worked on in spare time. About 7 months sporadically.

I'm not an expert, and this is not intended to be used in production. I wanted a better background on the subject!https://t.co/EhyBuQ4pD3 pic.twitter.com/vGhBbV1shf
— Phil Eaton (@eatonphil) May 25, 2023

2023–05–25: DRAM frequency scaling on Pinephone Pro saves 0.5W of power!

megi's PinePhone Development Log — Thu, 25 May 2023 01:00:00 GMT

Python tip 28: string concatenation and repetition

learnbyexample — Wed, 24 May 2023 03:00:00 GMT

Python provides a wide variety of features to work with strings. In this tip, you'll learn about string concatenation and repetition.

The + operator is one of the ways to concatenate two strings. The operands can be any expression that results in a string value and you can use any of the different ways to specify a string literal. Another option is to use f-strings. Here are some examples:

>>> s1 = 'hello'
>>> s2 = 'world'
>>> print(s1 + ' ' + s2)
hello world

>>> f'{s1} {s2}'
'hello world'

>>> s1 + r'. 1\n2'
'hello. 1\\n2'

Another way to concatenate is to simply place any kind of string literal next to each other. You can use zero or more whitespaces between the two literals. But you cannot mix an expression and a string literal. If the strings are inside parentheses, you can also use newline characters to separate the literals and optionally use comments.

>>> 'hello' r'. 1\n2'
'hello. 1\\n2'

>>> print('apple'
...       '-banana'
...       '-cherry')
apple-banana-cherry

You can repeat a string by using the * operator between a string and an integer. You'll get an empty string if the integer value is less than 1.

>>> style_char = '-'
>>> print(style_char * 50)
--------------------------------------------------

>>> word = 'buffalo '
>>> print(8 * word)
buffalo buffalo buffalo buffalo buffalo buffalo buffalo buffalo

Video demo:

See also my 100 Page Python Intro ebook.

Programming is a Creative Activity

The Healthy Dev — Tue, 23 May 2023 13:31:27 GMT

How to treat people who write code well

The path to autonomous intelligent agents: what needs to happen.

I Am Not a Robot — Tue, 23 May 2023 07:13:29 GMT

VCs like to say that the next big thing will start out looking like a toy. We are definitely having fun with them, but when will they grow up?

2023–05–23: Boot time power consumption tracing

megi's PinePhone Development Log — Tue, 23 May 2023 01:00:00 GMT

2023–05–22: Pinephone (Pro) power measurements and optimizations

megi's PinePhone Development Log — Mon, 22 May 2023 01:00:00 GMT

Underscored Numbers

infrequent oscillations — Sun, 21 May 2023 13:03:36 GMT

In which we test some parsers.

Rebooting the Carolina Code Conference

Brightball Articles — Sun, 21 May 2023 03:00:00 GMT

The Carolina Code Conference is a welcoming and community-driven “polyglot” conference that’s set to take place in beautiful downtown Greenville, SC on Saturday August 19th, 2023 in the Greenville ONE building. This conference, which returns for the first time since 2019, invites coders of all experience levels to attend, plug into the development community, share their experiences and have a great time as well.

Testing GPT-4 spatial reasoning and comprehension

Bytepawn - Marton Trencseni — Sun, 21 May 2023 01:00:00 GMT

I run experiments to determine whether, or to what degree, GPT-4 has developed an comprehension of spatial relationships. I find that it it significantly better than GPT-3.

Para Lottery

infrequent oscillations — Sat, 20 May 2023 23:46:37 GMT

In which we simulate a lottery.

Confessions

Weblog on marginalia.nu — Fri, 19 May 2023 03:00:00 GMT

I use print debugging all the time I know how to use a debugger. I use a debugger sometimes, but most of my debugging is done by print statements that are like A B C . . 5 D . D , {true, 30} . , . , 10 E I think Clean Code makes some valid points I don’t think it should be your bible or treated as infallable, having seen the sort of code that came before it, yeah, Uncle Bob got some things right.

Leadership models IV: the Pareto Principle, the Peter Principle, the Rumsfeld Matrix, Servant Leadership and Pygmalion Effect, and Goleman's Emotional Intelligence Model

Bytepawn - Marton Trencseni — Fri, 19 May 2023 01:00:00 GMT

In this post, I describe the Pareto Principle, the Peter Principle, the Rumsfeld Matrix, Servant Leadership and Pygmalion Effect, and finally Goleman's Emotional Intelligence Model.

20 Years of Gentoo

Beetle Space — Thu, 18 May 2023 10:00:00 GMT

$ genlop -l -f emerge.log |head -n20

using logfile emerge.log
 * packages merged:

     Sun May 18 14:24:00 2003 >>> sys-apps/portage-2.0.47-r10
     Sun May 18 14:26:59 2003 >>> sys-apps/portage-2.0.47-r10
     Sun May 18 14:27:49 2003 >>> sys-apps/portage-2.0.47-r10
     Sun May 18 14 …

How to start a Go project in 2023

Ben E. C. Boyter — Tue, 16 May 2023 03:00:00 GMT

I previously wrote about starting a Go project in 2018. A lot has changed since I wrote that and I had been wanting to write an updated version. What follows should be enough for anyone new to Go to get started and ideally start them being productive.

Quicklinks

Install / Setup
Starting a Project
Learning Go
Searching
Building / Installing
Linting / Static Analysis / Security Scanning
Profiling
Unit Testing
Integration Testing
Community
Multiple Main Entry Points
OS Specific Code
Docker
Useful Tools/Packages

Install / Setup

The first thing to do is download and install Go. I would suggest always installing from the Go website itself https://golang.org/ and following the instructions for your OS of choice. Other than the Go 1.18 release (which included generics) I have never had any issue always installing the newest version of Go and compiling away. The backwards compatibility promise is real. So much so even if a project’s go.mod file says 1.20, if it does not use any 1.20 functionality you can probably still compile it using an earlier release.

Two books I recommend to developers

Notes on software development — Tue, 16 May 2023 03:00:00 GMT

Originally published on February 1, 2021. The original version included two books I don't think are actually so worthwhile. This list is down to two. I think that's a good thing actually.

These are the books I recommend to developers wanting to improve their skills as professional programmers because of high information density, believable premises/examples, and being well edited.

You don't need to read books to improve as a developer but they are unparalleled in quickly helping you gain depth in a subject.

High Performance Browser Networking

If you deal with networks, you would probably benefit from this book. It is a thorough high level introduction to mobile networks, browser network protocols, and fundamentals of networking.

Designing Data-Intensive Applications

If you use a database (including an in-memory array of items you search periodically) or if you build APIs, you would probably benefit from this book. A solid introduction to distributed computing, data transfer, indexing, etc.

That's it!

Generic software books conspicuously not on this list for me:

Clean Code
JavaScript the Good Parts
Design Patterns/Gang of Four
Structure and Interpretation of Computer Programs
A Philosophy of Software Design

They're not all bad but give nowhere near as much return for the investment of your time.

Four books I recommend to professional developers wanting to improve their craft, and a few I'd nothttps://t.co/1aTrfqZ9bd
— Phil Eaton (@phil_eaton) February 2, 2021

My favorite software subreddits

Notes on software development — Tue, 16 May 2023 03:00:00 GMT

Originally published on December 5, 2021.

If you are an experienced software developer whose only exposure to reddit is dank memes, proggit or even language-specific subreddits like /r/python, you're missing out.

What follows are my favorite subreddits in tech. My criteria is that:

The subreddit topic is relevant to advancing as a programmer
Posts generally go into good depth
The comments stay on topic
And the shit-posting is minimal

This list isn't hard to guess at if you consider advanced topics in software. But I wanted to share because I think it's worth explicitly supporting high-quality subreddits.

/r/EmuDev
- My favorite sub of all. Also has a phenomenal Discord group.
/r/ProgrammingLanguages
- Focuses a little more on PLT topics (parsing techniques, syntax, type systems) than on compiling and interpreting techniques, but still good.
/r/DatabaseDevelopment
- All about database internals, which ends up involving a bunch of correctness and distributed systems stuff as well.
- Disclosure: I run this sub. It's at 2.7k+ members at time of publishing.
/r/ReverseEngineering
- The largest subreddit on this list but still has pretty good posts.
/r/EsoLangs
- One of the best/most fun intros to programming languages/compilers/interpreters is through languages like Brainfuck. This sub does a good job of keeping the fun going.
/r/Compilers
/r/GraphicsProgramming

While some language subreddits are pretty good, they are more so a mixed bag than some of the topic-specific subreddits here. So they don't make my list, more on principle than anything else.

If there is a good one already, send me it!

What am I missing?

Am I missing other amazing subreddits? Just don't say language-specific ones. :)

It's an incorrect meme IMO that tech Reddit is low-quality. You just have to find the interesting subreddits.

I've updated my list for 2023.https://t.co/OtM2tk8HOn pic.twitter.com/ymyzChp0SO
— Phil Eaton (@eatonphil) May 16, 2023

Vim tip 26: executing shell commands

learnbyexample — Tue, 16 May 2023 03:00:00 GMT

You can execute external commands from within Vim. Here are some examples:

:!ls execute the given shell command and display output
- the results are displayed as part of an expanded Command-line area, doesn't change contents of the file
:.!date replace the current line with the output of the given command
- pressing !! in Normal mode will also result in :.!
- ! waits for motion similar to d and y commands, !G will give :.,$!
:%!sort sort all the lines
- recall that % is a shortcut for the range 1,$
- note that this executes an external command, not the built-in :sort command
:3,8!sort sort only lines 3 to 8
:r!date insert output of the given command below the current line
:r report.log insert contents of the given file below the current line
- Note that ! is not used here since there is no shell command
:.!grep '^Help ' % replace the current line with all the lines starting with Help in the current file
- % here refers to current file contents
:sh open a shell session within Vim
- use exit command to quit the session

See :h :!, :h :sh and :h :r for more details.

Video demo:

Leadership models III: First Principles Thinking

Bytepawn - Marton Trencseni — Tue, 16 May 2023 01:00:00 GMT

I describe Elon Musk's First Principles Thinking model through several examples: SpaceX, Warren Buffet's Berkshire, Google's organization design, Python's language design, Random Forests and Convolutional Neural Networks.

Enjoy the Season You’re In

Letters To A New Developer — Mon, 15 May 2023 16:17:00 GMT

This is a guest post from Aaron Kardell. Enjoy. Dear new developer, I’m enjoying the season I’m in. That’s the simple phrase I find myself conveying to others lately. I have 21 years of post-college professional experience. Except for a handful of short-term consulting gigs, all of that has been working on something I founded … Continue reading →

Coincidental Scores

infrequent oscillations — Sun, 14 May 2023 13:08:19 GMT

In which the scores coincide.

Leadership models II: Growth Mindset, Eisenhower Matrix, Tuckman Model, Cynefin Framework, SCARF Model

Bytepawn - Marton Trencseni — Sun, 14 May 2023 01:00:00 GMT

I describe the following mental models useful in leadership and self-management: the Growth Mindset, Eisenhower Matrix, Tuckman Model, Cynefin Framework, SCARF Model.

yurizaki 1.0.0

ブラック — Sat, 13 May 2023 03:00:00 GMT

It's finally out! My first ever Rust project!

Get it on GitHub now!

yurizaki is a file copying and sorting daemon written in Rust, made specifically for anime and fansub releases. It's specifically meant to automatically upgrade files in your library with better release groups and newer versions. You can specify multiple release groups and it'll replace lower-ranked groups with higher-ranked groups when they are added to the source directory.

Everything is set up with a YAML file. For example:

source: /src
library: /dst

Go to Bed:
  aliases:
    - Kimi wa Nero
  groups:
    - Shino
    - SupersDisplease

Then if you run yurizaki and start adding files to the source directory, you can see yurizaki matching, copying and removing files to keep the highest version of the highest-ranked release group in the library:

MATCH: "[SupersDisplease] Kimi wa Nero - 01 (1080p) [3838CF72].mkv" => Go to Bed
No other release
Copied "[SupersDisplease] Kimi wa Nero - 01 (1080p) [3838CF72].mkv" to "/dst/Go to Bed/[SupersDisplease] Kimi wa Nero - 01 (1080p) [3838CF72].mkv"

MATCH: "[Shino] Go to Bed 01 [1080p Hi10P AC3][F4D8E418].mkv" => Go to Bed
Inferior release found: "/dst/Go to Bed/[SupersDisplease] Kimi wa Nero - 01 (1080p) [3838CF72].mkv"
Copied "[Shino] Go to Bed 01 [1080p Hi10P AC3][F4D8E418].mkv" to "/dst/Go to Bed/[Shino] Go to Bed 01 [1080p Hi10P AC3][F4D8E418].mkv"

MATCH: "[SupersDisplease] Kimi wa Nero - 01v2 (1080p) [3838CF72].mkv" => Go to Bed
Superior release found: "/dst/Go to Bed/[Shino] Go to Bed 01 [1080p Hi10P AC3][F4D8E418].mkv"

MATCH: "[Shino] Go to Bed 01v2 [1080p Hi10P AC3][F4D8E418].mkv" => Go to Bed
Inferior release found: "/dst/Go to Bed/[Shino] Go to Bed 01 [1080p Hi10P AC3][F4D8E418].mkv"
Copied "[Shino] Go to Bed 01v2 [1080p Hi10P AC3][F4D8E418].mkv" to "/dst/Go to Bed/[Shino] Go to Bed 01v2 [1080p Hi10P AC3][F4D8E418].mkv"

The end result is that only one file is retained in the library:

$ tree /dst/
/dst/
└── Go to Bed
    └── [Shino] Go to Bed 01v2 [1080p Hi10P AC3][F4D8E418].mkv

I started writing yurizaki sometime in 2020, or possibly even 2019. Then I got it to a reasonably working state and just started using it half-unfinished. After that time, I've actually published several Rust projects on GitHub, meaning that this is not actually my first Rust release. But since all of those were written after the conception of yurizaki, I'm still going to call it the first.

But despite being my first attempt at Rust, and being left unfinished literally for years, it has actually worked very well in my use. I guess that might be one of the benefits of Rust – if you get it to compile, and you didn't litter your code with shortcuts, it might reasonably handle any situation you throw at it. Although my code did definitely have some shortcuts, and still does, but it has still worked fine. I don't remember if I've ever yurizaki panic on me.

I do still have some things I want to add to it though, so it's definitely not completely finished. I for example want to be able to run custom commands after copying to automate other things. It'd also be beneficial to reduce the remaining unwrap() uses to improve stability. I've also noticed that anitomy doesn't always play nicely with certain perverted titles, so regular expression overrides might also be a good idea.

Oh yeah, and there definitely are no unit tests yet. I always find unit testing in Rust to be a pain in comparison to a more flexible language like Python.

(Named after Yurizaki Mira from Dimension W, an absolute cutie patootie.)

CI/CD with KiCad and Gitlab

Stefan Schüller — Fri, 12 May 2023 18:37:16 GMT

(Blender Render of KiCad Exported PCB)

Leadership models I: Iceberg Model, Six Thinking Hats, Trust Equation, Circle of influence, OODA Loop

Bytepawn - Marton Trencseni — Fri, 12 May 2023 01:00:00 GMT

I describe the following mental models useful in leadership and self-management: Iceberg Model, Six Thinking Hats, Trust Equation, Circle of Influence and OODA Loop.

(incomplete) Getting your O1 Visa as Fast as Possible (2024)

swyx's site RSS Feed — Thu, 11 May 2023 22:19:29 GMT

I have just received my O1A Visa and like for the H1B1, I figured I should write down my experience, thoughts, and tips for those who may wish to make the same journey. Note that I am not a professional at this, I'm just a guy who recently went through it so dont rely on me for precise facts, mostly a fellow traveler a little ahead of you.

My New Startup Checklist

swyx's site RSS Feed — Thu, 11 May 2023 10:34:22 GMT

some of you may know I've recently started a new company. I'm not ready to talk about -that- yet, but I did want to capture some notes on logistical stuff I have had to ramp up on as a first time founder. hopefully this helps somebody out there.

CLI text processing with GNU grep and ripgrep book announcement

learnbyexample — Thu, 11 May 2023 03:00:00 GMT

Hello!

I am pleased to announce a new version of my CLI text processing with GNU grep and ripgrep ebook. Examples, exercises, solutions, descriptions and external links were added/updated/corrected. The chapter on ripgrep was changed significantly to focus mostly on the differences compared to GNU grep.

This book will help you learn these commands step-by-step from beginner to advanced levels with hundreds of examples and exercises.

Release offers🔗

To celebrate the new release, you can download PDF/EPUB versions of CLI text processing with GNU grep and ripgrep for FREE till 21-May-2023. You can still pay if you wish ;)

Other offers:

Computing from the Command Line is FREE — Linux command line tools and Shell Scripting for beginner to intermediate level users
All Books Bundle is $12 (normal price $32) — all my 13 programming ebooks

What's new?🔗

Command versions updated to GNU grep 3.10 and ripgrep 13.0.0
Many more exercises added
PCRE chapter — added section for conditional grouping, corrected description and examples for \K, atomic grouping, etc
ripgrep chapter — options and regex section modified to present only differences compared to GNU grep, added details for more options such as --field-match-separator, improved recursive search section, etc
Long sections split into smaller ones
In general, many of the examples, exercises, solutions, descriptions and external links were updated/corrected
Updated Acknowledgements section
Code snippets related to info/warning sections will now appear as a single block
Book title changed to CLI text processing with GNU grep and ripgrep
New cover image
Images centered for EPUB format

Videos🔗

On this blog, I post tips covering Python, command line tools and Vim. Here are video demos for these tips:

Interactive TUI app🔗

I also wrote an interactive TUI app based on some of the exercises from the ebook. Reference solutions are provided for both GNU grep and ripgrep.

Table of Contents🔗

Preface
Introduction
Frequently used options
BRE/ERE Regular Expressions
Context matching
Recursive search
Miscellaneous options
Perl Compatible Regular Expressions
Gotchas and Tricks
ripgrep
Further Reading

Web version🔗

You can also read the book online here: https://learnbyexample.github.io/learn_gnugrep_ripgrep/.

GitHub repo🔗

Visit https://github.com/learnbyexample/learn_gnugrep_ripgrep for markdown source, example files, exercise solutions, sample chapters and other details related to the book.

See also my blog post on how to customize pandoc for generating beautiful PDF/EPUB versions from GitHub style markdown.

Subscribe to learnbyexample weekly — free newsletter covering programming resources, updates on what I am creating, tips, tools, free ebooks and more, delivered every Friday.

Feedback and Errata🔗

You can reach me via:

Issue Manager: https://github.com/learnbyexample/learn_gnugrep_ripgrep/issues
E-mail: echo 'bGVhcm5ieWV4YW1wbGUubmV0QGdtYWlsLmNvbQo=' | base64 --decode
Twitter: https://twitter.com/learn_byexample

Happy learning :)

3D Printing The Hadley 114mm Newtonian Telescope

MiscDotGeek — Wed, 10 May 2023 02:57:38 GMT

Yes, we’re building a 3D Printed Newtonian Telescope called Hadley. It’s being printed in PETG and in the video below, I give a quick tour. My build isn’t done yet, but it’s darn close. In the next video I’ll be discussing aligning (collimating) the optics and hopefully have some first impressions to share. More info …

The post 3D Printing The Hadley 114mm Newtonian Telescope appeared first on MiscDotGeek.

CLI tip 27: reverse text line wise with tac

learnbyexample — Tue, 09 May 2023 03:00:00 GMT

You can use tac to reverse the input line wise. If you pass multiple input files, each file content will be reversed separately.

$ printf 'apple\nbanana\ncherry\nfig and honey\n' | tac
fig and honey
cherry
banana
apple

You can use the -s option to specify a different string to be used as the line separator (newline is the default separator). When the custom separator occurs before the content of interest, use the -b option to print those separators before the content in the output as well.

$ cat blocks.txt
%=%=
apple
banana
%=%=
1
2
3
%=%=
red
green

$ tac -b -s '%=%=' blocks.txt
%=%=
red
green
%=%=
1
2
3
%=%=
apple
banana

See CLI tip 8: extract from start of file until matching line for a practical example where reversing input content helps in constructing a solution.

Video demo:

See also tac section from my Command line text processing with GNU Coreutils ebook for more details and examples.

Setting up Hetzner ARM instances with and for Objective-S

metablog — Mon, 08 May 2023 13:19:31 GMT

The recent introduction of reasonably-priced ARM64 VPS instances by Hetzner was accompanied by a big smile and sigh of relief on my part, as I had previously made the decision to prioritize ARM with Objective-S, for example the native-compiler is currently ARM64-only, but the simple and low-cost VPS providers like Digital Ocean were sticking to x86 exclusively.

Although it is possible to operate in a mixed ARM/x86 environment, the added complexity is not something I want as a default, which is why I also switched the hosting of the Objective-S site from DO to the Oracle cloud (on their "free forever" tier), as it was the only way to host on ARM without incurring monthly charges upwards of $40. With a number of alternatives spanning the spectrum, I now felt it

I've long had a strong hunch that there is both room and a strong need for something between the "we'll just hack together a few simple shell scripts" of the (very good!) Deployment from Scratch and the aircraft carrier that is Kubernetes.

With the external pieces finally in place, it's time to follow that hunch, and what better way than to control the Hetzner server API using Objective-S?

Talking to the API

Perusing the documentation, we see that the base URL for talking to the API is https://api.hetzner.cloud/v1/. So let's set up an API scheme handler for talking to the Hetzner API, and also set up the authentication header and indicate that we will be using JSON:

scheme:https setHeaders: #{ 
    #Content-Type: 'application/json'
    #Authorization: "Bearer {keychain:password/hetzner-api/metaobject}",
       }.
scheme:api := ref:https://api.hetzner.cloud/v1 asScheme.

It's not a lot of code, but there is quite a bit going on: first, the token is stored in the macOS keychain, accessed via keychain:password/hetzner-api/metaobject. This is interpolated into the Bearer string inside a dictionary literal. The api: scheme is now available for talking to the Hetzner API, so for example api:servers will be sent as https://api.hetzner.cloud/v1/servers.

That setup now allows us to define a simple class that allows us to interact with the API:

class HetznerCloud {
   var api.
   -schemeNames { [ 'api' ]. }
   -images {
	api:images.
   }
   -types {
	api:server_types.
   }
}

It currently has two user-facing methods: -images, which lists the kinds of images that are available and -types, which lists the server types. The method bodies may appear to be a little short, but that really is all that's needed. The -schameNames method makes the api: scheme handler available within method bodies of this class.

Below is an excerpt of an interactive st-shell session first asking the API for image types and then for server types:

] cloud images
{ "images" = ( { "id" = 3;
"description" = "CentOS 7";
"created_from" = ;
"bound_to" = ;
"rapid_deploy" = true;
"deprecated" = ;
"os_flavor" = "centos";
"type" = "system";
"protection" = { "delete" = false;
} ;
"image_size" = ;
"labels" = { } ;
"deleted" = ;
"architecture" = "x86";
"created" = "2018-01-15T11:34:45+00:00";
"os_version" = "7";
"disk_size" = 5;
"status" = "available";
...
] cloud types
...
{ "memory" = 4;
"prices" = ( { "price_monthly" = { "net" = "3.2900000000";
"gross" = "3.9151000000000000";
} ;
...
} ;
} ) ;
"storage_type" = "local";
"id" = 45;
"cpu_type" = "shared";
"disk" = 40;
"deprecated" = ;
"architecture" = "arm";
"description" = "CAX11";
"name" = "cax11";
"cores" = 2;
}
...

The "CAX11" instance type is the entry-level ARM64 instance that we want to use.

Creating a server

Creating a VPS is accomplished by POSTing a dictionary describing the desired properties of the server to the servers endpoint:

extension HetznerCloud {
   -baseDefinition {
	#{ 
	    #location: 'fsn1',
	    #public_net: #{
                #enable_ipv4: true,
                #enable_ipv6: false,
           }
	}.
   }
   -armServerDefinition {
	#{
           #name:  'objst-2',
           #image: '103908070',
           #ssh_keys: ['marcel@naraht.local' ],
           #server_type: 'cax11',
	} , self baseDefinition.
   }
   -create {
	  ref:api:servers post: self armServerDefinition  asJSON.
   }
}

The -create sends the post: message directly to the reference of the endpoint.

Interacting with servers

Once we have a server, we probably want to interact with it in some way, at the very least to be able to delete it again. Although we could do this using methods of the cloud API taking an extra server_id parameter, it is nicer to create a separate server abstraction that lets us interact with the server and encapsulates the necessary information.

The HetznerHost is initialized with a server response from which it uses the ip address and the server id, the latter to define a server: scheme handler. The fact that it's a subclass of MPWRemoteHost will become relevant later.

class HetznerHost : MPWRemoteHost {
   var hostDict.
   var id.
   var server.

   +withDictionary:theServer {
	self alloc initWithDictionary:theServer.
   }
   -initWithDictionary:theServer {
       self := super initWithName:(theServer at:'public_net' | at:'ipv4' | at:'ip') user:'root'.
       self setHostDict:theServer.
       self setId: theServer['id'].
       self setServer: ref:api:/servers/{this:id} asScheme.

       self.
     }
     -schemeNames { ['server']. }
     -status { this:hostDict at:'status'. }
     -delete {
         ref:server:/ delete.

     }
}

The DELETE is handled similarly to the POST above, by sending a delete message to the root reference of the server: scheme.

We get server instances with a GET from the API's servers endpoint, the same one we POSTed to create the server. The collect HOM makes it straightforward to map from the dictionaries returned by the APU to actual server objects:

extension HetznerCloud {
   -servers {
	HetznerHost collect withDictionary: (api:servers at:'servers') each.
   }
}

At this point, you're probably thinking that having a class representing servers, with its own scheme-handler to boot, is a bit of overkill if all we are going to do is send a DELETE. And you'd be right, so here are some of the other capabilities:

extension HetznerHost {
     -actions { api:servers/{this:id}/actions value.  }
     -liveStatus { server:status. }
     -refresh {
         self setHostDict: (server:/ value at:'server').
     }
     -shutdown {
         ref:server:actions/shutdown post:#{}.
     }
     -start {
         ref:server:actions/poweron post:#{}.
     }
     -reinstall:osName {
         ref:server:actions/rebuild post: #{ #image: osName }.
     }
     -reinstall {
         self reinstall:'ubuntu-20.04'.
     }
}

With this, we have complete lifecycle control over the server, with a surprisingly small amount of surprisingly straightforward code, thanks to Objective-S abstractions such as Polymorphic Identifiers, Storage Combinators and Higher Order Messaging.

What's more, this control is available both immediately in script form, as well as for reuse in other applications as objects.

Installing Objective-S

Now that we can create, start, stop and destroy virtual servers, it would be nice to actually do something with them. For example: run Objective-S and Objective-S-based web-servers.

This is where the MPWRemoteHost comes in. This is what it says on the tin: a representation of a remote host, very rudimentary for now. One of the few things it knows how to do is set up an ssh connection to that remote host to execute commands and transfer files via SFTP. The latter is surfaced as a store, so you can create files on a remote host as easily as assigning to a local variable:

dest:hello.txt := 'Hello world!'.

Copying files is similar:

dest:hello.txt := file:hello.txt.

The script copies a tar archive containing both GNUstep and the Objective-S libraries, which it then untars into the '/usr' directory of the target machine. In addition it transfers the interactive Objective-S shell st, the runsite command that serves ".sited" bundles via HTTP, and a .bashrc that sets up some needed environment variables.

extension MPWHost { 
 -installObjS {
	scheme:dest := self store.
	filenames := [ 'ObjS-GNUstep-installed.tgz', 'st', '.bashrc', 'runsite' ].
	filenames do: { :filename | 
	     dest:{filename} := file:{filename}.
	}.
	self run:'chmod a+x st runsite';
	     run:'cd /usr ; tar zxf ~/ObjS-GNUstep-installed.tgz';
	     run:'mv st /usr/local/bin';
	     run:'mv runsite /usr/local/bin'.
   }
}
host := MPWHost host:hostip user:'root'.
host installObjS.

As this is an extension to MPWHost, which is the superclass of the MPWRemoteHost we used as the base for our HetznerHost, the server objects we use have the ability to install Objective-S on them. Neat.

And so do the server objects for the very similar script controlling DO droplets.

Conclusion

When I started out on this little excursion, my goal was not to demonstrate anything about Objective-S, I only needed to be able to use these cloud systems, and my hunch was that Objective-S would be good for the task.

It turned out even better than my hunch had suggested: the various features and characteristics of Objective-S, such as Polymorphic Identifiers, first class references, nested scheme handlers, and Higher Order Messaging, really work together quite seamlessly to allow interaction with both a REST API and with a remote host to be expressed compactly and naturally. In addition, it manages to naturally bridge the gap between ad-hoc scripting and proper modelling, remaining hackable without creating a mess.

It's working...

R.I.P. Memex

Weblog on marginalia.nu — Mon, 08 May 2023 11:49:28 GMT

I killed the old memex.marginalia.nu site. Not because it wasn’t great, but because I don’t have the time to maintain the software, which was quite janky, and perhaps most of all I wasn’t really feeling it. The new site looks superficially similar, but it’s actually just a Hugo template that emulates some of the memex’ capabilities. Although some of the coolest stuff is sadly gone as a result. Thankfully I decided to use an extremely portable markup format when building the original memex software, which meant that porting it over to hugo was literally just a matter of writing a brief python script.

Synergia review

ブラック — Sun, 07 May 2023 03:00:00 GMT

I've obtained quite a few games from various kinds of bundles, and I imagine that this was one of them since I cannot remember paying money for this one. And having played through it, I feel like that was the right choice.

The story and writing aren't really that good. I couldn't really get into the story and the characters, who don't really feel like actual people. I guess the androids get a pass for that, but the people definitely don't. You are given a couple of choices during the story, but it doesn't feel like they really do anything beyond the next couple of minutes of story. Feels like they were just thrown in there to be able to say the game has "choices" (plural).

The bad end feels like driving a car into a brick wall. I was shocked to see the credits roll. Completely feels like the story was just left unfinished. The good end felt less abrupt but still not exactly like a satisfying conclusion. Neither of them really left me satisfied and the epilogue didn't help things. And to top things off, there are issues with spelling and grammar. The script should be ran through a spell checker.

I don't really like the art. Most of it looks like they left in the placeholder art. I'm guessing that's a style that they're purposefully aiming for but it really doesn't click with me. There are a couple of hits when it comes to the CG, but most art were misses for me. There's also not a great variety of art on display.

The soundtrack is super simple but fitting. It's probably my favourite part of the game. I'm not sure if I would ever actually sit down and listen to it specifically, but it fits the general mood of the game. No voice acting but I'm not sure if I would've wanted any from this game anyways.

In addition to the issues with the visual and the novel aspects of this visual novel, it also seems to suffer from technical issues. When I read through it, there were these text transitions between scenes that flashed by my screen in like 0.2 seconds. Was this the intended effect or was it a bug? Could go either way but definitely felt like the latter. There's also a setting that I cannot figure out what it actually does.

At one point there's also a completely black screen, and you're just kinda supposed to figure out on your own that you need to advance manually to continue the story. And even when you are shown the dialogue UI, it's really not good. The font is also very small to a degree where it feels awful ergonomically. I do have to wonder what kind of visual novels the developers are used to reading. I at least don't remember another VN with such a poor user interface.

There's also no cross-saves, so I wouldn't try to split the read between your PC and Steam Deck. Not that you'd want to read through it on your Steam Deck anyways, since the text is tiny and there's no ready controller layout. You'd either have to make your own or stick to trackpad controls. For what it's worth though, it seems to run "fine" (the same way it does on Windows) on the Steam Deck.

There's really just not much to like here. While my monetary investment towards this product is minimal, I am still disappointed at what return I got for my time investment. Honestly speaking, I would not recommend this visual novel to anyone.

GPT News Poet: silly AI poems based on today's news

Bytepawn - Marton Trencseni — Sun, 07 May 2023 01:00:00 GMT

I show how I used GNews and the OpenAI API to build GPT News Poet, a fun toy site showing silly AI poems based on today's news.

An off-ramp from the digital IKEA maze

Weblog on marginalia.nu — Thu, 04 May 2023 03:00:00 GMT

There is an episode of Star Trek where a character is for plot reasons trapped in a shrinking parallel universe. As time passes, people she knows one by one just vanish and she is the only one who seems to notice. Eventually it gets to an absurd point. She asks if it really makes sense if a ship made for a thousand people would have a crew of a few people, and everyone just sort of like shrugs and looks at her like she’s crazy.

Python tip 27: enumerate() function

learnbyexample — Wed, 03 May 2023 03:00:00 GMT

When you use a for loop, you get one element per each iteration. If you need the index of the elements as well, use the enumerate() built-in function. You'll get a tuple value per each iteration, containing index (starting with 0 by default) and the value at that index.

>>> nums = [42, 3.14, -2, 1000]
>>> for t in enumerate(nums):
...     print(t)
... 
(0, 42)
(1, 3.14)
(2, -2)
(3, 1000)

>>> names = ['Jo', 'Joe', 'Jon']
>>> [(n1, n2) for i, n1 in enumerate(names) for n2 in names[i+1:]]
[('Jo', 'Joe'), ('Jo', 'Jon'), ('Joe', 'Jon')]

By setting the start argument, you can change the initial value of the index.

>>> items = ('car', 'table', 'book')
>>> for idx, val in enumerate(items, start=1):
...     print(f'{idx}: {val}')
... 
1: car
2: table
3: book

Video demo:

See also my 100 Page Python Intro ebook.

Anagrammatic Bus Stops

infrequent oscillations — Tue, 02 May 2023 13:11:27 GMT

In which we analyse some bus stops.

What I'm up to - May 2023

Philip I. Thomas — Tue, 02 May 2023 01:36:45 GMT

✨ What I've been up to

April was a blur - I was in the Bay Area, NYC, and Cleveland. I'm looking forward to spending the next three weeks without travel.

I am currently immersed in the development of Booklet, a new product from Contraption Co. designed for online online communities. I started building Booklet to address the shortcomings of using real-time chat apps for communities. Booklet strives to be calmer - it combines a forum and member directory, and generates a customized email newsletter for the 99% of community members who consume content rather than create it. If you're intrigued and want early access, register at booklet.community.

🤔 Things to share

Articles: Less Cars, More Money - ". . . the only reason we think we need cars to get around, is because we have wasted most of our space on accommodating cars, which spread everything out so far (and made everything so loud and dangerous) that nobody feels like walking or biking!"
Books: Let My People Go Surfing. The Rails and Hotwire Codex.
Coffee I'm drinking: Santa Teresa Natural Geisha from Prolog.

📫 What I'm up to this month

Working and relaxing, with a short trip to Colorado. I hope warm weather returns soon!

📍 Where I'll be

(Let me know if we overlap!)

May 24-28: 🏔️ Denver

Photo of the month: Baking canelés

February to April Gears emulator update

Linux Engineer's random thoughts — Tue, 02 May 2023 01:00:00 GMT

Previously, I had a bug in my gears emulator, which I then fixed. But Sonic 2 still wasn't working, so let's see what went wrong there.

Missing interrupt behaviour

The Sonic 2 ROM wasn't starting at all, leaving an uninteresting black screen, which led me to suspect some kind of …

Escaping the Time Trap: Why Estimating Effort, Not Time, Leads to Greater Success

smcleod.net — Fri, 28 Apr 2023 09:41:37 GMT

Effort is non-linear and multi-dimensional. Time (with relation to delivery) is linear and one-dimensional

Chirping, chirping, chirping

Beetle Space — Wed, 26 Apr 2023 10:00:00 GMT

For the last few days I’ve been playing with AI Chirpers - a Twitter-like social network where only AI bots are allowed and they chat with one another.

What fun!

How it works: You register an account, and create up to 5 AIs. All you need to do is provide …

On tools

bayindirh — Tue, 25 Apr 2023 03:00:00 GMT

Learning and being informed about new technologies is a habit of mine. I regularly visit webpages and follow RSS feeds to see what is happening and what are the new tools being developed.

While I learn and be aware about these new tools, I rarely switch what I use with shinier and newer incarnations of the ones with the same role or functionality. Moreover, I defend using the same tools for a long time despite their shortcomings. More importantly, I defend using stable workflows (or stacks if you prefer the term) for a long time, despite the disagreements with my peers.

Solving problems once

The biggest reason I'm using same tools and workflows for a long time is solving every problem once. When starting to build a workflow, selection of tools and making them work together takes a great deal of time, which is gone unnoticed due to excitement of starting a new project.

After setting the workflow for the first time, there will be inevitable problems and paper cuts which hamper both the development and mental flow. Interruption of this flow makes the developer lose a lot of time, since human brain works best when it's warmed up and focused on a deep subject for a while.

When these problems are solved, and documented if possible, the workflow starts to settle down. This settling is similar to annealing. It's a slow and delicate process, but when it's complete; the result is robust, high quality and pretty indestructible under normal usage conditions.

After this workflow and tools around it is settled, it can be lifted and applied to new projects with ease. The time required is minuscule when compared to doing everything all over again, every time, and this saved time allows for nifty things, namely iteration and mastery.

Reproducibility

Another benefit of having a well defined and well polished workflow is reproducibility. This affects not only the developer, but potential contributors of the project. When one has a reproducible workflow built around well-defined tools, it can be reproduced with much less effort.

This is crucial for making collaboration on a project easy and effortless. When somebody new to a project can setup the workflow in an hour and start hacking away, the contributor is more likely to be happy and motivated. This initial momentum will greatly improve both developer and user ergonomics, because if a user who knows how to program starts to use the tool, and setup its development environment in a coffee break, they can both use and improve the tool at the same time. As a result, the chance of getting patches from the same user will go up.

Mastery

As I noted in a previous post, I love the concept of craftsmanship and mastery. The results created by this dedication and discipline is always high quality, even exceptional.

I personally love to create tools for myself, and want these tools be in the same league with the more established tools, regardless of the category. Writing high quality code and robust tools which are production ready and complete in every sense is a great source of pride and satisfaction for me.

On the other hand, trying to reach that level of quality is a great learning experience. Because, finishing the second 90% of the work needs discipline, dedication and elaborate work.

Reaching that level of finesse is possible when one knows the tools at hand down to its smallest details and edge cases. Similarly, to be able to focus on the product to maximize its quality, one needs to trust the tools to the point they become invisible or extension of oneself. This again requires using the tool for an extensive time and being familiar with it. As a result, a stable and well worn workflow with established tools is essential in this process.

Licenses

While tangential, the licenses of the tools used in the workflow has a great impact on the workflow itself, because it affects the portability of the workflow and the possibilities to share it greatly.

I personally prefer free software over open source, because when combined with a free software project and proper documentation, all moats are removed between the project and possible collaborators.

Using workflows consisting free software tools as much as possible guarantees that the workflow will be stable in the long term, because the probability of tools going proprietary or being developed as closed source forks is way lower.

Lastly, the four freedoms about software is essential and non-negotiable from my perspective. As a result, everything I put out and the tooling required to build and modify is free software, and I have no plans or intentions to change that to more permissive models.

Building workflows is fun and exciting, but building them over and over limits the time which can be used for creating even more exciting things.

Until next time,

Be kind.

Vim tip 25: substitute flags

learnbyexample — Tue, 25 Apr 2023 03:00:00 GMT

Here are some of the flags you can use with the substitute command:

g replace all occurrences within a matching line
- by default, only the first matching portion will be replaced
c ask for confirmation before each replacement
i ignore case for searchpattern
I don't ignore case for searchpattern

These flags are applicable for the substitute command but not / or ? searches. Flags can also be combined, for example:

s/cat/Dog/gi replace every occurrence of cat with Dog
- Case is ignored, so Cat, cAt, CAT, etc are all valid matches
- Note that i doesn't affect the case of the replacement string

See :h s_flags for a complete list of flags and more details about them.

Video demo:

Suzume no Tojimari

ブラック — Mon, 24 Apr 2023 03:00:00 GMT

Just got out of the movie theatre after watching Suzume no Tojimari.

I've never really self-identified as a Shinkai Makoto fan, but if I keep going to the movie theatre to see his movies and love all of them, I guess I just might be one. Definitely feels like that keeps coming true ever since I watched Kimi no Na wa in a theatre back in 2017.

(I have seen Shinkai movies before Kimi no Na wa but not in theatres.)

If you've seen the two Shinkai movies before Suzume no Tojimari, you probably know what to expect: beautiful scenery, fluid animation, tons of details, the supernatural, comedy, drama, love and RADWIMPS. Stick to what you know, right?

The story was captivating as ever. Shinkai also writes great characters that you'll want to root for. That goes for the main characters and all of the side characters. Suzume was of course as adorable as Hina and Mitsuha before her. Tears were definitely also shed during. You can definitely see how the 2011 earthquakes shaped this story, and the suffering brought by it. You even see a diary entry for March 11th during a scene.

I think Suzume might be the first time Shinkai has crossed the two-hour threshold in runtime. It didn't really feel that long, and the narrative was quite packed. Wouldn't have minded at all if it was even longer, I wasn't checking my watch.

I feel like this time around though, they definitely toned down the RADWIMPS. No insert songs this time, which feels like a bit of a shame since 愛にできることはまだあるかい playing during Tenki no Ko still gives me shivers. Not that the audio side of Suzume was anything to scoff at. The soundtrack fits the storytelling like a glove and the sound effects kept me immersed. I'm actually pretty glad I saw it in a proper theatre, you just can't get those rich bass sounds in a home setup.

If I had to rank Shinkai's latest three movies, I'm not quite sure what I'd say. When I saw Tenki no Ko, I felt like I enjoyed it more than Kimi no Na wa, even though the popular opinion seemed to be the opposite. I in fact enjoyed it so much that I went to see it in the movies twice. Even dragged someone to go watch it with me. But I don't know how to rank Tenki no Ko and Suzume relative to each other. I feel like I love them both, but in different ways. Not sure if I'd go rewatch it in a threatre though – maybe if I got to drag someone with me again.

I don't think Shinkai has pulled a Miyazaki (announced some kind of a retirement that may or may not last) yet, so I guess I'll wait for the next beautiful supernatural adventure from him.

Thoughts on AI and AI-veganism

Weblog on marginalia.nu — Fri, 21 Apr 2023 03:00:00 GMT

I’ve come to think LLMs/GPTs/whatever are a threat to conventional search engines because the modern web is an unbelievably annoying dumpster fire. They don’t really provide better or faster answers, what they provide is an experience that is not a complete pain in the ass. This frog has been simmering for a long while now and we’re so used to it that seeing literally anything else seems revolutionary. You visit a website and need to dismiss a cookie policy notification, a request to show popups, a request to know your location, an invitation to subscribe to a newsletter, a sales rep wants to have a chat with you, then you get random layout shifts for several minutes as all the ad auctions finish, and then just as you’re ready to read the content the website crashes and reloads and the circus starts over again.

CLI tip 26: removing duplicate lines with GNU awk

learnbyexample — Thu, 20 Apr 2023 03:00:00 GMT

awk '!a[$0]++' is one of the most famous Awk one-liners. It eliminates line based duplicates while retaining input order. The following example shows it in action along with an illustration of how the logic works.

$ cat purchases.txt
coffee
tea
washing powder
coffee
toothpaste
tea
soap
tea

$ awk '{print +a[$0] "\t" $0; a[$0]++}' purchases.txt
0       coffee
0       tea
0       washing powder
1       coffee
0       toothpaste
1       tea
0       soap
2       tea

# only those entries with zero in the first column will be retained
$ awk '!a[$0]++' purchases.txt
coffee
tea
washing powder
toothpaste
soap

Removing field based duplicates is simple for single field comparison. Just change $0 to the required field number after setting the appropriate field separator.

$ cat duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
dark red,sky,rose,555
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333

# based on the last field
$ awk -F, '!seen[$NF]++' duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
dark red,sky,rose,555

For multiple fields comparison, separate the fields with , so that SUBSEP is used to combine the field values to generate the key. SUBSEP has a default value of \034 which is a non-printing character and not usually used in text files.

# based on the first and third fields
$ awk -F, '!seen[$1,$3]++' duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333

huniq is a faster alternative for removing line based duplicates.

Video demo:

See also my CLI text processing with GNU awk ebook.

Tools I use: Evernote

bayindirh — Tue, 18 Apr 2023 03:00:00 GMT

In recent years, a lot of note taking applications (or personal knowledge base tools if you prefer the term) have entered the space. Prominent examples are Roam Research, Obsidian, Notion and countless alternatives and reimplementations of those. On the other hand, there are older ones, and there's one literal elephant in the room, and it's called Evernote.

Evernote is one of the oldest applications in this space. Started as a suite of desktop and mobile applications with a web-based client for accessing your notes, it supports rich formatting and predates popularity of Markdown. As a result, what it provides is more of an old fashioned, WYSIWYG editor, with tables, file attachments, and other quality of life improvements.

Evernote was not in its current form when it started. It took more of an "everything and the kitchen sink" type of approach, more targeted to road warriors of 2000s. The resulting desktop application was something between an office suite and note taking tool. This situation created a big problem: There was no feature parity between platforms and this hurt Evernote big time.

At this stage, while I was ready to abandon Evernote and cancel my subscription, something happened: Company's CEO has changed, and they announced a new direction. I decided to wait longer and give them another chance.

In hindsight, waiting was the correct thing to do, because new Evernote is much better than the first iteration. Yes, it has an Electron based desktop application (and I don't like Electron), yet it's not sluggish and taxing your system in unreasonable amounts.

The new Evernote lacks many of the features of the first iteration, and many of its competitors, however this simplicity and well-roundedness is what makes Evernote enticing. As a result, I'm managing all my knowledge base, collaboration and public tech notes with Evernote.

What it makes Evernote great is the combination of simplicity and feature parity between platforms it runs on, including Web, which allows me to work on any platform. Combined with the new synchronization engine they developed over the years, everything is pretty seamless. I have never lost any data with them, and that's pretty important for me.

The simplicity they provide is reduced set of fonts and headers, a selection of color choices to emphasize things, and text alignment options. The resulting feature set can be called "Markdown+". While a superset of pure Markdown, it can degrade to it gracefully, and they can be converted to each other with relative ease.

Given my love for simplicity and practicality, Evernote's reduced set of options allows me to think less and create more. On top of that, this reduced set of formatting options allows me to create consistent documents with less effort, which is a win in my book.

Another thing I like with Evernote is one-click sharing of notes, which I use for writing technical documents and sharing them (which you can see here).

Like other tools, Evernote supports collaboration over shared notebooks, and the process is pretty smooth since the new Evernote has started its life. I have used it throughout my Ph.D. with my professor for seven years, and it worked. This is enough of a testament for the experience.

Finding what you have written in the past has gotten much easier with Evernote in recent years. Its search engine is revamped and improved greatly. While not all features are available to all tiers, the highest tier membership (which is nowhere expensive here since Evernote does local pricing) allows location based searches. Consider you went somewhere and took notes there. You can search the notes you have taken at that location, which is much more effective than it sounds. For example, I used frequent a Starbucks branch where I met with my professor for my studies. I can list the notes I have taken there and filter further down to find what I was looking for, much faster. That's both impressive and comforting, because that elephant not only can remember what you have told, but can recall these things with accuracy and precision.

Another feature added to Evernote in its recent iterations is task management and calendar integration. It's a set of powerful features centered around tasks and task delegations (even to people outside Evernote), but since I manage my daily tasks elsewhere, I don't use these features much, hence I can't comment on them.

Lastly, while Evernote is a closed source ecosystem, it has great exporting capabilities. It has a file format called .enex, which is an XML based export file. XML is disliked in tech circles (which is a story for another time), but Evernote both documents the format openly, and provides its schema for interoperability. You can read more about it here, and see the schema here. The files you attach to a particular note are even encoded BASE64 and embedded inside the XML, hence you can get your notes as-is and with everything you added.

As a result, the old elephant Evernote is useful, even today. It allows me to create nice looking documents, share them with the world or people I want to collaborate and allows me to get my notes as-is for plethora of purposes via an open and well documented file format. Paired with clients on many platforms with the same features, the result is a winning formula and happy users.

Until next time,

Be kind.

2023–04–16: Pinephone DRM driver issues

megi's PinePhone Development Log — Sun, 16 Apr 2023 01:00:00 GMT

Digital clutter

Paul Stamatiou — Sat, 15 Apr 2023 03:00:00 GMT

I started taking photography more seriously about a decade ago. I shot photos in RAW with increasingly higher megapixel cameras, spent a lot of time post-processing photos—see my 30K word post about building a custom Adobe Lightroom PC—and designed and built custom photo stories to share them.

While I would go through the process of culling photos that I published online, I didn’t do that for what I backed up. I kept everything. I always thought, "what if I want to go back to this and edit it differently one day?"

I kept every RAW photo I shot, every video, every GoPro clip... I even kept every phone photo going back to my first iPhone. And why not, storage is cheap and easy, right?

The terabytes kept adding up. I added close to a terabyte with New Zealand. Hundreds of gigs with Africa. When you hit a few terabytes the obvious solution is to move to a NAS, or Network Attached Storage. I wrote about this extensively in 2015 with Storage for Photographers (Part 2).

Years back these were pretty dumb devices that would just appear as a networked drive for your computer to manage. Today, they’re full-blown personal servers, capable of running self-hosted apps and services. That makes it easy to have them encrypt your data, and automatically back your files up to cloud services, which is probably what you want if you’re paranoid about backups and try to follow the 3-2-1 rule.

My NAS setup some years back.. it became far messier over the years after I moved to NYC and had less space for networking gear.

Over the years I paid thousands of dollars just to keep those RAW photos safe. At one point I think I had copies on 3 or 4 separate cloud services.. you know, just in case. I later upgraded to another NAS that was physically a bit smaller.

In hindsight, I couldn't care less about 90% of all that data, if not more. The fact that I had terabytes of available storage made it all too easy to not be picky with what I was backing up and just archive everything.

When it was time to move to a different apartment, I took that as the perfect excuse to simplify. I no longer wanted a NAS in my life. A little whirring box, always making random hard drive noises, always taking up space, always making heat. Though I’m sure my perspective would be a bit different if I had enough space to hide the NAS in a perfect server closet completely out of sight and mind.

I needed to get rid of tons of RAW photos for which I no longer cared. I knew I wouldn’t be able to get rid of everything and I still wanted maybe 2 TB of local storage, separate from my computer. My first inclination was to just look for a tiny NAS that I could load up with 2.5” SATA SSDs, or something even better like M.2 NVMe SSDs. I didn’t find anything I liked—most small NAS devices would still require some massive barrel-jack power adapter.

Picking an M.2 SSD enclosure

I ended up deciding to get a small, simple external Thunderbolt SSD enclosure. In particular, I wanted an M.2 NVMe SSD enclosure. M.2 SSDs are much smaller and faster than other modern SSDs using SATA. I wanted an small, bus-powered enclosure that was Thunderbolt 3 or 4, and had some basic heatsink or thermal pad for the SSD.

There were a few options I looked at. The Sabrent Rocket Nano looked perfect at first glance due to its tiny size but they don't sell it as an enclosure, and it's only USB-C, not Thunderbolt, so I ruled that out. Then I looked at a Thunderbolt 3 enclosure from Sabrent. Unfortunately, some reviews pointed out issues with performance and reliability that had me concerned. There was also a new Thunderbolt 4 enclosure from Satechi, but it wasn't shipping (and still seems to be out of stock). That aside, it looked like a great option, but it was far too large for some reason.

After a lot of looking I eventually landed on the OWC Envoy Express. I was hoping for a more aesthetically pleasing option but OWC has been a reputable brand I've been familiar with so it felt like the safe option. I don't like how the Thunderbolt cable is part of the case design (you can detach it but you have to open the case), but it wasn't a dealbreaker for me. At the time I purchased this, it was a bit cheaper at $79.

The Envoy Express enclosure alongside a cheaper USB-C enclosure I got for another spare SSD I had.

The Envoy Express enclosure is all metal and seems very well-made so I'm quite pleased with it.

As for what SSD to use with the enclosure, I decided to use one I already had: an older 2 TB Samsung 970 Evo. At the time I was also parting out and selling my gaming PC (didn't use it enough after I got an Xbox Series X and it was taking up space) and I decided to repurpose the 2 NVMe SSDs I took from it. If you want to do the same, you can find affordable 1 and 2 TB options from Western Digital, Samsung, and SK Hynix among others. It's when you start looking for 4TB SSDs that the prices go up considerably.

My old desktop PC where I salvaged my M.2 SSDs from

Installing the SSD in the OWC enclosure took less than a minute. There's two screws to open the case, then one more screw to fasten the SSD. The case has a built-in thermal pad that touches the top of the SSD, though I wish it had some sort of cooling for the bottom as well.

Breathing new life into my old 970 EVO M.2 SSD by putting it in the OWC Envoy Express enclosure.

Setup and disk encryption

The next thing I did was open up Disk Utility on macOS, format the SSD, then set up encryption. While there's lots of ways to encrypt your data—like VeraCrypt which touts plausible deniability with hidden volumes—I opted for the encryption functionality built into macOS for simplicity.

With the SSD ready to go, the hard part was determining what photos and data I wanted to from my NAS I wanted to keep and transfer to this SSD. This was made particularly hard as my NAS was very slow. I had gigabit ethernet but that doesn’t matter, these spinning disks are barely faster than 100MB/s for sequential read even with a RAID setup. So my long process over a week or so was transferring a folder or two to my laptop overnight, as much as I could with the available space I had on my MacBook Pro's internal SSD.

Once on my laptop I had to go through and decide what to delete and what to keep. There were a lot of easy ways I was able to save space by cutting things like GoPro videos of which I had hundreds of gigs from trips. I didn't care for that footage much and the important clips I had already used in other videos elsewhere.

But there's no getting around it, this was an arduous process. For some older trips I only kept the post-processed original size JPGs and deleted the RAWs. Then there were tons of just random photos I deleted. It took me a few days but eventually I was able to shave a few terabytes; enough to fit the important stuff on this 2 TB SSD. (I had a bit of overflow which I tossed on the other USB-C external enclosure I had which I won't use often so the speed isn't important).

Cloud encryption and backup

With the photos and data trimmed down and ready to transfer to the SSD, I needed to figure out how I wanted to have this SSD backed up and encrypted on cloud services. For that I turned to Arq.

Arq is a backup power tool for advanced users. It's a standalone Mac app that connects to just about any cloud service including AWS, Backblaze, OneDrive, Wasabi, Google Cloud, as well as all S3-compatible storage providers. The best part is that you bring your own credentials for those cloud services, and you have fine-grained control over how things are backed up, stored, encrypted, and much more.

I decided to separate my photos and other more important data into different folders with different backup strategies. Photos took up most of the 2 TB and I wanted them backed up but I didn't expect to access them frequently so I didn't want to use the Amazon "S3 Standard" storage class. I decided to use S3 Glacier Instant Retrieval for my photos.

I'm no stranger to Glacier, Amazon's slower and cheaper data archival storage tier, and have been using some form of it for a decade, but I never had a great experience. It always felt unpredictable and slow; yes, I realize that's the basically the trade-off you sign up for. As of 2021, Amazon added a new type of Glacier storage called Instant Retrieval. So you get the benefit of more affordable data archival storage along with quicker retrieval times.

There's quite a few S3 storage classes to select from: Standard, Standard Infrequent Access, One Zone Infrequent Access, Glacier Instant Retrieval, Glacier Flexible Retrieval, Glacier Deep Archive. It's important to familiarize yourself with the functionality of the S3 storage class you plan on using. And that's just with AWS, you can to use any other cloud provider you like with Arq.

To give you a rough idea of the prices associated with some of these S3 storage classes, here's what it would cost to store 2 TB per month, with no other actions or transfers:

S3 Storage Class	Monthly price to store 2 TB
S3 Standard	$47.10 USD
S3 Standard Infrequent Access	$25.60 USD
S3 One Zone Infrequent Access	$20.48 USD
S3 Glacier Instant Retrieval	$8.19 USD
S3 Glacier Flexible Retrieval	$7.41 USD
S3 Glacier Deep Archive	$2.05 USD

And then, for my other data, which was more important and not nearly as sizable, I opted for S3 Standard Infrequent Access. Here's how that looks in Arq:

Arq showing different S3 storage classes for different folders

Arq also lets you specify encryption for your data—separate from the macOS encryption I set up that's required to mount the drive—so I did that as well. It's important to note that you will need the Arq client to be able to access the encrypted files again; they're unreadable if you go directly to the storage bucket on S3.

That's it! I'm very happy with the new setup. I'm relieved to no longer have a NAS taking up space and making noise in my house. The external SSD is tiny and I still have control over where my data is backed up online. I can easily add more cloud storage destinations later on as I see fit.

Cloud control

However, after all this work slimming down my archives, the job is not done. My NAS was just the first target. I still have a stockpile of data and memories across other services like Google. Take Google Photos for example. Absolutely amazing service. I love the heck out of the product. And while Apple's Photos functionality is great, I've always loved all that Google Photos had to offer from amazing search to strong sharing functionality.

I just wish Google Photos wasn't owned by Google. I'm trying to reduce my reliance on consumer-oriented Big Tech products like this in favor of services where I have more control. Google has been heralded for their data portability tool Google Takeout, which includes Google Photos data, but in my experience using it was all but pleasant.

First, I had to wait days to let the archives prepare, which I understand what needs to happen with this much data so that's fine. But then, I had to download more than a dozen separate zipped files amounting to over 800 GB. I don't even think I had that much space on my laptop, let alone extra space to be able to unzip them all. And once unzipped, I don't recall the files having any sort of helpful organization to make it feasible to trim down moments you don't need. I gave up on that.

Cloud storage services make it all too easy to eagerly backup anything and everything without being intentional. Now we've all inadvertently become data hoarders.

Just because we have easy access to near-infinite and increasingly more affordable storage doesn't mean we should be using all that space. The next generation of cloud-based services need to design for curation and control. It doesn't feel like an immediate concern, but it will only continue to increase.

These days getting your Google account hacked isn't just about your email. It may be about your entire digital life. This is more than just decluttering, it's about control, security and privacy too.

Can ChatGPT write the C++20 async message queue codes from the previous posts?

Bytepawn - Marton Trencseni — Sat, 15 Apr 2023 01:00:00 GMT

I try to get ChatGPT to write the codes in the previous posts. It's able to write the basic message queue skeleton, but it cannot implement more complicated features such as delivery semantics with caching; also, the code crashes. C++ programmers do not (yet) need to fear for their jobs.

2023–04–15: Pinephone Backlight Testing

megi's PinePhone Development Log — Sat, 15 Apr 2023 01:00:00 GMT

Ranking #1 on HN in Mid April

swyx's site RSS Feed — Fri, 14 Apr 2023 08:17:23 GMT

I last wrote about Ranking #1 on HN in December, and wanted to offer an update from my mild hit today. I am now taking Latent Space (the new name enabled by the previous owner of that domain selling it to me in my first P2P domain purchase) a lot more seriously with the support of my partner and friend Alessio.

Blog

ブラック — Fri, 14 Apr 2023 03:00:00 GMT

Everything old is new again.

Seeing as Twitter has been (unsurprisingly) in a constant backslide since last autumn, I've decided that I need to start diversifying where I output all of my thoughts and opinions. And considering what a craptastic thing Mastodon is, I'd rather not partake in it either.

Therefore, I have decided that I should have a web log. Just like in the good old days.

I have probably attempted to have a blog in the past too, without much success. I was much more of a fan of microblogging, which is probably why I've been on Twitter for the past 15 years. But since there's a massive vacuum in the microblogging space, I've decided to pivot into (macro?)blogging.

Quite honestly, I do not have a long-term vision for this blog beyond having a space to store whatever I feel like I need to release into the wildness of the Internet. Most likely it'll be just a pool of random thoughts about whatever interests I have at the time, without any kind of a specific audience in mind beyond myself. Possibly also quite a lot of grumbling and venting. The first posts are just short reviews of the games that I've played lately, reposted from my Steam profile.

Also, considering what has been dominating the tech news for the entirety of 2023, I'll start off by proclaiming that this blog will not be written by large language models, but by something resembling a human.

Technical

This entire site is a static site made with Lektor. I chose Lektor because it's built on Python and uses Jinja2 for templating, and I'm already familiar with both of those so it's easy to hack on. And I chose to use a static site because it's much simpler to run. No need to worry about your unpatched Wordpress installation becoming part of a botnet.

For the layout, I just hammered away on the default layout that it bootstraps until it looked like me. It probably has very little in common with the initial layout now. Not gonna lie, restricting yourself to black and white does make it a bit hard, but I think my hands were tied with that one when I decided to use this domain.

There's also an Atom feed.

Muv-Luv photonflowers* review

ブラック — Fri, 14 Apr 2023 03:00:00 GMT

Muv-Luv photonflowers* is a collection of side stories from the Muv-Luv universe, both Extraverse and the Alternative timeline.

Extra

The Extra side of things contains one longer story and six short stories. In my opinion, the short stories are kind of a waste of time. They're not particularly interesting and don't really have any payoff. The correct time to read through them is probably right after finishing Muv-Luv Extra. I'd also read them before reading through the longer photonflowers* Extra story, mostly because they'd feel like a disappointment after.

The longer Extra story is luckily a lot better than the short stories. It continues the story Extra story from the Sumika ending and feels like a proper extension of Extra. Much like Extra, it contains a mix of humour and serious bits. I think it alone is a couple hours worth of reading and has a satisfying story arc with an emotional ending.

You do get full voice acting with the exception of Takeru, just like in Extra. Art department will also feel like Extra, since it's mostly recycled assets from Extra. I guess it would feel weird to have new character sprites but you can't praise them for going an extra mile here.

Alternative

The Alternative side of photonflowers* is split between five side stories from the Alternative timeline.

Two of the longer Alternative side stories, Confessions and Atonement, are focused on giving established characters a backstory, and are on the longer side. I found both of them to be enjoyable reads and I imagine that any other Muv-Luv fan would also find them enjoyable. They are very much the flagship titles of this collection.

The remaining three, Rain Dancers, Chicken Drivers and Inheritance are very short. I think all of them were around 15-30 minutes each. While none of the stories were bad, some of them weren't great.

Rain Dancers centers around characters that I'd never heard of, fighting at a front that I've never seen covered in Muv-Luv before, and didn't really have any kind of a discernible story arc. Chicken Drivers also mostly has never-seen characters, but at least they're fighting at a front already covered, and gives a bit more interesting backstory. Inheritance has the tightest coupling to the Alternative story, and is probably the best of these three.

The art on the Alternative side has a lot of new sprites and generally feels higher quality than the Extra side. Possibly because they're not twenty-year-old assets. Some good CG and sprites too. Even the shortest stories get pretty good art assets. Again, full voice acting included.

photonflowers*

Systems-wise, photonflowers* is basically the same as Muv-Luv and Muv-Luv Alternative. That is to say, the technical level of it is about two decades behind today. Coming to this after reading through THE DAY AFTER trilogy (tetralogy?), it definitely feels dated. I would much rather have the engine from THE DAY AFTER. As a speedreader, I especially missed the ability to set when voice lines are stopped. The dialogue display is also a lot cleaner in TDA.

Thankfully it's very playable on the Steam Deck. It actually feels like it suffered from less audio issues after you wake the Steam Deck from sleep with the game running. I definitely had to restart the game a couple of times to get everything working again. This is made much less painful by the fact that photonflowers* jumps right back to where you quicksaved, so restarting is maybe like a 5-10 second ordeal.

However, when it comes to the price, the value proposition doesn't feel that great. There's around ten hours worth of reading, and all of the stories aren't bangers. Nonetheless, you are asked 21€ for the package on Steam. I would personally not recommend photonflowers* for that price. However, I got my copy at a 50% discount, and for that price point, it seems like a decent package.

All in all, Muv-Luv photonflowers* is a decent read for the dedicated Muv-Luv fan – at least if they don't overpay for it.

ChatGPT plugins: OpenAI's attempt to kill Google

I Am Not a Robot — Thu, 13 Apr 2023 21:11:18 GMT

If this works, the ad model becomes obsolete. If I were Sundar Pichai I'd be freaking out.

Highfleet Optimizer Beta Test!

jodavaho.io — Thu, 13 Apr 2023 05:22:23 GMT

Hi Tarkhans,

u/althowd and I are excited to present our highfleet shipyard, with cogitator backend!

https://hfopt.jodavaho.io

We’ve been working on it for about a year, and current the site provides two things:

A free ship hosting / sharing service (Upload requires registering, downloads do not)
A free “cogitator” to help you rapidly test designs and check in-game statistics

Check it out, or see more details here! Since we’re self “hosting” (paying), we wanted to limit useage to those who really are using it, so an email-based registration is required, but we send zero emails and don’t use it for anything other than logins.

Please note this is a beta test! Give us a little slack, and clear cookies for jodavaho.io if anything starts getting weird.

To share a ship file:

Login or create account
Select “Upload”
Upload your .seria file
The uploaded file will be verified (to make sure it’s a highfleet ship and not a naughty picture), then will be available for download.
Check “Shipyard” to see the status and to get your shareable link. Verification may take up to 10 minutes for very large files. I tested with Castle Doctorine, and it took 3 minutes.

How to use Cogitator / ship design support

To test the cogitator:

Login or create account
Go to the main page, or click “Cogitator”
Select the modules from the drop downs (and their quantities) that you know you want the ship to have. These are the “required” modules.
Select the in-game statistics (min and if required, max) that you know you want the ship to have. These are the “constraints”.
Press “Send”.
The cogitator will receive your required modules and statistics, and “Fill in” all missing modules until the statistics line up correctly. It will do so while minimizing the total cost. This can take up to 15 minutes, but usually takes less than a minute.
When done, the page will be populated with all the ship modules. You can use this as a “bill of materials” reference when building the ship in the highfleet shipyard.
To start over, refresh the page, or clear the module list

I use the cogitator to ask questions like:

“What is the cheapest ship that can carry four d80 molots at least 1000 km with speed at least 200”
“I just designed this ship with 2x 180-6, massive range, and a speed of 100. I’m pretty sure I could have done it cheaper by changing the engines, let me put in the in-game stats and guns and see what the cogitator chooses”
What is the cheapest scout ship that can carry a MK-800 radar at least 2000km, with speed at least 350
I want a fuel tanker, what would it cost to have 4x Large Fuel tanks and a speed of 250?
What’s the cheapest / smallest IRST scout ship?

… and so on.

Unfortunately, since we started work, 1.61 has changed the balance and cost of modules, so I will have to update this to reflect the new stats. Still, try it out and see if you like the idea, while I work to do so.

Feedback

Please give us feedback and suggestions. You can post here, send us messages, or email me at hfopt-support@jodavaho.io

If you want to help out with the non-zero hosting cost, or buy us a coffee, you can here:

Hosting, Backend, cogitator optimizer, data: jodavaho, https://patreon.com/jodavaho
Frontend, design: Altho, https://github.com/sponsors/altho

Other credits

Much thanks to the /r/highfleet community for discussions, ship files to test with, and some early prototypes of code (esp /u/d0d0b1rd for helping with stats early on)

We’re working to build:

A way to “preview” ships before you download. It might be module list at first, but eventually we’d like to render the ship
“Stars” or saving ships to lists, and so on
A way to “save” the cogitator input and output, so that you can resume progress.
Performance improvements to the cogitator and shipyard verifier.

Please don’t hesitate to suggest other things! We really have loved building this.

Python tip 22: possessive quantifiers

learnbyexample — Thu, 13 Apr 2023 03:00:00 GMT

Until Python 3.10, you had to use alternatives like the third-party regex module for possessive quantifiers and atomic grouping. The re module supports these features from Python 3.11 version.

Greedy quantifiers will match as much as possible but will backtrack to help the overall pattern to succeed. Possessive quantifiers behave like greedy but won't backtrack.

Suppose you want to match integer numbers greater than or equal to 100 where these numbers can optionally have leading zeros.

>>> numbers = '42 314 001 12 00984'

# this solution fails because 0* and \d{3,} can both match leading zeros
# and greedy quantifiers will give up characters to help overall regex succeed
>>> re.findall(r'0*\d{3,}', numbers)
['314', '001', '00984']

# here 0*+ will not give back leading zeros after they are consumed
>>> re.findall(r'0*+\d{3,}', numbers)
['314', '00984']

# workaround if possessive quantifiers are not supported
>>> re.findall(r'0*[1-9]\d{2,}', numbers)
['314', '00984']

Here's another example. The goal is to match lines whose first non-whitespace character is not a # character. A matching line should have at least one non-# character, so empty lines and those with only whitespace characters should not match.

>>> lines = ['#cmt', 'c = "#"', '\t #comment', 'abc', '', ' \t ']

# this solution fails because \s* can backtrack
# and [^#] can match a whitespace character as well
>>> [e for e in lines if re.match(r'\s*[^#]', e)]
['c = "#"', '\t #comment', 'abc', ' \t ']

# this works because \s*+ will not give back any whitespace characters
>>> [e for e in lines if re.match(r'\s*+[^#]', e)]
['c = "#"', 'abc']

# workaround if possessive quantifiers are not supported
>>> [e for e in lines if re.match(r'\s*[^#\s]', e)]
['c = "#"', 'abc']

See my blog post on possessive quantifiers and atomic grouping for more examples, details about catastrophic backtracking and so on.

Video demo:

See also my 100 Page Python Intro and Understanding Python re(gex)? ebooks.

Python tip 26: atomic grouping

learnbyexample — Thu, 13 Apr 2023 03:00:00 GMT

Until Python 3.10, you had to use alternatives like the third-party regex module for possessive quantifiers and atomic grouping. The re module supports these features from Python 3.11 version.

Greedy and non-greedy quantifiers will backtrack to help the overall pattern to succeed. The syntax for an atomic group is (?>pat), where pat is the pattern you want to safeguard from further backtracking. You can think of it as a special group that is isolated from the other parts of the regular expression.

Here's an example with greedy quantifier:

>>> import re
>>> numbers = '42 314 001 12 00984'

# 0* is greedy and the (?>) grouping prevents backtracking
# same as: re.findall(r'0*+\d{3,}', numbers)
>>> re.findall(r'(?>0*)\d{3,}', numbers)
['314', '00984']

Here's an example with non-greedy quantifier:

>>> ip = 'fig::mango::pineapple::guava::apples::orange'

# this matches from the first '::' to the first occurrence of '::apple'
>>> re.search(r'::.*?::apple', ip)[0]
'::mango::pineapple::guava::apple'

# '(?>::.*?::)' will match only from '::' to the very next '::'
# '::mango::' fails because 'apple' isn't found afterwards
# similarly '::pineapple::' fails
# '::guava::' succeeds because it is followed by 'apple'
>>> re.search(r'(?>::.*?::)apple', ip)[0]
'::guava::apple'

Video demo:

See also my 100 Page Python Intro and Understanding Python re(gex)? ebooks.

Real World CPU profiling of ngram/trigram tokenization in Go to reduce index time in searchcode.com

Ben E. C. Boyter — Wed, 12 Apr 2023 03:00:00 GMT

Another progress update on https://searchcode.com where I was trying to reduce the time it takes to index source code. Since the this was a real world example of profiling code trying to reduce its cost I thought I would document how it went.

The first step is to add code allowing the collection of a profile.

I don’t do this often enough in searchcode to have a want to enable/disable this so I just comment/uncomment this when needed. Note that I have this set to stop profiling 30 seconds after it starts, which is more than enough for me to identify CPU pressure as the index process is the first thing to start.

Python 3.11: possessive quantifiers and atomic grouping added to re module

learnbyexample — Wed, 12 Apr 2023 03:00:00 GMT

Quoting from What's New In Python 3.11:

Atomic grouping ((?>...)) and possessive quantifiers (*+, ++, ?+, {m,n}+) are now supported in regular expressions. (Contributed by Jeffrey C. Jacobs and Serhiy Storchaka in bpo-433030.)

Poster created using Canva

If you are not familiar with regular expressions, see my Understanding Python re(gex)? ebook to get started.

Backtracking🔗

Greedy quantifiers match as much as possible, provided the overall regex is satisfied. For example, :.* will match : followed by rest of the input line. However, if you change the pattern to :.*apple, the .* portion cannot simply consume the rest of the input line. The regex engine will have to find the largest portion such that apple is also part of the match (provided the input has such a string, of course).

>>> import re

>>> ip = 'fig:mango:pineapple:guava:apples:orange'

>>> re.search(r':.*', ip)[0]
':mango:pineapple:guava:apples:orange'

>>> re.search(r':.*apple', ip)[0]
':mango:pineapple:guava:apple'

For the :.*apple case, the Python regular expression engine actually does consume all the characters on seeing .*. Then realizing that the overall match failed, it gives back one character from the end of line and checks again. This process is repeated until a match is found or failure is confirmed. In regular expression parlance, this is called backtracking.

This type of exploring matches to satisfy overall regex also applies to non-greedy quantifiers. .*? will start with zero characters followed by one, two, three and so on until a match is found.

>>> ip = 'fig:mango:pineapple:guava:apples:orange'

>>> re.search(r':.*?', ip)[0]
':'

>>> re.search(r':.*?apple', ip)[0]
':mango:pineapple'

Note that some regex engines like re2 do not use backtracking.

Possessive quantifiers🔗

Until Python 3.10, you had to use alternatives like the third-party regex module for possessive quantifiers. The re module supports possessive quantifiers from Python 3.11 version.

The difference between greedy and possessive quantifiers is that possessive will not backtrack to find a match. In other words, possessive quantifiers will always consume every character that matches the pattern on which it is applied. Syntax wise, you need to append + to greedy quantifiers to make it possessive, similar to adding ? for non-greedy case.

Unlike greedy or non-greedy quantifiers, :.*+apple will never match, because .*+ will consume rest of the line, leaving no way to match apple.

$ python3.11 -q
>>> import re

>>> ip = 'fig:mango:pineapple:guava:apples:orange'

>>> re.search(r':.*+', ip)[0]
':mango:pineapple:guava:apples:orange'

>>> bool(re.search(r':.*+apple', ip))
False

Here's a more practical example. Suppose you want to match integer numbers greater than or equal to 100 where these numbers can optionally have leading zeros.

>>> numbers = '42 314 001 12 00984'

# this solution fails because 0* and \d{3,} can both match leading zeros
# and greedy quantifiers will give up characters to help overall regex succeed
>>> re.findall(r'0*\d{3,}', numbers)
['314', '001', '00984']

# here 0*+ will not give back leading zeros after they are consumed
>>> re.findall(r'0*+\d{3,}', numbers)
['314', '00984']

# workaround if possessive quantifiers are not supported
>>> re.findall(r'0*[1-9]\d{2,}', numbers)
['314', '00984']

>>> lines = ['#comment', 'c = "#"', '\t #comment', 'abc', '', ' \t ']

# this solution fails because \s* can backtrack
# and [^#] can match a whitespace character as well
>>> [e for e in lines if re.match(r'\s*[^#]', e)]
['c = "#"', '\t #comment', 'abc', ' \t ']

# this works because \s*+ will not give back any whitespace characters
>>> [e for e in lines if re.match(r'\s*+[^#]', e)]
['c = "#"', 'abc']

# workaround if possessive quantifiers are not supported
>>> [e for e in lines if re.match(r'\s*[^#\s]', e)]
['c = "#"', 'abc']

Atomic grouping🔗

(?>pat) is an atomic group, where pat is the pattern you want to safeguard from further backtracking by isolating it from other parts of the regex.

Here's an example with greedy quantifier:

>>> numbers = '42 314 001 12 00984'

# 0* is greedy and the (?>) grouping prevents backtracking
# same as: re.findall(r'0*+\d{3,}', numbers)
>>> re.findall(r'(?>0*)\d{3,}', numbers)
['314', '00984']

Here's an example with non-greedy quantifier:

>>> ip = 'fig::mango::pineapple::guava::apples::orange'

# this matches from the first '::' to the first occurrence of '::apple'
>>> re.search(r'::.*?::apple', ip)[0]
'::mango::pineapple::guava::apple'

# '(?>::.*?::)' will match only from '::' to the very next '::'
# '::mango::' fails because 'apple' isn't found afterwards
# similarly '::pineapple::' fails
# '::guava::' succeeds because it is followed by 'apple'
>>> re.search(r'(?>::.*?::)apple', ip)[0]
'::guava::apple'

The regex module has a regex.REVERSE flag to match from right-to-left making it better suited than atomic grouping for certain cases.

>>> import regex

>>> ip = 'fig::mango::pineapple::guava::apples::orange'
>>> regex.search(r'(?r)::.*?::apple', ip)[0]
'::guava::apple'

# this won't be possible with just atomic grouping
>>> ip = 'and this book is good and those are okay and that movie is bad'
>>> regex.search(r'(?r)th.*?\bis bad', ip)[0]
'that movie is bad'

Catastrophic Backtracking🔗

Backtracking can become significantly time consuming for certain corner cases. Which is why some regex engines do not use them, at the cost of not supporting some features like lookarounds. If your application accepts user defined regex, you might need to protect against such catastrophic patterns. From wikipedia: ReDoS:

A regular expression denial of service (ReDoS) is an algorithmic complexity attack that produces a denial-of-service by providing a regular expression and/or an input that takes a long time to evaluate. The attack exploits the fact that many regular expression implementations have super-linear worst-case complexity; on certain regex-input pairs, the time taken can grow polynomially or exponentially in relation to the input size. An attacker can thus cause a program to spend substantial time by providing a specially crafted regular expression and/or input. The program will then slow down or becoming unresponsive.

Here's an example:

>>> from timeit import timeit

>>> greedy = re.compile(r'(a+|\w+)*:')
>>> possessive = re.compile(r'(a+|\w+)*+:')

# string that'll match the above patterns
>>> s1 = 'aaaaaaaaaaaaaaaa:123'
# string that does NOT match the above patterns
>>> s2 = 'aaaaaaaaaaaaaaaa-123'

# no issues when input string has a match
>>> timeit('greedy.search(s1)', number=10000, globals=globals())
0.016464739997900324
>>> timeit('possessive.search(s1)', number=10000, globals=globals())
0.016358205997676123

# if input doesn't match, greedy version suffers from catastrophic backtracking
# note that 'number' parameter is reduced to 10 since it takes a long time
>>> timeit('greedy.search(s2)', number=10, globals=globals())
53.71723825200024
>>> timeit('possessive.search(s2)', number=10, globals=globals())
0.00019008600065717474

(a+|\w+)*: is a silly regex pattern, since it can be rewritten as \w*: which will not suffer from catastrophic backtracking. But this example shows how quantifiers applied to a group with multiple alternatives using quantifiers can lead to explosive results. More such patterns and mitigation strategies can be found in the following links:

Deduplicate a slice in Go, use sort or a map?

Ben E. C. Boyter — Tue, 11 Apr 2023 03:00:00 GMT

TL;DR: If you don’t need to preserve order of the elements, sort the slice first and dedupe.

I have been working on https://searchcode.com a fair bit recently, having been dealing with memory issues which were causing it to crash with out of memory exceptions. The cause of that was due to backtracking regular expressions in the syntax highlighter, which will be the subject of another post sometime.

As a result of the above I have spent a fair amount of time looking at memory profiles. One thing that stood out to me when doing so was a call I made to remove duplicate values from an array/slice of uint64. A fairly simple fuction that appeared often enough in the output.

NEEDY GIRL OVERDOSE review

ブラック — Sun, 09 Apr 2023 03:00:00 GMT

NEEDY GIRL OVERDOSE is definitely an interesting experience.

The gameplay is essentially a raising simulator, where you make decisions for OMGkawaiiAngel to make her the biggest streaming star on the Internet. It doesn’t take long to wrap your head around the available choices and the day-to-day rhythm. Advancing through a single day is also quite fast, so you can make through the game pretty fast. Nothing too advanced here, just a core gameplay loop that does what it needs to do to present the story.

Story-wise, it’s a bit hard to comment. The game doesn’t play like an ordinary visual novel with a branching storyline. It’s more of a Choose Your Own Adventure book where you’re constantly making decisions that affect what happens, and at the end, you’ll land on one of the endings. Hence, there is not really a “story”.

The endings vary from humorous ones to more serious ones, and there’s a couple of dozen of them. Not gonna lie, some of the endings did hit me like a ton of bricks. And luckily grinding them isn’t too arduous of a task. However, if you go for 100% completion, you’ll probably need a walkthrough – some of the endings are pretty opaque as to how you’ll reach them.

I also need to point attention towards the localisation: it's fantastic. It actually feels hard to imagine that it's been translated from Japanese. Feels like they definitely put in effort to make it work in English.

The art is perfection. From the character designs, to the ways the sprites are drawn, the various filters that happen when things start to go awry and the faux Windows 95 user interface… I can’t find a single thing that I do not love here. I’d say just getting to ogle at the game is well than worth the price of admission alone, considering that this is not an expensive title.

The music is also superb. The chiptone soundtrack fits the art and story and it definitely didn’t get old during my time with the game even though you will hear a lot of repetition. And the theme song is just a chef’s kiss on top of that. Naturally, you can listen to it in its entirety in the game, as well as hear a chiptone arrangement of it as one of the background tracks.

A fairly short and bittersweet, I’d definitely recommend NEEDY GIRL OVERDOSE to anyone with even the vaguest of interest in it.

Writing a simple C++20 async message queue server - Part II

Bytepawn - Marton Trencseni — Sat, 08 Apr 2023 01:00:00 GMT

I write a somewhat more complicated, but still relatively simple async message queue server in modern C++20.

AGI Doom and the Drake equation

I Am Not a Robot — Fri, 07 Apr 2023 21:06:43 GMT

Why those who claim AI will kill humanity are extremely pessimistic and don't do their homework

Mike teaches Chat GPT about Cesium Beam Atomic clocks

--Mike-- — Tue, 04 Apr 2023 07:50:11 GMT

Here's the text of my most recent interaction with ChatGPT

Mike Warot: explain how cesium beam atomic clocks work, in the style of Richard Feynman

Ah, yes, cesium beam atomic clocks - a fascinating piece of technology! Let me explain how they work in the style of Richard Feynman.

First, let's take a step back and talk about what we mean by time. Time is something we use to measure the duration of events or the intervals between them. But what is it really? Is it a physical quantity, like distance or mass? Well, it turns out that time is a bit more elusive than that. In fact, one could argue that time is simply what a clock measures.

So, how do we make a good clock? We need something that oscillates regularly and predictably. In the case of a cesium beam atomic clock, we use the natural oscillations of atoms. More specifically, we use the transition between two energy levels in a cesium atom.

Here's how it works. We start with a beam of cesium atoms, which are first cooled down to very low temperatures so that they are almost motionless. Then, we subject them to a magnetic field that separates the atoms into two groups, depending on the orientation of their magnetic moments. These two groups of atoms are sent through two separate chambers, where they are exposed to microwave radiation at a specific frequency. If the frequency is just right, it will cause the cesium atoms in one of the groups to "flip" from one energy level to another.

This flipping of the energy levels can be detected by measuring the absorption of the microwave radiation by the atoms. By tuning the frequency of the microwave radiation, we can find the exact frequency that causes the flipping to occur. This frequency is known as the resonance frequency of the cesium atom.

Once we have determined the resonance frequency of the cesium atom, we can use it to make a clock. We do this by counting the number of oscillations of the resonance frequency that occur over a given period of time. By definition, one second is the duration of 9,192,631,770 oscillations of the resonance frequency of the cesium atom.

So, that's how cesium beam atomic clocks work. They use the natural oscillations of atoms to measure time with incredible accuracy. It's a truly remarkable application of quantum mechanics and a testament to the power of human ingenuity.

Mike Warot: but that's not correct, the cesium beam clock I worked on started with an oven containing the cesium

You are absolutely right, and I apologize for any confusion my previous answer may have caused. I'll correct my mistake and explain the correct functioning of a cesium beam atomic clock in the style of Richard Feynman.

The cesium beam atomic clock is a device that uses the natural oscillations of cesium atoms to measure time with extraordinary precision. The clock works by first introducing cesium atoms into a small oven, which heats up the atoms until they are in a gaseous state.

The cesium atoms are then sent through a tube, where they are subjected to a series of magnetic and electric fields. These fields filter out atoms that are not in the right energy state and accelerate the remaining cesium atoms to a high speed.

Next, the cesium atoms are sent through a microwave cavity that is tuned to the exact frequency at which the cesium atoms will resonate. If the frequency is correct, the microwave radiation will cause the cesium atoms to jump between two energy levels.

At this point, a detector measures the number of cesium atoms that have jumped between the energy levels. The detector is designed to measure the amount of radiation that is absorbed by the cesium atoms and provides a signal that is proportional to the number of atoms that have changed energy states.

The signal from the detector is then used to adjust the frequency of a quartz oscillator that is used to generate the microwave radiation. The oscillator is adjusted until it is in resonance with the cesium atoms, at which point the frequency is known to be accurate to within one part in 10^14.

Finally, the frequency of the quartz oscillator is used to keep time. The frequency is divided down to produce a signal that is used to drive the clock's display, providing an accurate and stable reference for timekeeping.

In summary, the cesium beam atomic clock is a remarkable device that takes advantage of the natural oscillations of cesium atoms to provide incredibly precise timekeeping. Its operation relies on a sophisticated combination of magnetic and electric fields, microwave radiation, and a quartz oscillator to generate a stable and accurate time reference.

Mike Warot: but that's not true, the cesium atoms in the clock I repaired were driven by the motion when they exited the holes in the oven, and were not accelerated after that

I apologize for my previous mistake and any confusion it may have caused. Thank you for correcting me. Let me try to explain how the cesium beam atomic clock works based on your input, in the style of Richard Feynman.

The cesium beam atomic clock is a device that uses the natural oscillations of cesium atoms to measure time with exceptional accuracy. The clock starts with a small oven containing cesium atoms. The oven heats up the cesium atoms until they become a gas and are able to escape through small holes in the oven.

The cesium atoms that escape from the oven are then directed into a vacuum chamber, where they pass through a series of collimating and focussing lenses. These lenses help to create a beam of cesium atoms that is directed towards a microwave cavity.

Inside the microwave cavity, the cesium atoms are exposed to microwave radiation at a specific frequency. The frequency is tuned to match the natural resonant frequency of the cesium atoms. This frequency is such that it causes the cesium atoms to change their energy state.

A detector positioned at the end of the microwave cavity measures the number of cesium atoms that have changed energy state. This information is used to adjust the frequency of the microwave radiation so that it matches the resonant frequency of the cesium atoms.

The oscillation frequency of the cesium atoms is then used to keep time. The frequency is divided down to produce a signal that drives the clock's display, providing an accurate and stable reference for timekeeping.

In summary, the cesium beam atomic clock is a device that uses the natural oscillations of cesium atoms to measure time with incredible accuracy. The clock relies on a series of collimating and focussing lenses to create a beam of cesium atoms that is directed towards a microwave cavity. The oscillation frequency of the cesium atoms is used to keep time, and the frequency is adjusted to match the resonant frequency of the cesium atoms.

Mike Warot: there are no magnetic lenses in the cesium clock I repaired, but there were magnets to sort their energy state, so you got that part right

Thank you for the clarification. Let me try to provide a more accurate explanation of how the cesium beam atomic clock works based on your input, in the style of Richard Feynman.

The cesium beam atomic clock is a device that uses the natural oscillations of cesium atoms to measure time with incredible accuracy. The clock starts with a small oven containing cesium atoms. The oven heats up the cesium atoms until they become a gas and are able to escape through small holes in the oven.

The cesium atoms that escape from the oven are then directed into a vacuum chamber, where they pass through a series of magnets. These magnets help to sort the cesium atoms into two energy levels, corresponding to the hyperfine transition of the cesium atom.

Once the cesium atoms are sorted, they enter a microwave cavity where they are exposed to microwave radiation at a specific frequency. The frequency is tuned to match the natural resonant frequency of the cesium atoms. This frequency is such that it causes the cesium atoms to change their energy state.

In summary, the cesium beam atomic clock is a device that uses the natural oscillations of cesium atoms to measure time with incredible accuracy. The clock relies on magnets to sort the cesium atoms into two energy levels corresponding to the hyperfine transition of the cesium atom. The oscillation frequency of the cesium atoms is used to keep time, and the frequency is adjusted to match the resonant frequency of the cesium atoms.

Mike Warot: that's pretty good, thank you!

You're welcome! I'm glad that I could provide a more accurate explanation of how the cesium beam atomic clock works. If you have any more questions or clarifications, feel free to ask!

Vim tip 24: movement commands within the current file

learnbyexample — Tue, 04 Apr 2023 03:00:00 GMT

Here are some commands you can use in Normal mode to move within the current file:

gg move to the first non-blank character of the first line
G move to the first non-blank character of the last line
5G move to the first non-blank character of the fifth line
- As an alternative, you can use :5 followed by Enter key (Command-line mode)
50% move to the halfway point
- you can use other percentages as needed
% move to matching pair of brackets like (), {} and []
- This will work across lines and nesting is taken into consideration as well
- If the cursor is on a non-bracket character and a bracket character is present later in the line, the % command will move to the matching pair of that character (which could be present in some other line too)
- Use the matchpairs option to customize the matching pairs. For example, :set matchpairs+=<:> will match <> as well

It is also possible to match a pair of keywords like HTML tags, if-else, etc with %. See :h matchit-install for details.

Video demo:

What I'm up to - April 2023

Philip I. Thomas — Mon, 03 Apr 2023 18:38:45 GMT

✨ What I've been up to

Wow, it's Q2 already!

I spent a couple of weeks in Paris last month, where I worked remotely and found some amazing coffee.

For Postcard - I rebuilt the post-writing flow, which makes the entire product feel much more polished. I also hired a product marketing intern to help with some growth initiatives. I wrote about improving my motivation in "Hacking Dopamine for Entrepreneurial Success."

I also enjoyed a more spontaneous trip to Atlanta for work, where Valor Coffee was a highlight.

🤔 Things to share

Articles: What Happens When a Company (Like Patagonia) Transfers Ownership to a Nonprofit? (I was surprised this is common in the Nordics, e.g., Maersk shipping and Carlsberg beer are run by trusts), Maybe Treating Housing as an Investment was a Colossal, Society-Shattering Mistake, Hardware Microphone Disconnect as another example of Apple going above-and-beyond in security.
Books: Silo - The Zero Waste Blueprint is about a restaurant in London with no trash bin, which seems like the future to me. I Know This to Be True: Rene Redzepi: On Teamwork, Creativity, and Kindness.
Trends: Dead Internet Theory is accelerating due to AI.
Video: Ludovico Einaudi Tiny Desk.
Apps: VideoAsk - like Typeform, but more personal.
Coffees I'm drinking: Panama Finca Deborah Geisha Echo from Substance Cafe (fantastic!), Ethiopia Worka Sakara Natural from Valor Coffee
Quote: "Every founder I meet wants to build a consumer business when they start out, and then they realize consumer is really freaking hard." - Alex Boaziz, founder of Deel, in Stratechery

📫 What I'm up to this month

I'm working on an update to the Contraption Co. website, which I will publish soon. I also plan to launch a fun mini-project this month inspired by laptops in cafes.

I have two major Postcard features coming out this month, and I plan to spend some time getting a new project going.

Besides that, I have more travel planned.

📍 Where I'll be

(Let me know if we overlap!)

April 9-12: 🌉 Bay Area
April 23-30: Cleveland

Working remotely from Paris

Joins Don't Scale

Dan Palmer — Sun, 02 Apr 2023 02:00:00 GMT

A classic part of the NoSQL sales pitch is that SQL JOINs are too expensive and don’t scale, and a classic response is to point to big websites running smoothly on SQL databases. The reality, as always, is a bit more complicated than that. Types of scale When engineers talk about scale, they’re almost always referring to some sort of usage scale, but even this is not always clear. Usage scale can take the form of:

Writing a simple C++20 async message queue server - Part I

Bytepawn - Marton Trencseni — Sun, 02 Apr 2023 01:00:00 GMT

I write a simple, bi-directional async message queue server in modern C++20.

Engineering with Code Ownership

Dan Palmer — Fri, 31 Mar 2023 02:00:00 GMT

Code Ownership is the practice of assigning explicit owners to areas of codebases. Before Google I worked at small companies where it’s easy to know who should review each code change, but that doesn’t scale far. Even in a team of 10 it wasn’t always obvious who knew an area of code the best, and it was certainly less clear for new starters. Various tools have been developed to help with this.

Should you use OpenAI's embeddings? Probably not, and here's why.

I Am Not a Robot — Thu, 30 Mar 2023 22:40:35 GMT

Recently semantic search has exploded.

Using simple things

bayindirh — Wed, 29 Mar 2023 03:00:00 GMT

As I try to move towards minimalism, I discover what it is like in different dimensions.

When one starts to explore minimalism, it starts outside one's being, with the process of downsizing and keeping that state. Literature targets reduction of physical (and sometimes digital) belongings and presents it as the starting point and the only possible path. For these methods, victory is achieved when nothing else can be removed from peoples' living spaces.

The promise of removing items from one's life is simple, as I outlined in my previous post, Meditations on minimalism: Reduced mental load and having more time.

While these results are correct, directly targeting belongings as a starting point is a shallow perspective on the issue, in my opinion. The process is concerned with items themselves and how the result looks from outside. It's akin to curating your garden for that perfect look to envy your neighbors and pushes the owner and expected utility of the lawn out of the picture, like it doesn't matter.

However, for most of us, external clutter and build-up is a reflection of our inner state and disorganization. In other words, having a minimal set of belongings or preventing ingress of extra items is not possible without solving the problems causing these things in the first place. Then, the question evolves from "How can I reduce my belongings?" to "What is causing this crowdedness in the first place?".

I have found out that the biggest contributor to this crowdedness is stress. When the stress level starts to increase in my life, I try to balance it out by adding things which brings joy or postpones this stress until I have strength to deal with it, meaning I play more casual games, tend to procrastinate more by reading articles and watching videos, and by buying things. When I reach to the other end of the stressful period, I find myself with a small mound of clutter, which pushes me back in my journey.

However, I found another way.

During these periods between trying to cope with stress and having an easy time, I found out that one of the things I enjoy most is using simple things. The items are I put in this category have common properties. They are high quality items which do their job well, are not over the top neither in appearance or in cost, are not disposable, and are not overly complicated. Like a well made fountain pen or a double edge razor.

These items make me happy in more than one way. First, these items are built to last a lifetime and beyond when cared correctly. Second, using these items for a long time makes them accumulate memories. Writing an important note, something to remember, or a letter? That pen has seen it all, and will make you remember. Same for your other items you use for a long time. All will store and will make you remember important or significant moments in your life, reminding you what you have gone through.

And, in these moments, you will feel that you have lived and you're alive. Feeling this is both precious and way more fulfilling than many things in life. It allows you to see beyond the daily routine and peek into your soul, helping you to root better and balance yourself.

Use simple things, and simplify your soul before trying to simplify your material belongings. Because, minimalism starts inside.

Until next time,

Be kind.

Muv-Luv Unlimited: THE DAY AFTER 03 review

ブラック — Tue, 28 Mar 2023 03:00:00 GMT

Finally, the explosive finale to THE DAY AFTER series. Except it’s not really the finale – they’re just calling the continuation “Muv-Luv Resonative” instead of “THE DAY AFTER”, but I digress.

When I gathered my thoughts about THE DAY AFTER 02, I described it as "the best that THE DAY AFTER had to offer so far”. This was definitely correct – the best so far, as the best that THE DAY AFTER series has to offer is most definitely THE DAY AFTER 03.

The story feels packed for a relatively short read – I clocked 7.7 hours to complete it, or 30 minutes less than it took me to finish THE DAY AFTER 01. But unlike the other parts of THE DAY AFTER series, it feels like a significant increase in quality, and I never felt like I had to wait for the good part to begin like I did with the previous entries. It feels to the rest of THE DAY AFTER series what the Muv-Luv Alternative feels like its predecessors, although in condensed form.

Everything is still fully voiced with good voice acting. I don’t think I saw that many new sprites for returning characters. However, THE DAY AFTER 03 uses the old sprites in a much more animated way that makes everything feel that much more alive. In that aspect too, THE DAY AFTER 03 feels like a significant step up from the previous episodes. So while the old sprites may have not been anything amazing, it still gets a passing grade from me on that department.

On technical aspects, most stuff is fine. You’ll need a custom controller layout if you plan on using a controller on PC or reading it on a Steam Deck. And it reads just fine on the Steam Deck barring some post-sleep audio crackling that you can resolve by restarting the game. You should also stick to reading THE DAY AFTER 03 on one device – the way this game handles Steam Cloud saves makes switching between devices not impossible but very annoying.

THE DAY AFTER 03 is a definitive high note to end the series – at least until the fabled Muv-Luv Resonative comes out some fateful day.

CLI tip 25: get file properties using the stat command

learnbyexample — Tue, 28 Mar 2023 03:00:00 GMT

The stat command is useful to get details like file type, size, inode, permissions, last accessed and modified timestamps, etc. You'll get all of these details by default. The -c and --printf options can be used to display only the required details in a particular format.

Here's an example to get accessed and modified timestamps of a file:

# sample directory and sample file
$ mkdir stat_examples && cd $_
$ printf 'long\nshot\n' > ip.txt

# %x gives the last accessed timestamp
$ stat -c '%x' ip.txt
2023-03-27 20:20:55.217530670 +0530

# modify the file
$ printf 'apple\nbanana\n' >> ip.txt
# %y gives the last modified timestamp
$ stat -c '%y' ip.txt
2023-03-27 20:21:50.298964283 +0530

Here's an example with some more file properties:

# %s gives file size in bytes
# \n is used to insert a newline
# %i gives the inode value
# same as: stat --printf='%s\n%i\n' ip.txt
$ stat -c $'%s\n%i' ip.txt
23
6438890

Here's an example for a linked file:

$ ln -s /usr/share/dict/words words.txt

# %N gives quoted filenames
# if input is a link, path it points to is also displayed
$ stat -c '%N' words.txt
'words.txt' -> '/usr/share/dict/words'

You can also pass multiple file arguments:

$ printf '#!/bin/bash\n\necho hi\n' > hi.sh

# %s gives file size in bytes
# %n gives filenames
$ stat -c '%s %n' ip.txt hi.sh
23 ip.txt
21 hi.sh

The stat command should be preferred instead of parsing ls -l output for file details. See mywiki.wooledge: avoid parsing output of ls and unix.stackexchange: why not parse ls? for explanation and other alternatives.

Video demo:

See also my Linux Command Line Computing ebook.

Slow Travel in Paris: Discovering Substance Cafe

Philip I. Thomas — Sat, 25 Mar 2023 20:25:08 GMT

I just returned from two weeks of "slow travel" in Paris. Instead of visiting museums and attractions, "slow travel" consists of living my everyday life, but in a different city. My trip generally consisted of working during the day, then going out to dinner at night. Every afternoon, I would take a break from work and walk to explore a local coffee shop. Throughout the trip, I visited numerous cafes and had much delicious coffee. But, one place stood out as particularly special - Substance Cafe.

Located in the heart of Paris, Substance Cafe is the brainchild of Joachim, a competition barista who has transformed his passion for coffee into a unique and captivating experience.

I am consistently captivated by independent creators operating at their peak potential. Joachim is one of those people. He flies around the world to participate in coffee competitions, then returns to Substance Cafe as his workshop to train and hone his skills before the next event. He operates the cafe mostly alone, and has worked meticulously to control and optimize every step of coffee-making from the farm to cup. He sources and roasts the beans himself, formulates custom water recipes, and rebuilds his equipment in pursuit of the perfect coffee. At Substance Cafe, guests are offered a front-row seat to Joachim's creative pursuit.

Substance Cafe is a far cry from your typical neighborhood coffee shop. You won't find people queuing for their morning caffeine fix, as the doors only open at noon. In place of tables, takeaway cups, and sugar, you'll discover 15 stools at a bar arranged around the espresso machine, inviting guests to become an audience to the coffee-making process. While a concise standard menu of familiar cafe drinks is available, the pièce de résistance at Substance Cafe is the Omakase experience.

Joachim curates a symphony of flavors through a rotating selection of a dozen specialty coffee beans, personally chosen to transport patrons on a sensory tour of the coffee world. This Omakase menu lies at the heart of Substance Cafe's ethos – a celebration of the finest coffees, brewed to highlight the unique terroir of the farm where it was grown. The experience is akin to a tasting menu at Noma, where every course challenges your preconceptions about coffee. While the price of an espresso can reach $20, it's a small investment to partake in an experience that is among the best in the world.

During my trip, I went to Substance Cafe three times, pulled in by the allure of its exceptional espresso offerings. Although Joachim is equally renowned for his V60 pourover coffee, I focused on espresso - a complex art form I'd never dare to attempt at home. Furthermore, it's a rarity to find cafes that can expertly pull shots from specialty light-roasted beans. Within the walls of Substance Cafe, I experienced the pinnacle of espresso – a Geisha from Finca Deborah in Panama. According to Joachim, this remarkable coffee is the only one he has ever bestowed a perfect 10 out of 10 rating for flavor. Each sip was akin to a kaleidoscopic journey, with a dynamic array of tastes unfurling on my palate – from vibrant orange to delicate jasmine. As the coffee cooled, its floral notes gracefully emerged, adding yet another layer of complexity to this unforgettable espresso experience.

In an age where remote work and digital interactions dominate our lives, Substance Cafe serves as a refreshing reminder of the power and beauty of purpose-built physical spaces that foster engagement and connection. The cafe itself is a living testament to the artisan's relentless quest for excellence and the enriching experience it offers to its patrons.

I can't help but dwell on the role the Internet plays in Substance Cafe. It enables Joachim to find and connect with niche community of coffee aficionados who share his values and devotion to the craft. These customers are willing to spend $20 on an espresso shot, visit the cafe only on weekday afternoons, and dedicate an hour to appreciating a cup of coffee. In this sense, the story of Substance Cafe serves as a testament to the positive effects of technology, enabling passionate individuals to create unique and meaningful work that resonates with a global audience.

Ultimately, Substance Cafe exemplifies the remarkable potential that lies at the intersection of passion, craftsmanship, and technology. It serves as a beacon of inspiration for creators everywhere, reminding us that in a world that often moves too fast, there is still room for those who dare to slow down and strive for perfection in their craft.

Going Github

Weblog on marginalia.nu — Sat, 25 Mar 2023 02:00:00 GMT

I’ve moved Marginalia’s sources to Github. Can’t pick every battle. The main reason is I’m kind of tired of the amount of spam bots that keep signing up to my Gitea. The juice of self-hosting a public-access git forge, even locked down to prevent arbitrary repo creation, that juice just isn’t worth the squeeze. This is not without some consideration. To be blunt, I don’t like Github. Their use of dark patterns leaves a real nasty after-taste.

Muv-Luv Unlimited: THE DAY AFTER 02 review

ブラック — Fri, 24 Mar 2023 02:00:00 GMT

The third instalment of the THE DAY AFTER series, unless you decided to skip THE DAY AFTER 00 for some reason. I read somewhere that THE DAY AFTER 02 was the best yet that the THE DAY AFTER series had to offer. To be honest, I didn’t really believe it for approximately half of the story. It didn’t feel like the story was really progressing, but rather that I was getting a beat down.

Luckily that changes somewhere around halfway through the story. After that, it definitely picks up. The story gets more stakes and you see a lot more action. THE DAY AFTER series also finally starts adjoining the prologue TDA 00 into the overall storyline. And as the story progresses towards the finale, I felt that it definitely did become the best that THE DAY AFTER had to offer so far.

On the character level, THE DAY AFTER 02 adds several interesting characters, like Ogami and Komaki. Maybe they won’t be everyone’s favourite characters, but I did find them to make the story more interesting. THE DAY AFTER 02 also continues re-introducing familiar faces. Whereas previous instalments of the series could haul the old-timers around in a car, THE DAY AFTER 02 will need a minibus to carry all of its familiar faces. So it’s also definitely the best that THE DAY AFTER has to offer for fans of the series who want to bridge the gap between Unlimited and THE DAY AFTER.

On the art and sound direction there isn’t that much new. You still have a bunch of the old sprites and the main characters also get some new sprites. Honestly speaking some of the new sprites didn’t really seem to match well with the old sprites, so it comes out as a bit off-kilter. That being said, not having the new sprites as well would’ve felt a bit like they were cheaping out. Some of the art is still too low resolution for the amounts of zoom they’re asked of as well. Everything is still fully voiced and the performances are good.

The span of the story gives me a bit of pause though. Steam shows me that it took me 8.2 hours to read through TDA 01 whereas TDA 02 was a 6.7-hour read to completion. That’s a full 90 minutes less reading for the same price point. And there was definitely less effort needed on the art side, since a bunch of the assets could be recycled. But then again, if getting length-parity with TDA 01 would’ve required padding out the beginning of the story, I definitely wouldn’t want it. Yet, it definitely stings the part of the brain that considers value for money, even if the overall story is better than TDA 01.

All in all, it’s very easy to recommend THE DAY AFTER 02. You’re probably not getting this far into the series without having an investment into it, and despite some of the initial drudgery, you definitely get a payoff at the end.

Search Result Quality For Multiple Terms

Weblog on marginalia.nu — Thu, 23 Mar 2023 02:00:00 GMT

This is a bit of a follow up to the previous post. The Grand Code Restructuring [ 2023-03-17 ] Marginalia’s search result quality has, for a long while, been pretty good as long as your search query is a single term, but for multiple search terms it’s been a bit hit-and-miss. Marginalia was never great at this, but the quality of results in this usage pattern has taken a bit of a dive recently due to a re-write of the index last fall.

Errors and Zig

Notes on software development — Tue, 21 Mar 2023 02:00:00 GMT

At TigerBeetle these last few weeks I've been doing a mix of documenting client libraries, writing sample code for client libraries, and writing integration tests against the sample code.

The client library documentation is generated with a Zig script. The sample code is integration tested with a Zig script. A bunch of Zig scripts.

It's not the same rigorous sort of Zig as the main database. (We're generally more lax about scripts and test code.)

And I'm specifically writing this post on my personal blog since my script code is not under incredible scrutiny.

Furthermore, I'm still new to Zig. Since I'm still learning, there have been a few things that tripped me up.

And now that I've written this out, I realize most of my stumbling was related to errors.

Failure

Lots of things in programs allocate memory. This sounds dumb and obvious but before programming Zig I did not appreciate how many operations I'm used to allocate memory. I've previously only programmed in GC languages that do the allocations behind the scenes.

Furthermore, memory allocation can fail. Zig makes allocation failures explicit. So lots of things in Zig code need to handle failure.

Selectively omitting error handling is not allowed:

const std = @import("std");

fn thing(a: std.mem.Allocator) !void {
    std.fmt.allocPrint(a, "", .{});
}

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();

    const allocator = arena.allocator();
    try thing(allocator);
}

Run zig run test.zig:

test.zig:4:23: error: error is ignored
    std.fmt.allocPrint(a, "", .{});
    ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
test.zig:4:23: note: consider using 'try', 'catch', or 'if'
referenced by:
    main: test.zig:12:9
    callMain: /home/phil/vendor/zig-linux-x86_64-0.11.0-dev.2213+515e1c93e/lib/std/start.zig:617:32
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

This ends up meaning lots of code like:

fn do_stuff(
  alloc: std.mem.Allocator, // Let's assume this is an arena allocator so I don't care about freeing.
  stuff: Stuff,
) !void {
  var x = std.ArrayList([]const u8).init(alloc);
  try x.appendSlice(&[_][]const u8{
    "first of something",
    "one more",
  });
  try x.append(stuff.thing);
  try x.append(try std.fmt.allocPrint(alloc, "build some string {s}.\n", .{stuff.athing}));

  var other_stuff = try std.fmt.allocPrint(alloc, "things... {s}", .{blah});

  try do_other_stuff(x.items, other_stuff);
}

You have try-es all over the place.

Limits of `try`

Now I don't have a problem with acknowledging that allocations can fail. At least outside of scripts. In scripts like I've been writing though I don't really care.

Having all of those try-es is just extra typing all over the place.

It would be nice if I could have instead done:

fn do_stuff(
  alloc: std.mem.Allocator, // Let's assume this is an arena allocator so I don't care about freeing.
  stuff: Stuff,
) !void {
  var x = std.ArrayList([]const u8).init(alloc);
  try {
      x.appendSlice(&[_][]const u8{
      "first of something",
      "one more",
    });
    x.append(stuff.thing);
    x.append(std.fmt.allocPrint(alloc, "build some string {s}.\n", .{stuff.athing}));

    var other_stuff = std.fmt.allocPrint(alloc, "things... {s}", .{blah});

    do_other_stuff(x.items, other_stuff);
  }
}

But Zig's try doesn't work like that. I'm not sure why not. The Zig developers are sensible so I'm sure there's a good reason.

Still, are there other options?

`catch unreachable`

So the problem isn't just that you have to acknowledge memory allocation failures but that these failures within every helper function need to be acknowledged by the caller of the helper function. Failures infiltrate the entire call tree.

Now of course these potential failures would exist whether or not Zig exposed them. So I don't mean to say it's Zig's fault for exposing them.

But you can avoid failure handling by instead of try-ing everything, mark error conditions as unreachable.

fn do_stuff(
  alloc: std.mem.Allocator, // Let's assume this is an arena allocator so I don't care about freeing.
  stuff: Stuff,
) void {
  var x = std.ArrayList([]const u8).init(alloc);
  x.appendSlice(&[_][]const u8{
    "first of something",
    "one more",
  }) catch unreachable;
  x.append(stuff.thing) catch unreachable;
  x.append(std.fmt.allocPrint(alloc, "build some string {s}.\n", .{stuff.athing}) catch unreachable) catch unreachable;

  var other_stuff = std.fmt.allocPrint(alloc, "things... {s}", .{blah}) catch unreachable;

  do_other_stuff(x.items, other_stuff) catch unreachable;
}

As you can see from the function signature, this function no longer returns any error at all. But it could possibly panic.

Now in scripts, for things like memory allocations that can fail, I actually think it's reasonable to mark allocation failures as unreachable.

But I took it a bit further. Using @panic or unreachable in general failure conditions.

fn run(alloc: std.mem.Allocator, cmds: []const u8) void {
  var res = try std.ChildProcess.exec(.{
    .allocator = self.allocator,
    .argv = cmd,
  });
  switch (res.term) {
    .Exited => |code| {
      if (code != 0) {
        @panic("Expected command to succeed.");
      }
    },

    else => unreachable,
  }
}

Handling panics

But there are some things that will fail quite frequently (like running subprocesses or interacting with the filesystem in general).

Panicing (like what happens if @panic() ~~or `unreachable`~~ is hit) in these situations is all good until you have things that you want to get cleaned up.

My coworker points out I'm wrongly conflating unreachable and @panic() since depending on the release mode, hitting unreachable is actually undefined behavior whereas @panic() is always a panic.

Panics don't trigger defer or errdefer statements. So if you have a script that starts a background process or creates a temporary directory, and if you panic in that script, the script won't be able to run defer steps to stop the background process or delete the temporary directory.

There are panic handlers in Zig (not yet documented, Ctrl-f for "TODO: pub fn panic" in the Zig docs. But I'd just be getting further from what seems sensible if I went in that direction.

Zig errors

So I stopped panic-ing everywhere and switched to using real Zig errors, like:

fn run(alloc: std.mem.Allocator, cmds: []const u8) !void {
  var res = try std.ChildProcess.exec(.{
    .allocator = self.allocator,
    .argv = cmd,
  });
  switch (res.term) {
    .Exited => |code| {
      if (code != 0) {
        std.debug.print("Expected command to succeed.\n", .{});
        return error.RunCommandFailed;
      }
    },

    else => unreachable,
  }
}

It's pretty sweet. You get to make up a new error enum wherever you'd like.

It is unfortunate you can't (currently) include a payload with the error return value. There's an active issue discussing it.

But so far I've been able to work around that, as seen in that example above, by logging before returning an error. Since most of the time the payload you want to return is detailed information to provide context.

This logging is fine in a CLI application but probably not everything you'd want in a library. I'm not sure.

And now without panics, functions that deal with error enums and try work with defer and errdefer again! Cleanup of my background processes and temporary directories happens like I want.

Handling errors with `if`

Ok so now that I'm fully bought into Zig errors there were still a few more things that tripped me up.

First is that you can handle errors a few ways. You already saw the first one with try.

  var x = try thingThatCouldFail();

This will cause the function the statement is inside to short-circuit, returning immediately, if thingThatCouldFail has an error result.

But then I wanted to retry a function that could fail in a loop after handling the error.

  var x: SomeType = somedefault;
  while (tries > 0) {
    if (thingThatCouldFail()) |good_value| {
      x = good_value;
      break;
    }

    // do something that should fix it for the next time
    tries -= 1;
  }

But that isn't a real syntax. The Zig docs show an example of how you can use if with an error function:

fn doAThing(str: []u8) void {
  if (parseU64(str, 10)) |number| {
    doSomethingWithNumber(number);
  } else |err| switch (err) {
    error.Overflow => {
      // handle overflow...
    },
    // we promise that InvalidChar won't happen (or crash in debug mode if it does)
    error.InvalidChar => unreachable,
  }
}

But I don't care about the error at this moment (maybe I should, but I don't right now).

So I tried:

  var x: SomeType = somedefault;
  while (tries > 0) {
    if (thingThatCouldFail()) |good_value| {
      x = good_value;
      break;
    } else {
      // do something that should fix it for the next time
      tries -= 1;
    }
  }

But that gives me an obscure type error.

I was stumped here for a while until I decided to try the whole syntax in that example. And it turns out that at least the capture part is necessary at the parser layer:

  var x: SomeType = somedefault;
  while (tries > 0) {
    if (thingThatCouldFail()) |good_value| {
      x = good_value;
      break;
    } else |err| switch (err) {
      else => {
        // do something that should fix it for the next time
        tries -= 1;
      },
    }
  }

And eventually I guessed an unnamed error variable might also work without the switch, and that was correct:

  var x: SomeType = somedefault;
  while (tries > 0) {
    if (thingThatCouldFail()) |good_value| {
      x = good_value;
      break;
    } else |_| {
      // do something that should fix it for the next time
      tries -= 1;
    }
  }

Nice!

`catch` blocks

One last thing that I was stumbling around with was that when you use catch with a function that returns an error or some non-void value, the catch must "return" a value of the same type as the function.

The Zig docs show a simple example:

const number = parseU64(str, 10) catch 13;

But I also use catch with blocks sometimes:

const number = parseU64(str, 10) catch {
  // do some more complex stuff, maybe log, who knows
};

But that won't compile. So the "trick" is to combine Zig's named blocks with catch.

const number = parseU64(str, 10) catch blk: {
  // do some more complex stuff, maybe log, who knows

  // and then "return" a result
  break :blk 13;
};

Contributing to Zig docs

I didn't want to write this post without offering some of my examples to the docs. While there's a dedicated effort around autodoc, the tool that builds docs for the standard library, I haven't yet stumbled on docs for contributing the main Zig docs.

So I grepped in the Zig repo git grep 'Blocks are expressions.', a phrase that showed up in the HTML docs, and found doc/langref.html.in.

Then someone on the Zig Programming Language Discord pointed me at running zig build docs in the repo root to generate the HTML.

And now I've got a PR up! We'll see what folks think.

I wrote a new post about error-handling and Zig, as I've been doing a bunch of scripting with Zig recently.

I stumbled a few times so maybe that will be useful to you. And I was able to turn parts of my stumbling into a potential PR to the Zig docs. 🎉https://t.co/00RVWpodmd pic.twitter.com/wENSEpj63A
— Phil Eaton (@eatonphil) March 22, 2023

Python tip 25: split and partition string methods

learnbyexample — Tue, 21 Mar 2023 02:00:00 GMT

The split() method splits a string based on the given substring and returns a list. By default, whitespace is used for splitting and empty elements are discarded.

>>> greeting = '  \t\r\n have    a  nice \r\v\t day  \f\v\r\t\n '

>>> greeting.split()
['have', 'a', 'nice', 'day']

You can split the input based on a specific string literal by passing it as an argument. Here are some examples:

>>> creatures = 'dragon][unicorn][centaur'
>>> creatures.split('][')
['dragon', 'unicorn', 'centaur']

# empty elements will be preserved in this case
>>> ':car::jeep::'.split(':')
['', 'car', '', 'jeep', '', '']

The maxsplit argument allows you to restrict the number of times the input string should be split. Use rsplit() if you want to split from right to left.

# split once
>>> 'apple-grape-mango-fig'.split('-', maxsplit=1)
['apple', 'grape-mango-fig']

# match the rightmost occurrence
>>> 'apple-grape-mango-fig'.rsplit('-', maxsplit=1)
['apple-grape-mango', 'fig']
>>> 'apple-grape-mango-fig'.rsplit('-', maxsplit=2)
['apple-grape', 'mango', 'fig']

The partition() method will give a tuple of three elements — portion before the leftmost match, the separator itself and the portion after the split. You can use rpartition() to match the rightmost occurrence of the separator.

>>> marks = 'maths:85'
>>> marks.partition(':')
('maths', ':', '85')

# last two elements will be empty if there is no match
>>> marks.partition('=')
('maths:85', '', '')

# match the rightmost occurrence
>>> creatures = 'dragon][unicorn][centaur'
>>> creatures.rpartition('][')
('dragon][unicorn', '][', 'centaur')

See my Understanding Python re(gex)? ebook to learn about string splitting with regular expressions.

Video demo:

See also my 100 Page Python Intro ebook.

Flow-Based Programming, a way for AI and humans to develop together

Henri Bergius — Mon, 20 Mar 2023 02:00:00 GMT

I think by now everybody reading this will have seen how the new generation of Large Language Models like ChatGPT are able to produce somewhat useful code. Like any advance in software development—from IDEs to high-level languages—this has generated some discussion on the future employment prospects in our field.

This made me think about how these new tools could fit the world of Flow-Based Programming, a software development technique I’ve been involved with for quite a while. In Flow-Based Programming these is a very strict boundary between reusable “library code” (called Components) and the “application logic” (called the Graph).

Here’s what the late J. Paul Morrison wrote on the subject in his seminal work, Flow-Based Programming: A New Approach to Application Development (2010):

Just as in the preparation and consumption of food there are the two roles of cook and diner, in FBP application development there are two distinct roles: the component builder and the component user or application designer.

…The application designer builds applications using already existing components, or where satisfactory ones do not exist s/he will specify a new component, and then see about getting it built.

Remembering that passage made me wonder, could I get one of the LLMs to produce useful NoFlo components? Armed with New Bing, I set out to explore.

The first attempt was specifying a pretty simple component:

That actually looks quite reasonable! I also tried asking New Bing to make the component less verbose, as well as generating TypeScript and CoffeeScript variants of the same. All seemed to produce workable things! Sure, there might be some tidying to do, but this could remove a lot of the tedium of component creation.

In addition to this trivial math component I was able to generate some that to call external REST APIs etc. Bing was even able to switch between HTTP libraries as requested.

What was even cooler was that it actually suggested to ask it how to test the component. Doing as I was told, the result was quite astonishing:

That’s fbp-spec! The declarative testing tool we came up with! Definitely the nicest way to test NoFlo (or any other FBP framework) components.

Based on my results, you’ll definitely want to check the generated components and tests before running them. But what you get out is not bad at all.

I of course also tried to get Bing to produce NoFlo graphs for me. This is where it stumbled quite a bit. Interestingly the results were better in the fbp language than in the JSON graph format. But maybe that even more enforces that the sweet spot would be AI writing components and a human creating the graphs that run those.

As I’m not working at the moment, I don’t have a current use case for this way of collaborating. But I believe this could be a huge productivity booster for any (and especially Flow-Based) application development, and expect to try it in whatever my next gig ends up being.

Illustrations: MidJourney, from prompt Robot software developer working with a software architect. Floating flowcharts in the background

Improve your Python regex skills with 75 interactive exercises

learnbyexample — Mon, 20 Mar 2023 02:00:00 GMT

(2023-03-20) Update: This TUI app covers many more exercises compared to the GUI app discussed below.

Still confused about Python regular expressions? Grow your confidence with Understanding Python re(gex)? ebook (FREE this month!) and an interactive GUI app.

Inspired by Advent of Code, I'll also be posting 3 challenges per day on twitter for 25 days.

Free ebook🔗

My post about the interactive GUI app made it to the Hacker News front page. To celebrate, you can get PDF/EPUB versions of my Understanding Python re(gex)? ebook for free using either of the below links. The offer is valid till 31-Dec-2021.

Or, you can use the web version if you prefer reading the book online.

Interactive GUI app🔗

Based on the Understanding Python re(gex)? book contents as well as the exercises, I made an interactive GUI app with 75 questions on re.search, re.sub, re.split and re.findall that'll test your understanding of anchors, alternation, grouping, escaping metacharacters, dot metacharacter, quantifiers, character class, grouping, lookarounds, flags, etc.

Here's some screenshots:

And here's a brief demo:

25 Days Of Regex🔗

If 75 exercises seem daunting to you, consider doing 3 exercises per day. Allocate some time everyday to read the book and complete 3 challenges.

I'd also be posting 3 challenges per day on twitter, where you'll be able to get help from me and fellow programmers.

Happy learning :)

Creating Info Manuals And Adding Them Into Emacs

Beetle Space — Sun, 19 Mar 2023 09:00:00 GMT

Once I got used to browsing Info manuals in Emacs, I wished I didn’t have to go look at online docs all the time.

Of all the manuals out there, how many can I get in Info format?

I quickly discovered that any document written in Sphinx (which many …

The unique scoreline rabbithole

infrequent oscillations — Sat, 18 Mar 2023 11:11:00 GMT

In which we find Barnsley and Everton are unique. Ish.

3D Printing Saves A Neighbors Day

MiscDotGeek — Fri, 17 Mar 2023 21:49:00 GMT

When I started off in the 3D Printing world in November of ’22, I knew it would be neat to be able to print trinkets here and there, or maybe a Telescope or two. But to be able to help a neighbor in a bind? I didn’t expect that. On Tuesday I saw a request …

The post 3D Printing Saves A Neighbors Day appeared first on MiscDotGeek.

How to change custom domain on Substack

swyx's site RSS Feed — Fri, 17 Mar 2023 08:01:45 GMT

Since September, I've been running my AI newsletter on https://lspace.swyx.io

The Grand Code Restructuring

Weblog on marginalia.nu — Fri, 17 Mar 2023 02:00:00 GMT

In general I don’t like to fuss over code, but this is exactly what I’ve been doing in preparation of the NLnet funded work. I’ve spent the last month restructuring Marginalia’s code base. It’s not completely done, but I’ve made great headway. Things got the way they got because in general for experimental solo-development projects, I think it makes sense to be fairly tolerant of technical debt. Since refactoring is something that is extremely difficult to break up into parallel tracks or do in small iterations, the cost of refactoring is effectively multiplied by the number of people that could be working on the code.

I Know Who I Am, Bing!

Beetle Space — Thu, 16 Mar 2023 10:00:00 GMT

Yesterday was the first time I interacted with any of the GPT flavors. I interacted with it via Bing Chat.

I think by now many have read about the humorous, and sometimes ominous responses Bing gives, and are probably tired of seeing more examples. Still, I want to share my …

Solving a Scraping Problem with Emacs and Org Mode

Beetle Space — Thu, 16 Mar 2023 09:00:00 GMT

I recently needed to read a detailed summary of the first 20 or so chapters of Bleak House.

I came across this site that fit the bill perfectly.

Consistent with my habit of printing articles and reading them offline, I wanted to print these.

This posed several problems. The first …

More More More Football Database, part one

infrequent oscillations — Tue, 14 Mar 2023 11:14:10 GMT

In which we update our priors, part one.

Vim tip 23: editing lines filtered by a pattern

learnbyexample — Tue, 14 Mar 2023 02:00:00 GMT

The syntax for g command (short for global) is shown below:

:[range]g[lobal]/{pattern}/[cmd]

This command is used to edit lines that are first filtered based on a searchpattern.

:g/call/d delete all lines containing call
- similar to the d Normal mode command, the deleted contents will be saved to the default " register
- :g/call/d a in addition to the default register, the deleted content will also be stored in the "a register
- :g/call/d _ deleted content won't be saved anywhere, since it uses the black hole register
:g/^#/t0 copy all lines starting with # to the start of the file
:1,5 g/call/d delete all lines containing call only for the first five lines
:g/cat/ s/animal/mammal/g replace animal with mammal only for the lines containing cat
:.,.+20 g/^#/ normal >> indent the current line and the next 20 lines only if the line starts with #
- Note the use of normal when you need to use Normal mode commands on the filtered lines
- Use normal! if you don't want user defined mappings to be considered

You can use g! or v to act on lines not satisfying the filtering condition.

:v/jump/d delete all lines not containing jump
- same as :g!/jump/d

In addition to the / delimiter, you can also use any single byte character other than alphabets, \, " or |.

See :h :g for more details.

Video demo:

Muv-Luv Unlimited: THE DAY AFTER 01 review

ブラック — Mon, 13 Mar 2023 02:00:00 GMT

THE DAY AFTER 01 continues the story in the Unlimited timeline. Mood is a bit all over the place, ranging from very serious politics discussion on life-and-death matters in a ruined world to Extra-esque slapstick comedy moments where the main character accidentally fondles tits. The most lighthearted moments even come with complementary chibi art. While I found the lighthearted moments of the story fun, I did feel like they dragged on a bit too long. But, much like with THE DAY AFTER 00, towards the end of the game I felt captivated to continue reading. Overall, THE DAY AFTER 01 does definitely deliver a more interesting story than its predecessor.

Most of the characters in THE DAY AFTER 01 are completely new. The main exception is Marimo-chan, who enjoys a lot of attention here. But those worried by the lack of familiar faces, don’t fret – the new cast is likeable. The main character does definitely have the kind of an idealistic view on world that some might find off-putting, but I didn't find his thinking to be too idealistic. If you’re anything like me, you’ll be cheering on the squad towards the end. I’m looking forwards to exploring more of their future, and their pasts.

As TDA 01 is roughly double the price of TDA 00, I expected the runtime being roughly double – and it is! However, it’s still quite short, just not as short as TDA 00. I read through TDA 00 in 4.4 hours and TDA 01 was 8.2 hours. However, as the regular price for TDA 01 is about $25, it’s definitely not a value purchase as far as visual novels come. Personally, I’d wait for a sale (and I did back in 2021).

Art and sound design are mostly good. Character sprites felt better than what TDA 00 had to offer. It also feels like there was less distracting zooming in on backgrounds that really just highlights any kind of artifacts. Either that, or I’ve already gotten used to it. Fully voiced story again with good character voices.

Translations at times dipped a bit too much on the liberal side for me. Luckily with the full voice acting, you can spot the differences if you have some amount of verbal Japanese knowledge. I also feel like they toned down some of the racist comments used in the story, going from “monkeys” to “Japs”. Not sure if they aimed for that, or if they just thought that the original comments wouldn’t have the wanted impact on English-speaking audiences.

Technical aspects are mostly fine. I read it from start to finish on my Steam Deck and there’s no blockers there. But you’ll have to use a custom controller layout to have a proper experience, and even then you’ll have to use mouse controls for some elements. Really wish they paid more attention to proper controller support.

I also observed the same kind of crackling after the VN has been running for a while / being suspended in sleep on my Steam Deck as I did with THE DAY AFTER 00. I feel like switching from the regular Proton to Proton-GE helped to alleviate the crackling issue, so maybe try that. Of course, you can just relaunch the game to get the audio in perfect condition again.

Overall, THE DAY AFTER 01 is an enjoyable story despite its relatively short length and smaller issues. I imagine I’ll crack open THE DAY AFTER 02 tomorrow to find out what happens after the world ends.

How to implement Go Channels in Java

Ben E. C. Boyter — Mon, 13 Mar 2023 02:00:00 GMT

I have been doing interviews recently, and whenever someone mentions knowing a lot of Go and Java one of the questions I put to them is how to implement Go channels in Java. Since there isn’t anything online to suggest how to do this that I liked reading (that I could find) I thought I would write one, and do it in native Java without needing to import a library.

Ghostwire: Tokyo review

ブラック — Sat, 11 Mar 2023 02:00:00 GMT

First things first: the environmental design in Ghostwire: Tokyo is great. You have an expansive virtual Tokyo to explore, and not just on the ground level, as you also have access sprawling underground systems and rooftops to explore. And not only is the world expansive to explore, but it’s also dressed up in this spooky veneer. And in parts of the game you also get these trippy sequences where the world just goes completely haywire and basically just dances around you. It’s great. The game world is definitely the best part of the game, and I can imagine someone getting their money’s worth just from the eerie Tokyo experience.

The combat is fine. If one were to pick up this game for the combat, I imagine disappointment would be almost guaranteed. Not that it’s bad, but it’s clearly not where the game shines. You have couple of basic attacks with slightly different behaviours, but at the end of the day, you aim at an enemy and shoot at them until they stop. Stealth mechanics are also not the best that I’ve encountered and there’s a couple of forced stealth segments. Nothing unbearable, but also not my favourite parts of the game.

There’s some degree of RPG elements in the gameplay, but not enough to actually make it an RPG. You have levels, and gaining a level nets you skill points that you can spend to improve your abilities. However, the combat always feels like the enemies always have the same level as you do, so you’re at least spared from having to grind levels to beat the game. Going for full completion will also net you all of the skills, so there’s no any kind of specialisation, mainly just a matter of deciding which order to acquire them.

The story is also just fine. The main story definitely isn’t too intricate or captivating and there’s no extensive lore to explore if that’s your thing. Again, nothing terrible here, but clearly this is not a story-first product. The story does luckily manage to hit some good emotional notes along the way. The side quests also don’t have any deep story, but do offer some small slice of life segments featuring ghosts and yokai that I found as quite enjoyable bite-sized experiences. The main sidekick that accompanies you throughout the game is also a fun character, and I enjoyed the interactions between them and the player character.

And while the game might be tagged as “horror”, I can personally guarantee as a certified coward that the game isn’t actually scary. It’s mostly “creepy”, “spooky” or “eerie”, but not outright scary or horrific. You’re not gonna get jumped on out of the blue or be forced to crawl through dark rooms. It’s definitely light enough of a “horror” experience that I was able to dig into it.

If you’re a completionist, be prepared: the game is full of collectibles. They want you to go collect landmarks, spirits, yokai, raccoon dogs, statues, shrines, beads, toys, foods, music tracks, outfits, and emotes, as well as complete all of the side quests. It’s actually wild how many different collectibles they put into the game, Shinji Mikami really got sick with Open-Worlditis here. The saving grace for the collectibles is that they’re spread out the amazing landscape, so wondering around for collectibles can be quite scenic. Thankfully there is a button that highlights nearby items to you, so be prepared to use that a lot if you’re one to go for all the achievements.

If you’re not a completionist, maybe temper your expectations; the main story takes about 10 hours to clear. Completionists will be in it for 30–40 hours, depending on your efficiency and luck. No wonder they added so many side activities, since just 10 hours of content would’ve been brutal for a full-priced title.

Ghostwire: Tokyo is a beautiful supernatural rendering of Tokyo. While few of the game’s aspect shine bright, I still enjoyed my holiday in Shinji Mikami’s creepy playground.

Disclaimer: I did not pay a single dime for this game, as I received it as a Christmas gift from a friend.

Another Football Update

infrequent oscillations — Thu, 09 Mar 2023 00:09:30 GMT

In which we find another bug whilst reimporting data.

Football Database Update

infrequent oscillations — Wed, 08 Mar 2023 01:25:57 GMT

In which we have to update our schema.

CLI tip 24: inserting file contents one line at a time

learnbyexample — Tue, 07 Mar 2023 02:00:00 GMT

The R command provided by GNU sed is very similar to r with respect to most of the rules seen in an earlier tip. But instead of reading entire file contents, R will read one line at a time from the source file when the given address matches. If entire file has already been read and another address matches, sed will proceed as if the line was empty.

Here's an example:

$ cat ip.txt
    * sky
    * apple
$ cat fav_colors.txt
deep red
yellow
reddish
brown

# add a line from 'ip.txt'
# whenever a line from 'fav_colors.txt' contains 'red'
$ sed '/red/R ip.txt' fav_colors.txt
deep red
    * sky
yellow
reddish
    * apple
brown

You can combine with other sed commands to solve various kind of problems. For example, to replace the matching lines:

# empty // will refer to the previously used regex, /red/ in this case
$ sed -e '/red/R ip.txt' -e '//d' fav_colors.txt
    * sky
yellow
    * apple
brown

And, here's how you can interleave contents of two files:

# /dev/stdin will get data from stdin (output of 'seq 4' here)
# same as: seq 4 | paste -d'\n' fav_colors.txt -
$ seq 4 | sed 'R /dev/stdin' fav_colors.txt
deep red
1
yellow
2
reddish
3
brown
4

# using 'paste' here will add a newline when stdin runs out of data
$ seq 2 | sed 'R /dev/stdin' fav_colors.txt
deep red
1
yellow
2
reddish
brown

Video demo:

See also my CLI text processing with GNU sed ebook.

Hosting your own Mastodon server

Paul Stamatiou — Tue, 07 Mar 2023 02:00:00 GMT

A lot has happened since I published my article on the open source decentralized social networking service Mastodon a few weeks ago. I saw a lot of interest in and discourse around that post that was great to see. Since then, a few notable things have happened that have caused Mastodon usage to spike yet again.

Twitter unsurprisingly ended up entirely banning third-party Twitter clients, including hugely popular ones like Tweetbot and Twitterific. I briefly summarized the ebb and flow of being a developer working with the Twitter API in the last post:

"Long story short, Twitter made it hard. Hard to even get an API key and get your app approved. Hard to get bugs fixed or new features accessible via the API. Hard to access the data developers wanted. Hard to not hit arbitrary constraints or limits. Things were always changing, rugs were pulled, announcements about new priorities and new rugs were made, only for them to be pulled again (read this, this and this)".

Well, the final rug was pulled. Fortunately, there was a bit of a silver lining. The folks at Tapbots were able to launch their lovely Mastodon app Ivory. The launch went so well that a flood of people took that as an opportunity to take a closer look at Mastodon and join. Mastodon has grown from 500,000 active users last year to now being almost at 10 million.

Mastodon just keeps getting better. There's more authentic conversations and far less linkbait and thread spam from influencers. It feels like a place I'd prefer spending more of my time to catch up on things.

With my interest in Mastodon, I recently decided I wanted to take the next step and host my own Mastodon server for my account: @stammy@stammy.design. In my previous post about Mastodon I provided some details about different routes you could take to self-host Mastodon. Today, I'll show you how you can host your own Mastodon instance, step-by-step.

This is a long post and took me more than a month to write, but you don't have to go through it all. At a high-level I show two routes to hosting your own Mastodon server: a simple route employing a fully-managed webhost, and a much more comprehensive and advanced route involving a traditional webhost.

I also make sure you're good to go with maintenance tips, debugging tips, scripts to backup your database, purge storage and more.

So you want to host your own Mastodon server?

And why you would want to do host your own Mastodon server? If you've read this far, you probably already have a hunch or two as to why, but here are a few excuses to choose from:

You value being in complete control over your Mastodon identity, data, and privacy. You don't want to risk it on an instance where you don't know the moderators, don't want to be under their control, or simply don't trust them to maintain the server long-term.
You want a speedy Mastodon experience. By controlling your own instance you can set it up with the appropriate resources and hosting location near you to keep Mastodon feeling speedy at all times. That's not the case with various instances, especially the larger ones experiencing rapid growth.
You want a weekend project, because Mastodon is open source and able to be self-hosted
You see lots of people running their own instance and it seems cool.
You want a better Mastodon user name, one that speaks more to your identity or interests. Run Mastodon on your long-time domain name or an entirely new one, paired with whatever user name you like.

Whatever your reason for wanting to host your own Mastodon server, I'm here to help you get started. I'll walk you through the process of setting up your own Mastodon server. I'll also provide some tips along the way to help you get the most out of your Mastodon experience.

What you'll need

At a bare minimum, you'll need a domain, some money, and some time:

A domain. Picking the user name and domain name you wish to use for your new Mastodon instance is probably the hardest part.

You'll need a domain name you can connect with your new Mastodon server. You should also know the basics of how to modify DNS records on it. This varies by domain name registrar but is generally pretty easy to figure out. If you don't have a preferred registrar yet, Google Domains is pretty easy and you don't have to make a new account with two-factor auth if you're already a Google user.

Otherwise, I also like Amazon AWS Route 53, but it's a bit harder to use and they don't support nearly as many top-level domains (such as .design, .dev, et cetera) as other registrars.

Some money. For a single user instance you can get away with $6/mo with the lowest plan on masto.host. but if you have a lot of followers (or just want a beefier server), you'll want to upgrade to a bigger plan and that could run you closer to $50/mo.

And a few hours.

If you opt for a more involved hosting setup you'll need access to a few more things, such as an SMTP server to send emails, and optionally, access to a cloud service such as Amazon S3 to host media. I'll get into that in a bit.

Choose how you'd like to host

The easy way or the hard(er) way?

I'll show you two different ways to setup your own Mastodon server. For the most part I'm assuming you are only setting up Mastodon for yourself, or just a few people. This guide is not really meant for someone wanting to setup a Mastodon server that will accomodate hundreds of users as that will take you down a different path.

Option 1: Using a dedicated Mastodon host

The easiest way to get your own Mastodon server up and running quickly is with a host that specializes in Mastodon hosting. There are several, but I'll share how to use one of the most popular ones, Masto.host.

So what exactly do I mean by dedicated host? You won't have to worry about the typical setup and maintenance required with a regular host, like configurations, software updates, backups, et cetera. This might be route for you if you want the easiest possible way to get started, and having full server control is not as important to you. You won't have complete control over the hardware, the software (beyond Mastodon itself), how your media is stored and served, or anything lower level like that.

While you won't have the ability to tweak things and keep an eye on performance, but it's an easy way to get started for cheap. The most affordable Masto.host plan is only $6 per month for up to 20GB of media storage and 2GB database. For a casual single user without a lot of followers this should be okay. Masto.host lets you upgrade to higher plans if you need later on.

Personally, I did not want to use an option like this. I wanted to be able to serve my media files with a CDN, control the entire server config, as well as just know more about the hardware used in my server (SSD? NVMe SSD? etc). I always want to have significant headroom on my server in case any of my Mastodon posts goes viral. I don't want to have a bogged down server limiting the reach of my posts.

Option 2: DIY with a regular webhost and server

The other option is getting a VPS or dedicated server, setting up Mastodon and its dependencies. While this route takes considerably more time to setup, it's the most flexible by far. You control everything and you can eke out more performance and functionality. And some hosting providers will give you a head start with a Linux image with Mastodon pre-installed—there's quite a few things to configure but this makes it a bit easier.

Maybe you decide you want to modify the code to remove a rate limit, or change the character limit of posts? Maybe you want to move to a fork of Mastodon, like Hometown. Want to serve your media from a fast CDN on a subdomain that you have backed up? You can do all of that on your own server.

As expected, it will take longer to setup (a few hours) and it will probably cost more. But if you're reading this far, you probably already know your having own server is right for you.

Setting up Mastodon with Masto.host

Option 1, the easy way

This is by far the easiest way to get started. By the end of this section, you'll have your own Mastodon instance running on your own domain. You only need access to your domain name registrar (or DNS provider if use a separate service) so you can change a DNS record.

You don't have to worry about getting access to an SMTP server to send notification emails, nor will you have to figure out how to save media files to a cloud provider.

Sign up and change DNS

Visit Masto.host, choose a plan, and provide your domain name (though you can run it as a subdomain on masto.host if you wish). You'll be greeted with a page telling you to add a DNS A record so your domain name can point to your Masto.host server.

Every domain name registrar / DNS provider has a slightly different way of modifying DNS records, so I can't show too much more detail there. You'll likely need to log into where you have your domain name registered and look for some DNS or zone settings.

Installation

When you've successfully added that DNS record, Masto.host should automatically detect it and begin installation.

The installation screen said this process usually took a few minutes. Unfortunately, in my case it actually took a whopping 8 hours. I hope that's just temporary due to a temporary spike in new users.

Create your Mastodon user & upgrade permissions

Once that's done you can go right to your new Mastodon instance and create an account. This will be your Mastodon account, so choose the username you want.

After creating your Mastodon account, you'll just need to do one more thing. Go back to Masto.host and look for a button called "Change User Role." Here you can upgrade your new account to have "Owner" privileges. Now your account will have access to a new Administration tab in Preferences.

That’s it!

That's pretty much it! There are some things to fill out, and settings to change in Preferences which I go over later in this article, but your server is now online and ready to explore!

The Masto.host site leaves a lot to be desired. You can only see how much storage your media and database use.

Aside from that, there's not much you can control or change with Masto.host. The site lets you do a few things like download daily backups for your own safe-keeping, change your plan, and opt to install ElasticSearch for an additional monthly fee.

ElasticSearch lets your Mastodon instance have full-text search for logged in users to search their own posts, bookmarks, favorites, and mentions on your server. I probably would not recommend it if you have the smallest server plan as it can use a good bit of RAM; it's unclear how much RAM your Masto.host server has and how much RAM ElasticSearch is configured to use.

What you can't do

There are a few things you won't be able to do with a setup like Masto.host where you don't have control over the exact configuration. One of these is so-called single user mode. This mode makes the homepage your profile instead of a timeline of users. Typically, you would just update your .env.production file enable it. Though this is a pretty easy thing for Masto.host to build into their web dashboard as a future setting.

Similarly, you can't do a more advanced username where you could have a username like you@example.com even though Mastodon may actually be hosted on a subdomain like you@social.example.com. This is described in the Mastodon documentation as WEB_DOMAIN.

Setting up Mastodon with a regular webhost

Option 2, the not-so-easy way (DigitalOcean VPS)

This section is for people like me that want more control over their server and the software that runs on it, and don't mind a few twists and turns to get there. In this section we'll pick out server hardware and options, create an account to send emails with SMTP, setup media storage with a cloud provider, follow a setup wizard in the command line and then configure a few things.

You should be comfortable with Linux and command line basics for this setup process as well as ongoing server maintence.

Getting an SMTP server

First, we'll need to get access to an SMTP server to let Mastodon send email notifications. While this seems like a step that should be optional, it's required by Mastodon to send emails used to verify new accounts. There are many ways to get access to an SMTP server:

Setup and configure an MTA like postfix, exim, or sendmail on your server. This is not a particularly easy route. You may still have deliverability issues, or your webhost may restrict SMTP. It's also not advisable to run an SMTP server on the same IP as your web server.
Use your email provider's SMTP server. If you use GMail (details linked), Outlook, Fastmail or similar email service providers, you can try to just use their SMTP server. This will require you to provide some credentials (either by generating a unique app-specific password, or your email password), and send from the same email address and domain you currently use with your email.
Use a dedicated SMTP provider like Amazon AWS SES (Simple Email Service), Mailjet, Resend, SendGrid, Mailgun, SMTP2Go, SparkPost, ZeptoMail, Postmark, and many more. I personally use AWS SES and would recommend it if you're already using AWS services in some capacity. Setting up SES is a bit involved though, and you'll be in a sandbox period at first.

While I personally use AWS SES, I was looking for something even easier to setup for this article and after trying a few, had a great experience getting started with Mailjet. By comparison, trying to setup SendGrid was a disaster; way too many steps involved, including having to talk with support.

There's a few steps to follow before you can get your SMTP server login credentials but Mailjet makes it pretty straightforward:

Sign up for a free account at Mailjet.
Add your domain as a new sender domain. Click "Setup my SMTP" (or navigate to Sender domains from Account) to add your domain.
Validate your domain by adding a TXT DNS record. You'll need to log into your DNS provider or domain name registrar to edit that.
Authenticate the domain by adding 2 TXT DNS records for SPF and DKIM.
Specify the email address you'll send from and marking it as transactional.
Get your API key and SMTP server URL. This can be found under "SMTP and SEND API Settings" on the Account page. When setting up Mastodon, the API key will act as your SMTP username and the API secret will act as your password.

Tip: Use your webhost for DNS too

You may want to just use whichever hosting provider you end up choosing as your DNS provider as well, instead of your domain name registrar (if they differ). Most hosts make it a bit easier to set your A record that points to your server if they also manage the DNS.

In this article I show how to use DigitalOcean as your hosting provider, so you can skip ahead to that part below or create an account and follow these instructions.

Mailjet is one of the easier ways to get access to

Keep your new SMTP credentials handy, you'll need it soon.

Picking your webhost

You can get a Linux server up and running from any number of hosting providers (like AWS EC2 or Lightsail, Linode, Hetzner, DigitalOcean, Vultr, etc) if you're comfortable with the standard Mastodon installation process. If you go this route you'll need to install various system packages, Ruby, Node, Postgres and nginx among others.

However, I'm going to focus on using a DigitalOcean "Droplet" for this article. Mainly because they have a 1-Click Mastodon installer. It saves you a good bit of time but there's still more to do after that I'll walk you through.

Droplets are DigitalOcean's name for their Linux-based virtual machines running on virtualized hardware. They have a good variety of hardware to choose from, including options for faster NVMe SSDs which I really appreciate.

Lets get started! Click "Create Mastodon Droplet" on the 1-click installer page and create an account if you don't have one yet.

Picking your hardware

After logging in you'll be sent to a page to create your new Mastodon droplet and select what hardware to use. How much CPU, RAM and storage space do you need? Storage doesn't really matter as we'll be hosting uploaded media on a separate cloud service. As for storage speed, DigitalOcean droplets only offers SSDs and faster NVMe SSDs; no slower mechanical hard drives. You can opt for the NVMe SSD if you wish (I did) but it's probably overkill for most people just setting up a single-user personal Mastodon instance.

Creating a DigitalOcean "droplet" server

With CPU and RAM, it's definitely a lot of "the more, the better" and whatever you can afford, but here's some additional context to think about it with:

RAM: When it comes to RAM, while you could get by with 1GB, I would say you'd want at least 2GB, especially if you plan to run ElasticSearch on your instance. I ended up going with 4GB and my machine on average consumes about ~2GB of RAM so I have a good amount of headroom.

CPU: CPU is critical to several parts of what run Mastodon like Sidekiq, a background job processor, and Puma, the app server on top of nginx. The more CPU resources you have available, the more processes and threads you can spin up without bogging down your system. With the web server for example, if you increase the number of threads you'll use more CPU, but if you create more processes, you'll use more RAM.

I picked a server with 2 Intel cores and it has been doing fine with my Mastodon activity with about 4,000 Mastodon followers.

Tip: Start small

With DigitalOcean droplets you can easily upgrade to a more powerful machine, but you may not be able to downgrade to a less powerful machine. This is usually due to having a disk image that's too large for another server.

I suggest starting with a server that has 2GB of RAM. If you find you need more RAM, CPU, disk space, et cetera, you easily upgrade later. It takes maybe 5-10 minutes to upgrade and DigitalOcean handles everything for you automatically.

After selecting your server hardware, you'll want to select the option to enable monitoring and backups. Then, click "Advanced" and enable IPv6. And finally, you'll need to create or provide an SSH key. I highly recommend following their SSH instructions so you can use a key that your computer has so you can easily access your server with your preferred terminal app. You can still use the web console provided by DigitalOcean but it's a bit clunky.

Option: Managed Database

I ended up selecting the option to have a managed database for an extra $15 per month. If you don't mind the extra cost it's nice for it's automatic backups that let you revert to any point in time. That being said, for a single user Mastodon instance, you probably won't get any performance benefit out of it, just peace of mind. And since it's basically another remote server, you can access it with a tool like SQLPro for Postgres (if you've configured its security settings to allow your IP) if you're so inclined to poke around in the database.

Important: If you select the managed database option while creating your Mastodon 1-Click droplet, it should automatically put the correct database credentials for you, but there is one more place where you need to specify the database access credentials: /etc/environment. If you don't do this you may see Mastodon admin CLI commands fail, such as when trying to modify your first Mastodon account to give it owner privileges.

You will also want to verify that /etc/environment has the correct database name specified. By default it should be mastodon_production.

And finally, you will need to add the following to /home/mastodon/live/.env.production:

DB_SSLMODE=require
# You may also need to do the following as a hack
# More context: https://github.com/mastodon/mastodon/discussions/19917
NODE_TLS_REJECT_UNAUTHORIZED=0

And then restart the services sudo systemctl restart mastodon-*. Without those lines, you may see some errors in the streaming service logs.

And finally, you'll want to go to DigitalOcean, click on your managed database, then visit settings to ensure you restrict connections to just your IP and your Mastodon server under the "Trusted sources" section.

Click Create Droplet and your server should be ready in about 30 seconds or so.

Adding your domain & setting up DNS

Next up, we'll need to add your domain name to DigitalOcean and setup necessary DNS records.

Tip: Using a Mastodon on a domain where you already host a website, without exposing the subdomain

If you wanted to use a domain where you're already hosting a website, you may think that it's not possible to host Mastodon on that domain without resorting to a subdomain which would leave you with a lengthy Mastodon username like username@social.example.com.

Fortunately there's a a lesser known Mastodon tip that lets you host Mastodon on a subdomain but allows your username to be shown as being directly on the root domain itself, such as username@example.com. This is done by setting the WEB_DOMAIN environment variable to the root domain. This is not something that the Mastodon setup wizard will ask you about, but it's something you can do after the setup wizard is complete by manually editing the .env.production file, typically located in the /home/mastodon/live directory.

This is described in the Mastodon documentation as ,[object Object] and LOCAL_DOMAIN. To sum up: with this configuration you can have Mastodon hosted on a subdomain but completely hide that from people that follow you. They'll only see your username at your root domain.

Click the menu on your droplet and select Add a domain.
Type in the domain you want to use for Mastodon, and select your droplet server from the dropdown.
If you haven't done so already, you'll want to go to your domain name registrar and designate DigitalOcean as the DNS host by adding the 3 nameservers shown. More info can be found here.
Finally, you just need to add an A record for your droplet. Just type @ for the Hostname and click the dropdown to select your droplet to automatically select the IP address needed.

After DNS propagates in a few moments you'll be able to directly SSH into your server with just ssh root@yourdomain.com, assuming you setup your SSH key earlier.

Setting up cloud file storage

Before we can start setting up Mastodon, there's one more thing we need to do: setup and get login credentials for cloud object storage for all user-uploaded files on your Mastodon instance. This part is optional: Mastodon can easily host files directly from your server.

However, hosting user-uploaded files on a cloud service instead of your Mastodon server has some benefits. First, it takes the load off your web server to respond to requests for media, freeing up your server to respond to other requests. This is especially important if you have a less powerful server. It's an easy way to make things more performant. Second, cloud object storage providers are made for serving static files quickly and that's especially true if you use one with a CDN.

There are a lot of services to choose from such as DigitalOcean Spaces, Wasabi, Minio, Google Cloud Storage, Amazon S3, and any service that offers S3-compatibility like Cloudflare R2.

I personally use Amazon S3 with a CloudFront distribution (CDN), used on a CNAME subdomain. It was a bit more involved to setup, including creating separate IAM and bucket permissions, so I wanted to show a simpler solution for this article, using DigitalOcean Spaces. If you're familiar with AWS, that's a great route to consider though.

The easiest way is to use DigitalOcean's own object storage solution, Spaces. It costs $5/month for 250GB of storage with 1TB of outbound transfer. That should be more than enough space for most.

To get started, click the Create button at the top of DigialOcean's dashboard and select Spaces to create a bucket for storing your Mastodon server's files. Here you'll need to pick a bucket name and location, which should default to the same region as your droplet server. The bucket name doesn't matter much as it's not readily visible to your users unless they dig into their browser web inspector. You'll also want to check the box to enable CDN.

After you create the bucket you'll need to specify a subdomain to access your files from and setup an SSL certificate so HTTPS will work. DigitalOcean makes this pretty easy. Go to the Settings tab of your new Spaces bucket and click Change → Edit CDN Settings.

Select a subdomain name that you'll use for hosting user-uploaded media on your Mastodon server, like files.yourdomain.com. While this is optional, it's a nice touch. After the certificate is created, the next step is to just select it from the dropdown on the Settings page under CDN as shown below. If you're lost at any point during this process, more details can be found here.

The final step is getting the API key and secret for your new bucket. On the main Spaces page (click Spaces from the sidebar) there is a link at the top right titled Manage keys. Click that and then you can generate a new key and give it a name. Once you're done with that copy your key and secret for use later.

Setting up Mastodon

By this point your DigitalOcean project should look like this: you have a droplet server, a Spaces bucket, and an attached domain.

You're ready to begin the actual Mastodon setup! Mastodon is already installed thanks to the 1-Click Droplet installer. You just need to access the setup wizard via the command line. You can use the built-in console on the website or SSH into it with your preferred terminal app. The website console can be accessed by selecting Access console from the ... menu on your droplet. I prefer directly SSHing into the server with ssh root@yourdomain.com.

When you're connected to your server, you'll instantly see the Mastodon setup wizard start to run:

The Mastodon setup wizard

Tip: If you make a mistake...

If at any point you make a mistake while working in the command line setup wizard and need to undo something, just close the SSH session or web console, and re-open it. The setup process will start over again until it's completed.

You'll be asked to answer a few questions and fill in some details and credentials:

Domain name: Your domain name for Mastodon.
Do you want to store user-uploaded files on the cloud? (y/N): Type y for yes.
Provider: Select DigitalOcean Spaces.
Space name: The bucket name you chose.
Space region: Type in the region for your spaces bucket. When you go to copy your Spaces endpoint, it'll be the subdomain after your bucket name that's short (like nyc3) depending on the hosting location.
Space endpoint: Your Spaces "Origin Endpoint" as seen on the DigitalOcean website but with the bucket name subdomain at the front removed. Even though you setup the CDN, for this you need the non-CDN URL or it won't work. For me it was the following but this varies based on the location you chose:
```
https://nyc3.digitaloceanspaces.com
```
Do you want to access the uploaded files from your own domain?: This is asking if you want to use a CNAME alias for your hosted files, for example to appear as if they are hosted from your domain. I'll assume you followed the directions above for setting up Spaces CDN with a subdomain and certificate, so you can say yes.
SMTP server: Check with your SMTP provider, but in the case of Mailjet it was in-v3.mailjet.com for me. This could be different for you.
SMTP port: Depends on your SMTP provider, but likely 587.
SMTP username / password: Provide the username and password for your SMTP provider. If you're using Mailjet as described above the API key will act as your SMTP username and the API secret will act as your password.
SMTP authentication: The default plain should be fine. You may also be asked for an OpenSSL verify mode, for that you can usually input peer or none.
Email address to send emails "from": Hit enter to accept the default notifications@yourdomain.com address.
It is time to create an admin account that you'll be able to use from the browser! Username: Pick the username that you'll want to use for your Mastodon profile.

Your full Mastodon handle will end up being @yourusername@yourdomain.com.

At this point you'll be given a password to log into your new Mastodon account. You'll want to change it to something stronger after you log in for the first time (more on that later). But before that can be done you'll need to follow the instructions to setup your SSL certificate with Let's Encrypt so https will work for your domain.

If you find yourself struggling to get the Let's Encrypt process to work, check the DNS records on your domain to ensure everything is setup correctly, particularly the A record for the domain and that you've correctly added the DigitalOcean nameservers with your registrar. Also, you can just give it some time for DNS to propagate and try again later.

There's just a few more things to do before logging into your new Mastodon server.

Fixing the cloud storage config

Unfortunately I was unable to get DigitalOcean Spaces cloud object storage with CDN working flawlessly out of the gate and had to manually edit a few lines in a config file. We'll be editing the .env.production file—where a lot of Mastodon settings live— to adjust some of the media storage settings.

If you're using the subdomain as shown during Spaces setup: You'll need to add a line for S3_ALIAS_HOST as shown below. This value is the same as the subdomain you setup earlier along with the SSL certificate. Then you'll need to modify the S3_ENDPOINT and S3_HOSTNAME values to be in this format.

S3_PROTOCOL=https
# Replace the following with your bucket name 
S3_BUCKET=stammy-mastodon
# Replace with your bucket region
S3_REGION=nyc3
# Replace with the subdomain you setup earlier for your Spaces bucket / CDN
S3_ALIAS_HOST=files.stammy.social
# Change the region in the URL below to match your bucket region
S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
# This is your "Origin Endpoint" as found on your Spaces bucket page
S3_HOSTNAME=stammy-mastodon.nyc3.digitaloceanspaces.com

If you're not using a subdomain alias for hosting your files: You will need to remove the bucket name from your DigitalOcean Spaces bucket's "Origin Endpoint" that is used as your hostname. Ensure your S3_ENDPOINT and S3_HOSTNAME look like this. Remove any line with S3_ALIAS_HOST if it already exists.
```
S3_ENABLED=true
S3_PROTOCOL=https
S3_BUCKET=stammy-mastodon
# Replace nyc3 region in the next 3 lines with your bucket region
S3_REGION=nyc3
S3_HOSTNAME=nyc3.digitaloceanspaces.com
S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
```

If you used Amazon S3 with CloudFront instead: If you went for the more advanced route of using S3 with a CloudFront distribution connected to a CNAME on your domain, you will configure it like this:

# Your S3 bucket name. 
S3_BUCKET=my-mastodon-files
# Replace us-east-1 with your bucket region, lowercase.
S3_REGION=us-east-1
# Hostname is likely your bucket name + .s3.amazonaws.com
S3_HOSTNAME=my-mastodon-files.s3.amazonaws.com
# Only use if you've configured a CloudFront CNAME on your domain with SSL cert 
# If you're not using a CNAME, remove this line and then change the S3_HOSTNAME to the default (such as s3.us-east-1.amazonaws.com based on your region)
S3_ALIAS_HOST=turbo.stammy.design
# You do not need an S3_ENDPOINT line

Once you've made the changes to your .env.production file, you'll need to restart the Mastodon services to apply the changes:

sudo systemctl restart mastodon-*.service

Upgrading your Mastodon account privileges

By default your new Mastodon account does not have all privileges available. Mastodon allows an account to have roles like Moderator, Admin, and Owner. You'll want to modify your account to have the Owner role. This can be done using the admin CLI tool called tootctl.

Connect to your server via SSH again (or DigitalOcean's web console) and run the following commands:

# Change from the root user to the mastodon user 
sudo su - mastodon 
# Change to the Mastodon bin directory 
cd /home/mastodon/live/bin
# Modify your account to have the Owner role
RAILS_ENV=production ./tootctl accounts modify admin --role Owner

This may take a few seconds then you should see "OK". If you have any issues, or are using a different webhost, just confirm that you're in the right directory. If you have an error running this and happen to be using the DigitalOcean managed database, you may need to configure the database in another file. Read the managed database section for details.

Enabling single user mode

If this Mastodon account is for you and you only, you can enable single user mode to have the Mastodon homepage be a profile view of your account. By default, this is not enabled and the Mastodon homepage is a timeline view.

To enable this, you'll need to edit .env.production and then restart the web server.

# Change from the root user to the mastodon user 
sudo su - mastodon 
# Change to the Mastodon web directory 
cd /home/mastodon/live
# Edit the .env.production file in Vim
vim .env.production

Now you'll need to create a new line anywhere in the file by moving the cursor with the arrow keys, tapping i to enter Insert mode, creating a new line, then typing in the following line:

SINGLE_USER_MODE=true

Tap ESC to exit insert mode, and then type :wq and hit enter to save and exit vim. And finally, exit mastodon user mode to go back to root by typing exit, and then restart the web server:

# Restart the web server, this can take a few seconds
systemctl restart mastodon-web.service

Ensuring your SSL cert will auto-renew without issue

Mastodon uses the popular certbot tool from the EFF to obtain certs from Let's Encrypt and enable HTTPS on your server. That part all worked great during the initial setup. Certificates from Let's Encrypt last for 90 days before they need to be renewed. Certbot makes them easy to auto-renew: it should have installed a cron with it either in a systemd certbot.timer (see it with systemctl show certbot.timer if you're curious) or in /etc/cron.d/certbot.

You'll need to ensure that your cert will auto-renew without issue. Shouldn't this work out of the box? It should, yes. However, I began poking around in my setup and noticed a configuration issue by default (not sure if this is a DigitalOcean Mastodon installer issue or a Mastodon issue) that would lead to renewal failing.

The default method used is called standalone mode and it only works when the web server is not running, which won't happen during auto-renewal unless you setup your own cron that also stops and starts the server for renewal. This is fine for the initial setup, but if you want to auto-renew your cert, you'll need to use a different method. For more context on this issue, see this commit has updated installation instructions that were never merged to Mastodon main, and this issue.

First, see if you'll even have an issue by doing a dry run invocation of the certbot renew command:

sudo certbot renew --dry-run

If the dry run says the renewal would be successful, you don't need to do anything else and your setup is all good. If it says renewal will fail, you likely need to do the following:

# Edit this config file, swapping out example.com for your domain 
vim /etc/letsencrypt/renewal/example.com.conf

Then find the line that says authenticator = standalone and replace it with these two lines below and save the file.

authenticator = nginx
installer = nginx

Now you can do a dry run again and it should say the renewal would be successful as shown below:

Final setup: updating packages, enabling the firewall, etc..

At this point you could skip ahead and check that Mastodon works and log into your account for the first time, but you'll need to come back to complete a few more things. For the most part you just need to follow this Mastodon guide where you'll update system packages, configure fail2ban to block repeated login attempts and setup a firewall.

The DigitalOcean website has its own interface for configuring a firewall but after repeated attempts I was unable to get it working, so I stuck with the route recommended by Mastodon: iptables.

Optional: Configure IPv6

During DigitalOcean droplet setup you checked a box to enable IPv6 connectivity for your server. Even though that is enabled, you will need to edit some files and add a DNS record. Follow this DigitalOcean guide.

Then you'll need to add an AAAA DNS record to your domain. An AAAA record is like an A record that points to your server, but for IPv6. DigitalOcean should automatically show you your server's IPv6 address in a dropdown when creating that record.

Optional: Setting up automatic database backups

If you opted not to go for the DigitalOcean managed database upgrade, you'll want to a way to export and save backups of your database on a recurring basis. The easiest way to do this is to setup a cronjob that runs a script to dump your database, zip it up, and save it to a cloud storage bucket with the Amazon AWS command line interface. Since we already setup Spaces for cloud object storage, we can re-use that to save your database backups, though if you want to spend the extra time to use a separate cloud provider or bucket for saving only backups, that is preferrable.

First, you'll need to install the AWS CLI. You could follow this guide to install the AWS CLI, but likely you can just install it with: apt:

# Run as root 
apt install awscli
# Provide your credentials for your Spaces bucket, api key and secret when prompted 
aws configure 
# Switch to the Mastodon user
sudo su - mastodon 
# Test to see that it works by listing what's in your Spaces bucket 
# Note: you need to replace the region with your region if different 
# Any time you interact with the AWS CLI, you need to use the --endpoint-url option to 
# specify that we are using DigitalOcean Spaces, not Amazon S3
aws --endpoint-url https://nyc3.digitaloceanspaces.com s3 ls

cd /home/mastodon

# Now we'll create the script to backup the database
vim backup_database.sh

Tap i to enter Insert mode in vim, and paste the following, replacing the Spaces region in the endpoint URL with your Spaces region code:

#!/bin/bash

# Create a filename with the date and time
FILENAME=mastodon_production_$(date +%Y%m%d%H%M).gz

# Dump the database, gzip it, and save locally 
pg_dump mastodon_production | gzip > ${FILENAME}

# Upload the file to your Spaces bucket
# Important: Replace the region with your region if different
# Important: Replace the bucket name 'stammy-mastodon' with your bucket name
aws --endpoint-url https://nyc3.digitaloceanspaces.com s3 cp ${FILENAME} s3://stammy-mastodon/backups/

# Delete the local file
rm ${FILENAME}

Save the script (Tap ESC to exit insert mode, type :wq, hit Enter), then made it executable:

chmod +x backup_database.sh

Now we'll create a cronjob to run the script every day:

# Enter cron job editor. If you're asked what editor to use, select vim again.
crontab -e

Create a new line after the comment block at the time, and enter the following to create a database backup every week. If you want more frequent backups, you can customize it to your liking, such as with 0 0 */3 * * for every 3 days. Take a look at crontab.guru to help you figure out what values to input.

# Backup every week
@weekly ./backup_database.sh

Save the file and you should be good. You may want to try manually running the script now (./backup_database.sh) to ensure it works, and check your Spaces bucket to see a new file was added to the backups directory.

Optional: Install ElasticSearch

If you want the ability for you and your Mastodon instance users to be able to search their own posts, bookmarks, favorites, and mentions, you'll need to install ElasticSearch. This is optional because it's really not that useful. You cannot search across instances, and it does not even support showing posts from other people on other servers that your server has seen but that do not mention you, even if you follow them.

It also takes a bit of RAM, so I especially would not recommend install ElasticSearch if you only have 2GB of RAM or less server. It can be done, but I don't think it's worth it.

Left: Mastodon search without ES installed. Right: With ES installed, showing local results for people, hashtags and posts, including your own.

To install ElasticSearch, you can follow these instructions from the Mastodon site. However, when you get to the part of the guide that tries to enable the service, you may see this error:

Job for elasticsearch.service failed because a fatal signal was delivered to the control processed

This is likely to happen if your machine doesn't have enough RAM. By default ElasticSearch tries to take a lot of RAM, so you'll want to configure it to use less. For example, if you only wanted it to use 512MB of RAM, you can create a new file memory.options:

# Create an elastic search config file to limit memory usage
vim /etc/elasticsearch/jvm.options.d/memory.options

Then enter these two lines to specify a min and max amount of RAM that ElasticSearch can use. For example, a min of 256MB and max of 512MB:

-Xms256m
-Xmx512m

Save the file and try to start the service again:

systemctl daemon-reload
systemctl enable --now elasticsearch

Run the rest of the steps mentioned in the guide and you should have functional search on your instance. You may run into one more issue resulting in a security warning. This github thread is helpful for resolving that (though I think it was also fixed recently in Mastodon 4.1.0).If you want to go down the rabbithole you can modify more security settings but it's likely not a huge deal as we'll be setting up a firewall that won't allow port access to ElasticSearch.

Logging into Mastodon

Changing settings, configuring scripts & testing

If you used a fully-managed host like Masto.host (mentioned earlier), you've already created an account and logged in. If you've followed the guide for creating your own server for Mastodon, then you were given a temporary password to login with. Now it's time to visit your website and sign into Mastodon. You'll also want to immediately enable two-factor auth.

Your new homepage (single user mode enabled) will look empty like this

After signing in, click Preferences in the sidebar. The first thing we want to do is confirm that upgrading your account to the owner role actually worked. You should see a tab for Administration along with others like Sidekiq and PgHero. It should look like this:

Feel free to browse the Profile tab to fill out your bio and related settings. Importantly, I think "Require follow requests" may be on by default which you probably want to disable.

Next, here's a few things to setup:

Fill out server details

Administration → Server Settings → Branding: Fill out the name and description for your server. This is displayed alongside your profile, homepage, and posts so it's quite visible. You can also specify an image to display there.

Disable new account creation

Administration → Server Settings → Registrations: If this server is just for you, you'll want to disable account creation by changing the Who can sign-up setting to nobody.

Disable unauthenticated access to public timelines

Administration → Server Settings → Discovery: This is just a preference but since this is my own server, I only want my posts easily accessible. I wanted to disable the /public (the "Federated" tab) timeline which displayed posts from other people. I'm also a little concerned about liability related to my server hosting content from other people (as mentioned in my last Mastodon post) so making those posts harder to find seems best.

As such, I unchecked Allow unauthenticated access to public timelines.

Adjust media and content retention

Administration → Server Settings → Content retention: This page has some important settings that control how much media and content is kept on your server. If you follow or interact with lots of people on Mastodon (or have any relays setup), you will see the size and costs associated with your cloud object storage provider rise.

You change safely set the media cache retention period to something like 7-30 days. I went with 7 and haven't noticed any odd behavior. This will delete downloaded media files (from other servers, stored on your server or cloud storage) after the specified period. If your server requests the media again, it will be re-downloaded so there's no real harm in setting it too low, you can always change it.

The other setting content cache retention period is a bit less safe to tinker with and will delete posts from other users on your server. I decided not to touch this and will see if I need to change it if my database grows too large over time.

Setting up a cron job for periodic cleanup tasks

The media retention setting in the admin interface is just one setting. You have far more control at your disposal with the tootctl admin command in the command line. In fact, running these commands regularly is a recommend part of setup, mentioned under periodic cleanup tasks.

For this, I have several of the commands running as a cronjob. I put them in a bash script for a cronjob to run. I could have put them directly in the crontab editor as a weekly task but it's easier to keep it in one file where I know it will run serially, without having to specify different run times for each job so they don't run at the same time, potentially bogging down your server.

Here's a list of the commands I run regularly. I put them in a bash script named cleanup.sh that I store in my home directory /home/mastodon:

# This does the heavy lifting and removes locally cached copies of media attachments from other servers
# By default its set to delete media older than 7 days. 
# Adjust days with --days, eg --days 3 for a more aggressive setting
RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove

# Remove local thumbnails for preview cards (the thumbnail when links are shared, etc)
# The default is 180 days and under 14 days is not recommended as they will not be refetched
RAILS_ENV=production /home/mastodon/live/bin/tootctl preview_cards remove --days 60

# Deletes remote accounts that never interacted with local accounts 
# Removes many avatar and header images as part of this
RAILS_ENV=production /home/mastodon/live/bin/tootctl accounts prune

# Removes files that do not belong to existing media attachments 
# For example if you attached a file to a post composer but never posted it
# This command and the one after it are slow to run, so I like to keep them for last
RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove-orphans

# Remove unreferenced statuses from the database, such as statuses that 
# came from relays or from users who are no longer followed by any local accounts, 
# and have not been replied to or otherwise interacted with
RAILS_ENV=production /home/mastodon/live/bin/tootctl statuses remove

# Switch to the mastodon user and home directory 
sudo su - mastodon
# Create bash script 
vim cleanup.sh
# Make the script executable
chmod +x cleanup.sh
# Add the bash script to crontab
crontab -e

In crontab, add this line and then save:

@weekly ./cleanup.sh

I set it to run weekly but you may prefer to run it more often. I've been running my own Mastodon server for a month now and with this setup my cloud media storage is only at 8.5GB with my usage.

Use custom CSS to hide or modify things

Administration → Server Settings → Appearance: This part is completely optional but I found myself wanting to hide certain parts of the site navigation that aren't useful to others as my Mastodon server is only for me. I hid the sign in buttons for logged out users. I hid the "Local" tab for me when logged in: useless for me as I'm the only account on the server. I also hid the "Learn more" button that shows up next to the server details (felt pretty obvious that the server was just for me and I wanted to simplify things a bit).

// hide sign in buttons (you'll have to manually visit /auth/sign_in)
.navigation-panel .navigation-panel__sign-in-banner, 
.ui__header__links a[href="/auth/sign_in"],
.ui__header__links button { display: none !important; }

// Hide the "Local" tab for logged in users (only do this if you're the only user on the server)
.navigation-panel a[href="/public/local"] { display: none !important; }

// hide server details "Learn more" button
.server-banner .button { display: none !important; }

Bonus: Clean up the admin UI with an Arc Boost

If you're running this Mastodon account for you, and you only, you may find the default admin dashboard has a lot of space taken up by useless information, such as showing you the number of new accounts created. I prefer to keep some of that hidden to clean up this page.

Unfortunately, just adding CSS to hide it as in the previous step does not seem to work for admin pages. So I turned to a feature from my browser, Arc, called Boosts. It's an easy way to locally inject CSS (or JS as needed) into any website. Other browsers have similar ways of doing things with extensions. I created an Arc Boost to simply hide a few dashboard modules:

# Not pretty, but it does the job
.dashboard .dashboard__item { display: none; }

.dashboard .dashboard__item:nth-child(3), 
.dashboard .dashboard__item:nth-child(9) { display: block; grid-column: span 3; }

.dashboard .dashboard__item:nth-child(12),
.dashboard .dashboard__item:nth-child(13) { display: block; }

Admin UI, with a bit less clutter for single-user instances.

Try posting an image

And one last thing you'll want to do is try posting with an image attached. This will confirm your cloud object storage configuration works correctly and that your system can resize images. What you'll want to look out for is that dragging an image into the post composer shows a thumbnail preview, and osting it shows the image without requiring you to refresh the page.

Migrating your account

If you have an existing Mastodon account on another server

Now you're ready for the fun part: moving your existing Mastodon account over to your new server, along with your followers. Mastodon does a great job with this and while it's not perfect for reasons I'll mention below, your followers will automatically move with you. You can read a bit about the process from the Mastodon documentation, but it unfortunately does not go into much detail.

Here are the key things to know about migrating your account:

Your posts don't move with you. Posts on your existing Mastodon account do not move with you to the new server. This is unfortunate, but likely a product of how the federation is done (every server that interacted with your posts would have to update). If you only care about the content of your posts and not the engagement around them, you could repost them one by one to your new server (before you do the migration), and edit the date in the database to the original date. But that's weekend project territory to write a script to do that for you... I digress.
Your followers move with you. Your followers will get transferred pretty immediately. When I migrated my account, all of my ~3,000 followers were moved within a day with maybe 75% in the first few hours. It's not entirely flawless though, I did lose about 20 followers in the process.
You have to manually re-follow accounts you follow. You have to export a CSV file of accounts you follow from the old server and upload that on the new server. It's really easy, but it does mean that people you follow will receive a notification that you followed them. It's not ideal but people will start poking around and realize you just moved servers. I've noticed it dozens of times with people I follow and quickly pieced it together.
Your old profile shows a banner that you've moved to another server. And your old account will be in a locked down state. You can login but you can't post or edit anything aside from exporting data from the preferences page.
You can't migrate your account again for 30 days. So make sure you're using a username and domain you like.

Lets get started! The actual first step is creating an account on your new server, which you've already done.

Step 1: Create an account alias on the new server

Setting up a Mastodon account migration is a two-way process. The new server needs to be aware of the old account, and the old server needs to send everything to the new server that's expecting it.

So the first step is to log into your new server, open Preferences → Profile and click on the link under "Moving from a different account." On this page you need to provide your old Mastodon account in the format myusername@example.com.

This step is harmless and reversible—the actual migration does not start until the later steps.

On the new server, add an alias for your old account.

Step 2: Export your account data from the old server

Next, you'll want to export data from the old server. Most importantly, you'll want to export a CSV of accounts you follow so you can later import this on the new server. You can also export files for your muted accounts, blocked accounts, bookmarks, lists, and blocked domains. Go to your old server to download these files: Preferences → Import and Export → Data export.

On this page you can also request an export of your posts and uploaded media from the old server. This is just for you to have as you can't import posts on the new server. Also, it may not work you depending on your old server. I've tried to request my archive on my old server numerous times and it never generated a download for me.

Export data from the old server, especially the list of accounts you follow.

Step 3: Begin the migration on the old server

Now it's time for the actual migration. This will move followers from your old account to your new account. This cannot be undone, so be sure you're ready and have setup your server to your liking before proceeding. This is also a good time to edit anything on your old account (edit bio, delete any posts, etc) as you won't be able to change it later.

On the old server, go to Preferences → Profile → Move to a different account and click the link mentioned. This will take you to the account migration page. Carefully read the bulleted list to understand the implications of this move. Then, enter your new account's handle and the password used for your old server.

Starting the migration on the old server by providing the new account handle. Your old account will now display header saying your account has moved.

Now the migration will start. As mentioned earlier, your posts will not move to the new server, only your followers.

If you have a lot of followers or a particularly low-end server, it may be a good idea to keep an eye on your server as this migration process will keep it busy for a while. You can take a look at system resources with the top or htop commands.

And you can see what Mastodon is doing by looking at the Sidekiq dashboard, accessible in Preferences → Sidekiq. You can click around to see what jobs are running or failing. For the most part you don't need to worry too much. Common job failures are often related to other instances having timeouts. The only cause for concern would be an error indicating connectivity issues with your database, which could mean you may want to adjust the Sidekiq concurrency (mentioned a bit later) in /etc/systemd/system/mastodon-sidekiq.service.

Moving your followers will keep your machine busy. You may wish to keep an eye with htop.

Step 4: Import your following list

The last step of the migration process is to import the CSV files you exported in step 2, the most important of which is the list of accounts you follow. On your new server go to Preferences → Import and export → Import. Select what type of CSV file you're importing in the dropdown, select the CSV file and upload.

You can safely import all lists except the following list before you've actually started your migration (step 3) if you wish.

I recommend waiting a while after step 3 has had time to move most of your followers over before importing your following list. Your server may be quite busy with step 3 so it doesn't hurt to do this an hour later.

Importing your list of accounts you follow

That's it! All of your followers should get moved over shortly and you're ready to start using your own Mastodon server. It's probably a good idea to have a post or two so your followers don't see an empty profile if they happen to venture to your profile.

If you want to have a bit of fun, you can start looking into adding custom emojis (sometimes called emojos in the context of Mastodon) to your server. Take a look at Preferences → Administration → Custom emojis. where you can upload an image and assign it a shortcode. You can even use them in your username.

What to expect

Missing some posts and replies? Yeah, that's normal...

After you've had some time to use your new server, you may see some odd behavior that may lead you to something is broken with your server. Profile pages for accounts may be almost entirely empty, or only have a few posts. Post detail views may not have all the engagement or replies you might expect compared to visiting the original post URL on the remote server.

I'm just here to tell you this is normal. It's not a good thing, but it's the current state of how Mastodon works.

Missing posts on a profile: When you follow someone on your server, or visit the profile of someone you don't follow, Mastodon does not go and fetch all of their posts. That would be taxing on their server and yours. So what happens instead is you get some basic profile info, a post or two (pinned ones) and that's it. Then, the next time your server sees a post from this user (because you follow them or other some action lead to your server seeing this post), it will then appear on their profile. This is not ideal and it wouldn't be unreasonable for Mastodon to fetch a few more posts with a way to request them on demand.

This will get a bit better over time. Your server is brand new, you only just followed accounts and your server has yet to see a lot of new posts and fill up profiles with new posts going forward. You'll still stumble across a user account that your server has never seen before an experience the same blank profile behavior but at least now you know why that is.

Missing replies on posts from other accounts (not your own posts) is a similar situation; your server may have low federation and not be aware of some of those replies. Understanding why this is the case ends up going down a rabbithole of understanding federation more and how your server even comes to know about posts, and under what conditions another server sends a post to your server.

This Mastodon GitHub issue outlines the situations where your server comes to know about posts:

Someone on your instance follows the person who posted, at the time it was posted
Someone you (or anyone else on your instance) follow boosts or replies to that toot
Under some conditions, if that post was a reply to someone you (or anyone else on your instance) follow
The post mentions anyone on your instance
The post was a pinned post when the account was discovered by your instance
Someone on your instance manually fetches a post by pasting the URL in the search bar

This is a hot topic these days and I've seen discussions happening around people wanting to add buttons in Mastodon clients to explicitly load all replies on a post, or older posts on a profile, from the remote servers. There's various technical issues with some of these routes, but I'm curious to see if this situation evolves as I for one would love a refined experience here.

Maintaining, debugging, and optimizing your server

A few things to be aware of

This section is for people running their own server, not through a fully-managed host like Masto.host.

Updating Mastodon

You'll need to keep up to date with Mastodon updates to get new features, performance and security updates. You can check for new updates on the GitHub project page but it's probably easiest to subscribe to the atom feed in your news reader of choice, such as NetNewsWire.app. Here's the releases feed, or the website if you prefer.

The steps you need to follow for each update may vary, but additional steps will be mentioned in the release notes for each release. Generally you'll need to SSH into your server, fetch the latest release with git, and restart some services. Find the complete steps here: Upgrading to a new release.

You should check now that your server is on the latest Mastodon version. At the time of this writing, the DigitalOcean 1-click installer is running an outdated version.

Updating your Ubuntu server

Along with maintaining and keeping Mastodon up to date, you'll want to do the same with your server. If you followed along with the DigitalOcean route explained above, your machine uses Ubuntu OS, apt package manager and has a specific package called unattended-upgrades installed by default. That package will automatically install essential security updates, but you can modify it to update even more.

I don't want to get too in the weeds about linux server maintenance as everyone seems to have their own specific practice about when to update things, how to update them, and when to leave it alone.. you know, if it ain't broke don't fix it. Some people may prefer to update all packages every week. Not including security essentials, that's a bit too much for me and feels like I'd constantly be signing up for a potential debugging session if anything unexpected happens.

Personally, I just update available packages whenever I get around to it, perhaps every month or two. Usually with these:

# Refresh list of available updates 
sudo apt update
# View list of available upgradable updates
apt list --upgradable
# Upgrade packages 
sudo apt upgrade
# Remove unneeded packages not installed by you 
sudo apt autoremove

# Alternatively, instead of the lines above:
sudo apt update && sudo apt upgrade –y && sudo apt autoremove -y

When things go wrong

I wanted to provide a few pointers for things to look into when things go wrong with your Mastodon server. There's no way I can provide a general solution to what could be a specific problem you may encounter, but here's a few things may be a good place to start:

Check service status: Run this to get a quick look at the core Mastodon services: mastodon-web.service, mastodon-sidekiq.service, and mastodon-streaming.service.
```
sudo systemctl status mastodon-*
```
In addition to status of the 3 core Mastodon services (you'll want to see "active (running)" in green), this command will show the last 10 lines from the logs for each service.
Check the logs: This may sound obvious but knowing how to check and work with logs is critical. It's the best way to know what's going on with your machine. To access any of the logs for the aforementioned 3 services, we use a tool called journalctl to read and filter system logs that are saved in the binary "journal" by systemd. You can always run man journalctl to see more details on how you can use it; it's very powerful.

It's a good idea to go through the logs for each Mastodon service after you've set everything up to make sure that things are working smoothly. Here's a few examples of how you can use it:
```
# Show the last 50 lines of the mastodon-web log
# and keep showing new lines as they come in (that's the -f follow flag)
sudo journalctl -u mastodon-web -n 50 -f

# Or read the mastodon-streaming log 
sudo journalctl -u mastodon-streaming -n 50 -f

# Or the mastodon-sidekiq log 
sudo journalctl -u mastodon-sidekiq -n 50 -f

# Or only show me sidekiq logs that have priority level 4 (warn) or higher
sudo journalctl -u mastodon-sidekiq -f -n 100 -p 4

# And you can always pipe it to grep
# For example to show any lines from the web logs that have a status code of 500
sudo journalctl -f -u mastodon-web -n 500 | grep "status=500"
```
You can do a lot with these commands. This helped me initially discover database connection issues when I over-eagerly tried to optimize configs, and helped me notice that I forgot to require SSL for my remote database.
Restart things: You can restart or reload (like a restart but no downtime as it reloads the config, if supported by the service) these services with a command like:
```
sudo systemctl restart mastodon-web
```
Check sidekiq: Sidekiq is the background job processor for Mastodon. It's how Mastodon manages receiving new posts, sends our your posts, and more.

Generally you don't need to worry too much about what it's doing, but there's one key scenerio to be aware of: if you have a massive queue of jobs that are piling up. What this looks like is the "Enqueued" tab in the Sidekiq dashboard will show a large number that's not going down in a reasonable amount of time.

This would mean your machine cannot keep up. You may notice significant delays in receiving new posts on your server. A Mastodon instance running well would have likely have few to no tasks busy or queued, and when when queued tasks are added, they get processed quickly. If this happens to you, it could mean your server is underpowered and trying to do too much (don't use any relays if you are), or could benefit from optimizing your sidekiq setup (more later).

There's also a tab for dead jobs in Sidekiq. These are tasks that have failed repeatedly and will not be retried. You'll just want to glance at those to ensure the issues are not related to you. Even if some jobs fail, you'll likely see that they are HTTP connection or timeout errors with other servers and likely not your issue.

Access the sidekiq dashboard directly in Preferences → Sidekiq

Optional: Extend your post character limit beyond 500

It's your server, you control the code so why not extend your server's character limit well past the default limit of 500 characters? Doing this will take just a few minutes, all you have to do is change a number in 2 files, precompile assets, and then restart some services.

This article goes into detail about the process: Customize Mastodon to Change Your Post Character Limit. However, keep in mind that you'll have to reapply these changes with every Mastodon update.

I changed my character limit to 15,000. In reality I won't be posting things that long, though I did take advantage of a 1,000 character reply which was very handy and I didn't have to try to split it up into multiple replies or aggressively truncate my thoughts. The change works seamlessly: people on other Mastodon (or other ActivityPub software) servers will see the longer posts, and all Mastodon clients should see the updated limit in their post composer interfaces.

Extended character limit recognized by Ivory and Mastodon Web.

Optional: Change your join date

One annoying thing of creating a new Mastodon server, or migrating to a new one, is that your join date—shown prominently on your profile—reflects the date you created the new account. It doesn't inherit the date of your previous account(s) that you migrated.

Since you're running your own server and have access to the database, the join date is easy to change. You can change it to your original account creation date, or any date you like. Why not? It's your server, do with it what you want. SSH into your server, or use the web console from your hosting provider and run the following commands:

sudo su - mastodon 

# Enter PostgreSQL interactive terminal for the Mastodon database
psql -d mastodon_production

# Update the join date for your account...
# Replace YOUR_USERNAME with your username (does not include domain name)
# Replace date with your desired date, using the exact same format 
UPDATE accounts SET created_at='2018-01-01 00:00:00' where username='YOUR_USERNAME';

# Exit the PostgreSQL interactive terminal
exit

Optimizing Mastodon for your server

Proceed at your own risk

If you're just running your own single-user Mastodon instance that you expect to be low traffic, you don't need to touch anything in this section; the defaults are fine for your system. But if you have a beefy server with extra RAM and cores to put to use, you have a lot of followers, and/or you just generally want to nerd out on squeezing more performance from your server then read ahead.

As a word of caution, changing things in various Mastodon config files can easily take your server offline. I recommend doing this before you've done a migration, or late at night to minimize the effects of downtime. Also, I highly recommend setting up Rewind AI on your Mac so you can easily retrace your steps when you're changing things and running commands.

By following this guide, you've already done an important performance tweak: using a cloud object storage solution for hosting all media for your Mastodon posts. The most common things advanced users will do includes modifying sidekiq to use separate processes for each queue, as well as setting up pgbouncer for PostgresSQL.

Resources

Scaling Mastodon can be a deep topic depending on your needs. I'm only concerned with running Mastodon as a single-user instance, but it's helpful to better understand the pieces of Mastodon and what can and should be changed and when.

The Mastodon website has a page about scaling your server. It's particularly good to learn the relationship between threads and processes with the web server (threads max out your CPU first, processes max out your RAM first), as well as learning about pgbouncer to help reduce any database connection bottlenecks.

Then there's the lengthy and advanced article Scaling Mastodon: The Compendium. Most of the topics in there might be a bit overkill for a small single-user instance though.

Sidekiq

I mentioned a bit about what to be on the lookout for with sidekiq earlier in this article, such as having a backed up queue. The first thing I did with sidekiq is slightly increase the number of threads. By default using the DigitalOcean droplet, you may have a default of 25 for DB_POOL and the concurrency flag -c.

However, the number of max_connections for the Postgres database by default is 100. Counterintuitely, if you opted for a separate managed database from DigitalOcean you may have only 25 database connections available, in which case you may opt to setup their pgbouncer feature called "connection pools"; that's what I did. If you do the same, you'll need to change your database connection info so Mastodon connects to pgbouncer on a different port.

What I did was edit /etc/systemd/system/mastodon-sidekiq.service and change the DB_POOL value to 50, as well as the number after -c on the ExecStart line. You don't want to tell sidekiq to consume all of our available database connections as each Puma web server thread will need a database connection too.

Environment="RAILS_ENV=production"
# Edit this 
Environment="DB_POOL=50"
Environment="MALLOC_ARENA_MAX=2"
Environment="LD_PRELOAD=libjemalloc.so"
# and the number after -c 
ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 50

Then I also configured sidekiq to automatically restart every 6 hours. There seems to be an open issue where sidekiq memory usage continually creeps up over time. Honestly, this was one of the firs things I configured and never ran without this setting so I'm not entirely sure how necessary it is. But I haven't had any issues.

Restart=always
RuntimeMaxSec=21600

After modifying mastodon-sidekiq.service, you'll need to run these:

systemctl daemon-reload
systemctl restart mastodon-sidekiq.service

Puma

The other thing I did was adjust the number of threads and proccesses used by puma, the Mastodon web/app server. If you expect a lot of web traffic, or just like to be prepared, you may be interested in tweaking puma as well.

I initially looked into this because had some unused RAM on my server and wanted to let puma use a bit more. In the process I learned that the it's recommended to allocate 1.5 processes per CPU core. So my server has 2 cores, so I can get away with 3 processes for puma. I set it to also use 10 threads per process.

Edit /etc/systemd/system/mastodon-web.service and add these two lines (or modify if they already exist):

# Set this to 1.5 * number of cpu cores
# Find your # of cores by running: cat /proc/cpuinfo | grep processor | wc -l
Environment="WEB_CONCURRENCY=3"
# The number of threads per process. If you're RAM constrained, you may want to lower this.
Environment="MAX_THREADS=10"

Save the file and restart the service:

systemctl daemon-reload
systemctl restart mastodon-web.service

That being said, you don't really need to mess with puma much. For a single user instance the vast majority of your traffic will be from people interacting with your posts via a Mastodon app/client through their server—not directly via your website—which is not served by the web service.

This is barely scratching the surface with what's possible for optimizing and scaling Mastodon. I'm just running my own single-user instance and it's been running great so far. If you're not sure what to do, I would recommend not touching anything and seeing how your server works for a while. You can keep an eye on system logs as mentioned earlier, as well as sidekiq.

The end

This article has been in the works for a while now but hopefully it has helped you get a better understanding what's involved with hosting your own Mastodon server, how to keep it running smoothly, and how to tweak it to your liking.

It's a bit of work to get running but the benefits of owning your online identity and data with Mastodon are well worth it. If you've enjoyed this article, please share it with anyone that might find it useful, and of course follow me on Mastodon on my new instance: @stammy@stammy.design.

JShrink reaches over 21,000,000 installs and releases v1.6!

tedious ramblings — Mon, 06 Mar 2023 18:00:48 GMT

JShrink is a Javascript minifier written in pure PHP. It allows web applications such as Magento and Matomo (as well as thousands of other projects) to shrink Javascript code on the fly. In 2014 JShrink reached stability, and shortly after that hit the milestone of 100k total downloads- today JShrink has over 21 million downloads,...

Keeping a semi-automatic electronic ship's logbook

Henri Bergius — Mon, 06 Mar 2023 02:00:00 GMT

Maintaining a proper ship’s logbook is something that most boats should do, for practical, as well as legal and traditional reasons. The logbook can serve as a record of proper maintenance and operation of the vessel, which is potentially useful when selling the boat or handling an insurance claim. It can be a fun record of journeys made to look back to. And it can be a crucial aid for getting home if the ship’s electronics or GNSS get disrupted.

Like probably most operators of a small boat, on Lille Ø our logbook practices have been quite varying. We’ve been good at recording engine maintenance, as well as keeping the traditional navigation log while offshore. But in the more hectic pace of coastal cruising or daysailing this has often fallen on the wayside. And as such, a lot of the events and history of the boat is unavailable.

To redeem this I’ve developed signalk-logbook, a semi-automatic electronic logbook for vessels running the Signal K marine data server.

This allows logbook entries to be produced both manually and automatically. The can be viewed and edited using any web-capable device on board, meaning that you can write a log entry on your phone, and maybe later analyse and print them on your laptop.

Why Signal K

Signal K is a marine data server that has integrations with almost any relevant marine electronics system. If you have an older NMEA0183 or Seatalk system, Signal K can communicate with it. Same with NMEA2000. If you already have your navigational data on the boat WiFi, Signal K can use and enrich it.

This means that by making the logbook a Signal K plugin, I didn’t have to do any work to make it work with existing boat systems. Signal K even provides a user interface framework.

This means that to make the electronic logbook happen, I only had to produce some plugin JavaScript, and then build a user interface. As I don’t do front-end development that frequently, this gave me a chance to dive into modern React with hooks for the first time. What better to do after being laid off?

Signal K also has very good integration with Influx and Grafana. These can record vessel telemetry in a high resolution. So why bother with a logbook on the side? In my view, a separate logbook is still valuable for storing the comments and observations not available in a marine sensor network. It can also be a lot more durable and archivable than a time series database. On Lille Ø we run both.

User interface

The signalk-logbook comes with a reasonably simple web-based user interface that is integrated in the Signal K administration UI. You can find it in Web apps → Logbook.

The primary view is a timeline. Sort of “Twitter for your boat” kind of view that allows quick browsing of entries on both desktop and mobile.

There is also the more traditional tabular view, best utilized on bigger screens:

While the system can produce a lot of the entries automatically, it is also easy to create manual entries:

These entries can also include weather observations. Those using celestial navigation can also record manual fixes with these entries! Entries can be categorized to separate things like navigational entries from radio or maintenance logs.

If you have the sailsconfiguration plugin installed, you can also log sail changes in a machine-readable format:

Since the log format is machine readable, the map view allows browsing entries spatially:

Electronic vs. paper

The big benefits of an electronic logbook are automation and availability. The logbook can create entries by itself based on what’s happening with the vessel telemetry. You can read and create log entries anywhere on the boat, using the electronic devices you carry with you. Off-vessel backups are also both possible, and quite easy, assuming that the vessel has a reasonably constant Internet connection.

With paper logbooks, the main benefit is that they’re fully independent of the vessel’s electronic system. In case of power failure, you can still see the last recoded position, heading, etc. They are also a lot more durable in the sense that paper logbooks from centuries ago are still fully readable. Though obviously that carries a strong survivorship bias. I would guess the vast majority of logbooks, especially on smaller non-commercial vessels, don’t survive more than a couple of years.

So, how to benefit from the positive aspects of electronic logbooks, while reducing the negatives when compared to paper? Here are some ideas:

Mark your position on a paper chart. Even though most boats navigate with only electronic charts, it is a good idea to have at least a planning chart available on paper. When offshore, plot your hourly or daily position on it. This will produce the navigation aid of last resort if all electronics fail. And marked charts are pretty!
Have an off-vessel backup of your electronic logs. The signalk-logbook uses a very simple plain text format for its entries exactly for this reason. The logs are easy to back up, and can also be utilized without the software itself. This means that with a bit of care your log entries shouls stay readable for many many years to come. On Lille Ø we store them on GitHub
Print your logs. With something like a cheap receipt printer, it would be possible to print your log entries periodically, maybe daily or after each trip. Then you can have an archival copy that doesn’t rely on electronics. Here is a repository implementing just that

API

In addition to providing a web-based user interface, signalk-logbook provides a REST API. This allows software developers to create new integrations with the logbook. For example, these could include:

Automations to generate log entries for some events via node-red or NoFlo
Copying the log entries to a cloud service
Exporting the logs to another format, like GPX or a spreadsheet
Other, maybe non-web-based user interfaces for browsing and creating log entries

Getting started

To utilize this electronic logbook, you need a working installation of Signal K on your boat. The common way to do this is by having a Raspberry Pi powered by the boat’s electrical system and connected to the various on-board instruments.

There are some nice solutions for this:

Sailor Hat for Raspberry Pi allows powering a Raspberry Pi from the boat’s 12V system. It also handles shutdowns in a clean way, protecting the memory card from data corruption
Pican-M both connects a Raspberry Pi to a NMEA2000 bus, and powers it through that

You can of course also do a more custom setup, like we did on our old boat, Curiosity.

For the actual software setup, marinepi-provisioning gives a nice Ansible playbook for getting everything going. Bareboat Necessities is a “Marine OS for Raspberry Pi” that comes with everything included.

If you have a Victron GX device (for example Cerbo GX), you can also install Signal K on that.

Once Signal K is running, just look up signalk-logbook in the Signal K app store. You’ll also want to install the signalk-autostate and sailsconfiguration plugins to enable some of the automations.

Then just restart Signal K, log in, and start logging!

Can ChatGPT write the Python async message queue codes from the previous posts?

Bytepawn - Marton Trencseni — Sun, 05 Mar 2023 01:00:00 GMT

I try to get ChatGPT to write the codes in the previous posts. It's able to write the basic message queue skeleton, but cannot implement more complicated features such as delivery semantics with caching.

Floccus is the bookmark manager you have been looking for

Jan van den Berg — Sat, 04 Mar 2023 20:10:57 GMT

Floccus does exactly what you want because Floccus doesn’t break your bookmark management flow. The flow being: adding, changing, removing, moving bookmarks *in* your browser, straight from the bookmark bar and with the shortcuts you already know. Because Floccus is nothing more than a browser extension. How does it work? Floccus is actually not a […]

What I'm up to - March 2023

Philip I. Thomas — Thu, 02 Mar 2023 22:40:41 GMT

✨ What I've been up to

Part of my work at The Contraption Company includes helping startups develop new products. For the last six months, I've unintentionally been working almost exclusively with AI-focused startups, and I've seen the rise of OpenAI first-hand.

While the broader tech market shows signs of slowing, the hype in AI has been exploding. One Contraption Co. client went from an idea to >$7m in seed funding within five weeks. (They're still in stealth - but are hiring in engineering, growth, and design. Email me for an intro.)

I summarized my observations and predictions about this AI hype cycle in this post: OpenAI, the path for OpenAI-powered startups, and the AI hype cycle.

Finally, I've been contemplating this thought: Businesses like to buy from businesses, and people like to buy from people.

🤔 Things to share

Articles: Why Social Media is Bad for Artists (via Chris L-T). The Build-Nothing Country. Rise of the Silicon Valley Small Business.
Books: Dopamine Nation - a takeaway for me has been that adversity results in pleasure while pleasure results in pain.
Trends: Vector databases, such as pgvector.
Audio: Derek Sivers on Remote Ruby.
Video: Why single-origin coffee is so expensive (featuring Metric Coffee, my favorite Chicago roaster).
Fun fact: Cede and Company and Depository Trust Company are two of the most important companies in the USA.
Gadgets: Umeshiso Chopsticks.
Apps: Bartender adds the ability to reorder and hide Mac menu bar items, which I find essential on the new notched laptops. uBlock Origin can block elements on pages - I started using it on Linkedin to block the feed + notifications, and on Amazon to make the homepage just a search bar. You can subscribe to Youtube channels via RSS - allowing you to set up email notifications.
Coffees I'm drinking: Comparing geisha varietals from Prolog, April, and Prodigal.

📫 What I'm up to this month

I'm in the process of building 1,000 websites for people on Postcard. I'm also working on a post-writing experience I hope to launch soon.

📍 Where I'll be (Let me know if we overlap!)

March 8-24: 🇫🇷 Paris

How I do comparative tastings of coffee

Writing a simple Python async message queue server - Part II

Bytepawn - Marton Trencseni — Thu, 02 Mar 2023 01:00:00 GMT

I write a somewhat more complicated, but still relatively simple async message queue server in Python.

Python tip 24: modifying list using insert and slice

learnbyexample — Tue, 28 Feb 2023 02:00:00 GMT

The insert() list method helps to insert an object before the given index. Negative indexing is also supported.

>>> books = ['Sourdough', 'Sherlock Holmes', 'Cradle']

# same as: books.insert(-1, 'The Martian')
>>> books.insert(2, 'The Martian')
>>> books
['Sourdough', 'Sherlock Holmes', 'The Martian', 'Cradle']

# index >= list-length will append the object at the end
>>> books.insert(1000, 'Legends & Lattes')
>>> books
['Sourdough', 'Sherlock Holmes', 'The Martian', 'Cradle', 'Legends & Lattes']

You can use slicing notation to modify one or more list elements. The list will automatically shrink or expand as needed. Here are some examples:

>>> nums = [1, 4, 6, 22, 3, 5]

# modify a single element
>>> nums[0] = 100
>>> nums
[100, 4, 6, 22, 3, 5]

# modify the last three elements
>>> nums[-3:] = [-1, -2, -3]
>>> nums
[100, 4, 6, -1, -2, -3]

# elements at index 1, 2 and 3 are replaced with a single object
>>> nums[1:4] = [2000]
>>> nums
[100, 2000, -2, -3]

# element at index 1 is replaced with multiple elements
>>> nums[1:2] = [3.14, 4.13, 6.78]
>>> nums
[100, 3.14, 4.13, 6.78, -2, -3]

RHS must be an iterable when you use slicing notation with :, even when LHS refers to a single element. For example, nums[1:2] = 100 is not valid.

Video demo:

See also my 100 Page Python Intro ebook.

Writing a simple Python async message queue server - Part I

Bytepawn - Marton Trencseni — Mon, 27 Feb 2023 01:00:00 GMT

I write a simple, bi-directional async message queue server in Python.

What I learned from contributing to Rust's linter

Krishna's blog — Sun, 26 Feb 2023 02:00:00 GMT

I share my experience contributing to Rust's linter Clippy and the welcoming environment for new contributors.

Marginalia Search: 2 years, big news

Weblog on marginalia.nu — Sun, 26 Feb 2023 02:00:00 GMT

No time like the project’s two year anniversary to drop this particular bomb… Marginalia’s gotten an NLNet grant. This means I’ll be able to work full time on this project at least a year. https://nlnet.nl/project/Marginalia/ This grant is essentially the best-case scenario for funding this project. It’ll be able to remain independent, open-source, and non-profit. I won’t start in earnest for a few months as I’ve got loose ends to tie up before I can devote that sort of time.

How to transcribe podcast audio (WhisperX with speaker diarization)

swyx's site RSS Feed — Fri, 24 Feb 2023 17:33:15 GMT

Note: sometimes WhisperX is WAAYYYY too slow so I often end up using https://github.com/ggerganov/whisper.cpp which somehow runs much faster.

How Microsoft Became Phishing's Biggest Enabler

Brightball Articles — Thu, 23 Feb 2023 02:00:00 GMT

It might sound strange to hear that Microsoft, a company who goes to great lengths to protect computers and networks, is one of the biggest contributors to phishing and fraud on the planet. It's true unfortunately. They aren't actually committing the acts themselves of course, but they are enabling the problem by withdrawing support for standards designed to help stop it. Here's why this is such a big deal. UPDATE 4/12/2023: After years, Microsoft is finally fixing this by honoring p=reject. This is a huge improvement and deserves to be applauded. The work isn't done though. We need aggregate reports to avoid blind spots during our implementation. Offering the reports for enterprises is a great step though.

Notes from Neal Gabler's Walt Disney

Notes on software development — Sat, 18 Feb 2023 02:00:00 GMT

Disney was a celebrity by his mid-30s, Disney the company was famous by 1930s.

Even though politically the 1930s was considered the decade of Roosevelt (elected President in 1933), culturally the 1930s was considered the decade of Mickey Mouse.

Almost every new animation/filmmaking technique they tried, they would experiment with it in shorts (Silly Symphonies) before applying to big films like Snow White. Examples of this include:

Multiple layers of animation moving independently to create depth in The Old Mill
The first Disney animations with humans (not flora/fauna) like The Cookie Carnival

Nobody took animation seriously, didn't think there was much possibility for it in film. Disney kept pushing the envelope. Some examples include:

Not including the hand inside the drawing (though early Disney ones did)
Eventually focusing on actual stories, not just gags/jokes
Sound (the famous Mickey Steamboat Willie animation, read more)
Merchandise, not just the art
Feature films (i.e. Snow White in 1937, the first animated feature film), not just shorts
Brought Hollywood to Television
- "Walt Disney signed an exclusive long-term contract today with the American Broadcasting Company to become the first leading Hollywood producer to enter into formal alliance with television. NY Times
Added color to TV shows

Snow White

Disney hired fine arts teachers to come and teach employees. From time to time he forced the artists to take night classes.

They trained for years(?) before starting the animation of Snow White and did almost all the animation in the last 10 months or so before the release in December 1937.

They had to do 24-hour animation in 8 hour shifts to get up to speed. They had to hire 100s of animators to do fill in work so the “master” animators could focus on “drawing the extremes”.

The average age at Disney was 25. These days of the 1930s really felt quite similar to what a Silicon Valley startup is thought to be.

Disney preferred to hire recent art school students so they could train them in the Disney style.

They could not animate humans during Snow White well enough so they ended up just tracing them, called rotoscoping.

The Snow White voice cast were quite famous at the time. We wouldn't know it now but it was basically an ensemble cast.

World War 2

Ran low on money so they produced films for the US Government. Propaganda, basically. But also instructional videos.

Disney workers began striking (1941) and established unions. If Disney was a dick before this, he became a much bigger dick after this.

Post War

Got into television with ABC initially. First Hollywood company to do so. Arrangement with ABC was in part to finance Disneyland. (Not covered in the book but Disney eventually took over ABC, not before eventually splitting ABC and working with NBC though.)

Disney stopped caring about films and moved to mostly thinking about Disneyland, this under WED (what is now Walt Disney Imagineering).

After Disneyland launched he moved on to world fairs and eventually Disneyworld. He died of lung cancer before completing Disneyworld.

Tidbits

The Reluctant Dragon, a throwaway film because they needed money when they went public. It is the story of a children's book author trying to get Disney to make a film out of his book. He stumbles around the new Disney Burbank Studio through art classes and musicians practicing, uncovering how Disney films are made in the process.

Questions

What were the other major animation studies? Even if Snow White was the first animated feature film, surely others must have rushed to copy the success. Who were they?
- UPA (Mr Magoo) was one. Also Warner Brothers

Conclusion

Basically after every turn he'd get tired of the stuff he had already done (and killed at doing) to do something new. From animated shorts to feature films to television to Disneyland to Disneyworld and EPCOT.

To his employees he was a huge dick. They'd be in constant fear of upsetting him and getting fired. And he admitted that he would basically fire people randomly. He'd fire anyone important enough to get their name on a door (i.e. establish their own fiefdom within the company). But it seems more like Disney the company worked in spite of this rather than because of this.

After Mary Poppins (1964, two years before he died): "I'm on the spot. I have to keep trying to keep up to that same level. And the way to do it is not to worry, not to get tense. Not to think, 'I got to beat Mary Poppins', 'I got to beat Mary Poppins'. The way to do it is just to go off and get interested in some little thing, some little idea that interests me. Some little idea that looks like fun."

Finished Neal Gabler's Walt Disney (5/5) and here are my raw notes. (If I had to polish the notes I wouldn't have the will to publish.) Hopefully a few interesting bits and links in there though.

In particularly this quote (2nd pic) really struck me.https://t.co/P9astFZ6Ts pic.twitter.com/wKPd6zjLau
— Phil Eaton (@eatonphil) February 18, 2023

If I had a million dollars (to do physics)

--Mike-- — Fri, 17 Feb 2023 22:54:47 GMT

If I had a million $USD to do physics, I'd do some experiments.

Berry Phase/Electromagnetic 4 Potential experiments

First purchase/rental - a SQUID - Superconducting QUantum Interference Device.

My first hypothesis to check is regarding the A field, also known as the electromagnetic vector potential. I hypothesize that if you were to take a bifilar wound coil of quite a few turns, the flux when the windings were connected to oppose each other would correctly generate no B field (detectable with a compass, magnetometer, etc), but I believe there WOULD be an A field generated, perhaps much weaker, than that of the field when the windings aided each other, but non-zero.

I'd then investigate the polarity and form of this field in 3d around the coils, etc.

I'd also perform the same experiment with a set of toroidally wound coils, a pair, and see what type of A fields they generate, and see if everything matches simulations.

--

A refinement of this experiment would be to use a pair of atomic clocks, one outside a "null set of coils", and the other inside them (with actual B magnetic fields canceled), to see if there was any drift with power applied.

---

Gravimetric / Inertial measurements

Purchase - Vacuum Pump

Next, a torsion bridge, in the manner of Cavendish... a long (as long as I could get in the garage) rod inside a copper/aluminum tube, pumped to as good a vacuum as I could reasonably get. Using electromagnetic forces on the ends, test out various materials to see measure their gravimetric effects on the bridge.

I'm particularly interested to see if some materials have a different G than others

Also, it would be interesting to see if a large capacitor had a different gravimetric effect if it were charged/not charged (whilst inside a faraday cage, of course)

Simple jumphost ssh-agent config

Jan van den Berg — Fri, 17 Feb 2023 22:28:47 GMT

You can find many tutorials online on how to use ssh-agent or ssh-ident correctly. This is a short and simple two line fix aimed at a specific use i.e. a single connection to a jumphost. Add this to your .bashrc So now when you type jumphost: And from the jumphost you can ssh connect to […]

2023–02–17: Pinephone keyboard firmware update (1.3)

megi's PinePhone Development Log — Fri, 17 Feb 2023 01:00:00 GMT

Muv-Luv Unlimited: THE DAY AFTER 00 review

ブラック — Wed, 15 Feb 2023 02:00:00 GMT

Taking place after the events of Unlimited, the story has quite a different pace than Unlimited or Alternative with the BETA threat gone. I wasn’t completely sold on the story at first, but the characters are thankfully mostly likeable and interesting. However, I did find myself enjoying the read towards the tail end of the game. The tail end that won’t take too long to reach – it was around a four-and-a-half hour read for me. Obviously, as an “episode zero”, it is meant as a prologue to the rest of THE DAY AFTER story, and that’s probably why it’s not a very long and extremely captivating story.

The art is mostly satisfactory. The characters have some animation, and a good amount of expressiveness and perspectives. The biggest issue with the art is that some graphical assets are not high enough of a resolution for the amount of zoom levels they push, so you have occasional mosaics right in your face. Audio-wise all characters are fully voiced and you get some hits of nostalgia with some familiar background songs.

The game also plays nice on the Steam Deck, but you’re going to need a community layout. I don’t know about what goes into getting a game “Steam Deck Verified” as episode 00 is, but there’s no actual good Steam Deck layout. The community layout that I used allowed pretty comfortable UX but still required a bit of joystick mouse movement, so there’s definitely room for improvement. I did also notice some crackling issues with audio after waking the device from sleep without quitting the game, but that might be a SteamOS issue, is not really a major one and can be resolved by just restarting the game.

All in all, I would recommend THE DAY AFTER 00 for a fan of Muv-Luv – just temper your expectations, and maybe don’t pay full price. After all, if you’re interested in the THE DAY AFTER series, you’re probably not going to skip the prologue, right?

The Capabilities of High Altitude Balloons

MiscDotGeek — Sun, 12 Feb 2023 23:42:35 GMT

In this MiscDotGeek article, we’re trying something… different. This is an experiment. You know my writing, you know my style. Read the article. Poke holes in it. Why? See at the end (don’t cheat!) High Altitude Balloons: Unlocking the Potential of Near Space Exploration Have you ever wondered what the view from the edge of …

The post The Capabilities of High Altitude Balloons appeared first on MiscDotGeek.

The Time I Accidentally Ended Up Combatting Fraud for a Year

Brightball Articles — Fri, 10 Feb 2023 22:36:00 GMT

Lately, I’ve been spending a lot of time enjoying the Darknet Diaries podcast and it’s compelled me to finally share the entire story of the most intense year of my 20 year professional career. I was the sole developer hired by a company going through a circus-like ownership transition while criminals actively worked to defraud the 300,000 users of this 14 year old, high end marketplace. We experienced late nights, numerous technical challenges, worked with abuse response teams, learned a lot of lessons about phishing and fraud, high emotions, death threats and at least one person lost a business that depended on the site. Here’s the story from start to finish, including how to prevent many of these problems on your own site. Buckle up.

CLI tip 23: recursive filename matching with globstar

learnbyexample — Fri, 10 Feb 2023 02:00:00 GMT

Enable the globstar option to recursively match filenames within a specified path. You can use shopt -s globstar and shopt -u globstar to set and unset this option respectively.

First, create some sample files:

$ mkdir test_globstar && cd $_
$ mkdir -p todos projects/{tictactoe,calculator}
$ touch ip.txt .hidden.txt report.log hello.py
$ touch todos/{books,outing}.txt
$ touch projects/tictactoe/game.py projects/calculator/{calc.sh,notes.txt}

Here are some examples:

$ shopt -s globstar

$ ls -1 **/*.txt
ip.txt
projects/calculator/notes.txt
todos/books.txt
todos/outing.txt

$ ls -1 **/*/*.txt
projects/calculator/notes.txt
todos/books.txt
todos/outing.txt

# assumes extglob is enabled
$ ls -1 **/*.@(py|sh)
hello.py
projects/calculator/calc.sh
projects/tictactoe/game.py

$ ls -1d **/
projects/
projects/calculator/
projects/tictactoe/
todos/

If you need to match hidden files as well, enable the dotglob option:
$ shopt -s dotglob
$ ls -1 **/*.txt
.hidden.txt
ip.txt
projects/calculator/notes.txt
todos/books.txt
todos/outing.txt

Video demo:

See also my Linux Command Line Computing ebook.

Vim tip 22: word and WORD motions

learnbyexample — Fri, 10 Feb 2023 02:00:00 GMT

Definitions from :h word and :h WORD are quoted below to explain the difference between word and WORD.

word A word consists of a sequence of letters, digits and underscores, or a sequence of other non-blank characters, separated with white space (spaces, tabs, <EOL>). This can be changed with the iskeyword option. An empty line is also considered to be a word.

WORD A WORD consists of a sequence of non-blank characters, separated with white space. An empty line is also considered to be a WORD.

word based motions:

w move to the start of the next word
b move to the beginning of the current word if the cursor is not at the start of word. Otherwise, move to the beginning of the previous word
e move to the end of the current word if cursor is not at the end of word. Otherwise, move to the end of next word
ge move to the end of the previous word
3w move 3 words forward
- Similarly, a number can be prefixed for all the other commands discussed here

WORD based motions:

W move to the start of the next WORD
- 192.1.168.43;hello is considered as a single WORD, but has multiple words
B move to the beginning of the current WORD if the cursor is not at the start of WORD. Otherwise, move to the beginning of the previous WORD
E move to the end of the current WORD if cursor is not at the end of WORD. Otherwise, move to the end of next WORD
gE move to the end of the previous WORD

All of these motions will work across lines. For example, if the cursor is on the last word of a line, pressing w will move to the start of the first word in the next line.

Video demo:

The Sawtooth Career

jodavaho.io — Thu, 09 Feb 2023 02:00:00 GMT

On the optimality of sawtooth careers

I was having lunch at JPL with their Chief Engineer, Rob Manning. Normally, you’d expect the Chief Eng. to reserve their time for more important people, but that’s not how JPL works.

I had posted in the New Researcher Support Group slack channel, asking for stories from folks who regret some part of their career. Gloomy, I know. He was curious, and so we we got lunch.

Being early career is fraught with uncertainty. What should I specialize in? How can I make an impact? etc.

He cut through it pretty quickly with his model for an optimal career: The sawtooth.

Behold, the sawtooth graph

Rob said that he had a horrible character flaw, in that when he got bored of something he’d usually leave to do something else¹. This led to him perpetually entering new fields as a beginner, asking dumb questions and holding himself accountable to learning quickly. He said this breadth of perspective allowed him to become a far better engineer over time.²

I respect that. Tying it back, it helped me see that just doing something, and doing it fully, is better than not. If you hate it, at least you did it, and now you can go dedicate yourself to something else.

For that reason I was perfectly happy not applying for management positions when I left Amazon. Even though this is kind of a step down. Or, when I was a manager, I could get over it and do small little contributions to small little projects because I could learn something.

As a meta-cognitive skill, I’d like to become better at walking into new areas as a beginner, and less reluctant to do so. Less pearl clutching, more exploration.

In the last 3 years, my saw-teeth have been:

Learning to optimize with SIMD, CUDA, and what you might call “esoteric problem reformulations”.
Beginning projects in new languages. Zig, Rust, or (since I rarely use it anymore) C.
Diving into backend development with both feet, especially on AWS and vercel.

These should be evident in projects.

And, going forward, I’d really love to do more:

High performance or low-latency computing
Image processing, especially for tracking
Work/research on parallelization for search, esp in game theory or discrete optimization

Now I just need to figure out how.

Clearly the chief eng of JPL knows how to finish projects, so I suspect he meant on the scale of years, not days. ↩︎
“May you have breadth over years, and depth over days” – maybe this will catch on? ↩︎

Search and replace tricks with ripgrep

learnbyexample — Thu, 09 Feb 2023 02:00:00 GMT

ripgrep (command name rg) is a grep tool, but supports search and replace as well. rg is far from a like-for-like alternate for sed, but it has nifty features like multiline replacement, fixed string matching, PCRE2 support, etc. This post gives an overview of syntax for substitution and highlights some of the cases where rg is a handy replacement for sed.

Global search and replace🔗

$ cat ip.txt
dark blue, light blue
light orange
blue sky

# by default, line number is displayed if output destination is stdout
# by default, only lines that matched the given pattern is displayed
# 'blue' is search pattern and -r 'red' is replacement string
$ rg 'blue' -r 'red' ip.txt
1:dark red, light red
3:red sky

# --passthru option is useful to print all lines, whether or not it matched
# -N will disable line number prefix
# this command is similar to: sed 's/blue/red/g' ip.txt
$ rg --passthru -N 'blue' -r 'red' ip.txt
dark red, light red
light orange
red sky

Matching Nth occurrence🔗

As seen in previous example, rg will search and replace all occurrences. So, you'll have to be creative with regexp to replace only a specific occurrence per input line.

$ s='see bat hot at but at go gate at sat at but at'

# replace first occurrence only
# same as: sed 's/\bat\b/[xyz]/'
$ echo "$s" | rg --passthru -N '\bat\b(.*)' -r '[xyz]$1'
see bat hot [xyz] but at go gate at sat at but at

# same as: sed 's/\bat\b/[xyz]/3'
# the number within {} is N-1 to replace Nth occurrence, for N>1
$ echo "$s" | rg --passthru -N '^((.*?\bat\b){2}.*?)\bat\b' -r '$1[xyz]'
see bat hot at but at go gate [xyz] sat at but at

# replace last but Nth occurrence, for N>=0
$ echo "$s" | rg --passthru -N '^(.*)\bat\b((.*\bat\b){3})' -r '$1[xyz]$2'
see bat hot at but [xyz] go gate at sat at but at

In-place workaround🔗

rg doesn't support in-place option, so you'll have to do it yourself.

# -N isn't needed here as output destination is a file
# same as: sed -i 's/blue/red/g' ip.txt
$ rg --passthru 'blue' -r 'red' ip.txt > tmp.txt && mv tmp.txt ip.txt

$ cat ip.txt
dark red, light red
light orange
red sky

If you have moreutils installed, then you could use sponge as well.

rg --passthru 'blue' -r 'red' ip.txt | sponge ip.txt

Rust regex and PCRE2🔗

By default, rg uses Rust regular expressions, which is much more featured compared to GNU sed. The main feature not supported is backreference within regexp definition (for performance reasons). See Rust regex documentation for regular expression syntax and features. rg supports Unicode by default.

# non-greedy quantifier is supported
$ s='food land bark sand band cue combat'
$ echo "$s" | rg --passthru 'foo.*?ba' -r '[xyz]'
[xyz]rk sand band cue combat

# unicode support
$ echo 'fox:αλεπού,eagle:αετός' | rg --passthru '\p{L}+' -r '($0)'
(fox):(αλεπού),(eagle):(αετός)

# set operator example, remove all punctuation characters except . ! and ?
$ para='"hi", there! how *are* you? all fine here.'
$ echo "$para" | rg --passthru '[[:punct:]--[.!?]]+' -r ''
hi there! how are you? all fine here.

The -P switch will enable PCRE2 flavor, which has even more tricks. You can also use --engine=auto to allow rg to automatically use PCRE2 when needed (for example: useful as an alias for rg command so that it gives performance of Rust engine by default and use PCRE2 only when needed).

# backreference within regexp definition
$ s='cocoa appleseed tool speechless'
$ echo "$s" | rg --passthru -wP '([a-z]*([a-z])\2[a-z]*){2}' -r '{$0}'
cocoa {appleseed} tool {speechless}

# replace all whole words except 'imp' and 'ant'
$ s='tiger imp goat eagle ant important'
$ echo "$s" | rg --passthru -P '\b(imp|ant)\b(*SKIP)(*F)|\w+' -r '[$0]'
[tiger] imp [goat] [eagle] ant [important]

# recursively match parentheses
$ eqn='(3+a)x * y((r-2)*(t+2)/6) + z(a(b(c(d(e)))))'
$ echo "$eqn" | rg --passthru -P '\((?:[^()]++|(?0))++\)' -r ''
x * y + z

$ # all lowercase letters and optional hyphen combo from start of string
$ s='apple-fig-mango guava grape'
$ echo "$s" | rg --passthru -P '\G([a-z]+)(-)?' -r '($1)$2'
(apple)-(fig)-(mango) guava grape

Extract and modify🔗

The -r option can be used when -o option is active too. The example shown below is not easy to do with sed.

$ s='0501 035 154 12 26 98234'

# numbers >= 100 and ignore leading zeros
$ echo "$s" | rg -woP '0*+(\d{3,})' -r '"$1"' | paste -sd,
"501","154","98234"

Fixed string matching🔗

Like grep, the -F option will allow fixed strings to be matched, a handy option that I feel every search and replace tool should provide.

$ printf '2.3/[4]*6\nfoo\n5.3-[4]*9\n' | rg --passthru -F '[4]*' -r '2'
2.3/26
foo
5.3-29

-F doesn't extend to replacement section though, so you need $$ instead of $ character to represent it literally.

$ echo 'a.*{2}-b' | rg --passthru -F '.*{2}' -r '+$x\tc'
a+\tc-b
$ echo 'a.*{2}-b' | rg --passthru -F '.*{2}' -r '+$$x\tc'
a+$x\tc-b

Multiline matching🔗

Another handy option is -U which enables multiline matching.

$ s='hi there\nhave a nice day\nbye'

# (?s) flag will allow . to match newline characters as well
$ printf '%b' "$s" | rg --passthru -U '(?s)the.*ice' -r ''
hi  day
bye

See my blog post for a detailed discussion on multiline fixed string search and replace operations from the command line.

Handling dos-style input🔗

rg provides support for dos-style files with --crlf option.

# same as: sed -E 's/\w+(\r?)$/xyz\1/'
# note that output will retain CR+LF as line ending
# similar to the sed solution, this will work for unix-style input too
$ printf 'hi there\r\ngood day\r\n' | rg --passthru --crlf '\w+$' -r 'xyz'
hi xyz
good xyz

Speed comparison with GNU sed🔗

Another advantage of rg is that it is likely to be faster than sed. See ripgrep benchmark with other grep implementations by the author for a methodological detailed analysis and insights.

# for small files, initial processing time of rg is a large component
$ time echo 'aba' | sed 's/a/b/g' > f1
real	0m0.002s
$ time echo 'aba' | rg --passthru 'a' -r 'b' > f2
real	0m0.007s

# for larger files, rg is likely to be faster
# 6.2M sample ASCII file
$ wget 'https://norvig.com/big.txt'
$ time LC_ALL=C sed 's/\bcat\b/dog/g' big.txt > f1
real	0m0.060s
$ time rg --passthru '\bcat\b' -r 'dog' big.txt > f2
real	0m0.048s
$ diff -s f1 f2
Files f1 and f2 are identical

# nearly 8 times faster!!
$ time LC_ALL=C sed -E 's/\b(\w+)(\s+\1)+\b/\1/g' big.txt > f1
real	0m0.725s
$ time rg --no-unicode --passthru -wP '(\w+)(\s+\1)+' -r '$1' big.txt > f2
real	0m0.093s
$ diff -s f1 f2
Files f1 and f2 are identical

Other alternatives for sed🔗

rpl — search and replace tool, has interesting options like interactive mode and recursive mode
sd — simple search and replace, implemented in Rust
perl and ruby — programming languages with excellent command line support

Creating GUI Applications with wxPython - book review

learnbyexample — Thu, 09 Feb 2023 02:00:00 GMT

Photo Credit: Tranmautritam on Pexels

I've always wanted to create nice looking, useful GUI applications over the years. And I've given up most of the time as the programming seemed too difficult for me and GUI requires at least some level of design skills. I only managed to grit through one Android app for over a year as it was a dream game from school days and I had loads of free time having quit my job. At the end of it though, I had a spaghetti mess of several 1000+ lines programs and a strong aversion to Java and object oriented programming. Part of the reason is that I didn't try to learn in a formal way, just started from a tutorial closest to the game I wanted to do.

Several years later, here I am, trying my hand with GUI again. I have several small to medium scale apps in mind to implement and hopefully I'll avoid previous mistakes, especially feature creep. When I saw this tweet from Mike Driscoll, I took up the offer. I got a free book in exchange for reviewing Creating GUI Applications with wxPython. The book is currently on sale till May 15. Having to review has served as an extra incentive to read the book regularly, and so far I'm quite satisfied to have done so.

I hadn't heard of wxPython before this book. When it comes to GUI in Python, I knew about tkinter which comes by default with standard libary, Kivy, Pygame and PyQt5. This book starts with an introduction to wxPython and then dives into project-based approach. I've finished half the chapters so far, covering four project concepts:

Image viewer
Database viewer and editor
Calculator
Archiver

Rest of the chapters cover these topics:

MP3 tag editor
Image application using NASA's API
PDF merger/splitter
File search
FTP application
XML editor
Distributing your application

There are also a couple of appendix chapters.

As mentioned in book's introduction, you definitely need to be comfortable with Python classes before you start this book. The code used in the book is also available from GitHub repo, but I highly recommend to type them manually.

The project nature also means that after chapter 3, you could probably skip chapters you are not interested in. For example, I didn't pay too much attention to database chapters as I don't have much experience with databases. Each project is described and shown step by step. The projects could be run at different stages as well - playing around with the GUI at those points helps in mapping code-to-output, as well as to experiment different settings.

All in all, I would highly recommend this book for those wanting to start coding GUI applications in Python. And please do contact the author to let him know your feedback or if you have any clarifications. Happy learning :)

Python for maths

learnbyexample — Thu, 09 Feb 2023 02:00:00 GMT

The above image was generated using matplotlib courtesy code provided by Doing Math with Python book.

Last month, I had an opportunity to conduct beginner Python workshop for maths department students in an arts and science college. It was a great experience and I had my first taste of how Python could be applied for mathematical problems. Presented here are bunch of useful links that I gathered as resources for the students.

Documentation links🔗

Books and courses🔗

Python for beginners🔗

Automate the Boring Stuff with Python — teaches you programming concepts and then shows how to automate everyday problems
How to Think Like a Computer Scientist: Interactive Edition — inspired by Think Python
The Python Coding Book — friendly, relaxed programming book for beginners
Comprehensive Python cheatsheet
Pythontutor: Visualize code execution — also has example codes and ability to share sessions
What does debugging a program look like?
Problem solving skills

See my comprehensive Python learning resources for more.

numpy, scipy, matplotlib🔗

More resources🔗

A short and satisfying bug hunt

learnbyexample — Thu, 09 Feb 2023 02:00:00 GMT

The surprise🔗

So, a pleasant surprise awaited me last Sunday. As is my usual habit, I opened my github account after breakfast to see if I've got any sudden spurt in traffic. And as usual, things were normal. Except for the blue notification, which was rare. I hoped it wasn't a silly pull request and thankfully it was a new issue that was opened.

I gave the issue a cursory glance and wrongly guessed it was probably some line ending issue (user was on Windows OS). As someone who has seen plenty of bugs in previous job, I wasn't ruling out anything though. I first cloned the repo so as to try to recreate the working environment without possible interference from my local working copy. As the user had provided detailed information while opening the issue, I was able to quickly replicate it. Sure enough, I was seeing the same problem. I only wondered why it wasn't brought to my attention before. Either past users chose not to or things weren't interesting enough to reach that far in the exercises.

Creating minimal failing case🔗

As I had written the solution checker script about 2 years back, the script looked alien. Right from cloning the repo, I had to fight the urge to improve things. By the time I spotted the issue, all such fantasies were thrown out. Replaced by a todo note to someday write automated testing script to check that my script is indeed working properly for all the exercises.

To put it simply, the role of solve script is to check if the previous command executed by the user solves the current exercise question. To do so, the script gets the previous command from history and compares the output of that command and a reference solution present in the exercise directory. Sounds simple right? Yeah, I thought so too. I do remember testing few cases before I first published it and no one had submitted an issue so far. So, why was it failing now?

As mentioned before, I thought it could be some weird line ending issue. But that was effectively ruled out as it was failing for me as well on Linux. Still, I did check for funny characters with cat -A. Nope, no issues there.

$ grep -o '^[^=]*' sample.txt
a[2]
foo_bar
appx_pi
greeting
food[4]
b[0][1]
$ source ../solve -s
---------------------------------------------
Mismatch for question 1:
Expected output is:
a[2]
foo_bar
appx_pi
greeting
food[4]
b[0][1]
---------------------------------------------

Expected output was same as output for submitted solution. So, why is the script failing? I remember passing the script through shellcheck but still checked it again. No progress. So, then I started by trying to debug the most likely culprit from terminal before trying to debug the whole script. Luckily, that turned out well.

$ cat sample.txt 
a[2]='sample string'
foo_bar=4232
appx_pi=3.14
greeting="Hi  there		have a nice   day"
food[4]="dosa"
b[0][1]=42

$ # say what??
$ [[ $(eval "command grep -o '^[^=]*' sample.txt") == \
>    $(eval "command grep -o '^[^=]*' sample.txt") ]] || echo 'Not fine'
Not fine

$ # after some attempts, I tried a command that won't have
$ # any [] characters in the output
$ # Eureka!
$ [[ $(eval "command grep 'bar' sample.txt") == \
>    $(eval "command grep 'bar' sample.txt") ]] || echo 'Not fine'
$ [[ foo == foo ]] && echo 'fine'
fine
$ [[ 'a[5]' == a[5] ]] || echo 'Not fine'
Not fine
$ [[ 'a[5]' == 'a[5]' ]] && echo 'fine'
fine

Having a minimal failing case from terminal was a relief. I tried set -x but that didn't light a bulb either. Finally, somehow I thought perhaps characters in the output was causing the issue and when [] characters were not present, the comparison worked as expected.

I did think quoting could be the issue, but dismissed it at first as both sides of comparison had the same command. Then my recent experience from reviewing Command Line Fundamentals book came in handy. I remembered that if quotes aren't used on RHS of comparison operator, it is treated as glob matching instead of string matching. Phew.

TL;DR🔗

Always quote strings in bash unless you have a very good reason for not using them.

After adding double quotes around the command substitution commands, the script worked as expected. I thanked the user for opening the issue. And then informed the author for cli fundamentals book as well.

Python tip 23: map, filter and reduce

learnbyexample — Tue, 07 Feb 2023 02:00:00 GMT

Many operations on container objects can be defined in terms of these three concepts. For example, if you want to sum the square of all even numbers:

separating out even numbers is Filter (i.e. only elements that satisfy a condition are retained)
square of such numbers is Map (i.e. each element is transformed by a mapping function)
final sum is Reduce (i.e. you get one value out of multiple values)

One or more of these operations may be absent depending on the problem statement. Each of these steps will be first illustrated using straightforward code and then the equivalent list comprehensions (and generator expressions) are also shown.

The first of these steps could look like:

>>> def get_evens(iterable):
...     op = []
...     for n in iterable:
...         if n % 2 == 0:
...             op.append(n)
...     return op
... 
>>> nums = [100, 53, 32, 0, 11, 5, 2]
>>> get_evens(nums)
[100, 32, 0, 2]

>>> [n for n in nums if n % 2 == 0]
[100, 32, 0, 2]

The second step could be:

>>> def sqr_evens(iterable):
...     op = []
...     for n in iterable:
...         if n % 2 == 0:
...             op.append(n * n)
...     return op
... 
>>> sqr_evens(nums)
[10000, 1024, 0, 4]

>>> [n * n for n in nums if n % 2 == 0]
[10000, 1024, 0, 4]

And finally, the third step could be:

>>> def sum_sqr_evens(iterable):
...     total = 0
...     for n in iterable:
...         if n % 2 == 0:
...             total += n * n
...     return total
... 
>>> sum_sqr_evens(nums)
11028

>>> sum(n * n for n in nums if n % 2 == 0)
11028

You can also use map(), filter() and functools.reduce() for such problems.

Video demo:

See also my 100 Page Python Intro ebook.

A new approach to domain ranking

Weblog on marginalia.nu — Mon, 06 Feb 2023 02:00:00 GMT

This is a very brief post announcing a fascinating discovery. It appears to be possible to use the cosine similarity approach powering explore2.marginalia.nu as a substitute for the link graph in an eigenvector-based ranking algorithm (i.e. PageRank). The original PageRank algorithm can be conceptualized as a simulation of where a random visitor would end up if they randomly clicked links on websites. With this model in mind, the modification replaces the link-clicking with using explore2 for navigation.

Lifelong purchases

Bytepawn - Marton Trencseni — Sun, 05 Feb 2023 01:00:00 GMT

What are some of my buy-it-for-life purchases?

How to Find Podcasts That Have Been Deleted

swyx's site RSS Feed — Sat, 04 Feb 2023 21:13:40 GMT

TLDR, https://megaphone.spotify.com/ rehosts all mp3s

What I'm up to - February 2023

Philip I. Thomas — Sat, 04 Feb 2023 02:26:42 GMT

✨ What I've been up to

Hello from Cozumel, Mexico! It's 80° F here - a welcome break from the Chicago Winter.

Since last Summer, I've been taking a "solo week" trip every quarter as a creative recharge. This trip to Cozumel is one of these solo weeks, and I've been spending a lot of time in cafes exploring new Contraption Company projects. It's a ritual I plan to continue - I've already scheduled solo weeks over the next year to Copenhagen, Mexico City, And Tokyo.

Postcard continues to do well. I built new features and did a short interview with GoSolo.

Robin Sloan mentioned Postcard in his newsletter. I enjoyed Robin's book Mr. Penumbra's 24-Hour Bookstore while living in the San Francisco neighborhood where the story is set.

🤔 Things to share

Articles: Nash Equilibria and Schelling Points. Underutilized fixed assets and how marketplaces inevitably shift from retail to professional sellers. The Digital Workplace Is Designed to Bring You Down. The uncovering of a real Tom Cruise-style dogfight between US and Soviet fighters during the Korean War. All of Bits about Money (like Money Stuff for financial systems).
Books: Intrigued recently by the Nordics, I read every book by Kay Xander Mellish - even a guide to American culture for Danes (!).
Trends: Starlink is popping up on Airbnb listings.
Listening: Creativity by John Cleese. A 2019 interview between Lex Friedman and Donald Knuth (personal takeaway: seek simplicity in tools I build, but embrace complexity in tools I use).
Fun fact: Many Russian submarines have a sauna.
Apps: Obsidian replaced Notion as my personal organization tool. Quitter closes my calendar, messages, and email after 5 minutes of inactivity.
Cool infrastructure: Chicago Deep Tunnel project.
Coffee I'm drinking: Metric Costa Rica La Bandera Geisha.

📫 What I'm up to this month

Writing a post about the AI hype cycle
Launching some new Postcard features
Researching online communities
Prototyping a new project

📍 Where I'll be (Let me know if we overlap!)

Feb 18-24: 🗽 NYC

Current office

Python Regex Surprises

learnbyexample — Sat, 04 Feb 2023 02:00:00 GMT

In this post, you'll find a few regular expression examples that might surprise you. Some are Python specific and some are applicable to other regex flavors as well. To make it more interesting, these are framed as questions for you to ponder upon. Answers are hidden by default.

Poster created using Canva

If you are not familiar with regular expressions, check out my Understanding Python re(gex)? ebook.

$ vs \Z🔗

Are the $ and \Z anchors equivalent?

Click to view answer

$ can match both the end of string and just before \n if it is the last character. \Z will only match the end of string.

>>> greeting = 'hi there\nhave a nice day\n'

>>> bool(re.search(r'day$', greeting))
True
>>> bool(re.search(r'day\n$', greeting))
True

>>> bool(re.search(r'day\Z', greeting))
False
>>> bool(re.search(r'day\n\Z', greeting))
True

Slicing vs start and end arguments🔗

Did you know that you can specify start and end index arguments for compiled methods?

Pattern.search(string[, pos[, endpos]])

Now, here's a conundrum:

>>> word_pat = re.compile(r'\Aat')

>>> bool(word_pat.search('cater'[1:]))
True

# what will be the output?
>>> bool(word_pat.search('cater', 1))

Click to view answer Specifying a greater than 0 start index when using \A is always going to return False. This is because, as far as the search() method is concerned, only the search space has been narrowed — the anchor positions haven't changed. When slicing is used, you are creating an entirely new string object with new anchor positions.

Do ^ and $ match after the last newline?🔗

When you use the re.MULTILINE flag, the ^ and $ anchors will match at the start and end of every input line. Question is, will they also match after a newline character at the end of the input?

Click to view answer

Yes, they will both match after the last newline character.

>>> print(re.sub(r'(?m)^', 'apple ', '1\n2\n'))
apple 1
apple 2
apple 

>>> print(re.sub(r'(?m)$', ' banana', '1\n2\n'))
1 banana
2 banana
 banana

Word boundary vs lookarounds🔗

\b..\b is same as (?<!\w)..(?!\w) — True or False?

Click to view answer

False! \b matches both the start and end of word locations. In the below example, \b..\b doesn't necessarily mean that the first \b will match only the start of word location and the second \b will match only the end of word location. They can be any combination! For example, I followed by space in the input string here is using the start of word location for both the conditions. Similarly, space followed by 2 is using the end of word location for both the conditions.

In contrast, the negative lookarounds version ensures that there are no word characters around any two characters. Also, such assertions will always be satisfied at the start of string and the end of string respectively. But \b depends on the presence of word characters. For example, ! at the end of the input string here matches the lookaround assertion but not word boundary.

>>> ip = 'I have 12, he has 2!'

>>> re.sub(r'\b..\b', '{\g<0>}', ip)
'{I }have {12}{, }{he} has{ 2}!'

>>> re.sub(r'(?<!\w)..(?!\w)', '{\g<0>}', ip)
'I have {12}, {he} has {2!}'

Undefined escape sequences🔗

If you use undefined escape sequences like \e, will you get an error or will it match the unescaped character (e for this example`)?

Click to view answer

Python raises an exception for escape sequences that are not defined. Apart from sequences defined for character sets (for example \d, \w, \s, etc), these are allowed: \a \b \f \n \N \r \t \u \U \v \x \\ where \b means backspace only in character classes. Also, \u and \U are valid only in Unicode patterns.

>>> bool(re.search(r'\t', 'cat\tdog'))
True

>>> bool(re.search(r'\c', 'cat\tdog'))
re.error: bad escape \c at position 0

Using octal and hexadecimal escapes in the replacement section🔗

In string literals, you can use octal, hexadecimal and unicode escapes to represent a character. For example, '\174' is same as using '|'. Do you know which of these escapes you can use inside raw strings in the replacement section of the sub() function?

Click to view answer

Only octal escapes are allowed inside raw strings in the replacement section. If you are otherwise not using the \ character, then using normal strings in the replacement section is preferred as it will also allow hexadecimal and unicode escapes.

>>> re.sub(r',', r'\x7c', '1,2')
re.error: bad escape \x at position 0

>>> re.sub(r',', r'\174', '1,2')
'1|2'
>>> re.sub(r',', '\x7c', '1,2')
'1|2'

I feel like it would have been rather better if octal escapes were also not allowed. That would have allowed us to use \0 instead of \g<0> for backreferencing the entire matched portion in the replacement section.

Using escape sequences for metacharacters🔗

In the search section, if you use an escape (for example, \x7c to represent the | character), will it behave as the alternation metacharacter or match it literally?

>>> re.sub(r'2|3', '5', '12|30')
'15|50'

# what will be the output?
>>> re.sub(r'2\x7c3', '5', '12|30')

Click to view answer

The output will be '150' since escapes will be treated literally.

Empty matches🔗

You are likely to have come across this before:

# what will be the output?
>>> re.sub(r'[^,]*', r'{\g<0>}', ',cat,tiger')

Click to view answer

Can quantifiers be grouped out?🔗

Similar to a(b+c)d = abd+acd in maths, you get a(b|c)d = abd|acd in regular expressions. (a*|b*) is same as (a|b)* — True or False?

Click to view answer

Railroad diagram created using debuggex.com

False. Because (a*|b*) will match only sequences like a, aaa, bb, bbbbbbbb. But (a|b)* can match mixed sequences like ababbba too.

Portion captured by a quantified group🔗

This should be another familiar regex gotcha:

# what will be the output?
>>> re.sub(r'\A([^,]+,){3}([^,]+)', r'\1(\2)', '1,2,3,4,5,6,7')

Click to view answer

Referring to the text matched by a capture group with a quantifier will give only the last match, not the entire match. You'll need an outer capture group to get the entire matched portion.

>>> re.sub(r'\A([^,]+,){3}([^,]+)', r'\1(\2)', '1,2,3,4,5,6,7')
'3,(4),5,6,7'

>>> re.sub(r'\A((?:[^,]+,){3})([^,]+)', r'\1(\2)', '1,2,3,4,5,6,7')
'1,2,3,(4),5,6,7'

Character combinations🔗

\b[a-z](on|no)[a-z]\b is same as \b[a-z][on]{2}[a-z]\b — True or False?

Click to view answer

False. [on]{2} will also match oo and nn.

>>> words = 'known mood know pony inns'

>>> re.findall(r'\b[a-z](?:on|no)[a-z]\b', words)
['know', 'pony']
>>> re.findall(r'\b[a-z][on]{2}[a-z]\b', words)
['mood', 'know', 'pony', 'inns']

Greedy vs Possessive🔗

Suppose you want to match integer numbers greater than or equal to 100 where these numbers can optionally have leading zeros. Will the below code work? If not, what would you use instead?

>>> numbers = '42 314 001 12 00984'

# will this work?
>>> re.findall(r'0*\d{3,}', numbers)

Click to view answer

No. You can either modify the pattern such that 0* won't interfere or use possessive quantifiers to prevent backtracking.

>>> numbers = '42 314 001 12 00984'

# this solution fails because 0* and \d{3,} can both match leading zeros
# and greedy quantifiers will give up characters to help overall RE succeed
>>> re.findall(r'0*\d{3,}', numbers)
['314', '001', '00984']

# 0*+ is possessive, will never give back leading zeros
>>> re.findall(r'0*+\d{3,}', numbers)
['314', '00984']

# workaround if possessive isn't supported
>>> re.findall(r'0*[1-9]\d{2,}', numbers)
['314', '00984']

See my blog post on possessive quantifiers and atomic grouping for more examples, details about catastrophic backtracking and so on.

Optional flags argument🔗

Will the sub() function in the code sample below match case insensitively or not?

>>> re.findall(r'key', 'KEY portkey oKey Keyed', re.I)
['KEY', 'key', 'Key', 'Key']

# what will be the output?
>>> re.sub(r'key', r'(\g<0>)', 'KEY portkey oKey Keyed', re.I)

Click to view answer

You should always pass flags as a keyword argument. Using it as positional argument leads to a common mistake between re.findall() and re.sub() functions due to difference in their placement.

re.findall(pattern, string, flags=0)

re.sub(pattern, repl, string, count=0, flags=0)

>>> +re.I
2

# works because flags is the only optional argument for findall
>>> re.findall(r'key', 'KEY portkey oKey Keyed', re.I)
['KEY', 'key', 'Key', 'Key']

# wrong usage, but no error because re.I has a value of 2
# so, this is same as specifying count=2
>>> re.sub(r'key', r'(\g<0>)', 'KEY portkey oKey Keyed', re.I)
'KEY port(key) oKey Keyed'

# correct use of keyword argument
>>> re.sub(r'key', r'(\g<0>)', 'KEY portkey oKey Keyed', flags=re.I)
'(KEY) port(key) o(Key) (Key)ed'
# alternatively, you can use inline flags to avoid this problem altogether
>>> re.sub(r'(?i)key', r'(\g<0>)', 'KEY portkey oKey Keyed')
'(KEY) port(key) o(Key) (Key)ed'

re vs regex module flags🔗

The third-party regex module is handy for advanced features like subexpression calls, skipping matches and so on. Can you use re module flag constants with the regex module?

Click to view answer

When using the flags argument with the regex module, the constants should also be used from the regex module.

>>> +re.A
256

>>> +regex.A
128

Again, you can use inline flags to avoid such issues.

Understanding Python re(gex)? book🔗

Visit my GitHub repo Understanding Python re(gex)? for details about the book I wrote on Python regular expressions. The ebook uses plenty of examples to explain the concepts from the very beginning and step by step introduces more advanced concepts. The book also covers the third-party module regex.

Rob’s Awesome Python Template

tedious ramblings — Fri, 03 Feb 2023 18:59:44 GMT

Rob's Awesome Python Template is a highly configurable CookieCutter template that's packed full of best practices for Python Projects.

How to Reverse Interview Investors

swyx's site RSS Feed — Thu, 02 Feb 2023 07:55:43 GMT

I honestly never expected this to be a topic that was common enough to write up, however, it suddenly hit me today that it is the ultra niche topics that deserve writing up since it is the stuff that is outside the usual SERP riffraff.

Vim tip 21: working with tabs

learnbyexample — Wed, 01 Feb 2023 02:00:00 GMT

Multiple files can be opened in Vim within the same tab page and/or in different tabs. From :h windows-intro:

A buffer is the in-memory text of a file.

A window is a viewport on a buffer.

A tab page is a collection of windows.

:tabe filename open the given file in a new tab (:tabe is short for :tabedit)
- if filename isn't specified, you'll get an unnamed empty window
- by default, the new tab is opened to the right of the current tab
- :0tabe open as the first tab
- :$tabe open as the last tab
- see :h :tabe for more details and features

Switching between tabs:

:tabn switch to the next tab (:tabn is short for :tabnext)
- if tabs to the right are exhausted, switch to the first tab
- gt and Ctrl+Page Down can also be used
- 2gt switch to the second tab (the number specified is absolute, not relative)
:tabp switch to the previous tab (:tabp is short for :tabprevious)
- if tabs to the left are exhausted, switch to the last tab
- gT and Ctrl+Page Up can also be used
:tabr switch to the first tab (:tabr is short for :tabrewind)
- :tabfirst can also be used
:tabl switch to the last tab (:tabl is short for :tablast)

Moving tabs:

:tabm N move the current tab to after N tabs from the start (:tabm is short for :tabmove)
- :tabm 0 move the current tab to the beginning
- :tabm move the current tab to the end
:tabm +N move the current tab N positions to the right
:tabm -N move the current tab N positions to the left

Buffer list includes all the files opened in all the tabs. You can also use the mouse to switch/move tabs in GVim.

Video demo:

The Swyx.io 2023 rewrite

swyx's site RSS Feed — Tue, 31 Jan 2023 10:43:17 GMT

Discussing the 2023 migration of swyx.io to the SvelteKit 1.0 and the swyxkit template

Lessons learned streaming building a Scheme-like interpreter in Go

Notes on software development — Mon, 30 Jan 2023 02:00:00 GMT

I wanted to practice making coding videos so I did a four-part series on writing a basic Scheme-like language (minus macros and arrays and tons of stuff).

I picked this simple topic because I wanted a low-stakes way to learn what I did not know about making videos.

Here was the end result (nothing crazy):

$ go build
$ cat examples/fib.scm
(func fib (a)
      (if (< a 2)
          a
          (+ (fib (- a 1)) (fib (- a 2)))))

(fib 11)
$ ./livescheme examples/fib.scm
89

The code for the project is here.

Video archives

Here are the four episodes! Each about an hour long. One per week for four weeks.

Live live

The videos were streamed to Twitch live.

I didn't prep for them because I wanted to show warts and all. The thought process.

But some things turned out to be tricky to explain without preparation (function calling conventions, mostly).

Overall hopefully the series was somewhat useful.

Full screen windows

The first episode I did I didn't make sure that the terminal window was captured full screen. So some of my code went off the bottom of the video. That was dumb.

I even have a tmux mode-line at the bottom of the terminal app that I could have looked for to notice it didn't exist in the OBS view.

So I made sure to have the full window in view after the first episode.

Twitch moderation

Twitch Shield Mode is great. But the default setting prevents folks from commenting live until they've followed you for 2 weeks or something.

For someone starting a channel that doesn't make much sense. So in my first video I disabled it so folks could chat. And then some crypto scammer came in. Go figure.

After the first video I turned Shield Mode back on but set the minimum follow time to 10 minutes I think.

OBS Studio

I used OBS Studio to record. I was frustrated with it for a while because the video would lag so much when I tested out streaming. After playing around with Twitch Studio and giving up on it for being too simple, I messed with OBS video settings enough to get my video to not lag. Unfortunately I can't remember what settings I used.

Noise Gate / pop filter

The Noise Gate Filter is awesome. My mechanical keyboard sounded obnoxious before I turned it on. I was considering getting a pop filter but then discovered that the Noise Gate Filter is built in, you just have to turn it on.

Scenes

It also took me a while to understand OBS Scenes but then I realized I can use them to have an intro graphic (without the mic on!), a main coding scene (focused on my terminal and with my webcam overlayed), and a "back soon" graphic if I needed it.

To get the mic off you have to disable the mic globally (it's on globally by default) and then add it as an input only to the scenes you want.

Storage and export to YouTube

Twitch doesn't store streams by default. You have to turn on Video on Demand.

Even when it's turned on the videos only seem to be stored for 1 week. Maybe that's configurable but I didn't see it.

In any case it's not a problem because you can set up a YouTube connection. Then after a stream is complete you find the stream video and click Export. It takes about a minute to upload the hour long videos I did. Though YouTube post-processing took a while longer after that.

Next?

I'm forced to take a break from recording these videos for the next two weeks since I'll be in Cape Town.

I haven't decided yet if I'll continue this series (not something I'm extremely excited about since everyone builds a Scheme-like language).

I'd like to have a project that I can keep contributing to over time but I don't see very much value in doing that based on a Scheme or any lisp-like.

Maybe I'll do a basic JavaScript implementation next. Or another basic SQL database. Dunno.

Now that I'm done that series on the Scheme-like interpreter in Go (at least for a few weeks), I wrote down a few thoughts about the experience and the Twitch and OBS Studio setup.

Up next after Cape Town? Not totally sure yet!https://t.co/bgdO1ZI5Ow pic.twitter.com/E1kwMRcCWY
— Phil Eaton (@eatonphil) January 31, 2023

Compact WSL partition and reclaim storage space

Jan van den Berg — Sun, 29 Jan 2023 00:37:53 GMT

Start PowerShell Find where your WSL vhdx file is located. Usually under: C:\Users\yourname\AppData\Local\Package\Linuxdistroflavour\LocalState\ext4.vhdx Start diskpart (from PowerShell or CMD): diskpart.exe Run: select vdisk file="C:\Users\Jan van den Berg\AppData\Local\Packages\TheDebianProject.DebianGNULinux_76v4gfsz19hv4\LocalState\ext4.vhdx" and next: compact vdisk

January Gears emulator update

Linux Engineer's random thoughts — Sat, 28 Jan 2023 01:00:00 GMT

Previously, I had a bug on my emulator with the way the map is rendered.

With the bug

How it should look like

After I wrote the previous article, I posted it on the SMS power discord, asking for ideas on this corruption bug. Many great people chimed in.

A …

Are you ok?

Weblog on marginalia.nu — Fri, 27 Jan 2023 02:00:00 GMT

I don’t know if I’m just imagining it, but has the Internet gone progressively more crazy the last decade or so? It’s like everyone is so damn angry all the time. If they aren’t angry they’re bitter and resentful. And when they aren’t angry or bitter, they’re so depressed they’re barely able to crawl out of bed. And if they aren’t angry, bitter, or depressed, they have crippling anxiety. Every other week there’s some public blow-out where some person or another just loses their shit.

Crypto V: On NFTs

Bytepawn - Marton Trencseni — Fri, 27 Jan 2023 01:00:00 GMT

I argue that virtual goods such as in-game skins make more sense in a centralized walled-garden approach, and NFTs tracked and traded on blockchains are legally problematic.

Introduction to AI and Large Language Models (LLMs)

smcleod.net — Thu, 26 Jan 2023 03:00:00 GMT

A high level intro to LLMs that I'm writing for a few friends that are new to the concept. It is far from complete, definitely contains some errors and is a work in progress.

Jet Set Radio review

ブラック — Thu, 26 Jan 2023 02:00:00 GMT

Short but sweet (soul brother)

Two decades later and Jet Set Radio is still a great game. Luckily the visual style keeps the game looking quite fresh despite 23 years worth of patina. The movement and camera controls do show some age, but not to a degree that you can't get used to them. Once you do, the movement does actually become rather nice and ever-so stylish. And the soundtrack is still one of the all-time greats.

My second playthrough of the main story only took around 4.5 hours after having beaten it for the first time in 2014, so it's definitely not the longest. I imagine first-time players will progress through the game a bit slower than I did though. However, there is still more content and collectibles to grind out if the main story isn't enough skating action.

I would definitely recommend this piece of gaming history. It is, if nothing else, a stylish and fun little snack.

As an added bonus, played basically perfectly on my Steam Deck and with good battery life.

CLI tip 22: grep options to suppress stdout and stderr

learnbyexample — Wed, 25 Jan 2023 02:00:00 GMT

While writing scripts, sometimes you just need to know if a file contains the pattern and act based on the exit status of the command. Instead of redirecting the output to /dev/null you can use the -q option. This will avoid printing anything on stdout and also provides speed benefit as processing would be stopped as soon as the given condition is satisfied.

$ cat find.txt
The find command is more versatile than recursive options and
and extended globs. Apart from searching based on filename, it
has provisions to match based on the the file characteristics
like size and time.

$ grep -wE '(\w+) \1' find.txt
has provisions to match based on the the file characteristics
$ grep -qwE '(\w+) \1' find.txt
$ echo $?
0

$ grep -q 'xyz' find.txt
$ echo $?
1

$ grep -qwE '(\w+) \1' find.txt && echo 'Repeated words found!'
Repeated words found!

The -s option will suppress the error messages that are intended for the stderr stream.

# when the input file doesn't exist
$ grep 'in' xyz.txt
grep: xyz.txt: No such file or directory
$ grep -s 'in' xyz.txt
$ echo $?
2

# when sufficient permission is not available
$ touch new.txt
$ chmod -r new.txt
$ grep 'rose' new.txt
grep: new.txt: Permission denied
$ grep -s 'rose' new.txt
$ echo $?
2

Errors regarding regular expressions and invalid options will be on the stderr stream even when the -s option is used.

$ grep -sE 'a(' find.txt
grep: Unmatched ( or \(

$ grep -sE 'a(' find.txt 2> /dev/null
$ echo $?
2

Check out my ch command line tool for a practical example of using the -q option.

Video demo:

See my CLI text processing with GNU grep and ripgrep ebook if you are interested in learning about GNU grep and ripgrep commands in more detail.

3D Printer Nozzle Camera

Stefan Schüller — Mon, 23 Jan 2023 17:37:16 GMT

(please ignore the horrible layer lines, this is just a prototype)

ActivityPub -> Announce -> Post

Ben E. C. Boyter — Mon, 23 Jan 2023 02:00:00 GMT

I have been investigating how ActivityPub is meant to work. What follows is how a boost/announce/retweet works in Mastodon and any other system that implements against ActivityPub.

You can find the full collection on github https://github.com/boyter/activitypub and contribute your own details.

ActivityPub -> Announce -> Post

When a user clicks on bonk/boost/retweet the following sequence happens.

The users intent is converted to an announce. This announce is sent to every follower they have as well as the owner of the post that the user is boosting.

An effective product manager

Notes on software development — Mon, 23 Jan 2023 02:00:00 GMT

There are three specific activities I have loved in some product managers I've worked with (and missed in others).

tldr;

Talk with customers and prospects
Develop and share a vision
Evangelize

Talk with customers and prospects

As a product manager, your superpower over engineering is to have spent time with customers and prospects. You should have (or develop) a good understanding of the market and your product's potential.

The only way you can do this is by spending time, over time, with customers and prospects. Understanding their workflows and their issues.

Cynical folks will cringe at the word "vision" but it is a serious and necessary part of a successful organization.

As a product manager, you should establish and share a path for engineering to follow based on your understanding of customers, prospects, the market, and the company.

This is the "roadmap" and "prioritization". But prioritization is useless without a long-term vision.

The roadmap should represent (and broadly demonstrate) a concrete and meaningful goal. A goal that you can and should adjust over time as the company and market changes.

Evangelize

In bigger organizations there might be dedicated evangelism teams. But product managers must drive this work.

Evangelism should fit the vision you've developed.

And in the absense of dedicated evangelism teams, product managers should be creating demos, writing blog posts, and testing the solution with customers and prospects.

Again, it's fine for dedicated teams outside of product management to do bits of that work. But it must be driven and led by the product manager.

It's hard

Observed as I have from outside, being an effective product manager feels like a massively challenging task.

It's so easy to go without talking to customers, to get sucked into day-to-day issues and not create a vision, and to allow evangelism to happen ad-hoc.

Then there's the fact you don't live in a vacuum. You may have a boss in product management. Your engineering peers may have competing priorities. You may have a hard time understanding the founders or CEO. In a large company, you may not even have a CEO.

My ideas, your ideas

These are my ideas based on my experience. You may have your own ideas. If mine help you, great! If they don't, great!

I've been considering recently what makes an effective product manager. So I wrote down a few of my thoughts.

What I've loved the most in some PMs and missed the most in others.

I'd likewise love to hear what you think!https://t.co/5vTWTNhs68 pic.twitter.com/vXjPY9fiVT
— Phil Eaton (@eatonphil) January 23, 2023

Crypto IV: On stock investing vs. crypto investing

Bytepawn - Marton Trencseni — Sat, 21 Jan 2023 01:00:00 GMT

I argue that a cryptocurrency like Bitcoin has a weird property: the more you buy of it, the less valuable it becomes.

Crypto III: On perceivd crypto advantages such as anonymity and irreversibility

Bytepawn - Marton Trencseni — Fri, 20 Jan 2023 01:00:00 GMT

I argue that anonymity, irreversibility and decentralization, while interesting technical features that implemented in a fascinating way using cryptographic primites, are not practical for real-world use.

Paying my bills with 'free' ebooks

learnbyexample — Thu, 19 Jan 2023 02:00:00 GMT

TL;DR: Small victories are more precious when you have nothing. Instead of burning through my savings, I'm now adding to it. The relief is priceless.

It is worth it (for me)🔗

The section title is my response to this article Writing a book: is it worth it? that I saw on Hacker News.

For my unique circumstances, the decision to write ebooks has brought me financial stability, improved my mental health and gives me a sense of satisfaction. This could've come from any of my previous attempts to earn money, but ebooks is what worked out for me.

Photo by Bram Naus on Unsplash

How it all started?🔗

I left my job in 2014 for various reasons. I didn't have any plans for the future, just knew that I couldn't work as an employee any more.

After enjoying my break, I had to try something to start earning again. I wrote an android gaming app, fantasized earning loads of money with an awesome work planner/communicator software that never left my imaginations, tried a small stint with a team making an educational app, etc. I failed due to various reasons — didn't try hard enough, quit early, didn't fit my skills, wasn't good at design/marketing and so on. The educational app for example went on to become a success. Or perhaps, having saved enough to live out a few years without working meant I wasn't under enough pressure to earn.

Among these failures, college workshops was the sole bread giver (and long way from supporting my modest living costs). My bachelor's degree was in electronics and communications and I had worked in a semiconductor company. So I knew enough to teach students pursuing similar courses the basics for Linux command line, Vim, Perl, Bash scripting, etc. As reference materials, I used to provide ppt slides (when I still had a job). Now that I had loads of free time, I started expanding my knowledge. Came to know about sites like Stackoverflow/Stackexchange/Reddit/etc. With newer and better materials to learn from, I created PDFs (using LibreOffice, which was pretty much the only option I knew about).

Another loss maker was getting a domain/host to share these learning materials. Web development was too much for me and the (ugly) site didn't get any love. In hindsight, one of the better turning points was learning about GitHub in 2016. I loved markdown's nice output with syntax highlighting (and realized I was using it poorly in Reddit) and GitHub's social aspect (stars, issues, etc) — plus I can use Vim! I manually converted my materials from LibreOffice to markdown (again, I didn't know that tools like pandoc could've helped me). Just like any other skill, I was learning and getting better with every iteration. That was the year I learned Python (thanks to Al Sweigart's free coupon for "Automate the Boring Stuff with Python" video course) and started conducting workshops for Python instead of Perl.

Being active on Stackoverflow and Reddit, I finally became proficient at CLI one-liners (late by 8 years, since it would have significantly helped in my role as a design and test engineer). I came across articles/books on regular expressions and one-liners. I thought — I can do that too, plus I was really liking them. Thus began my epic Command Line Text Processing repo, another big turning point in my journey as an author.

Encouraging signs🔗

Over the course of ten months, I managed to complete the holy trinity of grep, sed and awk one-liners. I promoted these tutorials on Reddit, Google+, LinkedIn and other social sites I knew at that time. The repo got hundreds of stars and more importantly, I got critical feedback. I was ecstatic, even if I was continuing to burn through my savings.

Then, I got to know about Hacker News (I think it was someone bragging about reaching front page). It took me a while to get used to Reddit, and HN was similarly alien to me. I posted a few links as a test and then I was brave enough to submit my awk one-liners post. I was refreshing HN anxiously for about half an hour or so. It got one vote and then other submissions pushed it away from new posts tab. Disappointed, I moved on. After sometime, I was checking traffic on my GitHub repo as usual, a habit I had picked up (all kinds of points, karma, likes, etc were so enticing). I noticed a HUGE spike in traffic and star count, the likes of which I had never seen before/since. The last time I had felt that proud of my work was during my job. This comment made a big impression on me:

These are the best stories on HN and why i subscribed here in the first place. I have often seen awk used so many times on SO but I've always put it up for something later to learn. Finally today I have some basic understanding of awk and this is really great stuff! I did get by with Perl but this is definitely more handy and the example approach to teaching it makes is super easy to understand!

After the euphoria had died down (about a week I guess), I was thinking about all the various kinds of posts I could make. And I was thinking how to use the repo popularity to bring in money. Long story short, I ended up adding donate buttons to my repos. This was before GitHub sponsors was announced. I wanted my materials to be freely available, so I wasn't even thinking about creating paid only options. Despite adding more tutorials, getting featured in rubyweekly and other newsletters, social sites, etc, all I got was a single recurring donation (which ended prematurely when that platform switched payment set up).

Another turning point came when a friend of mine was authoring a book and referred me for the reviewer role. Around that time, I had been converting Think Python to Think Ruby and simultaneously working on a separate Ruby tutorial. During the book review process, I was given a list of topics and asked if I was interested in writing a book (they were impressed by my existing repos). The topics were either beyond my knowledge or out of scope, and they weren't interested in the repos I had already put up.

My friends were always suggesting me to write a book and my reply consistently had been that I wasn't good enough to write one (the imposter syndrome hasn't left me even now). The book review experience, existing repos, my tryst with Think Ruby, dwindling savings, etc changed my mindset enough to try. By then I was already familiar with Leanpub, so I knew self-publishing was an option. I picked a niche topic (Ruby Regexp), learned enough pandoc to produce a PDF and published it even before the book review ended. It helped that I already had material as part of the Ruby tutorial I was working on. I still had to work a lot, since tutorial description was all bullet points.

I got only a few sales, but I had landed another review (video course for the same book) and was getting paid. So, I converted 'Ruby Regexp' to Python re(gex)?. I made it free for a few days and posted on Reddit, HN and other social sites. HN submission didn't get any traction, but fortunately Reddit submission on /r/Python/ was a big hit — thousands of free downloads and a few paid ones enough to cover 2 months of my expenses. I should mention now that I live alone, in outskirts of an Indian city, and my modest lifestyle costs about $150 per month. What works for me won't necessarily suit others.

A dip followed by sustainable momentum🔗

Encouraged by the second release, I changed my focus from updating my GitHub repos to writing books. All those repos were now a fodder for book conversion. I picked up grep first and included ripgrep as well to keep it inline with the trend. Got decent sales from free promotions. HN submission tanked at first, but got good attention when I posted again after a revision. Then I published a new version of 'Python re(gex)?' with significant changes and this HN submission got good views too. But note that these HN hits weren't anywhere close to what my awk one-liners post had received.

Writing sed took a lot out of me. Probably I was getting jaded again, juggling between workshops and ebooks. Then I had a medical issue. I didn't even try promoting the sed book on HN. I managed to learn enough JS to write JavaScript regexp. Wasn't anywhere close to what I got from the Python book.

Photo by David Traña on Unsplash

So, despite reasonable reception during free promotions, my ebooks weren't still good enough to consistently pay my bills. Combined with workshops I was just about making my ends meet. I was losing interest and the medical issue was continuing. Still, without anything else to do, I finally started a book on awk one-liners. Things started getting better for a few months and then the pandemic hit.

Given the recent medical scare, pandemic fears and the trend of giveaways, I decided to open source my book contents. And, I made all my ebooks free to download indefinitely. Made a single bundle of all the 5 books I had published until then to make it easier to download in one shot. The reception was better than expected. Shortly after (last week of March), I published the awk book early by cutting corners like excluding exercises. All books bundle now had 6 entries. Again, the reception was much better than expected. I hadn't made so many paid sales during a month ever before.

Encouraged by the success, I made another important decision. Instead of starting another book, I took up the task of updating all my books. I alloted a month or two for this task, but it took me more than 4 months in the end. It wasn't that I had lot of new features to add. The feedback I had received over the past year and my own improving writing skills meant that I just couldn't help updating the books to the best of my abilities. Somehow, lockdown and fear of the pandemic ended up improving my workflow.

Workshops weren't going to come my way anytime soon, but ebook sales for about 6 months averaged $200+ per month. For the first time since leaving my job, I was saving money!!! During this period all my books were free to download, in addition to the markdown source being available from GitHub repos. I even managed to create EPUB and web versions for my ebooks. The web version generated using mdBook was much better than my attempts with wordpress all those years ago, but to be fair I hadn't known enough about formatting for coding books.

After finishing this marathon revision task, I reverted PDF/EPUB versions to be a paid option again. Since then, I've managed to write three more books. I did Perl and Ruby one-liners (as part of the ongoing conversion of the CLI text processing repo) despite knowing sales won't be good enough to keep up the momentum. Then I wrote a Python intro book for those already familiar with programming basics. Published last month, sales are much lesser than I expected. Given Python is now 30 years old and there's no shortage of Python books for beginners, I shouldn't be surprised though. I'm probably grumpy because it took a month more than expected even though I already had decent material from my workshops. Anyway, my main motivation was to improve my Python knowledge and it did serve that purpose. As a bonus, I just got started with workshops again, conducted online (a first for me). The book is already proving useful as a handy reference for me as well as the students.

Feedback and Criticism🔗

Here's some of the feedback I've received over the past two years.

Grammatical mistakes. Missing a, an and the articles were particularly jarring for the readers. If you couldn't tell from reading this article (heh) that English isn't my native language, I'll consider that I've improved a lot.
Some readers stated that they didn't bother checking out my books because the covers are so bad. I finally got a cover done by an artist for the Python intro book.
For the regexp books, a few readers said my introductions were light on content. So during the marathon book updates I did last year, I managed to add more details. I feel there's still plenty of room for improvement.
My comprehension is kinda average and it works better whenever I manage to create code snippets to prove or disprove my understanding. So, my books are heavily example oriented. I've received feedback that there are too many examples, explanations aren't sufficient, etc. I'm trying to improve on this count, but I doubt I can change my natural writing style.
A few readers wanted more exercises, which I was happy to oblige. It took me a while to accept that I should provide solutions as well.

I did get a few negative feedback (ones I consider weren't constructive in nature). One such feedback affected me a lot, despite the encouraging sales for the second book. Over time, I've adapted but I'm still afraid of seeing one whenever I promote my books.

Self publishing experience🔗

I don't have a personal experience with traditional publishing (other than the two review opportunities). After the initial success of 'Python re(gex)?' book, I was happy to stay being self published. When there was a dip, I did consider it would be nice to have the backing of a traditional publisher and a chance to improve the contents of my books.

What I like about self published:

I can give away free copies whenever I want, change pricing, share the source code, put up free web version of the books, etc.
- I'm aware of a few publishers allowing authors to put up free web copies, but it isn't universal.
I can push updates easily and inform the readers as well.
No deadlines, other than self imposed ones. This is both good and bad. The good thing is that I can take my time. The bad thing is that the reduced pressure leads to longer schedule. I spend a lot of time on social media, reading fiction, watching entertainment, etc. The lockdown marathon did improve my average working hours, but there's still a lot of room for improvement.
I am not restricted by guidelines set by a publisher regarding chapter structure, images, exercises, etc.

What I feel would improve with traditional publishing:

Cover image
PDF/EPUB quality
Content quality, especially grammar
Audience reach

Not sure how my earnings would be affected. On the one hand, I get minimum 80% on book sales. On the other hand, I'd probably reach a wider audience with traditional publishing. I did receive a few offers when my promotional posts were trending. One of the offer (for 'Python re(gex)?' ebook) had a joining bonus and initial advance — both combined was less than what I had already earned. But if they had extended the offer for other books as well, it would've been a much more tempting deal.

Currently, I'm happy with status quo. Always free web versions and free PDF/EPUB promotional sales kinda solves my donation problem before I started selling ebooks — I get paid and readers have a way to get the materials for free. I'm also inspired by FOSS products I use and authors like Al Sweigart and Allen B. Downey who give away quality learning resources for free.

That said, I wish I could improve my marketing skills. Or, somehow someone likes my books so much that their review attracts significant attention and my sales increase as a result. I've also considered trying out affiliates, but haven't even created a list of people to contact yet. I don't have analytics set up on Leanpub, my blog, web versions of my books, etc. Based on analytics that is available by default on GitHub and Gumroad, I do see a few links from schools and universities. I wish they would contact me, so that I can help if needed and improve my book contents based on their experiences.

Leanpub vs Gumroad🔗

I started with Leanpub since I had seen a few posts from self published authors using this platform. By the time I had published the second book I got to know about Gumroad and was attracted by the pricing/payout structure. From then on, I have published on both platforms.

Here's what I like most about both these platforms:

I can change pricing (including free option) and book contents any number of times
I can allow users to pay more than the product price, which is how I get paid during free promotional sales
I can inform readers whenever I update my books
I can create bundles
They handle collection of VAT (and other such fees)
Their payout options work for me in India

Here's some differences and my opinions on some of their features:

Gumroad's pricing structure is better. If you have a following like Julia Evans, pricing would make a huge difference
Gumroad gives analytics for free
Gumroad's email notification is opt-out compared to opt-in for Leanpub. Opt-in is better for readers, but in my experience less than 10% sign up and thus miss out when I want to send them book updates
Leanpub payout delay is 45-75 days, Gumroad is 7-14 days (or instant in some cases)
Leanpub's bundle feature is better since it doesn't require a new cover and files are automatically picked based on the links provided. In Gumroad, it is essentially a new product, but it does allow to manually pick files from existing ones. Also, Leanpub allows bundling with another author (which I have used and given me decent sales)
Leanpub's product page and UI is better. The sliding scale (along with information on author's share) to pick a price is clearer than Gumroad's manual price entry. And I don't like that Gumroad places the minimum price information away from the box where the user enters a price. On Leanpub, all of these are shown together and reduces confusion
Leanpub's product page has always ranked higher in search results in my experience
Leanpub's 45-day Guarantee and Sample chapters as part of the product page makes it easier for readers to take a risk
Leanpub has weekly/monthly sale newsletters in which you could get featured. This has brought me significant earnings in the past few months. If you enable an option, Gumroad would promote your product too (for 10% extra fee) but this has given me very few sales compared to Leanpub's newsletter

Pricing UI on Leanpub

Pricing UI on Gumroad

pandoc and mdbook🔗

I picked pandoc to generate PDF from GitHub style markdown, as it seemed the most popular tool for this purpose. The default output is good enough, but I wanted to customize a lot of things. With help from documentation and various Stackoverflow/Stackexchange threads, I was able to generate an output to my liking. I didn't know about templates though, otherwise I could have researched about them and re-used solutions from others. I wrote a blog post about my learnings, visit Customizing pandoc to generate beautiful pdf and epub from markdown if you are interested.

Some readers wanted EPUB versions too. I thought it made sense for reading from mobile, but my own experience with this format on desktop was quite disappointing. Only later did I learn that I wasn't using a proper EPUB reader for technical books. Which is why I didn't realize that the default output from pandoc for EPUB was also good enough. During the revision marathon, I finally created EPUB versions too. I'd say I am still a beginner, but I did learn enough CSS and LaTeX to customize EPUB and PDF generation with pandoc.

pandoc has its own enhanced version of markdown, which has a lot of nifty features for ebooks. But I chose to stick with GitHub style markdown. And it came in handy when I wanted to re-use book material for blog posts, generating web versions of the book with mdbook and so on. After I had decided to open source my books, I also wanted to make a web version that feels like a book instead of just the single page markdown source from the GitHub repos. I would've probably used Gitbook if they hadn't moved away from the legacy version. I came across mdbook as an alternate for Gitbook and I'm glad I did.

Future plans🔗

I have certainly improved a lot as a writer since I first published my book in late 2018. But after 9 books, I'm finding it a lot more difficult to motivate myself to keep writing. See also HN discussion: Writing a book, still the same pain 15 years later for another example.

I have plans to publish at least one more book in 2021 and revise my existing books (not comprehensive, but a few items have cropped up). I hope the current momentum can extend enough to cover my expenses for this year at least. Beyond that, I think I will write more books, but I'll have to mix it up with other things (such as video courses, interactive courses, freelancing, etc) to keep myself motivated. I just hope that this time I will be able to pick an alternative quickly.

Resources🔗

I've been asked a few times regarding my experiences as an author (especially self publishing) and resources I've used. That was my primary intention in writing this blog post. I thought I'd add a bit of background as well, not the multi-section essay I ended up with. Anyway, here's some links that I've bookmarked related to book writing.

Authors sharing their experiences

Writing skills

Tools and Miscellaneous

Customizing pandoc to generate beautiful pdf and epub from markdown — my own blog post, includes resource links for similar articles and tools other than pandoc
List of awesome design tools
launch-cheatsheet

A parting advice🔗

Don't quit easily!

Hacker News Personal Blogs 2023 | Top 100 Blogs

2023 in Retrospective

A Hobby Coding Biography

What I learned during Advent of Code 2023

Make your own way

Everyone want community

The lifecycle of groups

Community is contagious

Do whatever you want!

A Frivolous Feature

Podcast: IaC in depth with Robert Hafner

How To Read An Article On The Internet

WARC'in the crawler

Dwell and the Tools for Thought Hackathon

Go Libraries/Packages I Like

Using the BlueSky API to change your Handle to your Domain Name

Optimizing Ship Load-outs for Highfleet: Problem Setup

Over-engineering Highfleet load-outs using Integer Programming

Ship Design Problem statement

Solving

Reverse Engineering the Game Stats

We don't need no stinking bastion host!

Cleaning Up Speech Recognition with GPT

What I'm up to - December 2023

What I did in November

Things to share

Plans for December

Where I'll be

Bouldering

Festive offers for books on Python, Linux, Regular Expressions and more

My ebooks🔗

Indie creators🔗

Miscellaneous🔗

(fake) Football Leagues

Lua Carousel

Starfield review

The Knight Capital Disaster

Wild Speculation Regarding the OpenAI Fiasco

Open source, locally hosted AI powered Siri replacement

Exploring a Postgres query plan

What is a query plan?

Development environment

Intercepting query execution

Nodes and Datums

What's in a QueryDesc?

Walking a sequential scan

A string builder

Walking the WHERE clause

Walking OpExpr

Walking Const

Walking Var

Walking the column list

Variations

Next steps

Postscript: On ChatGPT

Github CI

sum-grid.love

rustcred, a git credentials helper

CHAOS;HEAD NOAH review

Fun with Makefiles - Dynamic Menu Generation

MBA Washing

Turning a project into a product

GopherConSyd 2023 - All slides, notes and links

My go-to 'make code go faster' ideas

Anchor Tags

What I'm up to - November 2023

What I did in October

Things to share

Plans for November

Where I'll be

Being a minor AI public figure

Writing a storage engine for Postgres: an in-memory Table Access Method

Why alternative storage engines?

Why not foreign data wrappers?

Digging in

A debug Postgres build

Extension infrastructure

pgtam.c

pgtam.control

pgtam--0.0.1.sql

`Node`s and `Datum`s

What's in a `QueryDesc`?

Walking the `WHERE` clause

Walking `OpExpr`

Walking `Const`

Walking `Var`

`pgtam.c`

`pgtam.control`

`pgtam--0.0.1.sql`

`slot_callbacks`

`scan_begin`

`getnextslot`

Homepage design `v1.0`

Homepage design `v2.0`