Hacker News Personal Blogs 2022 | Top 100 Blogs

Energy, Sustainability and Deployment Frequency

smcleod.net — Sat, 31 Dec 2022 03:03:41 GMT

Deployment / Delivery Frequency

I often end up needing to advocate for more frequent delivery/deployments with clients.

There’s the usual benefits commonly discussed such as improved feedback, reduced risk, well understood processes, maintainable dependencies etc… however what’s often missed entirely is how it relates to the health and sustainability of the team.

2022 Year in Review

Ben Overmyer's Site — Sat, 31 Dec 2022 02:00:00 GMT

2022 was a year of big changes. Some were foreseen and others came out of nowhere.

40th birthday trip to Hawai'i

At the end of February, we went to Hawai'i for my 40th birthday. Our trip was exclusively to the Big Island. We stayed in three different places - a small Bali-themed apartment, a treehouse in the rainforest, and a resort hotel. The treehouse was definitely both the scariest and the most interesting. I say "scariest" because the bottom floor was open and there's a ton of wildlife, including wild pigs. However, it was gorgeous. I would stay there again.

We went on a tour of a coffee plantation. I gained a newfound appreciation for Kona coffee because of that. Sadly, it's hard to get locally here in North Carolina. We also went to some interesting restaurants. My favorite was a sushi restaurant with only three tables that I managed to get a reservation at, despite reservations being hard to come by.

One day when we were staying at the treehouse we went to see a big volcano. It was gorgeous, and the walk was fun. We got a lot of great photos from the trip, and some of the volcano shots were among the best.

Moving to Wilmington, North Carolina

In January, I spent the month in Wilmington, North Carolina looking for a place to rent. A few days into my stay at my boss's condo on the beach, I found the perfect house and got everything signed and sorted out. I moved out of the condo and into the completely empty house. I bought a few things - really, just enough to survive for the month. Sarah visited for a couple weeks to see the place and explore the area.

Before Sarah visited, I'd had another panic attack in January - my second ever - and it made me nervous. Throughout the first half of 2022, I experienced symptoms that would be near to the start of a panic attack, but I'd learned how to stave them off.

The month of February we spent getting everything ready to move, and taking that trip to Hawai'i.

March was the month we moved from Minneapolis to Wilmington. We packed up the house, and then I took Mojave and Kalahari in my Seltos and drove to our new home. The cats didn't much care for the road trip - they spent the majority of it curled up together behind the center console. The drive took three days. I hope to never make such a long drive again, especially with cats in tow. Sarah drove down after the movers came and hauled away all of our stuff. This was the first time I had personally hired movers for an interstate move. It didn't go as well as I'd hoped, but it also didn't go as badly as I'd feared. The weirdest part was having to pay in cash instead of with a card.

Over the course of the next month or so, we got unpacked and moved in. Exploring Wilmington was not as much of a priority as establishing a sense of normalcy. We didn't go to the beach as much as we perhaps should have this first year in Wilmington. However, I did bike to the beach twice after we got our bicycles. That's a 45 minute bike ride, which is the longest I've ever taken, I think. The climate here on the coast is a welcome change from Minneapolis, especially now in the cold months. As I write this it's 48 degrees Fahrenheit, with the expected high being 66. Not bad for December 30.

Iron Arachne

Iron Arachne saw a great many updates over the course of the year. In June, I launched a dungeon generator for it. That was a huge project that I'd never quite figured out before and I was pretty proud of what I'd built. It wasn't as full-featured as others around the web, but it was mine. Writing it required a ton of small subsystems like room layout and distributing keys and monsters and whatnot. I added a ton of new sentient species. I rewrote the word generation system from scratch, and added a utility generator to the site to help me come up with new seed patterns for the generator. The region generator, too, saw a bunch of changes - in particular, adding nearby regions, including vassal states; as well as making name set generation for it configurable.

After awhile I noticed that the language generator in the code base wasn't complete and wasn't used anywhere. So, I started expanding that. It's not complete yet, and I haven't touched it in months. I intend to get back to it at some point, but it's not a priority. In November, I changed the build system to use Node.js 18, and then later in the month migrated it to Vite. At the end of November, I also moved all the navigation to a dedicated page. This change was a long time in coming and finally made the site usable on mobile.

At the end of 2021, I wrote that I'd update the culture, heraldry, and region generators. Well, I didn't touch the culture generator much, other than to flesh out some details and swap out the religion generator for it. I did add new charges to the heraldry generator. The region generator really saw the bulk of the changes. In addition to the dungeon generator, I also added an AD&D 2e character generator. That was the first time I'd ever written a complex generator as the result of a poll. I don't think I'll do that again any time soon; I really prefer working on generators of my own interest. In January I'd added a DCC character generator. That was more fun.

Finally, I moved the site from Netlify to Vercel, to give it a little more separation from my personal sites.

Updates to my personal website

My personal website saw a number of expansions and design changes this year. I redesigned it a bit after learning more about minimal CSS. The whole thing is designed to be quick to load even on much older and slower hardware. I added several books to my "books read" list, several blog posts, a few recipes, and a handful of new knowledge base articles.

I'm increasingly determined to make my site a proper storehouse of information and stories dear to me. It would be nice if someone out there found it useful, or at least interesting. Failing that, it's still useful to me.

Giving up alcohol

Probably one of the biggest personal changes happened on November 10. Tired of all the negative effects of drinking, I finally gave it up completely. I had become accustomed to drinking every other night, sometimes six or more drinks. Going sober was much easier than I'd expected. The positive effects were much more noticeable than I expected, too. The most noticeable was how much more energy I had all the time after a couple weeks. Also, caffeine's effects on me changed. Instead of needing caffeine to properly function in the morning, now it makes me jittery if I have the same amount that I used to.

I'm also dreaming again for the first time in years. Starting somewhere around 2015 or so, I stopped having dreams except only occasionally. Now I dream almost every night. They're the same kind of dreams I used to have - full of action and strangeness, like little movies that don't make a ton of sense. I'd forgotten how much fun they can be.

From individual contributor to manager to tech lead

For a brief period early in the year, I was one of two new managers on the Platform Engineering team. However, the higher-ups felt that was unnecessary and a waste of our technical talents, so we got turned into tech leads instead.

I don't handle any of the managerial duties, but I'm responsible for setting technical direction and leading standups. I didn't get a raise to go with the role change. My boss's boss assured me that was just because it fell outside the annual review cycle. I'm skeptical that I'll get a raise in the new year, though. While I make plenty, inflation means that if I don't get at least a raise matching the inflation rate, I'm actually getting a pay deduction each year. That feels kinda bad.

Also, when the legacy (but stable) side of the business merged with the "startup" side that I'd originally joined, my duties changed. I'm working with a lot of legacy tech, including Perl applications. I don't like that at all. It feels like a bait-and-switch; I was hired to work on the latest tech and explore new horizons, but now most of my time is spent modernizing legacy systems and trying to figure out how to use modern tools with ancient technology.

The pay and benefits are good, and I like the people I work with. However, I'm starting to feel stifled by some of the work, and I'm definitely not as engaged as I was before the merger.

AWS Re:invent 2022

The last time I went to Re:invent was 2016. The conference doubled in size; there was at least 60,000 people there this year. I tried to go to a dozen different sessions, but only got into one. I later made a presentation for my team about that session.

Going sober definitely had a big impact on my Re:invent experience. At Re:invent in 2015 and 2016, I drank a ton and went bar hopping with a bunch of strangers I met at the conference. This time around, I wasn't drinking at all, and a lot of the activities I used to enjoy seemed boring now. I also had a lot more energy for activities in the morning.

Re:invent had special areas and swag for people with AWS certifications, so I used the half-off voucher they gave me beforehand to get certified. This was the first certification I'd ever paid for. It took a couple weeks of studying, but I passed the exam with flying colors and got my certification. It was a very strange experience, studying for an exam. I hadn't done that since university. I didn't care for it at all.

End of my boycott of Blizzard

I wrote at greater length about this earlier this year. Since that post, I've played a ton of Overwatch 2 and World of Warcraft. I didn't play much Diablo 2 and uninstalled it. I did preorder Diablo IV, despite the controversy surrounding its launch date. Blizzard is definitely still having problems. I don't think I'll boycott them again, as it had no material effect on anything. We'll see how things turn out. There's still the Microsoft acquisition to consider... if it goes through.

Silver Gryphon Games comes back

In August, Kevin and I made the decision to revive Silver Gryphon Games. This time around, though, I was the one to submit all the registration paperwork and manage the company side. I'm hoping that by being the driving force I can avoid the deliverables problems we had before. We committed to fulfilling the Aethermancy Kickstarter that we abandoned years ago. Also, I'm writing the new edition of Aether. This is the first time I've had an active hand in Aether. While I'm much more interested in writing OSR games, Aether reminds me a lot of what the Palladium System could have been if it was... well, good. For the first time in a long time, I'm really excited about writing for a commercial RPG. Kevin's working on Aethermancy while I work on Aether. He's heavily involved in his retail games business, though, so he's much more distracted than I am. Hopefully I can do right with Aether by him.

Summary

This is by far my longest year-in-review post I've ever made. A ton happened in 2022. I imagine a lot is going to happen in 2023, too. Tomorrow I'll write about all my plans for the new year.

Here's a list of all the year-in-review posts I've made. I have the strange sense that I've written more of these, but they're either lost to time, or in physical journals. I might have to try and find them if that's the case.

Ranking #1 on HN in the December Dead Zone

swyx's site RSS Feed — Fri, 30 Dec 2022 02:50:58 GMT

For those who weren't aware (you would be if you were subscribed on email!), I started a separate AI blog, L-space Diaries, this year to 1) try out Substack in anger and 2) create a focused feed on a topic rather than a person (I also started DX Tips for devtools/devrel writing, but have been accordingly distracted).

2022: year in perspective

learnbyexample — Fri, 30 Dec 2022 02:00:00 GMT

TL;DR: Published two programming ebooks, wrote several blog posts, recorded plenty of Youtube videos, newsletter prospered, improved Twitter audience, read 100+ novels, and so on. Had an excellent year in terms of ebook sales 😇

Books published🔗

Vim Reference Guide — concise learning resource for beginner to intermediate level Vim users, published in March
Computing from the Command Line — Linux command line tools and Shell Scripting for beginner to intermediate level users, published in November

Workshops🔗

Offline workshops were back on menu this year. I got only one offer though. Surprisingly, it was for Python basics, despite students already having had a course in their first year. It was a nice experience for me, thanks to the enthusiasm shown by the students.

And it was good to see BarCamp Bangalore being organized again. Gave a talk about my ebook publishing experience (I had also written a blog post on this topic last year).

Blog posts🔗

Here are my favorite posts I wrote this year:

I continued posting weekly programming tips (Python, Linux, Vim) that are short and easy to digest and wrote some mini blog posts as well.

Tools🔗

During the last two months of the year, I learned a bit of Textual and wrote a couple of TUI apps:

Square Tic Tac Toe — form a square with 4 corners
Linux CLI Text Processing Exercises — test your CLI text processing skills

I found the framework much easier to use compared to my experience with Tkinter.

Youtube🔗

While working on the Vim Reference Guide, I felt that some of the commands really needed video demos for easier understanding. So, I gave myself another chance at recording videos. I kept them simple and short, and with consistent practice I did better than my attempts a few years back. I then extended my new found enthusiasm to programming tips, ebook promo videos, etc. Visit my youtube channel for interesting tech nuggets.

Here are some of the tools I use:

SimpleScreenRecorder — recording video, really simple to use
auto-editor — removing silent portions from video recordings
FFmpeg — video processing, padding for example (FFmpeg is also a major part of the auto-editor solution)
Canva — video thumbnails (I also use this app for ebook covers)

Book sales🔗

Revenue from ebook sales were almost 50% higher than last year!! As I wrote in the 2021 was a wild ride post, I've been paying more attention to marketing and seems like my efforts are paying off. Here's my Gumroad revenue chart for 2022:

The peaks were during the two ebook releases. Additionally, I had less than half the above revenue from Leanpub. Last year, sales from Leanpub and Gumroad were nearly the same.

Last November, I started learnbyexample weekly newsletter. I've managed to send an email every Friday without fail so far and I'm proud of that. Sometimes I had to schedule issues a week ahead. Currently about 600 subscribers and some readers are even paying me monthly despite being a free newsletter.

Building Twitter audience🔗

As part of marketing efforts last year, I started building my Twitter audience as well. Follower count was less than 400 in July and about 1100 in December last year. Now it has crossed 2900. I'm not focused on increasing follower count with plethora of engagement inducing tweets. Just trying to be consistent and promoting all sorts of interesting links I come across. That said, I'd like to try creating cool infographics (probably using Canva) next year.

Follow me on Twitter for interesting tech nuggets 😉

Fictional reading🔗

I enjoy reading fantasy and science-fiction novels. I read 100+ SFF books this year and recently wrote a post listing my favorites.

I also got a chance to beta read Tongue Eater, Soul Relic, The Umbral Storm and The Book of Zog. I find these a good way to give back to the writing community, having myself received plenty of support from strangers.

Goals for 2023🔗

I met most of my goals this year, so that's a nice feeling. Contributing to open source projects needs a lot more focus in the coming year. I'm not likely to publish a new ebook in 2023. Instead, I'm planning to update my existing books and that will probably take more than a year. Apart from catching up to new features and improving existing examples/exercises, I'll also focus on changing book titles and cover images. And, I'll likely create interactive apps for exercises.

I need to also find something other than books to keep me creatively busy. It has been more than 4 years since I first published an ebook and 6 years since I started writing programming tutorials.

Here's wishing you a very happy, healthy and prosperous 2023 👍 😇

Trouble with TypeScript Timeouts: How to Fix It

Donat Studios — Wed, 28 Dec 2022 22:46:19 GMT

The tl;dr is this:

You can fix the problem by replacing number with ReturnType<typeof setTimeout>

Similarly you can replace number with ReturnType<typeof setInterval> if you are seeing the problem with setInterval.

If you want to know more about it, read on.

I've run into this problem a number of times where I'm developing frontend TypeScript and after I import a package, I suddenly find throughout my project I'm receiving:

error TS2322: Type 'Timeout' is not assignable to type 'number'.

If you are working on a Node project rather than a front end project this can also manifest itself as the opposite

error TS2322: Type 'number' is not assignable to type 'Timeout'

Usually this happens because some dependency included @types/node under dependencies instead of devDependencies. When that happens, Node's global types leak into your project—even in frontend environments—causing setTimeout to return a Timeout object instead of a number.

The heart of the problem is that setTimeout() and setInterval() in Node API return a Timeout whereas in the HTML DOM Window API, they return an integer.

The ideal solution would be for the offending package to fix the problem themselves, and for you to update to the corrected version of the package. This is a hassle and might not be possible, with the issue going ignored for months if not years - as on Facebook's Jest package ^[1] ^[2].

For example, consider the following:

let x : number = setTimeout(() => console.log("foo"), 1000);

Your immediate reaction might be to either change the definition to let x : Timeout or eliminate the explicit type definition all together à la

let x = setTimeout(() => console.log("foo"), 1000);

Both of these options work.

The first option of setting it to Timeout suffers from two issues though. Firstly it's fragile - presumably you're not explicitly including @types/node so the change to the environment is a side effect you're coding around. Should the specific packages you include change, or said package correct this it will break. Second, it's ambiguous. Timeout isn’t a global type—it's specific to Node's type system—so using it assumes a Node environment even if you're working in the browser. That can confuse collaborators or tooling down the line.

The second option of implicit typing will work in a lot of cases just fine. There are many cases where you'd want for instance to define the variable in a higher scope than you are setting its value - as a trivial example

class Foo {
    private x? : number;

    start() {
        this.x = setTimeout(() => console.log("foo"), 1000);
    }

    stop() {
        clearTimeout(this.x);
    }
}

The answer is frustratingly simple. We can just use the ReturnType utility type to set our type to the return type of setTimeout. We can do this using ReturnType<typeof setTimeout>.

Our examples from above become

let x : ReturnType<typeof setTimeout> = setTimeout(() => console.log("foo"), 1000);

and

class Foo {
    private x?: ReturnType<typeof setTimeout>;

    start() {
        this.x = setTimeout(() => console.log("foo"), 1000);
    }

    stop() {
        clearTimeout(this.x);
    }
}

Similarly, if you are using setInterval you can utilize type ReturnType<typeof setInterval> for similar purposes.

Explore the fediverse, but use block like it's a machine gun in a zombie apocalypse

Ben E. C. Boyter — Wed, 28 Dec 2022 02:00:00 GMT

I didn’t really expect to have a post this late in the year but here we go. Lets go through some assumptions that were broken for me recently. Not just my own through perhaps some other peoples assumptions too.

I have been working with ActivityPub for a bit. I want to run my own server, beyond installing some 3rd party. I want to build my own from scratch. However this rapidly exposed some problems with testing ActivityPub.

Python tip 21: sorting iterables based on a key

learnbyexample — Wed, 28 Dec 2022 02:00:00 GMT

You can use the sort() method for sorting lists inplace. The sorted() function can be used to get a sorted list from any iterable.

The key argument accepts the name of a function (i.e. function object) for custom sorting. If two elements are deemed equal based on the result of the function, the original order will be maintained (stable sorting). Here are some examples:

# based on the absolute value of an element
# note that the input order is maintained for all three values of "4"
>>> nums = [-1, -4, 309, 4.0, 34, 0.2, 4]
>>> nums.sort(key=abs)
>>> nums
[0.2, -1, -4, 4.0, 4, 34, 309]

# based on the length of an element
>>> words = ('morello', 'irk', 'fuliginous', 'crusado', 'seam')
>>> sorted(words, key=len, reverse=True)
['fuliginous', 'morello', 'crusado', 'seam', 'irk']

Here are some examples using lambda expressions:

# sorting dictionaries based on values
>>> vehicles = {'bus': 10, 'car': 20, 'jeep': 3, 'cycle': 5}
>>> sorted(vehicles, key=lambda k: vehicles[k])
['jeep', 'cycle', 'bus', 'car']
>>> dict(sorted(vehicles.items(), key=lambda t: t[1]))
{'jeep': 3, 'cycle': 5, 'bus': 10, 'car': 20}

# based on file extension
>>> files = ('report.txt', 'hello.py', 'calc.sh', 'tictactoe.py')
>>> sorted(files, key=lambda f: f.rsplit('.', 1)[-1])
['hello.py', 'tictactoe.py', 'calc.sh', 'report.txt']

Creepy Website Similarity

Weblog on marginalia.nu — Mon, 26 Dec 2022 02:00:00 GMT

This is a write-up about an experiment from a few months ago, in how to find websites that are similar to each other. Website similarity is useful for many things, including discovering new websites to crawl, as well as suggesting similar websites in the Marginalia Search random exploration mode.

A link to a slapdash interface for exploring the experimental data.

The approach chosen was to use the link graph look for websites that are linked to from the same websites. This turned out to work remarkably well.

Crypto I: On cryptocurrencies, explained using FTX Tokens

Bytepawn - Marton Trencseni — Sat, 24 Dec 2022 01:00:00 GMT

The crypto space is a fascinating intersection of technology, economics and human psychology. However, I remain skeptical of the value of crypto tokens and NFTs. Here I explain my thought process using FTX Tokens as an example.

On Wizards and Sorcerers

Weblog on marginalia.nu — Fri, 23 Dec 2022 02:00:00 GMT

While this post is about programming, it also draws an extended analogy to dungeons and dragons, specifically two of its classes, that correspond to two attitudes toward programming.

In D&D, wizards study magic. They prepare their magic spells ahead of time. While they may learn a large number of magic spells, they need to prepare them ahead of time and can’t just cast them at will.

Wizard programmers prefer up-front design. They apply reason and logic to divide and conquer a large problem, they rely on building blocks like design patterns and algorithms. Wizards rely on explicit knowledge.

Old School Baroque

Ben Overmyer's Site — Thu, 22 Dec 2022 02:00:00 GMT

Lately there have been an increasing number of new tabletop role-playing games coming out that are distinctly different from the Dungeons & Dragons retroclones popularized categorized as Old School Rennaissance or Old School Revival. These games eschew simplicity in favor of complex or complicated rules, and often have remarkable layout and design or gonzo themes. However, they still are directly influenced by rulesets and design themes from several decades ago. I think we can characterize these as "Old School Baroque."

I did not coin this term. The first incidence I can find of it is on a forum post on RPGPub in 2019. A more codified version of it appears in a more recent blog article that carefully examines trends in OSR RPGs. While both the forum post and blog article use the term rather tongue-in-cheek, I think it's appropriate to describe this emerging category of games.

The Baroque period of history was dramatic and a reaction to the Reformation. It leaned heavily into art and sensory exploration. The word "Baroque" is derived from the Italian "barocco" and is roughly equated to an abberation in or avoidance of logic. Where OSR games strive hard to adhere to the rules and themes of 70s and 80s Dungeons & Dragons, Old School Baroque games instead throw caution and tradition to the wind.

One example of these games would be the much-vaunted Mörk Borg. The layout and design are heavily over-the-top and the game's setting is richly, even brutally, apocalyptic. It leans into a death metal aesthetic that cries out to the senses. The rules are clearly influenced by OSR games that came before it, but they're stretched and torn in ways that might offend more delicate OSR sensibilities. It is definitely more rules-light than other games in the Old School Baroque genre, though.

Another game in this vein is Errant, which presents itself as being "rules light, procedure heavy." That latter clause is what really makes it more of an OSB game than an OSR game. Well, that, and the hit-you-in-the-face aesthetic. While the aforementioned "procedures" are explicitly called out as not being rules, they are effectively rules in how to run the game. Even though Errant itself has a unified core mechanic, those procedures generally run all over the place.

This is a fascinating new genre. I'm looking forward to watching its evolution.

Announcing learnbyexample weekly newsletter

learnbyexample — Thu, 22 Dec 2022 02:00:00 GMT

Hello!

I'm excited to announce learnbyexample weekly newsletter, scheduled to be delivered every Friday.

This free newsletter will help you discover awesome programming resources. I'll primarily focus on resources related to Python, Linux, CLI tools, Regular Expressions and Vim. Sometimes, I'll also include other programming resources.

You can expect 5-15 links, usually categorized into the following sections:

Article of the week
Resources
Free programming books, courses and deals
Tip of the week
Tools
Curiosity Corner

Here are some of the resource links from the first issue:

How To Learn Stuff Quickly by Josh W. Comeau
Python pathlib Cookbook — 57+ Examples to Master It
Complete Guide to CSS Flex and Grid by Shruti Balasa (25% OFF for a week)
Carbon — Create and share beautiful images of your source code

After subscribing to learnbyexample weekly, you'll get a confirmation email followed by another email with the latest issue contents. You can also view the past issues from your Gumroad account.

Hope you find the newsletter useful. Let me know your feedback via email (learnbyexample.net@gmail.com) or twitter.

Happy learning :)

The Indie Python Extravaganza

learnbyexample — Thu, 22 Dec 2022 02:00:00 GMT

Hello!

You never know where a conversation between indie authors will lead to. A tweet about Leanpub Python book sales brought up giveaways that we indie authors tend to do. Long story short, the four of us ended up deciding to create The Indie Python Extravaganza bundle.

And guess what?! You can use this pytober coupon link to get the bundle for FREE (the offer is valid till 31-Oct-2021).

Bundle contents🔗

A collection of books that will help you to improve your knowledge of the Python programming language one page at a time. Join four indie authors in a journey from the basics of Python to the structure of production-ready systems, going through the core features of the language, some intermediate projects and a deep dive into regular expressions.

Coupon link

In this bundle, Mike will teach you the basics of Python with Python 101. Sundeep will then take the lead and help you to put your knowledge into practice with Practice Python Projects. Learn what NOT to do when writing your Python programs with Rodrigo in his Pydon'ts book! If you need to learn regular expressions, Sundeep has again your back with his Python re(gex)? book, and when you are ready to start working on production code, you'll have Clean Architectures in Python to help you!

Authors🔗

Leonardo Giordani: Blog, Twitter
Michael Driscoll: Blog, Twitter
Rodrigo Girão Serrão: Blog, Twitter
Sundeep Agarwal: Blog, Twitter

How can you help?🔗

Share the bundle link with your friends and colleagues interested in learning Python.

Your feedback on the book contents would be appreciated even more.

Happy learning 😇

Automating Excel with Python - book review

learnbyexample — Wed, 21 Dec 2022 02:00:00 GMT

In this post, I review Automating Excel with Python by Michael Driscoll. From the introduction chapter of this book:

The purpose of this book is to help you learn how to use Python to work with Excel. You will be using a package called OpenPyXL to create, read, and edit Excel documents with Python. While the focus of this book will be on OpenPyXL, you will also learn about other Python packages that you can use to interact with Excel using the Python programming language.

Book details🔗

Book cover

Amazon — Paperback, Kindle
Gumroad — PDF, EPUB, Mobi
Leanpub — PDF, EPUB, Mobi
GitHub — code examples and sample spreadsheets used in the book
Goodreads — book reviews

Review🔗

My very first job assignment (at a semiconductor company) required me to use spreadsheets for tabulating results of various experiments, adding charts, etc. I used to manually copy-paste the results generated from a Perl script. There were multiple sheets and my work was complicated enough to require multiple months of refinement, feature modifications, etc. Not sure if a library like OpenPyXL existed back then, but I think I should've at least asked/searched ways to automate the spreadsheet process.

Going through this book felt like someone wrote a book just for that project, albeit 13 years late. Here's a rough list of features that would've helped me:

Creating xlsx files with multiple sheets
Adding data
Formatting cells based on a known equation
Creating charts

Instructions and examples were clear and easy to follow. Snapshots were also shown for all the examples, so you can check if you've followed along as expected. While the book is best suited if you have MS Excel, most of the examples worked for me on LibreOffice Calc. Only the charts had major differences — some types weren't supported and x/y axis label/data were problematic as shown below:

Bar Chart in Excel (snapshot from the book)

Bar Chart in LibreOffice Calc (what I got on my machine)

Apart from the openpyxl module, the author also briefly covered how you can use pandas, xlsxwriter and gspread (for working with Google sheets). Some features were presented at the end as Appendix chapters.

Table of Contents🔗

Introduction
Chapter 1 - Setting Up Your Machine
Chapter 2 - Reading Spreadsheets with OpenPyXL
Chapter 3 - Creating a Spreadsheet with OpenPyXL
Chapter 4 - Styling Cells
Chapter 5 - Conditional Formatting
Chapter 6 - Creating Charts
Chapter 7 - Chart Types
Chapter 8 - Converting CSV to Excel
Chapter 9 - Using Pandas with Excel
Chapter 10 - Python and Google Sheets
Chapter 11 - XlsxWriter
Appendix A - Cell Comments
Appendix B - Print Settings Basics
Appendix C - Formulas

Feedback and Reviews🔗

All in all, I would highly recommend this book for those wanting to use Python for automating spreadsheets. I'd request you to post reviews after going through the book (they help us indie authors a lot). And please do contact the author to let him know your feedback or if you have any clarifications.

Happy learning :)

2021 was a wild ride

learnbyexample — Wed, 21 Dec 2022 02:00:00 GMT

TL;DR: Started and ended the year well, with a depressing period in the middle. Published three programming ebooks, several blog posts, started a newsletter, improved Twitter readership, read 80+ novels, and so on. Had a good year in terms of ebook sales 😇

Books published🔗

100 Page Python Intro — short, introductory guide for the Python programming language. Started writing last year, published in February
Practice Python Projects — five beginner to intermediate level projects inspired by real world use cases. Started writing last year (before "100 Page Python Intro"!), published in July
Command line text processing with GNU Coreutils — learn 20+ specialized text processing tools provided by the GNU coreutils package. Published in October

I also spent time updating all my existing books from February to May.

Workshops🔗

First and only workshop I conducted since the start of pandemic in 2020. And this was possible only because it was online. The topic was Python scripting introduction for Biotech students. Publishing "100 Page Python Intro" was timely for this workshop.

This took up most of my time during March/April along with updating existing books.

Blog posts🔗

I've been consistently writing books for the past three years, but I find it difficult to come up with ideas for my programming blog. This is partly due to not wanting to repeat content from my books. Here's my favorite posts I wrote this year:

I tried to be more consistent by posting short articles (see mini posts list), but lost interest. Starting a newsletter in November helped change my perspective about re-using content from my books. I've started posting tips and coding challenges that are short and easy to digest:

I was more consistent for my Escapist Reviews blog that I started late last year to review novels I read.

Book sales🔗

Had better sales compared to last year, which I really wasn't expecting. Especially when the average monthly sales was around $100 between May to September (my monthly expenses is around $150). This coincided with some health issues and the struggle to finish writing the "Practice Python Projects" book.

This led me to reading articles about better landing pages, building audience on social media, affiliates, etc. I still have a long way to go, but I feel these active efforts led to much improved sales in the last quarter of the year. I ended up deciding not to use affiliates though.

Here's my sales chart from Gumroad for this year (I had similar revenue from Leanpub):

There were plenty of reasons that led to the awesome last quarter sales. Here's some significant events I remember:

Joined hands with fellow Python authors for The Indie Python Extravaganza bundle (given away freely for a month)
- A Twitter discussion led to the giveaway idea, which resulted in creating this bundle
- Combined marketing efforts by all four of us gave significant paid sales too
Published "Command line text processing with GNU Coreutils"
- In addition to my usual practice of making a new book free, this time I offered All books bundle for $5 and a lot of users bought it
- Announcing the book on Reddit and Hacker News was well received
- I was beginning to improve my Twitter audience around that time, which helped a bit
- Got featured in Leanpub's monthly sales newsletter
- Jesse Smith on distrowatch.com wrote a lovely book review, which resulted in significant sales in December
Getting featured on Ruby weekly
Programming deals for the last week of November
- Helped a lot by commenting on Hacker News and getting featured in blog posts of fellow Python authors
Interactive GUI app for Python regex
- As part of 50 days of break from book writing, I worked on this Python app
- Made it to the front page of Hacker News yet again
"The Indie Python Extravaganza" bundle and some of my other books were featured in Leanpub's Boxing day sales
And I believe creating GitHub Readme helped as well

The biggest takeaway for me was to actively look for opportunities (small or big) instead of just relying on free offering during book launch (which is about once in four months).

During the 50 days break, the other significant project I started was learnbyexample weekly newsletter. This is still in early stages to point out any impact it will have on my book sales, but it certainly has been a pleasure so far to email an issue every Friday.

And as mentioned earlier, this led me to write programming blog posts consistently (tips and coding challenges).

Building Twitter audience🔗

I joined Twitter in 2015. My follower count was less than 400 in July. In December, I crossed 1100. This is far from being impressive (I know a few authors who added more than 15000 followers during that time period).

Being active on Twitter led me to awesome opportunities mentioned earlier in the Book sales section. The best tips I can give is to tweet consistently, interact with your readers and don't be afraid to participate in conversations initiated by top users. Oh, and reading articles/books about social media audience building would help too.

Follow me on Twitter for interesting tech nuggets 😉

Fictional reading🔗

I enjoy reading fantasy and science-fiction novels. I read 80+ SFF books this year and recently wrote a post listing my top 10 favorites.

I also got a chance to beta read The Siege of Skyhold and an ARC of Bastion. I find these a good way to give back to the writing community, having myself received plenty of support from strangers.

Goals for 2022🔗

Foremost goal is to continue taking care of physical/mental health. And I'd be more than happy if I manage yet another year with $250+ average monthly income.

Books:

I'm currently working on Vim Reference Guide ebook. Likely to publish in the first quarter
I started working on Command line text processing with Rust tools ebook even before "Command line text processing with GNU Coreutils", hope to publish in 2022
Have several more book topics in mind, but not sure if I'll start working on any of them. And it is possible that I'll come up with something else I fancy and work on it instead of already planned topics

Projects:

Interactive apps for exercises from other books, similar to the one I did for Python regex
Games for fun

Miscellaneous:

Continue to build an audience via Twitter, Newsletter, etc
Contribute to other open source projects

Here's wishing you a very happy, healthy and prosperous 2022 👍 😇

Finding similar API functions between Pytorch and Tensorflow with Doc2Vec

Bytepawn - Marton Trencseni — Wed, 21 Dec 2022 01:00:00 GMT

I use Doc2Vec to try to find pairs of similar API functions between Pytorch and Tensorflow.

Vim tip 19: working with buffers

learnbyexample — Tue, 20 Dec 2022 02:00:00 GMT

Multiple files can be opened in Vim within the same tab page and/or in different tabs. From :h windows-intro:

A buffer is the in-memory text of a file.

A window is a viewport on a buffer.

A tab page is a collection of windows.

:e refreshes the current buffer (:e is short for :edit)
:e filename open a particular file by its path, in the same window
:e # switch back to the previous buffer, won't work if that buffer is not named
Ctrl+6 switch back to the previous buffer, works even if that buffer is not named
- Ctrl+^ can also be used
:e #1 open the first buffer, and so on
:buffers show all buffers
- :ls or :files can also be used
:bn open the next file in the buffer list (:bn is short for :bnext)
- opens the first buffer if you are on the last buffer
:bp open the previous file in the buffer list (:bp is short for :bprevious)
- opens the last buffer if you are on the first buffer

Use :set hidden if you want to switch to another buffer even if there are unsaved changes in the current buffer. Instead of this setting, you can also use :hide edit filename to hide the current unsaved buffer. You'll still get an error if you try to quit Vim without saving such buffers, unless you use the ! modifier.

See :h 'autowrite' option if you want to automatically save changes when moving to another buffer.

See :h 22.4 and :h buffer-hidden for user and reference manuals on working with buffer list.

Video demo:

Writing as a Programmer

The Cranky Developer on Crater Moon Development — Sun, 18 Dec 2022 23:12:59 GMT

For programmers, writing is an essential skill to build shared understanding.

Testing GPT-3 spatial reasoning and comprehension

Bytepawn - Marton Trencseni — Sat, 17 Dec 2022 01:00:00 GMT

I run experiments to determine whether, or to what degree, GPT-3 has developed an comprehension of spatial relationships.

Markdown Tables on the Fedivee

infrequent oscillations — Fri, 16 Dec 2022 12:38:46 GMT

In which we see who supports Markdown tables correctly.

Make the world small again

Ben Overmyer's Site — Thu, 15 Dec 2022 02:00:00 GMT

The world has gotten too big.

A lot of online services that we've come to rely on are too intent on serving everyone. In the process, they serve no one well. Similarly, governments are trying very hard to generalize and set universal rules and policies that ultimately cause more problems than they solve. I won't talk about that latter issue in this post, though, as that would make this far longer than it needs to be. So, online services.

Media Services

Streaming services like Spotify, Netflix, and so on have increasingly retired media that I'd really like to continue to have available. The reasons are varied, but usually boil down to licensing changes. For the past decade or so, I've put up with this because I usually only watch a given movie or show once every decade or so. Music streaming services like Spotify and Apple Music offer a huge wealth of music, but... even with all of that available, I tend to stick to particular genres or artists. Also, I have some eclectic tastes in music, and some of these songs have never been available on streaming services, or have been retired after licensing changes.

Yesterday, I saw a post on Mastodon that stuck with me. Here it is, reproduced in text form because I hate screenshots of text:

Kids this is a reminder to collect physical copies of your favorite media. Corporations will axe your favorite show/movie/book in front of its family and make it a felony to retrieve it. Have a library.

The author is @lolennui@mstdn.social. It was posted on Dec 14, 2022 at 02:45. Here's a link to the post.

I think I've hit a tipping point where I agree fully with this sentiment. If you don't own something, it can disappear at any time, for any reason, and there's nothing you can do about it. A lot of these streaming services are starting to optimize for "waves" of content, and licensing titles for shorter periods of time.

I still want to digitally access all of this content. I don't really want to have a massive physical library, as that can take up a huge amount of space. My parents had a collection of over 400 VHS tapes and hundreds of books twenty years ago. They filled rooms with that. So that leaves me with building and operating my own private streaming service.

Which... is easier than you might think. I've set up a Jellyfin server and begun loading media that I already own into it. This works really well so far.

Now comes the long haul part, though - I need to build that library. I want to legitimately own all the media I load into that server, too, so piracy is right out. Movies and TV shows are going to have to come through DVDs and Blurays that I buy and rip from. Music, though, can come from all sorts of places, since I can buy MP3s directly. I'm particularly enjoying discovering new artists I like on Bandcamp and buying their albums.

Search Engines and Discovery

The mainstream search engines - Google, Bing, even DuckDuckGo - are getting worse and worse at finding meaningful and useful content. Twenty years of search engine optimization, clickbait, and design for algorithms has created a morass of content that increasingly looks designed for machines, not humans. The search engines encourage this, exacerbating the problem. When I search for something, the results are often recently "updated," very similar to a dozen other pages, and optimized to death. Trying to find a web page from the early 2000s that you know for a fact is still online? Good luck. The mainstream search engines probably don't even have it indexed anymore. They don't have infinite storage, and in order to limit their expenses, they cull the least popular (read: least algorithm-optimized) links from their indexes.

It's not all bad news. More and more people are beginning to realize this problem exists. New, independent search engines that index specific niches are popping up. They're small, but serve specific needs very well. One such search engine is Wiby, which specifically targets plain pages with no commercial content. It's really good for finding blog articles on topics that interest you. Another is Old'aVista, which indexes old personal websites.

Another thing that's starting to make a comeback is the web directory. Back in the late 90s and early 00s, web directories were the best way to find curated content that was directly relevant to your interests. They mostly disappeared once search engines hit their stride. Now that the web is too big for any one search engine to handle, though, web directories have an important use again.

I run one such web directory. It's called RPGGen, and it's meant for discovering websites that provide content generators for tabletop role-playing games. There are many others out there. For example:

Incidentally, I discovered all three of these through Wiby.

Oh, social media. You were once so full of promise. Now, though, all social media is good for is sound bites, short videos of people or animals doing stupid things, and text that triggers various parts of our lizard brains.

The original promise of social media - keeping in touch with people you care about - is all but forgotten.

As it turns out, though, we never really lost the tools we need to keep in touch. I've started using email and even physical mail again to reach out to people. If I have exciting news I want to share with people I care about, I do so directly with each person. It's not much more effort than posting on social media, and it's much more rewarding.

I still use social media, but I'm not nearly as active on it as most people are. I have an account on the major social media sites, but I never use those anymore. The accounts are just placeholders, really. I have one friend who insists on using Facebook Messenger still, and that's honestly probably the only reason I still have a Facebook account. Twitter... well, I didn't really use it anymore to begin with, but once Melon Tusk bought it, that killed the whole thing for me. Unfortunately I still have to use Twitter to log in to two services that require social media login. I should probably fix that.

The other use of social media is broadcasting our presence and thoughts to the world in general. There's a better alternative for that too: the personal website. If you don't have the technical skills to maintain a static site like the one I have here, there are options for you - Neocities is one such.

Summary: Solutions for Each

So to recap, here are my solutions for each of the above three categories of problems.

To replace streaming services, build your own library. Use Jellyfin for media. Consider leaning into a physical library, if you have the space for it.

To replace mainstream search engines, use independent search engines and web directories.

To replace social media, use email, letters, and phone calls. Make a personal website for your web presence.

Let's make the world smaller again. Just for us.

Fred Brooks' The Mythical Man-Month

Bytepawn - Marton Trencseni — Thu, 15 Dec 2022 01:00:00 GMT

I reflect on the core points of Fred Brooks' seminal book, The Mythical Man-Month, that I often recall and apply in my daily work.

Reintroducing Opel: Put All Your Pelican Posts in One Org File

Beetle Space — Wed, 14 Dec 2022 10:00:00 GMT

A few years ago I introduced opel - an Emacs package that allows one to write all their Pelican posts in one Org file. It was inspired by ox-hugo. I’ve since made significant updates.

Motivation

Why would one want this as opposed to writing one post per file?

You can …

CLI tip 20: expand and unexpand

learnbyexample — Wed, 14 Dec 2022 02:00:00 GMT

These two commands will help you convert tabs to spaces and vice versa. Both these commands support options to customize the width of tab stops and which occurrences should be converted.

The default expansion aligns at multiples of 8 columns (calculated in terms of bytes).

# 'apple' = 5 bytes, \t converts to 3 spaces
# 'banana' = 6 bytes, \t converts to 2 spaces
# 'a' and 'b' = 1 byte, \t converts to 7 spaces
$ printf 'apple\tbanana\tcherry\na\tb\tc\n' | expand
apple   banana  cherry
a       b       c

# 'αλε' = 6 bytes, \t converts to 2 spaces
$ printf 'αλε\tπού\n' | expand
αλε  πού

By default, the unexpand command converts initial blank (space or tab) characters to tabs. The first occurrence of a non-blank character will stop the conversion. By default, every 8 columns worth of blanks is converted to a tab.

# input is 8 spaces followed by 'a' and then more characters
# the initial 8 spaces is converted to a tab character
# 'a' stops any further conversion, since it is a non-blank character
$ printf '        a       b       c\n' | unexpand | cat -T
^Ia       b       c

# input is 9 spaces followed by 'a' and then more characters
# the initial 8 spaces is converted to a tab character
# remaining space is left as is
$ printf '         a       b       c\n' | unexpand | cat -T
^I a       b       c

Video demo:

See expand and unexpand chapter my Command line text processing with GNU Coreutils ebook for more examples, options, etc.

How good an astrophysicist is GPT-3?

Bytepawn - Marton Trencseni — Mon, 12 Dec 2022 01:00:00 GMT

Here I will show a "conversation" with GPT-3 to gauge how good an astrophysicist — or an illusion of an astrophysicist — it is. I will focus on cosmology questions, because that's most interesting part of the field.

Building TUIs with textual: first impressions

learnbyexample — Fri, 09 Dec 2022 02:00:00 GMT

Last week, I finally started exploring textual. The main motivation was to start implementing a few project ideas I've had in my todo list for years. I don't particularly have a preference between TUI (terminal user interface) and GUI (graphical user interface) for these projects. Seeing a few Textual demos on twitter (courtesy Will McGugan) over the past few months, I felt like exploring this framework first.

For my first app, I picked a 4x4 board game — like Tic Tac Toe but form a square instead of a line. I came up with this variation in high school and been fond of coding it since college days.

Installation and Tutorials🔗

The Getting started page of the documentation will give you all the relevant installation instructions. I used pip install 'textual[dev]' since the development mode has nice features like live editing. As I looked up the Devtools page to link here in this blog post, I found that there's a console command for print() based debugging! That would've been handy while I was working on the game — sigh, I should've been more proactive in exploring the documentation site.

In the Getting started page, you'll also be informed about python -m textual (builtin demo) and other examples in the GitHub repo.

After playing with the demo a bit, I went through the tutorial — shows how to build a Stopwatch app step-by-step.

The documentation also includes Guide, Reference, API, etc. I gave them a cursory glance and decided to start building my game.

I should note that while I got introduced to programming in school about 20 years ago, I don't have much experience with projects that need more than a few hundred lines. I'm good with command-line tools and text processing with scripting languages like Python. I had a horrible experience writing an Android app a few years back, mainly due to object-oriented programming and the complexity of the project. I've improved a bit since then, but still feel like a newbie when it comes to working with classes.

Building Square Tic Tac Toe board game🔗

Similar to the step-by-step Textual tutorial, I built the game by adding features incrementally. I tweeted my progress along with screenshots and recordings. Here's a summary:

Managed to place 16 buttons in a grid layout
Buttons now respond to clicking! And in response, the computer plays a random move
Recording below shows 3 games: User wins, AI wins, Tie
Added Easy/Hard modes — it is impossible to beat the AI in hard mode
Almost done! Layout is better now and starting new game is now a button instead of a shortcut keybinding
Cleaned up code a bit and posted on GitHub
Next step: write a blog post (this post!)

Visit my GitHub repo for the code, game rules and other details.

I had made a GUI version of this game using tkinter last year. I copied most of the game logic from there, so I didn't much struggle with object-oriented programming in this case. Here's a sample screenshot from the finished code:

What I liked🔗

As mentioned before, Textual supports live editing mode. The command is textual run --dev script.py and this helps you experiment with CSS. I found this very helpful while trying out layout combinations, margin, padding, etc.

The default colors were great too. I didn't have to think about choosing colors (except for setting background color for header and game status). The framework even provides an easy way to allow users to switch between dark and light themes! Though, I haven't yet figured out how to set light theme as the default (I worked around by explicitly adding a call to the theme toggle method).

Overall, the code was significantly shorter compared to the tkinter version I did last year. That version had a few more features, but I'd say Textual felt much easier to reason about. I remember having to spend days shifting through stackoverflow threads and tkdocs to get the GUI version working.

What gave me trouble🔗

Struggling with layout isn't new for me. I started with 4x4 grid for the board, which was fairly straightforward. Problems arose when I wanted to add status text area to the left and control buttons to the right. Placing them left/right was easy to do with dock in CSS. But, I couldn't get them to align well — too much spacing around the 4x4 board. I was trying to give 50% to 60% for the board and the remaining evenly divided for the other two elements. After some experimentation, what worked was giving 20% to status, 25% to control and not assigning a width value for the board.

I initially used a button for the status because I couldn't find a textbox widget (edit: Textual now has a Label widget). I knew that Static widget can display text, but I didn't find how to dynamically change the text from that documentation page. I thought I'll have to make a custom widget, but when I went to Widgets guide, I found that Static already has an update() method!

I probably missed something (or perhaps part of the roadmap), but I found it strange to have a single on_button_pressed() method to handle on click event for every Button widget. I'd prefer a way to bind a method to the buttons, like tkinter provides.

Next steps🔗

As mentioned before, I have several projects in my todo list. The next one I want to try is an app for interactive exercises for my ebooks. Last year, I made one for Python regular expressions using tkinter.

Strong Opinions & Eroding Discourse

bayindirh — Fri, 09 Dec 2022 02:00:00 GMT

I have been observing a trend both in blogs and general internet discourse, but instead of pointing fingers, I decided to sit back, observe and assume that I'm the man who was yelling at the cloud. However, an article, and comments on the orange site convinced me that I'm not the only one with this observation, hence I decided to note it here.

We're arguably going through the second big shift in computing history with a renewed interest in programming due to exponential rise of computing in our daily lives in visible and invisible ways. Sustaining this growth requires a much bigger brain power, and not falling behind in this global technology race. In turn, this fuels discourse and information sharing, and in turn, an evolved version of the flame wars of the 80s and 90s is taking place.

The reason for this is simple, in my opinion. We’re building more sophisticated and ambitious things, which results in bigger and more sensational outcomes. People involved in and excited by these technologies hype it knowingly or unknowingly, and when someone opposes their views, they get upset and answer back forcefully. These sparks ignite the already flammable atmosphere, and flames ensue.

While flame wars look like a natural outcome, its existence is detrimental to neutral and healthy discourse on the bigger scale. Eliminating this phenomenon is impossible in my opinion but, minimizing its occurrence is highly beneficial.

The key to reducing flame wars is a simple but hard to master technique: “listen (read) intently and talk (write) clearly and calmly”. Applying this requires a level head and is not always possible, I know. Also, some people are very good at intentionally triggering the emotions of the other party. However, going above and beyond this simple shouting matches is beneficial for both arguing and spectating parties, because it maximizes the chances for learning something, and leaving the discussion in a better place.

Next time, please try to reread the words of the person who’s answering you slowly and intently if you find yourself angrier than you should be, and answer ten minutes later if you can help it.

You’ll be amazed how this one weird trick can change your conversations for the better.

Until next time,

Be kind.

Rendering LaTeX Formulae in mastodon.el

Beetle Space — Thu, 08 Dec 2022 10:00:00 GMT

On Mastodon I follow a number of folks at the Mathstodon instance. Because that instance is about mathematics, they’ve enabled the rendering of LaTeX formulae in their web interface.

However, since I am not on that particular instance, all the toots with equations show up as ugly LaTeX when …

Python tip 20: saving and loading json

learnbyexample — Wed, 07 Dec 2022 02:00:00 GMT

JSON (JavaScript Object Notation) is one of the ways you can store and retrieve data necessary for functioning of an application. For example, my projects Python regex exercises and Linux CLI text processing exercises need to load questions and save user progress. You might wonder why not just a plain text file? I needed dict in the code anyway and JSON offered seamless transition. Also, this arrangement avoided having to write extra code and test it for potential parsing issues.

The json builtin module is handy for such purposes. Here's an example of saving a dict object:

>>> import json
>>> marks = {'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79}
>>> with open('marks.json', 'w') as f:
...     json.dump(marks, f, indent=4)
...

In the above example, indent is used for pretty printing. Here's how the file looks like:

$ cat marks.json
{
    "Rahul": 86,
    "Ravi": 92,
    "Rohit": 75,
    "Rajan": 79
}

And here's an example of loading a JSON file:

>>> with open('marks.json') as f:
...     marks = json.load(f)
... 
>>> marks
{'Rahul': 86, 'Ravi': 92, 'Rohit': 75, 'Rajan': 79}

See docs.python: json for documentation, more examples, other methods, caveats and so on.

Video demo:

See also my 100 Page Python Intro ebook.

Abstraction is Expensive

Speculative Branches — Wed, 07 Dec 2022 02:00:00 GMT

As you build a computer system, little things start to show up: maybe that database query is awkward for the feature you are building, or you find your server getting bogged down transferring gigabytes of data in hexadecimal ASCII, or your app translates itself to Japanese on the fly for hundreds of thousands of separate users. These are places where your abstractions are misaligned - your app would be quantitatively better if it had a better DB schema, a way to transfer binary data, or native internationalization for your Japanese users. Each of these misalignments carries a cost.

First Day in Dragonflight

Ben Overmyer's Site — Mon, 05 Dec 2022 02:00:00 GMT

So after coming home from a business trip, I finally got to play the new World of Warcraft expansion yesterday. It's so much fun! The last time I remember thoroughly enjoying the initial experience of a WoW expansion was Wrath of the Lich King over ten years ago.

The starting area of the Dragon Isles is absolutely gorgeous. That the expansion encourages exploration makes it even better. There are little things to discover all over the place. The new dragonriding mechanic - basically Flight 2.0 - is really enjoyable and dovetails into the exploration. The Dragon Isles are definitely designed with dragonriding in mind, because there are pillars, canyons, towers, and all other manner of interesting obstacles and terrain to fly around, through, and in.

Something that caught me completely by surprise was the changes to professions. They added a level of depth that is on par with Final Fantasy XIV crafting, and may even draw some inspiration from Star Wars: Galaxies in some limited ways. Even the gathering professions got a little bit of depth added to them, though they at least are still not as complex as FFXIV or SWG. It's going to take me awhile to learn the new systems, and I adore that I have to do that. It makes crafting much less boring and much more rewarding.

The story thus far is less cataclysmic than previous expansions' stories. Yes, there's a big bad, and letting them win would be very not-good. But it doesn't feel like I just got done saving the world and now I have to save it again; instead, the villain story is only part of a larger narrative around restoring the dragonflights to their former glory and their former home.

All of this, in one day. I'm really looking forward to playing more.

2022–12–02: Pinephone kernel news and some bits about the keyboard, too

megi's PinePhone Development Log — Fri, 02 Dec 2022 01:00:00 GMT

Is it worth writing about?

Notes on software development — Thu, 01 Dec 2022 02:00:00 GMT

You acquire a skill or experience through time and effort, then downplay the impact of writing and sharing the learning process.

Professionals seem naturally to imagine a high bar for what is worth writing about.

I think that's misguided. This article is not criticism of folks with these beliefs, but rather encouragement for folks looking for a reason to write.

There are (at least) a few concrete reasons to write about what you've learned, even when you don't think it's novel.

To practice writing

This is the easiest reason. While practice does not imply improvement, you cannot improve without practice.

Every time you learn something is a chance to write down both what you've learned and also how you learned it.

For professional developers this chance happens all the time. Daily, really. But most developers, even those who want to write more, let the opportunity slip.

Providing variety

When I learn a topic I normally go through dozens of posts, papers, docs, videos or books to find a version that clicks. If I can. I prefer to start with blog posts and often there are not blog posts on the subject. Books, docs, videos, and academic papers aren't often as accessible.

Even if you're writing about a popular topic, there's still a chance your post gets through to someone in a way other posts do not.

For programmers there are notorious topics you can avoid if you'd like ("What is a monad", "Why is lisp interesting", "Kubernetes sucks"). Or not. I've fallen into those traps.

Additionally, as you gain experience as a programmer (or product manager, or whatever), your perspective and approach becomes both more interesting and more valuable.

I don't recall ever thinking: "I wish they'd write less". But I'm always wishing some folks wrote more, or at all.

Some folks with experience, writing about widely varied topics in software include:

But experience need not be a prerequisite. Experts (who don't practice explaining) easily forget how they came to their current understanding. A beginner's experience is valuable for everyone who is not a beginner, sometimes also for beginners.

To cement understanding

Finally, honest writing forces you to either understand the dark corners of what you've learned or to ask for help in these dark corners.

I have repeatedly wrestled with topics in software only to be further forced to explain why (or how) when I write.

And it has often forced me to restructure code or ideas in ways that are easier to explain. I think that's a pretty valuable act for the long-term.

Bad faith

There's a bad faith argument that you sometimes see. Here's a variation that comes to mind.

The internet is already full of crap. People who aren't experts are just making it worse.

I hope you ignore these comments. :) If there's a quality problem that is genuinely causing harm, that's for search engines and trade organizations to deal with.

In the extreme

Simon Willison's TIL site is the most prolific version of this I've ever seen. I don't know if I personally aspire to Simon's level, but I think it's worth seeing.

Topics

Some topics I think are always worth writing about and sharing:

Your process, failures and successes, to figuring something out
How to hack on some major open source project
In-depth comparison of projects or approaches, down to source code, benchmarks, and architecture when relevant
Building minimal versions of some production system
How some major systems works under the hood, down to the code
Mistakes you made in structuring organizations, or production architecture, or testing, etc.
How to get the dang configuration right for testing Electron apps in Github Actions

For programming posts specifically: I strongly encourage you to include or walk through working code. Have tests. And have the code build process hooked up to GitHub Actions or SourceHut CI or whatever. This helps ensure your work is still relevant over time.

When you write

Write to explain and teach. When you don't understand something, call out that you don't understand it. That's not a bad thing, and the internet is normally happy to help.

Don't shy away from showing code, showing things that broke, showing the ugly process. It's encouraging for others to see.

End goal

Well, ideally we have fewer clickbait "5 best React alternatives" articles and more thoughtful pieces intended to teach and educate with a bit of rigor.

It's better for individuals and for companies. It's better for the internet.

Community

If you want a community of folks where you can find encouragement to write and eyes to review drafts, check out the #writing-and-drafts channel on the Software Internals Discord.

Is it worth writing about?

Well if you come to me I'm almost surely going to say yes. Poor Betteridge.

Wrote a short post as a bit of encouragement to folks who want to write more but imagine a high bar for what's worthwhile.

tldr; if you ask me it's almost always going to be a yes. And I think there's a path toward a higher-quality internet.https://t.co/Nn6BvXhNdZ pic.twitter.com/KELvsxnr2w
— Phil Eaton (@phil_eaton) December 1, 2022

Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

Bytepawn - Marton Trencseni — Wed, 30 Nov 2022 01:00:00 GMT

One of the most common Data Science tasks in a business setting is timeseries forecasting. I was curious what methods and libraries other Data Scientists use, so I posted an "Ask HN" on Hacker News. The post generated 89 comments, most of them high-quality. This is my summary of the discussion.

Vim tip 18: moving within long lines

learnbyexample — Tue, 29 Nov 2022 02:00:00 GMT

Here are Normal mode commands you can use to move within long lines that are spread over multiple screen lines:

g0 move to the beginning of the current screen line
g^ move to the first non-blank character of the current screen line
g$ move to the end of the current screen line
gj move down by one screen line, prefix a count to move down by that many screen lines
gk move up by one screen line, prefix a count to move up by that many screen lines
gm move to the middle of the current screen line
- Note that this is based on the screen width, not the number of characters in the line!
gM move to the middle of the current line
- Note that this is based on the total number of characters in the line

See :h left-right-motions for more details.

Video demo:

What problem are we trying to solve?

CRAIG KERSTIENS — Mon, 28 Nov 2022 21:23:56 GMT

If you want to seem like the smartest person in the room, wait for a break in conversation, after sitting quiet for 15 minutes, and ask “What problem are we trying to solve here?” It works every time.

We’ve all been there. You walk into a meeting, there are 10 people in it. It gets rolling, different folks chiming in. A few people seem to be taking personal notes, they always do. There is the person that scheduled the meeting, but they’re not the person that usually makes decisions. It’s primarily going back and forth between 3 individuals with 7 others watching on. You’re 15 minutes in, and while there has been lively discussion already… you find yourself back on the original point from minute 1 seemingly lost 14 minutes and you’re unsure to what.

A healthier environment ideally has an agenda sent out ahead of time. Based on the agenda there is a clear structure planned for the meeting which presumably points to a clear goal. Even without an agenda an alternative structure would be a clearly stated goal in the invite. Within the meeting you SCQA-it up live. Huge bonus points if the meeting invite includes a link to the notes doc where meeting notes will be transcribed by an explicit scribe for the meeting, sent out after for folks to agree/confirm as a good record of the meeting.

As a facilitator as good as you may be, you’re still going to end up in the first situation. Every time wait 10-15 minutes, then ask the question. It seems to be more effective after you’ve had the detour vs. leading off with it in the first 1 minute. Even better internalize the question to yourself vs. just making yourself look good by asking it (which will happen).

Adding More Images to this Blog

Beetle Space — Sun, 27 Nov 2022 10:00:00 GMT

I suck at art. As such, my blog has been mostly text with few images.

Recently I started playing with Stable Diffusion on my PC, and it occurred to me I can use it to generate some artwork for the blog. As a result, I’ve added images to various …

Fall/Winter 2022 Update… Let the 3d Printing Begin!

MiscDotGeek — Fri, 25 Nov 2022 20:25:24 GMT

Hey everyone, I’m very sorry for the long absence. It’s been a while since I’ve written, and so much has changed. In August, I took my 1988 Suburban, loaded it up with a 12 foot UHaul trailer, and dragged all my belongings from rural Western Washington to rural Western Nevada, about 30 miles from Reno. …

The post Fall/Winter 2022 Update… Let the 3d Printing Begin! appeared first on MiscDotGeek.

Festive deals for books on Python, Linux, JavaScript, Regular Expressions and more

learnbyexample — Fri, 25 Nov 2022 02:00:00 GMT

Hello!

Here are some exciting deals for my programming ebooks as well as from other creators.

My ebooks🔗

Offers valid till 30-Nov-2022:

All 13 Books Bundle — $10 (normal price $28)
Practice Python Projects — FREE (normal price $10)
JavaScript RegExp — FREE (normal price $10)
Learn by example Python bundle — $3 (normal price $15)

Indie creators🔗

Python books by Michael Driscoll — $10 off for all books, 20% off for Teach Me Python
Python Morsels — save up to $108 a year on Python Morsels, until Nov 28 (skill-honing system that helps developers deepen their Python skills)
- see also author's blog post for comprehensive links to other Python deals
Boost Your Django DX and Speed Up Your Django Tests — 50% off (plus further 50% off based on GDP) until Nov 28
- see also author's blog post for comprehensive links to other Django-related deals
Python, Git, and Pandas courses — 40% off
Practical Guide to Technical Blogging is 34% off and Python To Projects bootcamp is 20% off
Complete Guide to CSS Flex and Grid — 60% off on all versions of the eBook (4 days starting from Nov 24)
Explain Ideas Visually — 50% OFF

Miscellaneous🔗

NoStarch Press — Holiday Gift Guide, 35% off until Nov 28
The Pragmatic Bookshelf — 40% off on all ebooks and audio books
Manning Publications — save 50% when you buy 2 or more eBooks, liveProjects, or liveVideos
Leanpub Monthly Sale — offers for programming books, bundles and courses
Real Python Giveaway — a chance to win one of three prizes, until Nov 25
InfoSec Hack Friday — InfoSec related software/tools
The Cyber Plumber's Lab Guide and Interactive Access — 50% OFF
Huge list of awesome deals — tools, productivity, books, courses, etc

Happy learning :)

Do You Trust Your Tax Preparer?

Beetle Space — Wed, 23 Nov 2022 10:00:00 GMT

For most of my life, I’ve done my own taxes - using paper and pen. If you have the standard W-2 income, it’s fairly easy.

Thrice, though, my tax situation got complicated enough that I paid a tax preparer to do the taxes.

First

I had gotten married the …

Regarding social media, web presence, and dumpster fires

Ben Overmyer's Site — Wed, 23 Nov 2022 02:00:00 GMT

So, some rich sociopath bought Twitter and seems hell-bent on destroying it.

I haven't used Twitter in years, except to occasionally check the status of a misbehaving SaaS. So, Twitter's implosion doesn't really bother me. It did prompt me to log back into my Mastodon account and start using it actively. So in that sense, Twitter's implosion has affected me.

Tons of other people have migrated to the Fediverse as a direct result of the Twitter dumpster fire. And by tons, I mean hundreds of thousands. I've never seen Mastodon so active. It's causing no end of strain and hardship for Mastodon server administrators. However, it also is having an effect that I've wanted for years - decentralization of social networks is expanding into the mainstream.

The last time I saw anything like this was when Google decided to murder Google+. G+ had been host to a huge and active tabletop role-playing game community, and when the service died, that community scrambled to find a replacement. Back then, the community seemed to divide between just blogging, using Facebook, and using MeWe. The latter two are centralized services, but the first was decentralized, and I was happy to see more people blogging.

Now, with the mass adoption of Mastodon, it seems the needle is moving even futher into decentralization. At least, this is true for the TTRPG community. I love this. I've been blogging since the early 2000s... maybe since the late 90s, if you count sporadic updates to a static home page.

Hopefully this is a trend and not just a one-off event.

Something that still needs to be addressed, though, is the discovery side of things. Modern search engines like Google and Bing are getting worse and worse at actually finding small websites that don't care about SEO. Why should personal websites care about SEO, anyway? Their focus should be on their own identity. Discovery should be a community problem, rather than an individual's. My own contribution to this is a web directory called RPGGen. Given how enormous the tabletop role-playing game web community has gotten, maybe we need a dedicated search engine that only crawls role-playing game sites. That would be an interesting project.

CLI tip 19: extended globs

learnbyexample — Wed, 23 Nov 2022 02:00:00 GMT

The Bash shell provides extglob option for advanced pattern matching of filenames. These will help you apply regexp like quantifiers, provide alternate patterns and negation. From man bash:

Extended glob	Description
`?(pattern-list)`	Matches zero or one occurrence of the given patterns
`*(pattern-list)`	Matches zero or more occurrences of the given patterns
`+(pattern-list)`	Matches one or more occurrences of the given patterns
`@(pattern-list)`	Matches one of the given patterns
`!(pattern-list)`	Matches anything except one of the given patterns

Extended globs are disabled by default. You can use shopt -s extglob and shopt -u extglob to set and unset this option respectively.

Here are some examples (visit globs.sh to get the script used below).

$ source globs.sh
$ ls
100.sh   f1.txt      f4.txt    hi.sh   math.h         report-02.log
42.txt   f2_old.txt  f7.txt    ip.txt  notes.txt      report-04.log
calc.py  f2.txt      hello.py  main.c  report-00.log  report-98.log

# one or more digits followed by '.' and then zero or more characters
$ ls +([0-9]).*
100.sh  42.txt

# same as: ls *.c *.sh
$ ls *.@(c|sh)
100.sh  hi.sh  main.c

# not ending with '.txt'
$ ls !(*.txt)
100.sh   hello.py  main.c  report-00.log  report-04.log
calc.py  hi.sh     math.h  report-02.log  report-98.log

# not ending with '.txt' or '.log'
$ ls *.!(txt|log)
100.sh  calc.py  hello.py  hi.sh  main.c  math.h

Video demo:

See also my Linux Command Line Computing ebook.

A Programmer-Friendly I/O Abstraction Over io_uring and kqueue

Notes on software development — Wed, 23 Nov 2022 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Gonna use a personal trick/hack? Use it often

CRAIG KERSTIENS — Tue, 22 Nov 2022 18:11:56 GMT

What do I mean by trick/hack? This can be super flexible, but I’ll toss out a few of my own personal ones.

One of the rules of my product management philosophy, is if you’re going to use a tool or trick then use it often. This is in fact one of my favorite interview questions for PMs. If you ever interview with me, be prepared it’s coming. It applies for engineering managers and others in leadership roles as well, but I’ve found especially get for PMs.

Getting a response from team emails

It’s very common to have a team@ alias within your company, whether for the whole team or for smaller teams. As a PM you may need to email sales@ and get feedback of requested features, or feedback on the roadmap. Chances are you’ll send some good time and effort crafting this email, being concise but also giving enough context for them to give you an informed response. If you’re good you’ve fed the email through hemingway app before you sent it and gotten it down to grade level 7 or lower. You send it middle on a Tuesday around 10am so it’s not lost in Monday triage, and not missed on Thursday that aligns to a long holiday weekend. And then… silence. Nothing. Nada. Zip. Zilch.

I’m not really sure the reason for crickets, perhaps at best they assumed a bunch of their peers got back to you privately. At worst they’re apathetic and care strictly on closing their next deal and not giving feedback that helps long term growth. But I’m not here to dissect why. How do you get a response?

Slightly restructure the email to instead of going to the sales@ alias, be targeted to individuals. Pull the emails of everyone on the list, make the opening a generic “Hi,”, and have it go directly to the individuals via a mail merge.

From experience have had CEOs (who I forgot to omit) email me back when they were out on vacation apologizing for slow response (of hours) and that they’d get me feedback before weeks end.

Turn a group email into a love fest

As much as 1:1 emails are great and will solicit a response. Often you need to send out an announcement to a team. “We’ve got a big announcement tomorrow”. You’ll put the same work and effort into that team email as you did in the above, and again the usual response is crickets.

My teams know I do this, and yet it still works. Get 1-2 other leaders (not necessarily management) but morale leaders to know it’s coming and chime in on a reply-all to the team@ list with positive comments and praise. The same applies also to new hire announcements that goes to team@. You’re going to end up with folks not wanting to be left out from piling on. It’ll start to get flow of feedback public and private coming in about what is good on the communication and give you a sense of where it can improve.

What are your hacks?

These are just a few, as I keep writing up some longer form of my personal product philosophies will layer in more of these nuggets. But every good PM I’ve ever worked with has had their own bag of tricks. Again it’s one of my favorite interview questions, I’ve stolen many great tricks from others over the years. But curious, even without needing for us to be in an interview what are yours? @craigkerstiens

Building a custom code search index in Go for searchcode.com

Ben E. C. Boyter — Tue, 22 Nov 2022 02:00:00 GMT

Abstract TL/DR

I present what I belive is a unique index for indexing and searching source code. It copies ideas from Bing bitfunnel implementation to create a very fast, memory efficient trigram index over source code.

searchcode.com is now using a custom built index written by yours truly
It indexes 180-200 million documents and 75 billions lines of code
The index works using bloom filters sharded by unique document trigrams
It borrows some of the core ideas of bitfunnel used in microsoft’s bing
The use of trigrams inside a bloom filter search is as far as I can tell unique
It lowered the index search times from many seconds to ~40ms across searchcode
End user searches still take around 300 ms to process though
It also improved search relevance and reliability
Architecture wise things became simpler, as the new index sits on a single machine not four
It uses caddy as the reverse proxy and redis as a level 2 cache
The index sits entirely in memory on a 16 core 5950x CPU and 128 GB RAM machine
It currently processes close to a million searches every day
I also gave the site a new look and feel which is much better than what it looked like previously
I had an absolute blast learning and building it

What Happened?

On Monday 21st November 2022 I updated the DNS entries for searchcode to point at the new searchcode server running a custom implemented index, ending the reliance on sphinx/manticore in order to power searches. Since being released I have observed it run over 1 million searches a day with a rolling average runtime of ~40 ms for replacement index.

AWS may have broken your Cloudfront API for nearly a month

Tom Forbes — Sun, 20 Nov 2022 23:42:54 GMT

tl;dr : If you rely on the x-forwarded-for header with Cloudfront and have enabled Origin Shield, between October the 10th 2022 and November the 2nd 2022 the value of this header may have been incorrect for a percentage of requests. If your API relies on knowing the clients IP address in any way it ...

Deep Into Technical Debt

The Cranky Developer on Crater Moon Development — Sat, 19 Nov 2022 20:00:00 GMT

Technical debt is anything done today knowing it will make code harder to change later

On the impending loss of Twitter, and the value of social graphs

--Mike-- — Fri, 18 Nov 2022 19:50:47 GMT

Well, with the impending demise of Twitter, it looks like I'll be following Dave Rogers, and the rest of the world, on RSS again, along with everyone else. I've got a list... it's about 250 entries long, that will make the start of my RSS feeds.

Twitter had an immediacy that RSS lacks, having solved the problem of feedback by outsourcing the policing of spam, etc. to a staff that has been let go.

I like seeing what the people I follow are saying, but the idea of retweeting at a click was seductive, but a bit too easy because you never say WHY you retweet someone, you're just blindly sharing an idea without adding any new context.

I can't help but feel that if Aaron Swartz were still with us, he'd have figured out something more powerful, open source, and wonderful to replace Twitter at this time.

In losing Twitter, I lose links to the comments of friends now gone like Michael O'Connor Clarke, and they likely aren't archived anywhere else, except the Internet Archive of his blog.

I've suffered my own health issues, and this loss (or threat) of Twitter just serves to remind me of the things I'm thankful for, friends and family I've shared this journey with.

I've yet to pick out an RSS reader....I'll post here when I do.

[Update] RSSowl seems to be a usable, locally installed and run Desktop OS RSS Reader. It's only quirk is that it needs the 32 bit Java runtime, regardless of your 64 bit OS, etc.

Zsh Configuration and Plugins - Part Two

smcleod.net — Fri, 18 Nov 2022 07:32:43 GMT

My Z-Shell configuration, scripts and hacks

Fred Brooks

The Cranky Developer on Crater Moon Development — Fri, 18 Nov 2022 06:59:33 GMT

Frederick P.Brooks Jr., April 19, 1931 – November 17, 2022

Python tip 19: manipulating string case

learnbyexample — Wed, 16 Nov 2022 02:00:00 GMT

Here are five string methods you can use for changing the case of characters. Word level transformation is determined by consecutive occurrences of alphabets, not limited to separation by whitespace characters.

>>> sentence = 'thIs iS a saMple StrIng'

>>> sentence.capitalize()
'This is a sample string'

>>> sentence.title()
'This Is A Sample String'

>>> sentence.lower()
'this is a sample string'

>>> sentence.upper()
'THIS IS A SAMPLE STRING'

>>> sentence.swapcase()
'THiS Is A SAmPLE sTRiNG'

The string.capwords() method is similar to title() but also allows a specific separator (default is whitespace).

>>> import string
>>> phrase = 'this-IS-a:colon:separated,PHRASE'

# every word is transformed
>>> phrase.title()
'This-Is-A:Colon:Separated,Phrase'

# colon character is used as the text boundary
>>> string.capwords(phrase, ':')
'This-is-a:Colon:Separated,phrase'

Video demo:

See also my 100 Page Python Intro ebook.

Infosys leaked FullAdminAccess AWS keys on PyPi for over a year

Tom Forbes — Wed, 16 Nov 2022 01:00:00 GMT

You can check out their website for a lot of buzwords , but it’s clear from all the stock photos that they take security Very Seriously Indeed ™️. However, from what I’ve found recently, it seems that Infosys use the following Comprehensive Management-Endorsed Proficiently Driven Cybersecurity Strat...

What is the Highfleet Ship Optimizer?

jodavaho.io — Mon, 14 Nov 2022 10:00:00 GMT

Building a highfleet ship optimizer

Me and Altho built, essentially, a cogitator for your dieselpunk legos game.

Highfleet (koshutin.com) is a game by Konstantin Koshutin, of Hammerflight fame. In that game you have to bulid, peice by peice, huge rocket-propelled fleets to fight above a primitive desert society. The attention to detail is immaculate. Radio communications, electronic intelligence / warfare, and strike fleet management are just the tip of the iceburg. It’s great fun, for engineers especially.

What gets most people addicted, however, is the ship editing. You can create and build ships from a suprisingly few number of modules that become highly individualized / specialized. This gif (from koshutin.com) shows the design loop perfectly:

Highfleet ship design from koshutin.com

But seriously, check out that page, the game is beautiful. The most challenging part of building a good ship, is keeping the number of extra modules low, and selecting the appropriate modules to keep the cost low and performance high.

I’ve been working on an optimizer to help. This codebase selects modules from the game Highfleet, such that desired in-game statistics are satisfied. It does so optimally and fast, using state-of-the-art solver library SCIP.

This project is free, and hosted on hfopt.jodavaho.io if you’d like to give it a shot.

Most use cases are one of the following:

I have a number of weapons and sensor that I’d like to include, but don’t know the best set of engines and support modules to optimally include them
I have a scout ship in mind, one that can go a certain speed and travel a certain range with a few sensors.
I’m curious what is the cheapest ship that can accomodate a given weapon or sensor, given a speed, weight, range, or other constraint

As always, please feel free to file issues here with questions, suggestions, or bugs.

I especially need help verifying the stats of the output modules, once combined. The math behind the game is notoriously opaque, and all the stats used were compiled from reddit.com/r/highfleet, esp /u/d0d0b1rd and /u/twiglard.

Computing from the Command Line: sales report

learnbyexample — Mon, 14 Nov 2022 02:00:00 GMT

I've previously written about events and strategies that led to increased ebook sales during the last quarter of 2021.

Very pleased to inform that I continue to see more than expected sales during release week. My 13th ebook Computing from the Command Line was published on November 1st. Here's how the sales looked on Gumroad during the first ten days:

I used to offer my ebooks for free on release. For the past few releases, I have also added heavily discounted ebook bundles which seems to be the major factor in increased paid sales I'm seeing. Luck certainly plays a role too in reaching users through social media. Here are some of the ways I promoted my latest ebook:

Announcement post on Gumroad and sending an email to existing readers (1000+ users opened the email as per Gumroad analytics)
Pinned tweet — more than 300 link clicks as per Twitter analytics
Posting on /r/commandline/, /r/linux/, /r/linux4noobs/ and /r/FreeEBOOKS/
Show HN post on Hacker News — wasn't lucky this time to reach front page
Promo video on youtube
Mentioned in my learnbyexample weekly newsletter
And of course, I wrote a release post on this blog and also mentioned it on my GitHub Readme

Apart from Gumroad, 400+ readers downloaded the ebook from Leanpub and I got a few paid sales as well. I wrote about pros and cons of Gumroad/Leanpub here.

PS: Make sure to read the rules and be a regular user before self-promoting your content on the social media platforms mentioned above.

Writing a SQL database, take two: Zig and RocksDB

Notes on software development — Sun, 13 Nov 2022 02:00:00 GMT

For my second project while learning Zig, I decided to port an old, minimal SQL database project from Go to Zig.

In this post, in ~1700 lines of code (yes, I'm sorry it's bigger than my usual), we'll create a basic embedded SQL database in Zig on top of RocksDB. Other than the RocksDB layer it will not use third-party libraries.

The code for this project is available on GitHub.

Here are a few example interactions we'll support:

$ ./main --database data --script <(echo "CREATE TABLE y (year int, age int, name text)")
echo "CREATE TABLE y (year int, age int, name text)"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (2010, 38, 'Gary')")
echo "INSERT INTO y VALUES (2010, 38, 'Gary')"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (2021, 92, 'Teej')")
echo "INSERT INTO y VALUES (2021, 92, 'Teej')"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (1994, 18, 'Mel')")
echo "INSERT INTO y VALUES (1994, 18, 'Mel')"
ok

# Basic query
$ ./main --database data --script <(echo "SELECT name, age, year FROM y")
echo "SELECT name, age, year FROM y"
| name          |age            |year           |
+ ====          +===            +====           +
| Mel           |18             |1994           |
| Gary          |38             |2010           |
| Teej          |92             |2021           |

# With WHERE
$ ./main --database data --script <(echo "SELECT name, year, age FROM y WHERE age < 40")
echo "SELECT name, year, age FROM y WHERE age < 40"
| name          |year           |age            |
+ ====          +====           +===            +
| Mel           |1994           |18             |
| Gary          |2010           |38             |

# With operations
$ ./main --database data --script <(echo "SELECT 'Name: ' || name, year + 30, age FROM y WHERE age < 40")
echo "SELECT 'Name: ' || name, year + 30, age FROM y WHERE age < 40"
| unknown               |unknown                |age            |
+ =======               +=======                +===            +
| Name: Mel             |2024           |18             |
| Name: Gary            |2040           |38             |

This post is standalone (except for the RocksDB layer which I wrote about here) but it builds on a number of ideas I've explored that you may be interested in:

This project is mostly a port of my SQL database from scratch in Go project, but unlike that series this project will have persistent storage via RocksDB.

And unlike that post, this project is written in Zig!

Let's get started. :)

Components

We're going to split up the project into the following major components:

Lexing
Parsing
Storage
- RocksDB
Execution
Entrypoint (main)

Lexing takes a query and breaks it into an array of tokens.

Parsing takes the lexed array of tokens and pattern matches into a syntax tree (AST).

Storage maps high-level SQL entities like tables and rows into bytes that can be easily stored on disk. And it handles recovering high-level tables and rows from bytes on disk.

Invisible to users of the Storage component is RocksDB, which is how the bytes are actually stored on disk. RocksDB is a persistent store that maps arbitary byte keys to arbitrary byte values. We'll use it for storing and recovering both table metadata and actual row data.

Execution takes a query AST and executes it against Storage, potentially returning result rows.

These terms are a vast simplification of real-world database design. But they are helpful structure to have even in a project this small.

Memory Management

Zig doesn't have a garbage collector. Mitchell Hashimoto wrote bindings to Boehm GC. But Zig also has a builtin Arena allocator which is perfect for this simple project.

The main function will create the arena and pass it to each component, where they can do allocations as they please. At the end of main, the entire arena will be freed at once.

The only other place where we must do manual memory management is in the RocksDB wrapper. But I've already covered that in a separate post.

Zig Specifics

I'm not going to cover the basics of Zig syntax. If you are new to Zig, read this first! (It's short.)

Now that we've got the basic idea, we can start coding!

Types (`types.zig`, 10 LoC)

Let's create a few helper types that we'll use in the rest of the code.

pub const String = []const u8;

pub const Error = String;

pub fn Result(comptime T: type) type {
    return union(enum) {
        val: T,
        err: Error,
    };
}

That's it. :) Makes things a little more readable.

Lexing (`lex.zig`, 308 LoC)

Lexing turns a query string into an array of tokens.

There are a few kinds of tokens we'll define:

Keywords (like CREATE, true, false, null)
- Syntax (commas, parentheses, operators, and all other builtin symbols)
Strings
Integers
Identifiers

And not listed there but important to skip past is whitespace.

Let's turn this into a Zig struct!

const std = @import("std");

const Error = @import("types.zig").Error;
const String = @import("types.zig").String;

pub const Token = struct {
    start: u64,
    end: u64,
    kind: Kind,
    source: String,

    pub const Kind = enum {
        // Keywords
        select_keyword,
        create_table_keyword,
        insert_keyword,
        values_keyword,
        from_keyword,
        where_keyword,

        // Operators
        plus_operator,
        equal_operator,
        lt_operator,
        concat_operator,

        // Other syntax
        left_paren_syntax,
        right_paren_syntax,
        comma_syntax,

        // Literals
        identifier,
        integer,
        string,
    };

    pub fn string(self: Token) String {
        return self.source[self.start..self.end];
    }

Using an enum helps us with type safety. And since we're storing location in the token, we can build a nice debug function for when lexing or parsing fails.

    fn debug(self: Token, msg: String) void {
        var line: usize = 0;
        var column: usize = 0;
        var lineStartIndex: usize = 0;
        var lineEndIndex: usize = 0;
        var i: usize = 0;
        var source = self.source;
        while (i < source.len) {
            if (source[i] == '\n') {
                line = line + 1;
                column = 0;
                lineStartIndex = i;
            } else {
                column = column + 1;
            }

            if (i == self.start) {
                // Find the end of the line
                lineEndIndex = i;
                while (source[lineEndIndex] != '\n') {
                    lineEndIndex = lineEndIndex + 1;
                }
                break;
            }

            i = i + 1;
        }

        std.debug.print(
            "{s}\nNear line {}, column {}.\n{s}\n",
            .{ msg, line + 1, column, source[lineStartIndex..lineEndIndex] },
        );
        while (column - 1 > 0) {
            std.debug.print(" ", .{});
            column = column - 1;
        }
        std.debug.print("^ Near here\n\n", .{});
    }
};

And similarly, let's add a debug helper for when we're dealing with an array of tokens.

pub fn debug(tokens: []Token, preferredIndex: usize, msg: String) void {
    var i = preferredIndex;
    while (i >= tokens.len) {
        i = i - 1;
    }

    tokens[i].debug(msg);
}

Token <> String Mapping

Before we get too far from Token definition, let's define a mapping from the Token.kind enum to strings we can see in a query.

const Builtin = struct {
    name: String,
    kind: Token.Kind,
};

// These must be sorted by length of the name text, descending, for lexKeyword.
var BUILTINS = [_]Builtin{
    .{ .name = "CREATE TABLE", .kind = Token.Kind.create_table_keyword },
    .{ .name = "INSERT INTO", .kind = Token.Kind.insert_keyword },
    .{ .name = "SELECT", .kind = Token.Kind.select_keyword },
    .{ .name = "VALUES", .kind = Token.Kind.values_keyword },
    .{ .name = "WHERE", .kind = Token.Kind.where_keyword },
    .{ .name = "FROM", .kind = Token.Kind.from_keyword },
    .{ .name = "||", .kind = Token.Kind.concat_operator },
    .{ .name = "=", .kind = Token.Kind.equal_operator },
    .{ .name = "+", .kind = Token.Kind.plus_operator },
    .{ .name = "<", .kind = Token.Kind.lt_operator },
    .{ .name = "(", .kind = Token.Kind.left_paren_syntax },
    .{ .name = ")", .kind = Token.Kind.right_paren_syntax },
    .{ .name = ",", .kind = Token.Kind.comma_syntax },
};

We'll use this in a few lexing functions below.

Whitespace

Outside of tokens, we need to be able to skip past whitespace.

fn eatWhitespace(source: String, index: usize) usize {
    var res = index;
    while (source[res] == ' ' or
        source[res] == '\n' or
        source[res] == '\t' or
        source[res] == '\r')
    {
        res = res + 1;
        if (res == source.len) {
            break;
        }
    }

    return res;
}

All lexing functions will look like this. They'll take the source as one argument and a cursor to the current index in the source as another.

Keywords

Let's handle lexing keyword tokens next. Keywords are case insensitive. I don't think there's a builtin case insensitive string comparison function in Zig. So let's write that first.

fn asciiCaseInsensitiveEqual(left: String, right: String) bool {
    var min = left;
    if (right.len < left.len) {
        min = right;
    }

    for (min) |_, i| {
        var l = left[i];
        if (l >= 97 and l <= 122) {
            l = l - 32;
        }

        var r = right[i];
        if (r >= 97 and r <= 122) {
            r = r - 32;
        }

        if (l != r) {
            return false;
        }
    }

    return true;
}

Unfortunately it only supports ASCII for now.

Now we can write a simple longest-matching-substring function. It is simple because the keyword mapping we set up above is already ordered by length descending.

fn lexKeyword(source: String, index: usize) struct { nextPosition: usize, token: ?Token } {
    var longestLen: usize = 0;
    var kind = Token.Kind.select_keyword;
    for (BUILTINS) |builtin| {
        if (index + builtin.name.len >= source.len) {
            continue;
        }

        if (asciiCaseInsensitiveEqual(source[index .. index + builtin.name.len], builtin.name)) {
            longestLen = builtin.name.len;
            kind = builtin.kind;
            // First match is the longest match
            break;
        }
    }

    if (longestLen == 0) {
        return .{ .nextPosition = 0, .token = null };
    }

    return .{
        .nextPosition = index + longestLen,
        .token = Token{
            .source = source,
            .start = index,
            .end = index + longestLen,
            .kind = kind,
        },
    };
}

That's it!

Integers

For integers we read through the source until we stop seeing decimal digits. Obviously this is a subset of what people consider integers, but it will do for now!

fn lexInteger(source: String, index: usize) struct { nextPosition: usize, token: ?Token } {
    var start = index;
    var end = index;
    var i = index;
    while (source[i] >= '0' and source[i] <= '9') {
        end = end + 1;
        i = i + 1;
    }

    if (start == end) {
        return .{ .nextPosition = 0, .token = null };
    }

    return .{
        .nextPosition = end,
        .token = Token{
            .source = source,
            .start = start,
            .end = end,
            .kind = Token.Kind.integer,
        },
    };
}

Strings

Strings are enclosed in single quotes.

fn lexString(source: String, index: usize) struct { nextPosition: usize, token: ?Token } {
    var i = index;
    if (source[i] != '\'') {
        return .{ .nextPosition = 0, .token = null };
    }
    i = i + 1;

    var start = i;
    var end = i;
    while (source[i] != '\'') {
        end = end + 1;
        i = i + 1;
    }

    if (source[i] == '\'') {
        i = i + 1;
    }

    if (start == end) {
        return .{ .nextPosition = 0, .token = null };
    }

    return .{
        .nextPosition = i,
        .token = Token{
            .source = source,
            .start = start,
            .end = end,
            .kind = Token.Kind.string,
        },
    };
}

Identifiers

Identifiers for this project are alphanumeric characters. We could support more by optionally checking for double quote enclosed strings. But I'll leave that as an exercise for the reader.

fn lexIdentifier(source: String, index: usize) struct { nextPosition: usize, token: ?Token } {
    var start = index;
    var end = index;
    var i = index;
    while ((source[i] >= 'a' and source[i] <= 'z') or
        (source[i] >= 'A' and source[i] <= 'Z') or
        (source[i] == '*'))
    {
        end = end + 1;
        i = i + 1;
    }

    if (start == end) {
        return .{ .nextPosition = 0, .token = null };
    }

    return .{
        .nextPosition = end,
        .token = Token{
            .source = source,
            .start = start,
            .end = end,
            .kind = Token.Kind.identifier,
        },
    };
}

`lex`

Now we can pull together all these helper functions in a public entrypoint for lexing.

It will loop through a query string, eating whitespace and checking for tokens. It will continue until it hits the end of the query string. If it ever can't continue it fails.

pub fn lex(source: String, tokens: *std.ArrayList(Token)) ?Error {
    var i: usize = 0;
    while (true) {
        i = eatWhitespace(source, i);
        if (i >= source.len) {
            break;
        }

        const keywordRes = lexKeyword(source, i);
        if (keywordRes.token) |token| {
            tokens.append(token) catch return "Failed to allocate space for keyword token";
            i = keywordRes.nextPosition;
            continue;
        }

        const integerRes = lexInteger(source, i);
        if (integerRes.token) |token| {
            tokens.append(token) catch return "Failed to allocate space for integer token";
            i = integerRes.nextPosition;
            continue;
        }

        const stringRes = lexString(source, i);
        if (stringRes.token) |token| {
            tokens.append(token) catch return "Failed to allocate space for string token";
            i = stringRes.nextPosition;
            continue;
        }

        const identifierRes = lexIdentifier(source, i);
        if (identifierRes.token) |token| {
            tokens.append(token) catch return "Failed to allocate space for identifier token";
            i = identifierRes.nextPosition;
            continue;
        }

        if (tokens.items.len > 0) {
            debug(tokens.items, tokens.items.len - 1, "Last good token.\n");
        }
        return "Bad token";
    }

    return null;
}

That's it for lexing! Now we can do parsing.

Parsing (`parse.zig`, 407 LoC)

Parsing takes an array of tokens from the lexing stage and discovers the tree structure in them that maps to a predefined syntax tree (AST).

If it can't discover a valid tree from the array of tokens, it fails.

Let's set up the basics of the Parser struct:

const std = @import("std");

const lex = @import("lex.zig");
const Result = @import("types.zig").Result;

const Token = lex.Token;

pub const Parser = struct {
    allocator: std.mem.Allocator,

    pub fn init(allocator: std.mem.Allocator) Parser {
        return Parser{ .allocator = allocator };
    }

    fn expectTokenKind(tokens: []Token, index: usize, kind: Token.Kind) bool {
        if (index >= tokens.len) {
            return false;
        }

        return tokens[index].kind == kind;
    }

Expressions

Expressions are at the bottom of the syntax tree.

They can be:

Literals (like strings, integers, booleans, etc.)
Or binary operations

Let's define these in Zig:

    pub const BinaryOperationAST = struct {
        operator: Token,
        left: *ExpressionAST,
        right: *ExpressionAST,

        fn print(self: BinaryOperationAST) void {
            self.left.print();
            std.debug.print(" {s} ", .{self.operator.string()});
            self.right.print();
        }
    };

    pub const ExpressionAST = union(enum) {
        literal: Token,
        binary_operation: BinaryOperationAST,

        fn print(self: ExpressionAST) void {
            switch (self) {
                .literal => |literal| switch (literal.kind) {
                    .string => std.debug.print("'{s}'", .{literal.string()}),
                    else => std.debug.print("{s}", .{literal.string()}),
                },
                .binary_operation => self.binary_operation.print(),
            }
        }
    };

Now we can attempt to parse either of these from an array of tokens.

    fn parseExpression(self: Parser, tokens: []Token, index: usize) Result(struct {
        ast: ExpressionAST,
        nextPosition: usize,
    }) {
        var i = index;

        var e: ExpressionAST = undefined;

        if (expectTokenKind(tokens, i, Token.Kind.integer) or
                expectTokenKind(tokens, i, Token.Kind.identifier) or
                expectTokenKind(tokens, i, Token.Kind.string))
            {
                e = ExpressionAST{ .literal = tokens[i] };
                i = i + 1;
        } else {
                return .{ .err = "No expression" };
        }

        if (expectTokenKind(tokens, i, Token.Kind.equal_operator) or
                expectTokenKind(tokens, i, Token.Kind.lt_operator) or
                expectTokenKind(tokens, i, Token.Kind.plus_operator) or
                expectTokenKind(tokens, i, Token.Kind.concat_operator))
            {
                var newE = ExpressionAST{
                    .binary_operation = BinaryOperationAST{
                        .operator = tokens[i],
                        .left = self.allocator.create(ExpressionAST) catch return .{
                            .err = "Could not allocate for left expression.",
                        },
                        .right = self.allocator.create(ExpressionAST) catch return .{
                            .err = "Could not allocate for right expression.",
                        },
                    },
                };
                newE.binary_operation.left.* = e;
                e = newE;

                switch (self.parseExpression(tokens, i + 1)) {
                    .err => |err| return .{ .err = err },
                    .val => |val| {
                        e.binary_operation.right.* = val.ast;
                        i = val.nextPosition;
                    },
                }
        }

        return .{ .val = .{ .ast = e, .nextPosition = i } };
    }

Basically, we assume it's a literal expression unless we see an operator after it. If there's an operator after it we call parseExpression recursively and return a binary expression.

Important to note: this skips both implicit operator precedence and explicit precedence via parenthesis.

`SELECT`

A SELECT query's structure has a FROM table name, a comma-separated list of expressions, and an optional WHERE section with another expression for the where.

    pub const SelectAST = struct {
        columns: []ExpressionAST,
        from: Token,
        where: ?ExpressionAST,

        fn print(self: SelectAST) void {
            std.debug.print("SELECT\n", .{});
            for (self.columns) |column, i| {
                std.debug.print("  ", .{});
                column.print();
                if (i < self.columns.len - 1) {
                    std.debug.print(",", .{});
                }
                std.debug.print("\n", .{});
            }
            std.debug.print("FROM\n  {s}", .{self.from.string()});

            if (self.where) |where| {
                std.debug.print("\nWHERE\n  ", .{});
                where.print();
            }

            std.debug.print("\n", .{});
        }
    };

To parse it we look for:

SELECT
Then a comma separated list of ExpressionASTs
Then a FROM
Then optionally a WHERE
- And then another ExpressionAST

With the help of expectTokenKind and parseExpression it is not too difficult, but a little verbose, to write.

    fn parseSelect(self: Parser, tokens: []Token) Result(AST) {
        var i: usize = 0;
        if (!expectTokenKind(tokens, i, Token.Kind.select_keyword)) {
            return .{ .err = "Expected SELECT keyword" };
        }
        i = i + 1;

        var columns = std.ArrayList(ExpressionAST).init(self.allocator);
        var select = SelectAST{
            .columns = undefined,
            .from = undefined,
            .where = null,
        };

        // Parse columns
        while (!expectTokenKind(tokens, i, Token.Kind.from_keyword)) {
            if (columns.items.len > 0) {
                if (!expectTokenKind(tokens, i, Token.Kind.comma_syntax)) {
                    lex.debug(tokens, i, "Expected comma.\n");
                    return .{ .err = "Expected comma." };
                }

                i = i + 1;
            }

            switch (self.parseExpression(tokens, i)) {
                .err => |err| return .{ .err = err },
                .val => |val| {
                    i = val.nextPosition;

                    columns.append(val.ast) catch return .{
                        .err = "Could not allocate for token.",
                    };
                },
            }
        }

        if (!expectTokenKind(tokens, i, Token.Kind.from_keyword)) {
            lex.debug(tokens, i, "Expected FROM keyword after this.\n");
            return .{ .err = "Expected FROM keyword" };
        }
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.identifier)) {
            lex.debug(tokens, i, "Expected FROM table name after this.\n");
            return .{ .err = "Expected FROM keyword" };
        }
        select.from = tokens[i];
        i = i + 1;

        if (expectTokenKind(tokens, i, Token.Kind.where_keyword)) {
            // i + 1, skip past the where
            switch (self.parseExpression(tokens, i + 1)) {
                .err => |err| return .{ .err = err },
                .val => |val| {
                    select.where = val.ast;
                    i = val.nextPosition;
                },
            }
        }

        if (i < tokens.len) {
            lex.debug(tokens, i, "Unexpected token.");
            return .{ .err = "Did not complete parsing SELECT" };
        }

        select.columns = columns.items;
        return .{ .val = AST{ .select = select } };
    }

That's it!

`CREATE TABLE`

A CREATE TABLE query's structure has a table name and a list of comma separated identifier pairs for column name and kind.

    const CreateTableColumnAST = struct {
        name: Token,
        kind: Token,
    };

    pub const CreateTableAST = struct {
        table: Token,
        columns: []CreateTableColumnAST,

        fn print(self: CreateTableAST) void {
            std.debug.print("CREATE TABLE {s} (\n", .{self.table.string()});
            for (self.columns) |column, i| {
                std.debug.print(
                    "  {s} {s}",
                    .{ column.name.string(), column.kind.string() },
                );
                if (i < self.columns.len - 1) {
                    std.debug.print(",", .{});
                }
                std.debug.print("\n", .{});
            }
            std.debug.print(")\n", .{});
        }
    };

To parse it we look for:

CREATE TABLE
Followed by an identifier (the table name)
Followed by open parenthesis
Followed by a comma separated list of identifier pairs
Followed by close parenthesis

    fn parseCreateTable(self: Parser, tokens: []Token) Result(AST) {
        var i: usize = 0;
        if (!expectTokenKind(tokens, i, Token.Kind.create_table_keyword)) {
            return .{ .err = "Expected CREATE TABLE keyword" };
        }
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.identifier)) {
            lex.debug(tokens, i, "Expected table name after CREATE TABLE keyword.\n");
            return .{ .err = "Expected CREATE TABLE name" };
        }

        var columns = std.ArrayList(CreateTableColumnAST).init(self.allocator);
        var create_table = CreateTableAST{
            .columns = undefined,
            .table = tokens[i],
        };
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.left_paren_syntax)) {
            lex.debug(tokens, i, "Expected opening paren after CREATE TABLE name.\n");
            return .{ .err = "Expected opening paren" };
        }
        i = i + 1;

        while (!expectTokenKind(tokens, i, Token.Kind.right_paren_syntax)) {
            if (columns.items.len > 0) {
                if (!expectTokenKind(tokens, i, Token.Kind.comma_syntax)) {
                    lex.debug(tokens, i, "Expected comma.\n");
                    return .{ .err = "Expected comma." };
                }

                i = i + 1;
            }

            var column = CreateTableColumnAST{ .name = undefined, .kind = undefined };
            if (!expectTokenKind(tokens, i, Token.Kind.identifier)) {
                lex.debug(tokens, i, "Expected column name after comma.\n");
                return .{ .err = "Expected identifier." };
            }

            column.name = tokens[i];
            i = i + 1;

            if (!expectTokenKind(tokens, i, Token.Kind.identifier)) {
                lex.debug(tokens, i, "Expected column type after column name.\n");
                return .{ .err = "Expected identifier." };
            }

            column.kind = tokens[i];
            i = i + 1;

            columns.append(column) catch return .{
                .err = "Could not allocate for column.",
            };
        }

        // Skip past final paren.
        i = i + 1;

        if (i < tokens.len) {
            lex.debug(tokens, i, "Unexpected token.");
            return .{ .err = "Did not complete parsing CREATE TABLE" };
        }

        create_table.columns = columns.items;
        return .{ .val = AST{ .create_table = create_table } };
    }

`INSERT INTO`

And last we've got INSERT INTO. This tree has table name and a list of expressions to insert into the table.

    pub const InsertAST = struct {
        table: Token,
        values: []ExpressionAST,

        fn print(self: InsertAST) void {
            std.debug.print("INSERT INTO {s} VALUES (", .{self.table.string()});
            for (self.values) |value, i| {
                value.print();
                if (i < self.values.len - 1) {
                    std.debug.print(", ", .{});
                }
            }
            std.debug.print(")\n", .{});
        }
    };

We parse it by looking for:

INSERT INTO
Followed by a table name
Followed by VALUES
Followed by open parenthesis
Followed by a comma-separated list of expressions
Followed by a close parenthesis

    fn parseInsert(self: Parser, tokens: []Token) Result(AST) {
        var i: usize = 0;
        if (!expectTokenKind(tokens, i, Token.Kind.insert_keyword)) {
            return .{ .err = "Expected INSERT INTO keyword" };
        }
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.identifier)) {
            lex.debug(tokens, i, "Expected table name after INSERT INTO keyword.\n");
            return .{ .err = "Expected INSERT INTO table name" };
        }

        var values = std.ArrayList(ExpressionAST).init(self.allocator);
        var insert = InsertAST{
            .values = undefined,
            .table = tokens[i],
        };
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.values_keyword)) {
            lex.debug(tokens, i, "Expected VALUES keyword.\n");
            return .{ .err = "Expected VALUES keyword" };
        }
        i = i + 1;

        if (!expectTokenKind(tokens, i, Token.Kind.left_paren_syntax)) {
            lex.debug(tokens, i, "Expected opening paren after CREATE TABLE name.\n");
            return .{ .err = "Expected opening paren" };
        }
        i = i + 1;

        while (!expectTokenKind(tokens, i, Token.Kind.right_paren_syntax)) {
            if (values.items.len > 0) {
                if (!expectTokenKind(tokens, i, Token.Kind.comma_syntax)) {
                    lex.debug(tokens, i, "Expected comma.\n");
                    return .{ .err = "Expected comma." };
                }

                i = i + 1;
            }

            switch (self.parseExpression(tokens, i)) {
                .err => |err| return .{ .err = err },
                .val => |val| {
                    values.append(val.ast) catch return .{
                        .err = "Could not allocate for expression.",
                    };
                    i = val.nextPosition;
                },
            }
        }

        // Skip past final paren.
        i = i + 1;

        if (i < tokens.len) {
            lex.debug(tokens, i, "Unexpected token.");
            return .{ .err = "Did not complete parsing INSERT INTO" };
        }

        insert.values = values.items;
        return .{ .val = AST{ .insert = insert } };
    }

`AST`

Finally we can define the top-level SQL AST as being the union of the above three query types.

    pub const AST = union(enum) {
        select: SelectAST,
        insert: InsertAST,
        create_table: CreateTableAST,

        pub fn print(self: AST) void {
            switch (self) {
                .select => |select| select.print(),
                .insert => |insert| insert.print(),
                .create_table => |create_table| create_table.print(),
            }
        }
    };

And we can implement parse by switching on the current token.

    pub fn parse(self: Parser, tokens: []Token) Result(AST) {
        if (expectTokenKind(tokens, 0, Token.Kind.select_keyword)) {
            return switch (self.parseSelect(tokens)) {
                .err => |err| .{ .err = err },
                .val => |val| .{ .val = val },
            };
        }

        if (expectTokenKind(tokens, 0, Token.Kind.create_table_keyword)) {
            return switch (self.parseCreateTable(tokens)) {
                .err => |err| .{ .err = err },
                .val => |val| .{ .val = val },
            };
        }

        if (expectTokenKind(tokens, 0, Token.Kind.insert_keyword)) {
            return switch (self.parseInsert(tokens)) {
                .err => |err| .{ .err = err },
                .val => |val| .{ .val = val },
            };
        }

        return .{ .err = "Unknown statement" };
    }
};

Perfect. For today. :)

Storage (`storage.zig`, 338 LoC)

Next we're going to switch contexts completely and think about how tables and rows will get serialized into bytes that can be stored on disk.

The storage layer will define a few general helpers for correctly serializing and deserializing strings and numbers:

const std = @import("std");

const RocksDB = @import("rocksdb.zig").RocksDB;
const Error = @import("types.zig").Error;
const Result = @import("types.zig").Result;
const String = @import("types.zig").String;

pub fn serializeInteger(comptime T: type, buf: *std.ArrayList(u8), i: T) !void {
    var length: [@sizeOf(T)]u8 = undefined;
    std.mem.writeIntBig(T, &length, i);
    try buf.appendSlice(length[0..8]);
}

pub fn deserializeInteger(comptime T: type, buf: String) T {
    return std.mem.readIntBig(T, buf[0..@sizeOf(T)]);
}

pub fn serializeBytes(buf: *std.ArrayList(u8), bytes: String) !void {
    try serializeInteger(u64, buf, bytes.len);
    try buf.appendSlice(bytes);
}

pub fn deserializeBytes(bytes: String) struct {
    offset: usize,
    bytes: String,
} {
    var length = deserializeInteger(u64, bytes);
    var offset = length + 8;
    return .{ .offset = offset, .bytes = bytes[8..offset] };
}

Then we'll define the Storage struct itself. Under the hood it will use RocksDB to store and recover data on disk.

pub const Storage = struct {
    db: RocksDB,
    allocator: std.mem.Allocator,

    pub fn init(allocator: std.mem.Allocator, db: RocksDB) Storage {
        return Storage{
            .db = db,
            .allocator = allocator,
        };
    }

Now let's think about storage entities.

Values

The fundamental unit in the database is a value, or cell. It can be either a boolean, an integer, a string, or null.

    pub const Value = union(enum) {
        bool_value: bool,
        null_value: bool,
        string_value: String,
        integer_value: i64,

        pub const TRUE = Value{ .bool_value = true };
        pub const FALSE = Value{ .bool_value = false };
        pub const NULL = Value{ .null_value = true };

Since all values are strings in the original query, we'll provide a fromIntegerString that we can use to convert.

        pub fn fromIntegerString(iBytes: String) Value {
            const i = std.fmt.parseInt(i64, iBytes, 10) catch return Value{
                .integer_value = 0,
            };
            return Value{ .integer_value = i };
        }

Next we'll define functions to cast values to boolean.

        pub fn asBool(self: Value) bool {
            return switch (self) {
                .null_value => false,
                .bool_value => |value| value,
                .string_value => |value| value.len > 0,
                .integer_value => |value| value != 0,
            };
        }

To strings.

        pub fn asString(self: Value, buf: *std.ArrayList(u8)) !void {
            try switch (self) {
                .null_value => _ = 1, // Do nothing
                .bool_value => |value| buf.appendSlice(if (value) "true" else "false"),
                .string_value => |value| buf.appendSlice(value),
                .integer_value => |value| buf.writer().print("{d}", .{value}),
            };
        }

And to integers.

        pub fn asInteger(self: Value) i64 {
            return switch (self) {
                .null_value => 0,
                .bool_value => |value| if (value) 1 else 0,
                .string_value => |value| fromIntegerString(value).integer_value,
                .integer_value => |value| value,
            };
        }

And finally the storage layer's core concern: serialization...

        pub fn serialize(self: Value, buf: *std.ArrayList(u8)) String {
            switch (self) {
                .null_value => buf.append('0') catch return "",

                .bool_value => |value| {
                    buf.append('1') catch return "";
                    buf.append(if (value) '1' else '0') catch return "";
                },

                .string_value => |value| {
                    buf.append('2') catch return "";
                    buf.appendSlice(value) catch return "";
                },

                .integer_value => |value| {
                    buf.append('3') catch return "";
                    serializeInteger(i64, buf, value) catch return "";
                },
            }

            return buf.items;
        }

And deserialization.

        pub fn deserialize(data: String) Value {
            return switch (data[0]) {
                '0' => Value.NULL,
                '1' => Value{ .bool_value = data[1] == '1' },
                '2' => Value{ .string_value = data[1..] },
                '3' => Value{ .integer_value = deserializeInteger(i64, data[1..]) },
                else => unreachable,
            };
        }
    };

We use a simple, space-inefficient scheme for encoding/decoding to bytes that can be written to disk.

Rows

Now that we've got values, we can define rows in terms of values. And we can provide a few helper functions for getting cells by field name.

    pub const Row = struct {
        allocator: std.mem.Allocator,
        cells: std.ArrayList(String),
        fields: []String,

        pub fn init(allocator: std.mem.Allocator, fields: []String) Row {
            return Row{
                .allocator = allocator,
                .cells = std.ArrayList(String).init(allocator),
                .fields = fields,
            };
        }

        pub fn append(self: *Row, cell: Value) !void {
            var cellBuffer = std.ArrayList(u8).init(self.allocator);
            try self.cells.append(cell.serialize(&cellBuffer));
        }

        pub fn appendBytes(self: *Row, cell: String) !void {
            try self.cells.append(cell);
        }

        pub fn get(self: Row, field: String) Value {
            for (self.fields) |f, i| {
                if (std.mem.eql(u8, field, f)) {
                    // Results are internal buffer views. So make a copy.
                    var copy = std.ArrayList(u8).init(self.allocator);
                    copy.appendSlice(self.cells.items[i]) catch return Storage.Value.NULL;
                    return Storage.Value.deserialize(copy.items);
                }
            }

            return Value.NULL;
        }

        pub fn items(self: Row) []String {
            return self.cells.items;
        }

        fn reset(self: *Row) void {
            self.cells.clearRetainingCapacity();
        }
    };

Since values are serialized with length prefixes, we can serialize a row by concatenating all the values together.

Since we must map to keys and values for RocksDB, we give each row a key prefix that is the table name. And then we give it a random suffix to distinguish it from other rows in the table. A more intelligent design would use the table's primary key as the suffix but we don't support primary keys yet. (See also, the section on "Mapping SQL to key-value storage" in What's the big deal about key-value databases like FoundationDB and RocksDB?.)

    fn generateId() ![]u8 {
        const file = try std.fs.cwd().openFileZ("/dev/random", .{});
        defer file.close();

        var buf: [16]u8 = .{};
        _ = try file.read(&buf);
        return buf[0..];
    }

    pub fn writeRow(self: Storage, table: String, row: Row) ?Error {
        // Table name prefix
        var key = std.ArrayList(u8).init(self.allocator);
        key.writer().print("row_{s}_", .{table}) catch return "Could not allocate row key";

        // Unique row id
        var id = generateId() catch return "Could not generate id";
        key.appendSlice(id) catch return "Could not allocate for id";

        var value = std.ArrayList(u8).init(self.allocator);
        for (row.cells.items) |cell| {
            serializeBytes(&value, cell) catch return "Could not allocate for cell";
        }

        return self.db.set(key.items, value.items);
    }

RowIter

Reading rows will be slightly different from writing rows since reading rows will use an iterator. We will wrap the RocksDB iterator so the consumer of Storage only needs to deal with Rows and Values.

    pub const RowIter = struct {
        row: Row,
        iter: RocksDB.Iter,

        fn init(allocator: std.mem.Allocator, iter: RocksDB.Iter, fields: []String) RowIter {
            return RowIter{
                .iter = iter,
                .row = Row.init(allocator, fields),
            };
        }

        pub fn next(self: *RowIter) ?Row {
            var rowBytes: String = undefined;
            if (self.iter.next()) |b| {
                rowBytes = b.value;
            } else {
                return null;
            }

            self.row.reset();
            var offset: usize = 0;
            while (offset < rowBytes.len) {
                var d = deserializeBytes(rowBytes[offset..]);
                offset += d.offset;
                self.row.appendBytes(d.bytes) catch return null;
            }

            return self.row;
        }

        pub fn close(self: RowIter) void {
            self.iter.close();
        }
    };

It does the opposite of what writeRow did in terms of deserializing cells one after another. Again, this works because each cell is length-prefixed.

Next we must provide the interface for actually getting a RowIter. The only condition for the RowIter at the moment is that it contains all rows in the table.

Since we wrote each row with a table name prefix, we can recover it by iterating over all rows with that prefix.

    pub fn getRowIter(self: Storage, table: String) Result(RowIter) {
        var rowPrefix = std.ArrayList(u8).init(self.allocator);
        rowPrefix.writer().print("row_{s}_", .{table}) catch return .{
            .err = "Could not allocate for row prefix",
        };

        var iter = switch (self.db.iter(rowPrefix.items)) {
            .err => |err| return .{ .err = err },
            .val => |it| it,
        };

        var tableInfo = switch (self.getTable(table)) {
            .err => |err| return .{ .err = err },
            .val => |t| t,
        };

        return .{
            .val = RowIter.init(self.allocator, iter, tableInfo.columns),
        };
    }

Tables

Finally we've got tables. We must store table metadata: its name, columns and column types.

    pub const Table = struct {
        name: String,
        columns: []String,
        types: []String,
    };

We will use a tbl_ prefix instead of row_ prefix for table metadata. But we'll otherwise encode with the same length-prefixed concatentations.

    pub fn writeTable(self: Storage, table: Table) ?Error {
        // Table name prefix
        var key = std.ArrayList(u8).init(self.allocator);
        key.writer().print("tbl_{s}_", .{table.name}) catch return "Could not allocate key for table";

        var value = std.ArrayList(u8).init(self.allocator);
        for (table.columns) |column, i| {
            serializeBytes(&value, column) catch return "Could not allocate for column";
            serializeBytes(&value, table.types[i]) catch return "Could not allocate for column type";
        }

        return self.db.set(key.items, value.items);
    }

And the opposite for decoding.

    pub fn getTable(self: Storage, name: String) Result(Table) {
        var tableKey = std.ArrayList(u8).init(self.allocator);
        tableKey.writer().print("tbl_{s}_", .{name}) catch return .{
            .err = "Could not allocate for table prefix",
        };

        var columns = std.ArrayList(String).init(self.allocator);
        var types = std.ArrayList(String).init(self.allocator);
        var table = Table{
            .name = name,
            .columns = undefined,
            .types = undefined,
        };
        // First grab table info
        var columnInfo = switch (self.db.get(tableKey.items)) {
            .err => |err| return .{ .err = err },
            .val => |val| val,
            .not_found => return .{ .err = "No such table" },
        };

        var columnOffset: usize = 0;
        while (columnOffset < columnInfo.len) {
            var column = deserializeBytes(columnInfo[columnOffset..]);
            columnOffset += column.offset;
            columns.append(column.bytes) catch return .{
                .err = "Could not allocate for column name.",
            };

            var kind = deserializeBytes(columnInfo[columnOffset..]);
            columnOffset += kind.offset;
            types.append(kind.bytes) catch return .{
                .err = "Could not allocate for column kind.",
            };
        }

        table.columns = columns.items;
        table.types = types.items;

        return .{ .val = table };
    }
};

And that's it for storage! Again, we're building on top of the RocksDB layer I already wrote about. If you want to see how that works, go for it!

If you just want the rocksdb.zig file, grab it from here.

Execute (`execute.zig`, 210 LoC)

Now that we've got a storage layer and an AST from our parser, we can execute the query on top of the storage!

A better implementation might translate the AST to bytecode and implement a bytecode interpreter for expression evaluation. But we'll build a tree-walking interpreter instead.

const std = @import("std");

const Parser = @import("parse.zig").Parser;
const RocksDB = @import("rocksdb.zig").RocksDB;
const Storage = @import("storage.zig").Storage;
const Result = @import("types.zig").Result;
const String = @import("types.zig").String;

pub const Executor = struct {
    allocator: std.mem.Allocator,
    storage: Storage,

    pub fn init(allocator: std.mem.Allocator, storage: Storage) Executor {
        return Executor{ .allocator = allocator, .storage = storage };
    }

In general we'll make query responses optional. They can be empty or they can be an array of an array of strings (rows and cells) and an array of strings (column names).

    const QueryResponse = struct {
        fields: []String,
        // Array of cells (which is an array of serde (which is an array of u8))
        rows: [][]String,
        empty: bool,
    };
    const QueryResponseResult = Result(QueryResponse);

Expressions

For execution we start again at the bottom with expressions. There are literals.

    fn executeExpression(self: Executor, e: Parser.ExpressionAST, row: Storage.Row) Storage.Value {
        return switch (e) {
            .literal => |lit| switch (lit.kind) {
                .string => Storage.Value{ .string_value = lit.string() },
                .integer => Storage.Value.fromIntegerString(lit.string()),
                .identifier => row.get(lit.string()),
                else => unreachable,
            },

And there are a handful of binary operations.

            .binary_operation => |bin_op| {
                var left = self.executeExpression(bin_op.left.*, row);
                var right = self.executeExpression(bin_op.right.*, row);

                if (bin_op.operator.kind == .equal_operator) {
                    // Cast dissimilar types to serde
                    if (@enumToInt(left) != @enumToInt(right)) {
                        var leftBuf = std.ArrayList(u8).init(self.allocator);
                        left.asString(&leftBuf) catch unreachable;
                        left = Storage.Value{ .string_value = leftBuf.items };

                        var rightBuf = std.ArrayList(u8).init(self.allocator);
                        right.asString(&rightBuf) catch unreachable;
                        right = Storage.Value{ .string_value = rightBuf.items };
                    }

                    return Storage.Value{
                        .bool_value = switch (left) {
                            .null_value => true,
                            .bool_value => |v| v == right.asBool(),
                            .string_value => blk: {
                                var leftBuf = std.ArrayList(u8).init(self.allocator);
                                left.asString(&leftBuf) catch unreachable;

                                var rightBuf = std.ArrayList(u8).init(self.allocator);
                                right.asString(&rightBuf) catch unreachable;

                                break :blk std.mem.eql(u8, leftBuf.items, rightBuf.items);
                            },
                            .integer_value => left.asInteger() == right.asInteger(),
                        },
                    };
                }

                if (bin_op.operator.kind == .concat_operator) {
                    var copy = std.ArrayList(u8).init(self.allocator);
                    left.asString(&copy) catch unreachable;
                    right.asString(&copy) catch unreachable;
                    return Storage.Value{ .string_value = copy.items };
                }

                return switch (bin_op.operator.kind) {
                    .lt_operator => if (left.asInteger() < right.asInteger()) Storage.Value.TRUE else Storage.Value.FALSE,
                    .plus_operator => Storage.Value{ .integer_value = left.asInteger() + right.asInteger() },
                    else => Storage.Value.NULL,
                };
            },
        };
    }

`SELECT`

To execute a SELECT query we first validate the requested table and requested fields.

    fn executeSelect(self: Executor, s: Parser.SelectAST) QueryResponseResult {
        switch (self.storage.getTable(s.from.string())) {
            .err => |err| return .{ .err = err },
            else => _ = 1,
        }

        // Now validate and store requested fields
        var requestedFields = std.ArrayList(String).init(self.allocator);
        for (s.columns) |requestedColumn| {
            var fieldName = switch (requestedColumn) {
                .literal => |lit| switch (lit.kind) {
                    .identifier => lit.string(),
                    // TODO: give reasonable names
                    else => "unknown",
                },
                // TODO: give reasonable names
                else => "unknown",
            };
            requestedFields.append(fieldName) catch return .{
                .err = "Could not allocate for requested field.",
            };
        }

Then grab an iterator for rows in the table.

        var rows = std.ArrayList([]String).init(self.allocator);
        var response = QueryResponse{
            .fields = requestedFields.items,
            .rows = undefined,
            .empty = false,
        };

        var iter = switch (self.storage.getRowIter(s.from.string())) {
            .err => |err| return .{ .err = err },
            .val => |it| it,
        };
        defer iter.close();

And finally we iterate through all rows and add rows to the response if there is no WHERE condition or if we evaluate the WHERE condition successfully.

When we add rows to the response, we need to actually evaluate the expression for each column in the SELECT AST.

        while (iter.next()) |row| {
            var add = false;
            if (s.where) |where| {
                if (self.executeExpression(where, row).asBool()) {
                    add = true;
                }
            } else {
                add = true;
            }

            if (add) {
                var requested = std.ArrayList(String).init(self.allocator);
                for (s.columns) |exp| {
                    var val = self.executeExpression(exp, row);
                    var valBuf = std.ArrayList(u8).init(self.allocator);
                    val.asString(&valBuf) catch unreachable;
                    requested.append(valBuf.items) catch return .{
                        .err = "Could not allocate for requested cell",
                    };
                }
                rows.append(requested.items) catch return .{
                    .err = "Could not allocate for row",
                };
            }
        }

        response.rows = rows.items;
        return .{ .val = response };
    }

`INSERT INTO`

Inserting is pretty simple, we just evaluate the VALUES passed and write them to storage.

    fn executeInsert(self: Executor, i: Parser.InsertAST) QueryResponseResult {
        var emptyRow = Storage.Row.init(self.allocator, undefined);
        var row = Storage.Row.init(self.allocator, undefined);
        for (i.values) |v| {
            var exp = self.executeExpression(v, emptyRow);
            row.append(exp) catch return .{ .err = "Could not allocate for cell" };
        }

        if (self.storage.writeRow(i.table.string(), row)) |err| {
            return .{ .err = err };
        }

        return .{
            .val = .{ .fields = undefined, .rows = undefined, .empty = true },
        };
    }

`CREATE TABLE`

Similarly to INSERT INTO, but without any expression evaluation, we map the CreateTableAST to Storage entities and write them to storage.

    fn executeCreateTable(self: Executor, c: Parser.CreateTableAST) QueryResponseResult {
        var columns = std.ArrayList(String).init(self.allocator);
        var types = std.ArrayList(String).init(self.allocator);

        for (c.columns) |column| {
            columns.append(column.name.string()) catch return .{
                .err = "Could not allocate for column name",
            };
            types.append(column.kind.string()) catch return .{
                .err = "Could not allocate for column kind",
            };
        }

        var table = Storage.Table{
            .name = c.table.string(),
            .columns = columns.items,
            .types = types.items,
        };

        if (self.storage.writeTable(table)) |err| {
            return .{ .err = err };
        }
        return .{
            .val = .{ .fields = undefined, .rows = undefined, .empty = true },
        };
    }

For both CREATE TABLE and INSERT INTO there is more validation we could do. Exercise for the reader and whatnot. :)

`execute`

Finally we can switch on the AST and call the appropriate execution function.

    pub fn execute(self: Executor, ast: Parser.AST) QueryResponseResult {
        return switch (ast) {
            .select => |select| switch (self.executeSelect(select)) {
                .val => |val| .{ .val = val },
                .err => |err| .{ .err = err },
            },
            .insert => |insert| switch (self.executeInsert(insert)) {
                .val => |val| .{ .val = val },
                .err => |err| .{ .err = err },
            },
            .create_table => |createTable| switch (self.executeCreateTable(createTable)) {
                .val => |val| .{ .val = val },
                .err => |err| .{ .err = err },
            },
        };
    }
};

And now we're ready to put it all together in main!

`main` (`main.zig`, 144 LoC)

First we set up our arena allocator.

const std = @import("std");

const RocksDB = @import("rocksdb.zig").RocksDB;
const lex = @import("lex.zig");
const parse = @import("parse.zig");
const execute = @import("execute.zig");
const Storage = @import("storage.zig").Storage;

pub fn main() !void {
    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();

Then we parse CLI arguments. Importantly we need to grab a location on disk for RocksDB to store data. And we need a query to execute.

    var debugTokens = false;
    var debugAST = false;
    var args = std.process.args();
    var scriptArg: usize = 0;
    var databaseArg: usize = 0;
    var i: usize = 0;
    while (args.next()) |arg| {
        if (std.mem.eql(u8, arg, "--debug-tokens")) {
            debugTokens = true;
        }

        if (std.mem.eql(u8, arg, "--debug-ast")) {
            debugAST = true;
        }

        if (std.mem.eql(u8, arg, "--database")) {
            databaseArg = i + 1;
            i += 1;
            _ = args.next();
        }

        if (std.mem.eql(u8, arg, "--script")) {
            scriptArg = i + 1;
            i += 1;
            _ = args.next();
        }

        i += 1;
    }

    if (databaseArg == 0) {
        std.debug.print("--database is a required flag. Should be a directory for data.\n", .{});
        return;
    }

    if (scriptArg == 0) {
        std.debug.print("--script is a required flag. Should be a file containing SQL.\n", .{});
        return;
    }

Next we read the file passed for the query.

    const file = try std.fs.cwd().openFileZ(std.os.argv[scriptArg], .{});
    defer file.close();

    const file_size = try file.getEndPos();
    var prog = try allocator.alloc(u8, file_size);

    _ = try file.read(prog);

And pass the query to the lexer.

    var tokens = std.ArrayList(lex.Token).init(allocator);
    const lexErr = lex.lex(prog, &tokens);
    if (lexErr) |err| {
        std.debug.print("Failed to lex: {s}", .{err});
        return;
    }

    if (debugTokens) {
        for (tokens.items) |token| {
            std.debug.print("Token: {s}\n", .{token.string()});
        }
    }

    if (tokens.items.len == 0) {
        std.debug.print("Program is empty", .{});
        return;
    }

Pass the tokens to the parser.

    const parser = parse.Parser.init(allocator);
    var ast: parse.Parser.AST = undefined;
    switch (parser.parse(tokens.items)) {
        .err => |err| {
            std.debug.print("Failed to parse: {s}", .{err});
            return;
        },
        .val => |val| ast = val,
    }

    if (debugAST) {
        ast.print();
    }

Initialize storage.

    var db: RocksDB = undefined;
    var dataDirectory = std.mem.span(std.os.argv[databaseArg]);
    switch (RocksDB.open(allocator, dataDirectory)) {
        .err => |err| {
            std.debug.print("Failed to open database: {s}", .{err});
            return;
        },
        .val => |val| db = val,
    }
    defer db.close();

    const storage = Storage.init(allocator, db);

And execute and print results. :)

    const executor = execute.Executor.init(allocator, storage);
    switch (executor.execute(ast)) {
        .err => |err| {
            std.debug.print("Failed to execute: {s}", .{err});
            return;
        },
        .val => |val| {
            if (val.rows.len == 0) {
                std.debug.print("ok\n", .{});
                return;
            }

            std.debug.print("| ", .{});
            for (val.fields) |field| {
                std.debug.print("{s}\t\t|", .{field});
            }
            std.debug.print("\n", .{});
            std.debug.print("+ ", .{});
            for (val.fields) |field| {
                var fieldLen = field.len;
                while (fieldLen > 0) {
                    std.debug.print("=", .{});
                    fieldLen -= 1;
                }
                std.debug.print("\t\t+", .{});
            }
            std.debug.print("\n", .{});

            for (val.rows) |row| {
                std.debug.print("| ", .{});
                for (row) |cell| {
                    std.debug.print("{s}\t\t|", .{cell});
                }
                std.debug.print("\n", .{});
            }
        },
    }
}

build.zig

Finally, finally, tie it all together with build.zig.

const version = @import("builtin").zig_version;
const std = @import("std");

pub fn build(b: *std.build.Builder) void {
    const exe = b.addExecutable("main", "main.zig");
    exe.linkLibC();
    exe.linkSystemLibraryName("rocksdb");

    if (@hasDecl(@TypeOf(exe.*), "addLibraryPath")) {
        exe.addLibraryPath("./rocksdb");
        exe.addIncludePath("./rocksdb/include");
    } else {
        exe.addLibPath("./rocksdb");
        exe.addIncludeDir("./rocksdb/include");
    }

    exe.setOutputDir(".");

    if (exe.target.isDarwin()) {
        exe.addRPath(".");
    }

    exe.install();
}

Grab RocksDB, build it, and build our CLI.

$ git clone https://github.com/facebook/rocksdb
$ ( cd rocksdb && make shared_lib -j8 )

# ONLY IF YOU ARE ON A MAC
$ cp rocksdb/*.dylib . # ONLY IF YOU ARE ON A MAC
# DONE ONLY IF YOU ARE ON A MAC

$ zig build

And give it a go. :)

$ ./main --database data --script <(echo "CREATE TABLE y (year int, age int, name text)")
echo "CREATE TABLE y (year int, age int, name text)"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (2010, 38, 'Gary')")
echo "INSERT INTO y VALUES (2010, 38, 'Gary')"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (2021, 92, 'Teej')")
echo "INSERT INTO y VALUES (2021, 92, 'Teej')"
ok
$ ./main --database data --script <(echo "INSERT INTO y VALUES (1994, 18, 'Mel')")
echo "INSERT INTO y VALUES (1994, 18, 'Mel')"
ok

# Basic query
$ ./main --database data --script <(echo "SELECT name, age, year FROM y")
echo "SELECT name, age, year FROM y"
| name          |age            |year           |
+ ====          +===            +====           +
| Mel           |18             |1994           |
| Gary          |38             |2010           |
| Teej          |92             |2021           |

# With WHERE
$ ./main --database data --script <(echo "SELECT name, year, age FROM y WHERE age < 40")
echo "SELECT name, year, age FROM y WHERE age < 40"
| name          |year           |age            |
+ ====          +====           +===            +
| Mel           |1994           |18             |
| Gary          |2010           |38             |

# With operations
$ ./main --database data --script <(echo "SELECT 'Name: ' || name, year + 30, age FROM y WHERE age < 40")
echo "SELECT 'Name: ' || name, year + 30, age FROM y WHERE age < 40"
| unknown               |unknown                |age            |
+ =======               +=======                +===            +
| Name: Mel             |2024           |18             |
| Name: Gary            |2040           |38             |

From Here

As mentioned, this project is a vast simplification and there are plenty of bugs and subpar design choices. But hopefully it helps to make database development feel a little less intimidating!

If you liked this, here are some other things you might want to check out!

And of course, other posts on this blog. :)

Lastly, a few resources that helped me out while hacking on this:

Zig Documentation
Browsing the source code (and tests!!) of standard library data structures
Zig Programming Language Discord's #zig-help channel
- Friendly and helpful crowd :)

Spent a month hacking on it and happy to finally have this post out.

Let's build a basic SQL database in Zig on top of RocksDB. 😃https://t.co/fkSnaEKsya pic.twitter.com/adfpMvvvOn
— Phil Eaton (@phil_eaton) November 14, 2022

Blackbird: A reference architecture for local-first connected mobile apps

metablog — Sat, 12 Nov 2022 11:51:45 GMT

Wow, what a mouthful! Although this architecture has featured in a number of my other writings, I haven't really described it in detail by itself. Which is a shame, because I think it works really well and is quite simple, a case of Sophisticated Simplicity.

Why a reference architecture?

The motivation for creating and now presenting this reference architecture is that the way we build connected mobile apps is broken, and none of the proposed solutions appear to help. How are they broken? They are overly complex, require way too much code, perform poorly and are unreliable.

Very broadly speaking, these problems can be traced to the misuse of procedural abstraction for a problem-space that is broadly state-based, and can be solved by adapting a state-based architectural style such as in-process REST and combining it with well-known styles such as MVC.

More specifically, MVC has been misapplied by combining UI updates with the model updates, a practice that becomes especially egregious with asynchronous call-backs. In addition, data is pushed to the UI, rather than having the UI pull data when and as needed. Asynchronous code is modelled using call/return and call-backs, leading to call-back hell, needless and arduous transformation of any dependent code into asynchronous code (see "what color is your function") that is also much harder to read, discouraging appropriate abstractions.

Backend communication is also an issue, with newer async/await implementations not really being much of an improvement over callback-based ones, and arguably worse in terms of actual readability. (They seem readable, but what actually happens is different enough that the simplicity is deceptive).

Overview

The overall architecture has four fundamental components:

The model
The UI
The backend
The persistence

The main objective of the architecture is to keep these components in sync with each other, so the whole thing somewhat resembles a control loop architecture: something disturbs the system, for example the user did something in the UI, and the system responds by re-establishing equilibrium.

The model is the central component, it connects/coordinates all the pieces and is also the only one directly connected to more than one piece. In keeping with hexagonal architecture, the model is also supposed to be the only place with significant logic, the remainder of the system should be as minimal, transparent and dumb as possible.

memory-model := persistence.
persistence  |= memory-model.
ui          =|= memory-model. 
backend     =|= memory-model.

Graphically:

Elements

Blackbird depends crucially on a number of architectural elements: first are stores of the in-process REST architectural style. These can be thought of as in-process HTTP servers (without the HTTP, of course) or composable dictionaries. The core store protocol implements the GET, PUT and DELETE verbs as messages.

The role of URLs in REST is taken by Polymorphic Identifiers. These are objects that can reference identify values in the store, but are not direct pointers. For example, they need to be a able to reference objects that aren't there yet.

Polymorphic Identifiers can be application-specific, for example they might consist just of a numeric id,

MVC

For me, the key part of the MVC architectural style is the decoupling of input processing and resultant output processing. That is, under MVC, the view (or a controller) make some change to the model and then processing stops. At some undefined later time (could be synchronous, but does not have to be) the Model informs the UI that it has changed using some kind of notification mechanism.

In Smalltalk MVC, this is a dependents list maintained in the model that interested views register with. All these views are then sent a #changed message when the model has changed. In Cocoa, this can be accomplished using NSNotificationCenter, but really any kind of broadcast mechanism will do.

It is then the views' responsibility to update themselves by interrogating the model.

For views, Cocoa largely automates this: on receipt of the notification, the view just needs invalidate itself, the system then automatically schedules it for redrawing the next time through the event loop.

The reason the decoupling is important to maintain is that the update notification can come for any other reason, including a different user interaction, a backend request completing or even some sort of notification or push event coming in remotely.

With the decoupled M-V update mechanism, all these different kinds of events are handled identically, and thus the UI only ever needs to deal with the local model. The UI is therefore almost entirely decoupled from network communications, we thus have a local-first application that is also largely testable locally.

Blackbird refines the MVC view update mechanism by adding the polymorphic identifier of the modified item in question and placing those PIs in a queue. The queue decouples model and view even more than in the basic MVC model, for example it become fairly trivial to make the queue writable from any thread, but empty only onto the main thread for view updates. In addition, providing update notifications is no longer synchronous, the updater just writes an entry into the queue and can then continue, it doesn't wait for the UI to finish its update.

Decoupling via a queue in this way is almost sufficient for making sure that high-speed model updates don't overwhelm the UI or slow down the model. Both these performance problems are fairly rampant, as an example of the first, the Microsoft Office installer saturates both CPUs on a dual core machine just painting its progress bar, because it massively overdraws.

An example of the second was one of the real performance puzzlers of my career: an installer that was extremely slow, despite both CPU and disk being mostly idle. The problem turned out to be that the developers of that installer not only insisted on displaying every single file name the installer was writing (bad enough), but also flushing the window to screen to make sure the user got a chance to see it (worse). This then interacted with a behavior of Apple's CoreGraphics, which disallows screen flushes at a rate greater than the screen refresh rate, and will simply throttle such requests. You really want to decouple your UI from your model updates and let the UI process updates at its pace.

Having polymorphic identifiers in the queue makes it possible for the UI to catch up on its own terms, and also to remove updates that are no longer relevant, for example discarding duplicate updates of the same element.

The polymorphic identifier can also be used by views in order to determine whether they need to update themselves, by matching against the polymorphic identifier they are currently handling.

Backend communication

Almost every REST backend communication code I have seen in mobile applications has created "convenient" cover methods for every operation of every endpoint accessed by the application, possibly automatically generated.

This ignores the fact that REST only has a few verbs, combined with a great number of identifiers (URLs). In Blackbird, there is a single channel for backend communication: a queue that takes a polymorphic identifier and an http verb. The polymorphic identifier is translated to a URL of the target backend system, the resulting request executed and when the result returns it is placed in the central store using the provided polymorphic identifier.

After the item has been stored, an MVC notification with the polymorphic identifier in question is enqueued as per above.

The queue for backend operations is essentially the same one we described for model-view communication above, for example also with the ability to deduplicate requests correctly so only the final version of an object gets sent if there are multiple updates. The remainder of the processing is performed in pipes-and-filters architectural style using polymorphic write streams.

If the backend needs to communicate with the client, it can send URLs via a socket or other mechanism that tells the client to pull that data via its normal request channels, implementing the same pull-constraint as in the rest of the system.

One aspect of this part of the architecture is that backend requests are reified and explicit, rather than implicitly encoded on the call-stack and its potentially asynchronous continuations. This means it is straightforward for the UI to give the user appropriate feedback for communication failures on the slow or disrupted network connections that are the norm on mobile networks, as well as avoid accidental duplicate requests.

Despite this extra visibility and introspection, the code required to implement backend communications is drastically reduced. Last not least, the code is isolated: network code can operate independently of the UI just as well as the UI can operate independently of the network code.

Persistence

Persistence is handled by stacked stores (storage combinators).

The application is hooked up to the top of the storage stack, the CachingStore, which looks to the application exactly like the DictStore (an in-memory store). If a read request cannot be found in the cache, the data is instead read from disk, converted from JSON by a mapping store.

For testing the rest of the app (rather than the storage stack), it is perfectly fine to just use the in-memory store instead of the disk store, as it has the same interface and behaves the same, except being faster and non-persistent.

Writes use the same asynchronous queues as the rest of the system, with the writer getting the polymorphic identifiers of objects to write and then retrieving the relevant object(s) from the in-memory store before persisting. Since they use the same mechanism, they also benefit from the same uniquing properties, so when the I/O subsystem gets overloaded it will adapt by dropping redundant writes.

Consequences

With the Blackbird reference architecture, we not only replace complex, bulky code with much less and much simpler code, we also get to reuse that same code in all parts of the system while making the pieces of the system highly independent of each other and optimising performance.

In addition, the combination of REST-like stores that can be composed with constraint- and event-based communication patterns makes the architecture highly decoupled. In essence it allows the kind of decoupling we see in well-implemented microservices architectures, but on mobile apps without having to run multiple processes (which is often not allowed).

The World Is Run By People No Smarter Than You

swyx's site RSS Feed — Fri, 11 Nov 2022 17:38:07 GMT

This post was written as a reflection at the first Dev Writers Retreat. It's been really weird doing this in the PermaParty city while the world seemingly falls apart outside. Here's my attempt to make sense of it.

Vim tip 17: setting options

learnbyexample — Tue, 08 Nov 2022 02:00:00 GMT

From :h options.txt:

Vim has a number of internal variables and switches which can be set to achieve special effects. These options come in three forms:

boolean can only be on or off

number has a numeric value

string has a string value

Here are examples for each of these forms:

:set cursorline highlight the line containing the cursor
:set history=200 increase default history from 50 to 200
:set ww+=[,] allow left and right arrow keys to move across lines in Insert mode
- += allows you to append to an existing string value

Usage guidelines:

set {option} switch on the given boolean setting
- :set expandtab use spaces for tab expansion
set {option}! toggle the given boolean setting
- :set expandtab! if previously tabs were expanded, it will be turned off and vice versa
- set inv{option} can also be used
set no{option} switch off the given boolean setting
- :set noexpandtab disable expanding tab to spaces
set {option}? get the current value of the given option (works for all three forms)
- :set expandtab? output will be expandtab or noexpandtab depending on whether it is switched on or off
set {option} get the current value of number or string option
- for example, try :set history or :set ww

See :h options.txt for complete list of usage guidelines and available options.

Video demo:

The best ideas come AFK

Weblog on marginalia.nu — Mon, 07 Nov 2022 02:00:00 GMT

I get my best ideas when I’m not working.

This seems paradoxical, but past a point, the more I work on a project the slower it seems to go. I’ll find changes to do, but lose any sort of vision.

If I’m not programming at all, I rarely get good ideas as well.

There appears to be some magic stoichiometric mixture where I work on a project for a while, then force myself to take a break somewhere far away from any keyword for a day or two, the ideas start to roll in at a pace where I can barely keep up to write them down.

Elite speak GUID script

Ben E. C. Boyter — Mon, 07 Nov 2022 02:00:00 GMT

Some time ago I wrote about creating leet speak guid’s as a way of having fun while working with sitecore. https://boyter.org/2014/05/unique-guid/

I recently realised I wanted to have this around so I could create more interesting guids for one reason or another, but couldn’t find the script I was using back then. So I created it again, and its now on github as a gist for myself and anyone else to use https://gist.github.com/boyter/6f5b7c1b375e79449d01916f5828e1db feel free to use it as you wish!

Github Not-So-Reusable Actions

smcleod.net — Sun, 06 Nov 2022 06:52:27 GMT

Github Actions Reusable Workflows vs Composite Actions

YAML Anchors and Aliases

smcleod.net — Sun, 06 Nov 2022 06:52:27 GMT

Using YAML Anchors and Aliases to make config files more DRY

Zsh Configuration and Plugins - Part One

smcleod.net — Sun, 06 Nov 2022 01:32:43 GMT

My Z-Shell configuration, scripts and hacks

Return to Blizzard

Ben Overmyer's Site — Fri, 04 Nov 2022 02:00:00 GMT

A few weeks ago, I made the decision to rescind my ban on playing Blizzard games. In part this was because Overwatch 2 was imminent, and I missed the particular style of gameplay that only Overwatch had. But it was also because I know people who work at Blizzard, and boycotting Blizzard games appears to be wholly ineffective at assisting change.

This unlocking of the gate meant several things. I could play Overwatch 1 for one more day before the launch of the "sequel." Overwatch 2 completely replaced the original game, so the first one is gone now. I also needed to preorder the collector's edition of the next WoW expansion if I could. I own every expansion collector's edition - though not the CE for the original game, sadly. That, to my surprise, I accomplished easily by ordering through Blizzard's online store. I wonder if demand is down further than I expected. Finally, I installed the Diablo 2 remake and played it for several hours. I had preordered it prior to my ban, but the ban went into effect before its release.

Now, I've played Overwatch 2 for many hours, and I'm levelling my main in WoW to get ready for the release of the Dragonflight expansion later this month. I have until November 15 to reach level 60, and I'm level 58 now. Should be doable unless I get distracted by other things.

CLI tip 18: inserting file contents using GNU sed

learnbyexample — Wed, 02 Nov 2022 02:00:00 GMT

The r command accepts a filename as argument and when the address is satisfied, entire contents of the given file is added after the matching line. This is a robust way to add multiline text literally.

$ cat ip.txt
    * sky
    * apple
$ cat fav_colors.txt
deep red
yellow
reddish
brown

# space between r and filename is optional
# adds entire contents of 'ip.txt' after each line containing 'red'
$ sed '/red/r ip.txt' fav_colors.txt
deep red
    * sky
    * apple
yellow
reddish
    * sky
    * apple
brown

The e flag is the easiest way to insert file contents before the matching lines. Similar to the r command, the output of an external command (cat in the below example) is inserted literally.

$ sed '/red/e cat ip.txt' fav_colors.txt
    * sky
    * apple
deep red
yellow
    * sky
    * apple
reddish
brown

See Adding content from file chapter from my GNU sed ebook for many more examples, gotchas, details about the R command and so on.

Video demo:

See also my CLI text processing with GNU sed ebook.

How to add Tailwind 3 to Docusaurus 2 in 2022

swyx's site RSS Feed — Tue, 01 Nov 2022 04:35:34 GMT

We use Docusaurus at work, and while it shipped v2 this year it still has (as of v2.3) not shipped with any Tailwind support at all. Googled and found this post which was almost everything I needed, but required some stuff in the comments for it to work.

Encourage Your Peers To Contribute To Open Source

smcleod.net — Sun, 30 Oct 2022 09:00:00 GMT

Contributing to Open Source is important to the quality and maintainability of the software and engineering communities we rely on every day - so why is it that so many developers/engineers never participate?

A minimal RocksDB example with Zig

Notes on software development — Sun, 30 Oct 2022 03:00:00 GMT

I mostly programmed in Go the last few years. So every time I wanted an embedded key-value database, I reached for Cockroach's Pebble.

Pebble is great for Go programming but Go does not embed well into other languages. Pebble was inspired by RocksDB (and its predecessor, LevelDB). Both were written in C++ which can more easily be embedded into any language with a C foreign function interface. Pebble also has some interesting limitations that RocksDB does not, transactions for example.

So I've been wanting to get familiar with RocksDB. And I've been learning Zig, so I set out to write a simple Zig program that embeds RocksDB. (If you see weird things in my Zig code and have suggestions, send me a note!)

This post is going to be a mix of RocksDB explanations and Zig explanations. By the end we'll have a simple CLI over a durable store that is able to set keys, get keys, and list all key-value pairs (optionally filtered on a key prefix).

$ ./kv set x 1
$ ./kv get x
1
$ ./kv set y 22
$ ./kv list x
x = 1
$ ./kv list y
y = 22
$ ./kv list
x = 1
y = 22

Basic stuff!

You can find the code for this post in the rocksdb.zig file on Github. To simplify things, this code is only going to work on Linux. And it will require Zig 0.10.x.

RocksDB

RocksDB is written in C++. But most languages cannot interface with C++. (Zig cannot either, as far as I understand). So most C++ libraries expose a C API that is easier for other programming languages to interact with. RocksDB does this. Great!

Now RocksDB's C++ documentation is phenomenal, especially among C++ libraries. But if there is documentation for the C API, I couldn't find it. Instead you must trawl through the C header file, the C wrapper implementation, and the C tests.

There was also a great gist showing a minimal RocksDB C example. But it didn't cover the iterator API for fetching a range of keys with a prefix. But with the C tests file I was able to figure it out, I think.

Let's dig in!

Creating, opening and closing a RocksDB database

First we need to import the C header so that Zig can compile-time verify the foreign functions we call. We'll also import the standard library that we'll use later.

Aside from build.zig below, all code should be in main.zig.

const std = @import("std");

const rdb = @cImport(@cInclude("rocksdb/c.h"));

Don't read anything into the `@` other than that this is a compiler builtin. It's used for imports, casting, and other metaprogramming.

Now we can build our wrapper. It will be a Zig struct that contains a pointer to the RocksDB instance.

const RocksDB = struct {
    db: *rdb.rocksdb_t,

To open a database we'll call rocksdb_open() with a directory name for RocksDB to store data. And we'll tell RocksDB to create the database if it doesn't already exist.

    fn open(dir: []const u8) struct { val: ?RocksDB, err: ?[]u8 } {
        var options: ?*rdb.rocksdb_options_t = rdb.rocksdb_options_create();
        rdb.rocksdb_options_set_create_if_missing(options, 1);
        var err: ?[*:0]u8 = null;
        var db: ?*rdb.rocksdb_t = rdb.rocksdb_open(options, dir.ptr, &err);
        if (err != null) {
            return .{ .val = null, .err = std.mem.span(err) };
        }
        return .{ .val = RocksDB{ .db = db.? }, .err = null };
    }

Finally, we close with rocksdb_close():

    fn close(self: RocksDB) void {
        rdb.rocksdb_close(self.db);
    }

The RocksDB aspect of this is easy. But there's a bunch of Zig-specific details I should (try to) explain.

Return types

Zig has a cool error type. try/catch in Zig work only with this error type and subsets of it you can create. error is an enum. But Zig errors are not ML-style tagged unions (yet?). That is, you cannot both return an error and some dynamic information about the error. So the usefulness of error is limited. It mostly only works if the errors are a finite set without dynamic aspects.

Zig also doesn't have multiple return values. But it does have optional types (denoted with ?) and it has anonymous structs.

So we can do a slightly less safe, but more informational, error type by returning a struct with an optional success value and an optional error.

That's how we get the return type struct { val: ?RocksDB, err: ?[]u8 }.

This is not very different from Go, certainly no less safe, and I'm probably biased to use this as a Go programmer.

Felix Queißner points out to me that there are tagged unions in Zig that would be more safe here. Instead of struct { val: ?RocksDB, err: ?[]u8 } I could do union(enum) { val: RocksDB, err: []u8 }. When I get a chance to play with that syntax I'll modify this post.

Optional pointers

The next thing you may notice is ?*rdb.rocksdb_options_t and ?*rdb.rocksdb_t. This is to work with Zig's type system. Zig expects that pointers are not null. By adding ? we are telling Zig that this value can be null. That way the Zig type system will force us to handle the null condition if we try to access fields on the value.

In the options case, it doesn't really matter if the result is null or not. In the database case, we handle null-ness it by checking the error value if (err) |errStr|. If this condition is not met, we know the database is not null. So we use db.? to assert and return a value that, in the type system, is not null.

Zig strings, C strings

Another thing you may notice is var err: ?[*:0]u8 = null;. Zig strings are expressed as byte arrays or byte slices. []u8 and []const u8 are slices that keep track of the number of items. [*:0]u8 is not a byte slice. It has no length and is only null-delimited. To go from the null-delimited array that the C API returns to the []u8 (slice that contains length) in our function's return signature we use std.mem.span.

This StackOverflow post was useful for understanding this.

Structs

Anonymous structs in Zig are prefixed with a .. And all struct fields, anonymous or not, are prefixed with ..

So .{.x = 1} instantiates an anonymous struct that has one field x.

Struct fields in Zig cannot not be instantiated, even if they are nullable. And when you initialize a nullable value you don't need to wrap it in a Some() like you might do in an ML.

One thing I found surprising about Zig anonymous structs is that instances of the anonymous type are created per function and two anonymous structs that are structurally identical but referenced in different functions are not actually type-equal.

So this doesn't compile:

$ cat test.zig
fn doA() struct { y: u8 } {
  return .{.y = 1};
}

fn doB() struct { y: u8 } {
  return doA();
}

pub fn main() !void {
  _ = doB();
}
$ zig build-exe test.zig
test.zig:5:15: error: expected type 'test.doB__struct_2890', found 'test.doA__struct_3878'
    return doA();
           ~~~^~
test.zig:1:10: note: struct declared here
fn doA() struct { y: u8 } {
         ^~~~~~~~~~~~~~~~
test.zig:4:10: note: struct declared here
fn doB() struct { y: u8 } {
         ^~~~~~~~~~~~~~~~
test.zig:4:10: note: function return type declared here
fn doB() struct { y: u8 } {
         ^~~~~~~~~~~~~~~~
referenced by:
    main: test.zig:8:9
    callMain: /whatever/lib/std/start.zig:606:32
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

You would need to instantiate a new anonymous struct in the second function.

$ cat test.zig
fn doA() struct { y: u8 } {
  return .{.y = 1};
}

fn doB() struct { y: u8 } {
  return .{ .y = doA().y };
}

pub fn main() !void {
  _ = doB();
}

Uniform function call syntax

Zig seems to support something like uniform function call syntax where you can either call a function with arguments or you can omit the first argument by prefixing the function call with firstargument.. I.e. x.add(y) and add(x, y).

In the case of this code it would be RocksDB.close(db) vs db.close() assuming db is an instance of the RocksDB struct.

Like Python, the use of self as the name of this first parameter of a struct's methods is purely convention. You can call it whatever.

The point is that we always expect the user to var db = RocksDB.open() for open() and allow the user to do db.close() for close().

Let's move on!

Setting a key-value pair

We set a pair by calling rocksdb_put with the database instance, some options (we'll leave to defaults), and the key and value strings as C strings.

    fn set(self: RocksDB, key: [:0]const u8, value: [:0]const u8) ?[]u8 {
        var writeOptions = rdb.rocksdb_writeoptions_create();
        var err: ?[*:0]u8 = null;
        rdb.rocksdb_put(
            self.db,
            writeOptions,
            key.ptr,
            key.len,
            value.ptr,
            value.len,
            &err,
        );
        if (err) |errStr| {
            return std.mem.span(errStr);
        }

        return null;
    }

The only special Zig thing is there is key.ptr to satisfy the Zig / C type system. The type signature key: [:0]const u8 and value: [:0]const u8 makes sure that the user passes in a null-delimited byte slice, which is what the RocksDB API expects.

Getting a value from a key

We set a pair by calling rocksdb_get with the database instance, some options (we'll again leave to defaults), and the key as a C string.

    fn get(self: RocksDB, key: [:0]const u8) struct { val: ?[]u8, err: ?[]u8 } {
        var readOptions = rdb.rocksdb_readoptions_create();
        var valueLength: usize = 0;
        var err: ?[*:0]u8 = null;
        var v = rdb.rocksdb_get(
            self.db,
            readOptions,
            key.ptr,
            key.len,
            &valueLength,
            &err,
        );
        if (err) |errStr| {
            return .{ .val = null, .err = std.mem.span(errStr) };
        }
        if (v == 0) {
            return .{ .val = null, .err = null };
        }

        return .{ .val = v[0..valueLength], .err = null };
    }

One thing in there to call out is that we can go from a null-delimited value v to a standard Zig slice []u8 by slicing from 0 to the length of the value returned by the C API.

Also, rocksdb_get is only used for getting a single key-value pair. We'll handle key-value pair iteration next.

Iterating over key-value pairs

The basic structure of RocksDB's iterator API is that you first create an iterator instance with rocksdb_create_iterator(). Then you either rocksdb_iter_seek_to_first() or rocksdb_iter_seek() (with a prefix) to get the iterator ready. Then you get the current iterator entry's key with rocksdb_iter_key() and value with rocksdb_iter_value(). You move on to the next entry in the iterator with rocksdb_iter_next() and check that the current iterator value is valid with rocksdb_iter_valid(). When the iterator is no longer valid, or if you want to stop iterating early, you call rocksdb_iter_destroy().

But we'd like to present a Zig-only interface to users of the RocksDB Zig struct. So we'll create a RocksDB.iter() function that returns a RocksDB.Iter with an RocksDB.Iter.next() function that will return an optional RocksDB.IterEntry.

We'll start backwards with that RocksDB.Iter struct.

`RocksDB.Iter`

Each iterator instance will store a pointer to a RocksDB iterator instance. It will store the prefix requested (which is allowed to be an empty string). If the prefix is set though, we'll only iterate while the iterator key has the requested prefix.

    const IterEntry = struct {
        key: []const u8,
        value: []const u8,
    };

    const Iter = struct {
        iter: *rdb.rocksdb_iterator_t,
        first: bool,
        prefix: []const u8,

        fn next(self: *Iter) ?IterEntry {
            if (!self.first) {
                rdb.rocksdb_iter_next(self.iter);
            }

            self.first = false;
            if (rdb.rocksdb_iter_valid(self.iter) != 1) {
                return null;
            }

            var keySize: usize = 0;
            var key = rdb.rocksdb_iter_key(self.iter, &keySize);

            // Make sure key is still within the prefix
            if (self.prefix.len > 0) {
                if (self.prefix.len > keySize or
                    !std.mem.eql(u8, key[0..self.prefix.len], self.prefix))
                {
                    return null;
                }
            }

            var valueSize: usize = 0;
            var value = rdb.rocksdb_iter_value(self.iter, &valueSize);

            return IterEntry{
                .key = key[0..keySize],
                .value = value[0..valueSize],
            };
        }

Finally we'll wrap the rocksdb_iter_destroy() method:

        fn close(self: Iter) void {
            rdb.rocksdb_iter_destroy(self.iter);
        }
    };

`RocksDB.iter()`

Now we can write the function that creates the RocksDB.Iter. As previously mentioned we must first instantiate the RocksDB iterator and then seek to either the first entry if the user doesn't request a prefix. Or if the user requests a prefix, we seek until that prefix.

fn iter(self: RocksDB, prefix: [:0]const u8) struct { val: ?Iter, err: ?[]const u8 } {
        var readOptions = rdb.rocksdb_readoptions_create();
        var it = Iter{
            .iter = undefined,
            .first = true,
            .prefix = prefix,
        };
        if (rdb.rocksdb_create_iterator(self.db, readOptions)) |i| {
            it.iter = i;
        } else {
            return .{ .val = null, .err = "Could not create iterator" };
        }

        if (prefix.len > 0) {
            rdb.rocksdb_iter_seek(
                it.iter,
                prefix.ptr,
                prefix.len,
            );
        } else {
            rdb.rocksdb_iter_seek_to_first(it.iter);
        }
        return .{ .val = it, .err = null };
    }
};

And now we're done a basic Zig wrapper for the RocksDB API!

`main`

Next we write a simple command-line entrypoint that uses the RocksDB wrapper we built. This is not the prettiest code but it gets the job done.

pub fn main() !void {
    var openRes = RocksDB.open("/tmp/db");
    if (openRes.err) |err| {
        std.debug.print("Failed to open: {s}.\n", .{err});
    }
    var db = openRes.val.?;
    defer db.close();

    var args = std.process.args();
    _ = args.next();
    var key: [:0]const u8 = "";
    var value: [:0]const u8 = "";
    var command = "get";
    while (args.next()) |arg| {
        if (std.mem.eql(u8, arg, "set")) {
            command = "set";
            key = args.next().?;
            value = args.next().?;
        } else if (std.mem.eql(u8, arg, "get")) {
            command = "get";
            key = args.next().?;
        } else if (std.mem.eql(u8, arg, "list")) {
            command = "lst";
            if (args.next()) |argNext| {
                key = argNext;
            }
        } else {
            std.debug.print("Must specify command (get, set, or list). Got: '{s}'.\n", .{arg});
            return;
        }
    }

    if (std.mem.eql(u8, command, "set")) {
        var setErr = db.set(key, value);
        if (setErr) |err| {
            std.debug.print("Error setting key: {s}.\n", .{err});
            return;
        }
    } else if (std.mem.eql(u8, command, "get")) {
        var getRes = db.get(key);
        if (getRes.err) |err| {
            std.debug.print("Error getting key: {s}.\n", .{err});
            return;
        }

        if (getRes.val) |v| {
            std.debug.print("{s}\n", .{v});
        } else {
            std.debug.print("Key not found.\n", .{});
        }
    } else {
        var prefix = key;
        var iterRes = db.iter(prefix);
        if (iterRes.err) |err| {
            std.debug.print("Error getting iterator: {s}.\n", .{err});
        }
        var iter = iterRes.val.?;
        defer iter.close();
        while (iter.next()) |entry| {
            std.debug.print("{s} = {s}\n", .{ entry.key, entry.value });
        }
    }
}

Notably, the main function must be marked pub. The struct and struct methods we wrote would need to be marked pub if we wanted them accessible from other files. But since this is a single file, pub doesn't matter. Except for main.

Now we can get into building.

Building

First we need to compile the RocksDB library. To do this we simply git clone RocksDB and run make shared_libs.

Compiling RocksDB

$ git clone https://github.com/facebook/rocksdb
$ ( cd rocksdb && make shared_lib -j8 )

This may take a while, sorry.

`build.zig`

Next we need to write a build.zig script that tells Zig about this external library. This was one of the harder parts of the process, but building and linking against foreign libraries is almost always hard.

$ cat build.zig
const version = @import("builtin").zig_version;
const std = @import("std");

pub fn build(b: *std.build.Builder) void {
    const exe = b.addExecutable("main", "main.zig");
    exe.linkLibC();
    exe.linkSystemLibraryName("rocksdb");

    exe.addLibraryPath("./rocksdb");
    exe.addIncludePath("./rocksdb/include");

    exe.setOutputDir(".");
    exe.install();
}

Felix Queißner's zig build explained series was quite helpful.

Now we just:

$ zig build

And run!

$ ./main list
$ ./main set x 12
$ ./main set xy 300
$ ./main list
x = 12
xy = 300
$ ./main get xy
300
$ ./main list xy
xy = 300

Not bad!

I wrote a new post on using RocksDB with Zig! There weren't a lot of good examples of the C API and it was good practice for learning Zig.

Also sets me up for integrating it in a (WIP) port of my toy SQL database from Go to Zig. (This time with storage!)https://t.co/zquojV974G pic.twitter.com/gtAsB6Wrhi
— Phil Eaton (@phil_eaton) October 31, 2022

Carbon Dating HTML

Weblog on marginalia.nu — Thu, 27 Oct 2022 03:00:00 GMT

One of the more common feature requests I’ve gotten for Marginalia Search is the ability to search by date. I’ve been a bit reluctant because this has the smell of a a surprisingly hard problem. Or rather, a surprisingly large number of easy problems.

The initial hurdle we’ll encounter is that among structured data, pubDate in available in RDFa, OpenGraph, JSON+LD, and Microdata.

A few examples:

<meta property="datePublished" content="2022-08-24" />
<meta itemprop="datePublished" content="2022-08-24" />
<meta property="article:published_time" content="2022-08-24T14:39:14Z" />
<script type="application/ld+json">
{"datePublished":"2022-08-24T14:39:14Z"}
</script>

So far not so that bad. This is at least a case where the web site tells you that here is the pub-date, the exact format of the date may vary, but this is solvable.

Contemplating Randomness

Speculative Branches — Thu, 27 Oct 2022 03:00:00 GMT

I have recently been immersed in the theory and practice of random number generation while working on Arbitrand, a new high-quality true random number generation service hosted in AWS. Because of that, I am starting a sequence of blog posts on randomness and random number generators. This post is the first of the sequence, and focuses what random number generators are and how to test them.

Formally, random number generators are systems that produce a stream of bits (or numbers) with two properties:

Python tip 18: arbitrary number of arguments

learnbyexample — Wed, 26 Oct 2022 03:00:00 GMT

The print() function can accept zero or more values separated by a comma. Here's how the function arguments are shown in help(print):

print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

Here are some examples with varying number of arguments passed to the print() function:

>>> print()

>>> print('hello')
hello
>>> print(42, 22/7, -100)
42 3.142857142857143 -100

You can write your own functions to accept arbitrary number of arguments as well. The packing syntax is similar to sequence unpacking. A * prefix to an argument name will allow it to accept zero or more values. Such an argument will be packed as a tuple data type and it should always be specified after positional arguments (if any). args is often used as the variable name for this purpose. Here's an example:

>>> def many(x, *args):
...     print(f'{x = }; {args = }')
... 
>>> many()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: many() missing 1 required positional argument: 'x'
>>> many(1)
x = 1; args = ()
>>> many(1, 'two', 3)
x = 1; args = ('two', 3)

Here's a more practical example:

>>> def sum_nums(*args):
...     total = 0
...     for n in args:
...         total += n
...     return total
... 
>>> sum_nums()
0
>>> sum_nums(3, -8)
-5
>>> sum_nums(1, 2, 3, 4, 5)
15
>>> sum_nums(*range(1, 6))
15

Use ** prefix to accept arbitrary number of keyword arguments. See also docs.python: Arbitrary Argument Lists.

Video demo:

See also my 100 Page Python Intro ebook.

Scaling doesn't scale

Weblog on marginalia.nu — Tue, 25 Oct 2022 03:00:00 GMT

By which I mean there are deeply problematic assumptions in the very notion of scaling: Scaling changes the rules, and scaling problems exist in both directions. If what you are doing effortlessly scales up, it almost always means it’s egregiously sub-optimal given your present needs.

These assertions are all very abstract. I’ll illustrate with several examples, to try and build an intuition for scaling. You most likely already know what I’m saying is true, but you may need reminding that this is how it works.

Functions on the Heap

The Cranky Developer on Crater Moon Development — Fri, 21 Oct 2022 23:39:18 GMT

The history of object-oriented programming started with a simple buf very technical and low-level idea.

Marginalia's Index Reaches 100,000,000 Documents

Weblog on marginalia.nu — Fri, 21 Oct 2022 03:00:00 GMT

A very brief note to announce reaching a long term goal and major milestone for marginalia search.

The search engine now indexes 106,857,244 documents!

The previous record was a bit south of seventy million. A hundred million has been a pie-in-the-sky goal for a very long time. It’s seemed borderline impossible to index a that many documents on a PC. Turns out it’s not. It’s more than possible.

Twice this may even be technically doable, but is way past the pain point of sheer logistics. It’s already a real headache to deal with this much data.

Making Work Visible - Avoid DMs

smcleod.net — Thu, 20 Oct 2022 10:00:00 GMT

We create more value by having conversations in public instead of behind closed doors.

The Best Of - 2022 Edition

smcleod.net — Wed, 19 Oct 2022 22:00:00 GMT

Near the end of each year I note down a summary of the best apps I’ve enjoyed using throughout the year, here’s 2022.

Vim tip 16: terminal mode

learnbyexample — Tue, 18 Oct 2022 03:00:00 GMT

Terminal mode is one way to use shell commands from within Vim.

:terminal open a new terminal window as a horizontal split
- opens above the current window unless splitbelow option is set
:vertical :terminal open a new terminal window as a vertical split
- opens to the left of the current window unless splitright option is set

Here are some shortcuts to navigate between windows and change modes:

Ctrl+w followed by w or Ctrl+w move to the next window
- helps you to easily switch back and forth if you have one text editing window and one terminal window
- see the Splitting tip for more such commands
Ctrl+w followed by N goes to Terminal-Normal mode which will help you to move around using Normal mode commands, copy text, etc (note that you need to use uppercase N here)
- Ctrl+\ followed by Ctrl+n another way to go to Terminal-Normal mode
- :tnoremap <Esc> <C-w>N map Esc key to go to Terminal-Normal mode
Ctrl+w followed by : go to Command-line mode from terminal window

Depending on your shell, you can use the exit command to end the terminal session. Ctrl+d might work too.

There are lot of features in this mode, see :h terminal.txt for more details.

Video demo:

XPiphany

The Cranky Developer on Crater Moon Development — Mon, 17 Oct 2022 17:07:07 GMT

I thought XP was chaos. Turns out, I was both right and wrong

Unit Testing Randomness

The Cranky Developer on Crater Moon Development — Mon, 17 Oct 2022 07:23:00 GMT

Testing a (pseudo)random number generator

XCheck at Meta: Why it exists and how it works

Krishna's blog — Sun, 16 Oct 2022 03:00:00 GMT

The article discusses the XCheck system at Meta (formerly Facebook) and its purpose in preventing accidental enforcement from affecting accounts that are known to be good.

FOSDEM talks and emulation

Linux Engineer's random thoughts — Sun, 16 Oct 2022 01:00:00 GMT

In 2019 I gave a talk at FOSDEM in Brussels on the "music portal", a device I built for my children as a toy that plays music. I wrote about the first version in awk here. The talk revolved around the specifics of the Go language and how it was …

Future Security Threats - Example One - The Threat from Stable Diffusion

--Mike-- — Thu, 13 Oct 2022 19:40:37 GMT

I'm playing with Stable Diffusion on my PC, to see what kind of things it comes up with, and learn a bit more about the nature of AI. I've been trying to visualize the words of my friend Lloyd Smith, who well before his passing often said in jest

When I die... I want to be sat upright in a chair at a card table, on the surface of the moon, with a royal flush in my hands, and a shit eating grin on my face. That will give the astronomy students something fun to look for.

There have been quite a few thought provoking outputs, some of which worked out really well

But in browsing through them this morning, out of the 500+ images I've generated on this theme, I saw this

Which immediately brought back to mind the NDC Keynote Presentation by Laura Bell I watched last night, which addressed future security threats. One of the possibilities was the threat of curated data sets... and here it was in my own PC.

I don't know how, but somewhere in this curated set of data, copyrights might have been broken. The future is here.

CLI tip 17: common and unique lines

learnbyexample — Wed, 12 Oct 2022 03:00:00 GMT

Consider these sample input files that are already sorted and the default output from comm:

$ paste colors_1.txt colors_2.txt
Blue    Black
Brown   Blue
Orange  Green
Purple  Orange
Red     Pink
Teal    Red
White   White

$ comm colors_1.txt colors_2.txt
        Black
                Blue
Brown
        Green
                Orange
        Pink
Purple
                Red
Teal
                White

The following comm options will help you construct solutions to get common and unique lines:

-1 suppress lines unique to the first file
-2 suppress lines unique to the second file
-3 suppress lines common to both the files

# common lines
$ comm -12 colors_1.txt colors_2.txt
Blue
Orange
Red
White

# lines unique to colors_2.txt
$ comm -13 colors_1.txt colors_2.txt
Black
Green
Pink

If the input files are not already sorted, or if you want to preserve the order of input lines, you can use awk instead:

# common lines
$ awk 'NR==FNR{a[$0]; next} $0 in a' colors_1.txt colors_2.txt
Blue
Orange
Red
White

# lines unique to colors_2.txt
$ awk 'NR==FNR{a[$0]; next} !($0 in a)' colors_1.txt colors_2.txt
Black
Green
Pink

You can also use grep -Fxf colors_1.txt colors_2.txt (add -v for unique lines) but this wouldn't scale well for larger input files.

Video demo:

See also my Linux Command Line Computing ebook.

A database without dynamic memory allocation

Notes on software development — Wed, 12 Oct 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

The Absolute Best Way to Run Multiple npm Scripts in Parallel in 2022

swyx's site RSS Feed — Wed, 12 Oct 2022 01:18:03 GMT

Just a quick tutorial and explanation of how best to set up concurrently with named and colored log output since I had to look it up today.

What is artificial intelligence and what isn't?

I Am Not a Robot — Mon, 10 Oct 2022 18:37:40 GMT

Most people who talk about AI haven't spent much time thinking about what it means. What exactly is intelligence as an attribute of a software system?

Checking back in on Blizzard

Ben Overmyer's Site — Mon, 10 Oct 2022 03:00:00 GMT

It's been over a year since I swore off of Blizzard games. A lot has changed in that time. Many people have left the company, and they've made several changes that appear on the surface to indicate a better culture. They've also made some questionable decisions. I decided to give Blizzard another chance.

This decision point happened a little over a week ago. I was talking with Sarah about the news of the impending release of Overwatch 2. Despite having avoided Blizzard games for a year, I never stopped checking the headlines on Polygon about what Blizzard Entertainment was up to. Mostly it was like watching a train wreck in slow motion. The release of Diablo Immortal was somehow worse than anyone could have predicted, with one of the most awful monetization schemes known to man. The last few major leaders for Overwatch left. There was some good, though. World of Warcraft's team actually seemed to be making positive progress.

As I was mulling all this over with Sarah, I realized that enough had changed that I might be willing to give them another shot. It dawned on me, too, that bearing a grudge against a game company may not be the best way to encourage internal change at that game company. So the next day, after giving it a second round of thought, I decided to rescind my personal ban on Blizzard games and give it another go.

I reinstalled Overwatch for the last full day of the game's life before it was shut down to make way for Overwatch 2. I even tried to livestream it, though it was only the next morning that I realized the stream had no sound. Playing Overwatch was like stretching muscles long unused - no game had ever filled that niche for me, though I tried Paladins for awhile. I enjoyed being back in it. It felt like greeting an estranged friend for the first time in a long time.

I even bought the Collector's Edition for the new World of Warcraft expansion, since I own the CEs of every expansion. I only lack the CE for the original game. I'm waiting until later to begin playing that again, though. The expansion releases at the end of November, and while I still have a lot of the Shadowlands content to play through before it, I'm in no hurry.

Diablo 2: Resurrected finally landed on my PC, too. I had preordered it before the ban went into effect, so I never played it. The graphics are impressive and it feels like my memories of the original. I really do hate managing potions, though. That never changed.

Overwatch 2's launch was plagued by server problems, long queues, and a DDoS attack. I got to play it a little, and it seems to have the broad strokes down. There are tons of tiny little issues that irk me, though. Many of the new "legendary" skins are only variations of the basic skins, not complete reimaginings like all of the old legendaries. Further, they're extremely expensive compared to the old skins. I doubt I'll be collecting any of the new skins, or really even participating much in the new cosmetics shop. Too little value for too big of an expense. The gameplay itself, though, is pretty solid. I like the new Push mode.

So, the games are fun enough. But what about the company's culture? For that, I have to reach out to some friends. We'll see what I learn. If the culture is changing then this may not be a temporary return for me.

Estimating mathematical constants with Monte Carlo simulations

Bytepawn - Marton Trencseni — Sun, 09 Oct 2022 01:00:00 GMT

I use simple Monte Carlo simulations to estimate some mathematical constants: √2, ϕ, e and π.

2022–10–08: Pinephone UART HW Issue

megi's PinePhone Development Log — Sat, 08 Oct 2022 01:00:00 GMT

Python tip 17: counting frequency of items

learnbyexample — Thu, 06 Oct 2022 03:00:00 GMT

One of the ways to count the frequency of items is to make use of the dict.get() method:

>>> vehicles = ['car', 'jeep', 'car', 'bike', 'bus', 'car', 'bike']
>>> hist = {}
>>> for v in vehicles:
...     hist[v] = hist.get(v, 0) + 1
... 
>>> hist
{'car': 3, 'jeep': 1, 'bike': 2, 'bus': 1}

And here's a solution using the built-in collections module:

>>> from collections import Counter

>>> vehicles = ['car', 'jeep', 'car', 'bike', 'bus', 'car', 'bike']
>>> Counter(vehicles)
Counter({'car': 3, 'bike': 2, 'jeep': 1, 'bus': 1})

>>> Counter('abracadabra')
Counter({'a': 5, 'b': 2, 'r': 2, 'c': 1, 'd': 1})

See stackoverflow: using a dictionary to count items and stackoverflow: count frequency of elements for more ways to solve this problem.

Video demo:

See also my 100 Page Python Intro ebook.

Implementing truly safe semaphores in rust

The NeoSmart Files — Mon, 03 Oct 2022 23:11:34 GMT

Discuss this article on r/rust or on Hacker News. Low-level or systems programming languages generally strive to provide libraries and interfaces that enable developers, boost productivity, enhance safety, provide resistance to misuse, and more — all while trying to reduce … Continue reading →

The post Implementing truly safe semaphores in rust first appeared on The NeoSmart Files.

Common patterns in technical interviewing

Bytepawn - Marton Trencseni — Sat, 01 Oct 2022 01:00:00 GMT

I will attempt to enumerate all the categories of questions commonly asked in technical interview loops, and my experience with them.

Go install loop. The "clang" command requires the command line developer tools.

Ben E. C. Boyter — Wed, 28 Sep 2022 03:00:00 GMT

Recently on my M1 Macbook I ran into the following issue when trying to compile some Go code with C dependencies.

The “clang” command requires the command line developer tools.

Being a Mac this also prompted me with a button I could click to install them. Naturally I clicked it, only to be prompted with the following.

Vim tip 15: moving within current line

learnbyexample — Tue, 27 Sep 2022 03:00:00 GMT

Here are some of the Normal mode commands for moving within the current line:

0 move to the beginning of the current line (i.e. column number 1)
- you can also use the Home key
^ move to the beginning of the first non-blank character of the current line (useful for indented lines)
$ move to the end of the current line
- you can also use the End key
- 3$ move to the end of 2 lines below the current line
g_ move to the last non-blank character of the current line
3| move to the third column character
- | is same as 0 or 1|

Video demo:

How good a particle physicist is GPT-3?

Bytepawn - Marton Trencseni — Sat, 24 Sep 2022 01:00:00 GMT

Here I will show a "conversation" with GPT-3 to gauge how good a particle physicist — or an illusion of a particle physicist — it is.

Mike's law of capabilities

--Mike-- — Fri, 23 Sep 2022 16:31:23 GMT

Any sufficiently popular computing environment contains an ad hoc, informally-specified, bug-ridden, slow implementation of capability based security.

In a recent Hacker News thread[1], I quoted from the source article

Continuing a legacy is an honorable task, except in programming.

This is the most important of the items in the list. The "upgrade or you're unclean" ethos has some very deep roots. It goes all the way down to the security model of our operating systems.

It was perfectly reasonable to assume that a user knew what they were doing with their own files back in the days when Unix was written. That wasn't assumed in the world of mainframes, which were designed to multiplex hardware across tens, hundreds, or even thousands of jobs. In mainframes, your job is explicitly given resources to use and a run time environment. No matter what went wrong, a job run for Acme Inc couldn't corrupt the files of Bob's Distributor.
We don't have that... we've been lead to believe that we just have to be pure enough, our code has to be perfect enough, and defect free, in order to have safety. We blame everything except the OS, mostly because we don't want to cast doubt on the wisdom of Linus, and the sanctity of Linux.
We recreate the mainframe model, over and over, in part, not in whole. It's like Greenspun's 10th rule[2].

Any sufficiently popular computing environment contains an ad hoc, informally-specified, bug-ridden, slow implementation of capability based security. That's why we have Virtual Machines, Containers, and now WASM... which I hope actually manages to fend off the stupidity of letting an application decide which files to access.

We need to make better choices. Legacies are a choice, lots of old stuff is tossed by kids all the time, the legacies are the good things kept from the past. Keeping legacies alive is a choice, let's chose wisely.

[1] https://news.ycombinator.com/item?id=32935466
[2] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

Data Science Culture Doc

Bytepawn - Marton Trencseni — Fri, 23 Sep 2022 01:00:00 GMT

I wrote a Culture Doc for the Data Science team I lead.

CLI tip 16: transpose tables

learnbyexample — Wed, 21 Sep 2022 03:00:00 GMT

GNU datamash has plenty of nifty features for field based operations. Here's an example of transposing comma delimited data:

$ cat scores.csv 
Name,Maths,Physics,Chemistry
Ith,100,100,100
Cy,97,98,95
Lin,78,83,80
Er,60,70,90

$ datamash -t, transpose <scores.csv 
Name,Ith,Cy,Lin,Er
Maths,100,97,78,60
Physics,100,98,83,70
Chemistry,100,95,80,90

And here's an alternate solution using tr, wc and pr:

# divide input into five parts and join them vertically
$ seq 10 | pr -5ts,
1,3,5,7,9
2,4,6,8,10

# tr converts input table into single field per line
# wc calculates number of rows and pr does the rest
$ tr ',' '\n' <scores.csv | pr -$(wc -l <scores.csv)ts,
Name,Ith,Cy,Lin,Er
Maths,100,97,78,60
Physics,100,98,83,70
Chemistry,100,95,80,90

See also unix.stackexchange: How to process an x-column text file to get a y-column one? for many more ways to deal with such problems. See my blog post for examples and resource links on the GNU datamash command.

Video demo:

See also my Linux Command Line Computing ebook.

Talk Notes for The End of Localhost (Infobip Shift 2022)

swyx's site RSS Feed — Mon, 19 Sep 2022 15:13:40 GMT

I returned to Zadar!

Getting Senpai To Notice You

swyx's site RSS Feed — Sun, 18 Sep 2022 15:05:24 GMT

How to break the cold start problem in content creation as a new entrant to any field, and getting the leaders of that field to at least read your writing and know your name.

A minimal distributed key-value database with Hashicorp's Raft library

Notes on software development — Sat, 17 Sep 2022 03:00:00 GMT

When I wrote the "build a distributed PostgreSQL proof of concept" post I first had to figure out how to use Hashicorp's Raft implementation.

There weren't any examples I could find in the Hashicorp repo itself. And the only example I could find was Philip O'Toole's hraftd. It's great! However, I have a hard time following multi-file examples in general.

So I built my own single-file example. It's not perfect but it helped me get started and may help you too. We'll walk through that code, ~260 lines of Go, in this post.

The key-value database will only be able to set keys, not delete them. But it will be able to overwrite existing entries. And it will expose this distributed key-value database over an HTTP API.

Here's a sample interaction it will be able to support.

Terminal 1:

$ ./raft-example --node-id node1 --raft-port 2222 --http-port 8222

Terminal 2:

$ ./raft-example --node-id node2 --raft-port 2223 --http-port 8223

Terminal 3, tell 1 to have 2 follow it:

$ curl 'localhost:8222/join?followerAddr=localhost:2223&followerId=node2'

Terminal 3, now add a key:

$ curl -X POST 'localhost:8222/set' -d '{"key": "x", "value": "23"}' -H 'content-type: application/json'

Terminal 3, now get the key from either server:

$ curl 'localhost:8222/get?key=x'
{"data":"23"}
$ curl 'localhost:8223/get?key=x'
{"data":"23"}

Let's make it happen!

Eine kleine background

Raft is an algorithm for managing a replicated (basically append-only) log over a cluster of nodes. When you combine this with a state machine you get a stateful, distributed application. Log entries act as commands for the state machine. When a node in the Raft cluster crashes, it is brought up to date by sending (also called "replaying") all commands in the log through the state machine.

This can be made more efficient by implementing an application-specific concept of state snapshots. But since snapshots are just an optimization, we'll skip it entirely to keep this application simple.

If you want the details, just read the Raft paper! It is surprisingly accessible, especially as a user.

Our app

In our distributed key-value application, commands will be a serialized struct with a key and a value. The state machine will take each struct and set the key to the value in memory. Thus after replaying the entire log (and continuing to apply future log entries), each node will have an in-memory key-value store that is up to date with all other nodes in the cluster.

Note that although each node's key-value store will only be in memory, it will be backed by the durable append-only log! So with, Raft each in-memory key-value store will still be durable.

Let's get things set up in a file, main.go.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net"
    "net/http"
    "os"
    "path"
    "sync"
    "time"

    "github.com/hashicorp/raft"
    "github.com/hashicorp/raft-boltdb"
)

The state machine

The state machine acts on an in-memory key-value store.

type kvFsm struct {
    db *sync.Map
}

There are three operations this Raft library wants us to implement on our state machine struct.

Apply

The Apply operation is sent to basically-up-to-date nodes to keep them up to date. An Apply call is made for each new log the leader commits.

Each log message will contain a key and value to store in the in-memory key-value store.

type setPayload struct {
    Key   string
    Value string
}

func (kf *kvFsm) Apply(log *raft.Log) any {
    switch log.Type {
    case raft.LogCommand:
        var sp setPayload
        err := json.Unmarshal(log.Data, &sp)
        if err != nil {
            return fmt.Errorf("Could not parse payload: %s", err)
        }

        kf.db.Store(sp.Key, sp.Value)
    default:
        return fmt.Errorf("Unknown raft log type: %#v", log.Type)
    }

    return nil
}

Here we're reading a log in a custom format. Later on down in the HTTP server we'll write the part that submits that log in this custom format.

The Raft library just cares that logs are (opaque) bytes. Whatever format works.

Restore

The Restore operation reads all logs and applies them to the state machine.

It looks very similar to the Apply function we just wrote except for that this operates on an io.ReadCloser of serialized log data rather than the high-level raft.Log struct.

And most importantly, and unlike the Apply function, Restore must reset all local state.

func (kf *kvFsm) Restore(rc io.ReadCloser) error {
    // Must always restore from a clean state!!
    kf.db.Range(func(key any, _ any) bool {
        kf.db.Delete(key)
        return true
    })

    decoder := json.NewDecoder(rc)

    for decoder.More() {
        var sp setPayload
        err := decoder.Decode(&sp)
        if err != nil {
            return fmt.Errorf("Could not decode payload: %s", err)
        }

        kf.db.Store(sp.Key, sp.Value)
    }

    return rc.Close()
}

The io.ReadCloser represents the latest snapshot or the beginning of time if there are no snapshots.

Snapshot

We won't implement this. But to satisfy the Go interface we must have empty some functions.

type snapshotNoop struct{}

func (sn snapshotNoop) Persist(_ raft.SnapshotSink) error { return nil }
func (sn snapshotNoop) Release()                          {}

func (kf *kvFsm) Snapshot() (raft.FSMSnapshot, error) {
    return snapshotNoop{}, nil
}

I think this is a correct noop. If we implemented a real snapshot we'd serialize the current key-value state, and raft.SnapshotSink.Write() it to the raft.SnapshotSink. That sink, in turn, is what is passed (as an io.ReadCloser) to the Restore method above.

So it must be that when we do not call raft.SnapshotSink.Close(), as the docs suggest, no snapshot gets recorded.

Since we aren't implementing snapshots, the Raft library must be doing its own serialization, writing each message's bytes directly to some sink.

If I'm wrong, feel free to correct me.

That's it for the state machine!

Raft node initialization

In order to start the Raft library behavior for each node, we need a whole bunch of boilerplate for Raft library initialization.

Each Raft node needs a TCP port that it uses to communicate with other nodes in the same cluster.

Each node starts out in a single-node cluster where it is the leader. Only when told to (and given the address of other nodes) does there become a multi-node cluster.

Each node also needs a permanent store for the append-only log. The Hashicorp Raft library suggests boltdb.

func setupRaft(dir, nodeId, raftAddress string, kf *kvFsm) (*raft.Raft, error) {
    os.MkdirAll(dir, os.ModePerm)

    store, err := raftboltdb.NewBoltStore(path.Join(dir, "bolt"))
    if err != nil {
        return nil, fmt.Errorf("Could not create bolt store: %s", err)
    }

    snapshots, err := raft.NewFileSnapshotStore(path.Join(dir, "snapshot"), 2, os.Stderr)
    if err != nil {
        return nil, fmt.Errorf("Could not create snapshot store: %s", err)
    }

    tcpAddr, err := net.ResolveTCPAddr("tcp", raftAddress)
    if err != nil {
        return nil, fmt.Errorf("Could not resolve address: %s", err)
    }

    transport, err := raft.NewTCPTransport(raftAddress, tcpAddr, 10, time.Second*10, os.Stderr)
    if err != nil {
        return nil, fmt.Errorf("Could not create tcp transport: %s", err)
    }

    raftCfg := raft.DefaultConfig()
    raftCfg.LocalID = raft.ServerID(nodeId)

    r, err := raft.NewRaft(raftCfg, kf, store, store, snapshots, transport)
    if err != nil {
        return nil, fmt.Errorf("Could not create raft instance: %s", err)
    }

    // Cluster consists of unjoined leaders. Picking a leader and
    // creating a real cluster is done manually after startup.
    r.BootstrapCluster(raft.Configuration{
        Servers: []raft.Server{
            {
                ID:      raft.ServerID(nodeId),
                Address: transport.LocalAddr(),
            },
        },
    })

    return r, nil
}

Now let's dig into how nodes learn about each other.

An HTTP API

This key-value store application will have an HTTP API serving two purposes:

Cluster management: telling a leader to add followers
Key-value storage: setting and getting keys

type httpServer struct {
    r  *raft.Raft
    db *sync.Map
}

Cluster management

In this library, the leader is told to add other nodes as its follower. (This feels backwards to me, but it is what it is!)

For this, the library requires a node ID and its internal TCP port for Raft messages.

These will both be parameters we give each node later on when the node process is started.

func (hs httpServer) joinHandler(w http.ResponseWriter, r *http.Request) {
    followerId := r.URL.Query().Get("followerId")
    followerAddr := r.URL.Query().Get("followerAddr")

    if hs.r.State() != raft.Leader {
        json.NewEncoder(w).Encode(struct {
            Error string `json:"error"`
        }{
            "Not the leader",
        })
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    err := hs.r.AddVoter(raft.ServerID(followerId), raft.ServerAddress(followerAddr), 0, 0).Error()
    if err != nil {
        log.Printf("Failed to add follower: %s", err)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
    }

    w.WriteHeader(http.StatusOK)
}

Key-value storage

This part of the HTTP API exposes setting and getting.

Set

Setting is where, instead of modifying the local database directly, we pass a message to the Raft cluster to store a log that contains the key and value.

Since we read log messages in kvFsm.Apply and kvFsm.Restore as a JSON encoding of the setPayload struct we created, we must write log messages like so as well. Or, specifically in this case, we just expect that the user passes a JSON body that matches the setPayload struct.

Then we call Apply on the Raft instance with the log message to get this process going.

func (hs httpServer) setHandler(w http.ResponseWriter, r *http.Request) {
    defer r.Body.Close()
    bs, err := io.ReadAll(r.Body)
    if err != nil {
        log.Printf("Could not read key-value in http request: %s", err)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    future := hs.r.Apply(bs, 500*time.Millisecond)

    // Blocks until completion
    if err := future.Error(); err != nil {
        log.Printf("Could not write key-value: %s", err)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    e := future.Response()
    if e != nil {
        log.Printf("Could not write key-value, application: %s", e)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    w.WriteHeader(http.StatusOK)
}

I'm not completely sure if `future.Response()` is supposed to be called from inside the `future.Error()` error block. You can see the docs for yourself.

Get

If we wanted to be completely consistent we would need to pass a read message through to the Raft cluster and check its result for a key's value. We'd need to implement that read message in the state machine.

But if we don't care strongly about consistency for reads we can just read the local in-memory store, skipping the Raft cluster.

func (hs httpServer) getHandler(w http.ResponseWriter, r *http.Request) {
    key := r.URL.Query().Get("key")
    value, _ := hs.db.Load(key)
    if value == nil {
        value = ""
    }

    rsp := struct {
        Data string `json:"data"`
    }{value.(string)}
    err := json.NewEncoder(w).Encode(rsp)
    if err != nil {
        log.Printf("Could not encode key-value in http response: %s", err)
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
    }
}

And that's it for the server!

Configuration

Let's throw together a quick helper for grabbing configuration from the CLI.

When the process is started, each node must be configured with a Raft-level TCP address, a Raft-level unique node ID, and an HTTP address (for our application).

type config struct {
    id       string
    httpPort string
    raftPort string
}

func getConfig() config {
    cfg := config{}
    for i, arg := range os.Args[1:] {
        if arg == "--node-id" {
            cfg.id = os.Args[i+2]
            i++
            continue
        }

        if arg == "--http-port" {
            cfg.httpPort = os.Args[i+2]
            i++
            continue
        }

        if arg == "--raft-port" {
            cfg.raftPort = os.Args[i+2]
            i++
            continue
        }
    }

    if cfg.id == "" {
        log.Fatal("Missing required parameter: --node-id")
    }

    if cfg.raftPort == "" {
        log.Fatal("Missing required parameter: --raft-port")
    }

    if cfg.httpPort == "" {
        log.Fatal("Missing required parameter: --http-port")
    }

    return cfg
}

And finally, the main() that brings it all together.

main

func main() {
    cfg := getConfig()

    db := &sync.Map{}
    kf := &kvFsm{db}

    dataDir := "data"
    err := os.MkdirAll(dataDir, os.ModePerm)
    if err != nil {
        log.Fatalf("Could not create data directory: %s", err)
    }

    r, err := setupRaft(path.Join(dataDir, "raft"+cfg.id), cfg.id, "localhost:"+cfg.raftPort, kf)
    if err != nil {
        log.Fatal(err)
    }

    hs := httpServer{r, db}

    http.HandleFunc("/set", hs.setHandler)
    http.HandleFunc("/get", hs.getHandler)
    http.HandleFunc("/join", hs.joinHandler)
    http.ListenAndServe(":"+cfg.httpPort, nil)
}

Build it.

$ go mod init raft-example
$ go mod tidy
$ go build

And give it a shot. :)

Terminal 1:

$ ./raft-example --node-id node1 --raft-port 2222 --http-port 8222

Terminal 2:

$ ./raft-example --node-id node2 --raft-port 2223 --http-port 8223

Terminal 3, tell 1 to have 2 follow it:

$ curl 'localhost:8222/join?followerAddr=localhost:2223&followerId=node2'

Terminal 3, now add a key:

$ curl -X POST 'localhost:8222/set' -d '{"key": "x", "value": "23"}' -H 'content-type: application/json'

Terminal 3, now get the key from either server:

$ curl 'localhost:8222/get?key=x'
{"data":"23"}
$ curl 'localhost:8223/get?key=x'
{"data":"23"}

And we're golden!

Following up on that "build a distributed postgres" post I wanted to write down a shorter intro to building a stateful, distributed application using Hashicorp's Raft library.

So, here's a new blog post!

Also, try reading the Raft paper! It's not bad 😀https://t.co/C4S3uzxm0W pic.twitter.com/L3Wwawe0UC
— Phil Eaton (@phil_eaton) September 19, 2022

Profile on Livecycle.io Devx Project

swyx's site RSS Feed — Thu, 15 Sep 2022 19:44:19 GMT

I was interviewed for Livecycle's DevX interview series here. Reproducing for posterity.

Visualizing how S3 deletes 1 billion objects with Athena and Rust

Tom Forbes — Thu, 15 Sep 2022 02:00:00 GMT

A few weeks ago I had the chance to delete 1 petabyte of data spread across 1 billion objects from S3. Well, actually 940 million, but close enough to the click-baitable 1 billion. I thought it would be interesting challenge to try and visualize the execution of these deletions and possibly gain som...

Python tip 16: delete list elements using index or slice

learnbyexample — Wed, 14 Sep 2022 03:00:00 GMT

The pop() method removes the last element of a list by default. You can pass an index to delete that specific item and the list will be automatically re-arranged. Return value is the element being deleted.

>>> primes = [2, 3, 5, 7, 11]
>>> primes.pop()
11
>>> primes
[2, 3, 5, 7]

>>> student = ['learnbyexample', 2022, ['Linux', 'Vim', 'Python']]
>>> student.pop(1)
2022
>>> student[-1].pop(1)
'Vim'
>>> student
['learnbyexample', ['Linux', 'Python']]

To remove multiple elements using slicing notation, use the del statement. Unlike the pop() method, you won't get the elements being deleted as the return value.

>>> books = ['cradle', 'mistborn', 'legends & lattes', 'sourdough']
>>> del books[-1]
>>> books
['cradle', 'mistborn', 'legends & lattes']
>>> del books[:2]
>>> books
['legends & lattes']

>>> student = ['learnbyexample', 2022, ['Linux', 'Vim', 'Python']]
>>> del student[-1][1]
>>> student
['learnbyexample', 2022, ['Linux', 'Python']]

Video demo:

See also my 100 Page Python Intro ebook.

Text-to-image software: a piano or a digital camera?

I Am Not a Robot — Wed, 14 Sep 2022 00:56:58 GMT

Some technologies make it possible for extremely talented people to create new forms of expression. Are we witnessing one?

Developing Raycast Extensions

Dan Palmer — Tue, 13 Sep 2022 02:00:00 GMT

I’ve just started using Raycast, an application launcher for macOS. Like every other launcher before it, it does a lot more than just launch applications, and most of that functionality comes from extensions. Also like several other launchers before it, I decided to have a go at writing an extension and see what the process is like.

Back in the mid-2000s I was an avid user of Quicksilver. It was the first launcher I used, and was quite extensible, but extensions were native code (typically Objective-C) written against the Apple developer APIs, in Xcode, bundled up and injected into Quicksilver in a sort of plugin model. Writing extensions was cumbersome, poorly documented, and had a relatively long feedback cycle.

Introduction to Micro-Optimization

Speculative Branches — Sun, 11 Sep 2022 03:00:00 GMT

A modern CPU is an incredible machine. It can execute many instructions at the same time, it can re-order instructions to ensure that memory accesses and dependency chains don't impact performance too much, it contains hundreds of registers, and it has huge areas of silicon devoted to predicting which branches your code will take. However, if you have a tight loop and you are interested in optimizing the hell out of it, the same mechanisms that make your code run fast can make your job very difficult. They add a lot of complexity that can make it hard to figure out how to optimize a function, and they can also create local optima that trap you into a less efficient solution.

Write Your Own Task Queue

Dan Palmer — Sat, 10 Sep 2022 02:00:00 GMT

This is not a tutorial on how to write your own task queue, but rather an attempt to convince you that you should write your own.

What’s a “task queue” in this context? For the purposes of this post, a task queue is a system for performing work out of band from a user interaction, often at some later time. Typically this is a core component of many web apps, and is used for performing long running tasks or things that can fail and may need to be retried like sending emails.

10x-ing Svelte (Svelte Summit 2022 Talk Notes)

swyx's site RSS Feed — Fri, 09 Sep 2022 11:50:53 GMT

Some show notes for my Svelte Summit talk for those who are looking for all the references and cut content.

Decompiling Java, Excel and the Vista TCP/IP stack

Ben E. C. Boyter — Fri, 09 Sep 2022 03:00:00 GMT

Working for a large corporate company as a mid level developer is generally rather boring. Churn out some CRUD applications. Write ETL as businesses live on data (usually processed in excel). Meetings that are meant to resolve things but just result in more meetings.

So in short, it is usually boring. However occasionally you get some sort of interesting problem, especially if you gain a reputation for actually solving them.

Tools, craftsmanship and mastery

bayindirh — Fri, 09 Sep 2022 03:00:00 GMT

There are things which mesmerizes me, with no end. One of them is watching humble people who are good at what they're doing. Being calm, confident and efficient while doing high quality work has a particular charm to it, regardless of the discipline they're in.

Honestly, I have been studying these people on and off for quite some time. It's not a scientific study, but more of an informed observation, and I think I have found the things which makes these people tick.

Most important of all is, these people are devoted to their crafts. They like what they're doing with a well refined love, with a deep respect added at healthy amounts. When they start working on something, this feeling emanates from them. A deep concentration with a subtle sense of joy, combined with unwavering professionalism and focus for high quality results become palpable in the air around them.

Almost all these people are doing their crafts for a long time. They are used to ins and outs and peculiar details of their crafts. They don't get puzzled when something goes different than they expect, because they know what to do. They adjust and continue. This is rooted on experience per my observations. They know what might happen and what might not, and are mentally prepared at every step.

Another common thing among these people are tool usage habits. The tools they use are well worn. They might be handed down from a generation earlier, or are using what they got for a long time. In every case the craft and the tool has evolved and transformed together, expanding what's possible with the tool and the craft itself. Even another person who get the very same tool can't use it the same way, because the tool at hand requires mastery and experience to wield that efficiently and productively.

Last but not the least, these craftsman solve some of their problems once, and for all. Tools get fixed, methods solidified, procedures perfected. They internalize the things they do in a manner which limits them in numbers but maximizes in flexibility. They can get a new technique if what they have doesn't fit the bill, but they'll try to evolve what they have first. As a result, these people build a deep wisdom, appreciation and feeling for what they do, how they do and the results they get. This makes them distinguished individuals with a different sense about what they do, the results they get, the knowledge they embody and transfer to others.

These kinds of people deserve to be observed by hours on end with delight and a keen interest on what they do, and how they achieve what they achieve.

And honestly, I aspire to be one of these people.

Maybe we should all aspire to be like that at some point, I have no idea.

Until next time,

Be kind.

Vim tip 14: horizontal and vertical splits

learnbyexample — Tue, 06 Sep 2022 03:00:00 GMT

You can have multiple windows within the same tab page.

:split filename open file for editing in a new horizontal window, above the current window
- you can also use :sp instead of :split
- :set splitbelow open horizontal splits below the current window
:vsplit filename open file for editing in a new vertical window, to the left of the current window
- you can also use :vs instead of :vsplit
- :set splitright open vertical splits to the right of the current window

Here are some shortcuts to navigate between windows:

Ctrl+w followed by w switch to the below/right window for horizontal/vertical splits respectively
- Ctrl+w followed by Ctrl+w also performs the same function
- switches to the first split if you are on the last split
Ctrl+w followed by W switch to the above/left window for horizontal/vertical splits respectively
- switches to the last split if you are on the first split
Ctrl+w followed by hjkl or arrow keys, switch in the respective direction
Ctrl+w followed by t or b switch to the top (first) or bottom (last) window
Ctrl+w followed by HJKL (uppercase), moves the current split to the farthest possible location in the respective direction

If filename is not provided, the current one is used.

Vim adds a highlighted horizontal bar containing the filename for each split.

Video demo:

The God Endpoints will continue until morale improves

swyx's site RSS Feed — Tue, 06 Sep 2022 01:33:58 GMT

a brief meditation on why we keep trying to build God Endpoints and why we will fail until we figure out interfaces.

How good a Data Scientist is GPT-3? - Part II

Bytepawn - Marton Trencseni — Sat, 03 Sep 2022 01:00:00 GMT

I have further "conversations" with GPT-3, this time asking more difficult questions about real-world Data Science projects I have personally worked on.

Paradigm Lost (CascadiaJS 2022 Talk Notes)

swyx's site RSS Feed — Thu, 01 Sep 2022 23:22:47 GMT

Some show notes for my CascadiaJS talk for those who are looking for all the references and cut content.

CLI tip 15: text generation with printf and brace expansion

learnbyexample — Wed, 31 Aug 2022 03:00:00 GMT

You can use brace expansion for generating a sequence of numbers and alphabets. printf helps you to display multiple arguments using the same format specifier. For example:

$ echo {1..3}
1 2 3
$ echo {1..2}{a..b}
1a 1b 2a 2b

$ printf '%s\n' apple banana cherry
apple
banana
cherry

Combining the two, you can generate multiple lines of text. Here are some examples:

$ printf '%s\n' id_{3..1}
id_3
id_2
id_1

$ printf '%s\n' item_{100..120..4}
item_100
item_104
item_108
item_112
item_116
item_120

Here's a practical example:

# the string before %.s is repeated based on the number of arguments
$ printf 'x %.s' a b c
x x x 
$ printf -- '- %.s' {1..5}
- - - - - 

# same as: seq 10 | paste -d, - - - - -
$ seq 10 | paste -d, $(printf -- '- %.s' {1..5})
1,2,3,4,5
6,7,8,9,10

$ n=5
$ seq 10 | paste -d, $(printf -- '- %.s' $(seq $n))
1,2,3,4,5
6,7,8,9,10

$ n=2
$ seq 10 | paste -d, $(printf -- '- %.s' $(seq $n))
1,2
3,4
5,6
7,8
9,10

See this stackoverflow thread for other alternatives, avoiding printf for large numbers, etc.

Video demo:

See also my Linux Command Line Computing ebook.

Selecting parts of a podcast

infrequent oscillations — Tue, 30 Aug 2022 21:42:33 GMT

In which we filter some RSS.

What I learned from Magnus Carlsen

I Am Not a Robot — Mon, 29 Aug 2022 20:59:53 GMT

A short review of the podcast episodes I listened to in the past week, so you know which to skip

Update to Update to Portals

infrequent oscillations — Mon, 29 Aug 2022 10:00:00 GMT

In which we update our updated portals.

Some of the more annoying Go bugs I have dealt with

Ben E. C. Boyter — Mon, 29 Aug 2022 03:00:00 GMT

Thought I would write done some of the more annoying/interesting software bugs I have had to deal with recently.

Both were dealing with an archive system where large amounts of video/audio/images are uploaded into, then searchable. It allows the editing of metadata for each type of content, as well sending to external systems.

Problem #1

When a record is being edited a flag “This is being edited” is added to the record and removed on save. However randomly the flag would not be removed, and sometimes additional flags would appear. This was implemented as a “cheap” version of optimistic locking that worked within the application requirements.

What if your Index Page was Smart?

swyx's site RSS Feed — Sat, 27 Aug 2022 03:50:52 GMT

Let's rethink the humble Index Page from first principles.

Mark Zuckerberg explains how he's building the Metaverse. Does he know why?

I Am Not a Robot — Fri, 26 Aug 2022 20:50:21 GMT

I just listened to an interview with Mark Zuckerberg on the Joe Rogan podcast.

Python tip 15: string transliteration

learnbyexample — Wed, 24 Aug 2022 03:00:00 GMT

The str.translate() method accepts a table of codepoints (numerical value of a character) mapped to another character or codepoint. Map to None for characters that have to be deleted. You can use the ord() built-in function to get the codepoint of characters. Or, you can use the str.maketrans() method to generate the mapping for you.

>>> ord('a')
97
>>> ord('A')
65

>>> greeting = 'have a nice day'
# map 'a' to 'A', 'e' to 'E' and 'i' to None
>>> greeting.translate({97: 65, 101: 'E', 105: None})
'hAvE A ncE dAy'

# first and second arguments specify the one-to-one mapping of characters
# third argument is optional, specifies characters to be deleted
>>> str.maketrans('ae', 'AE', 'i')
{97: 65, 101: 69, 105: None}
>>> greeting.translate(str.maketrans('ae', 'AE', 'i'))
'hAvE A ncE dAy'

The string module has a collection of constants that are often useful in text processing. Here's an example of deleting punctuation characters:

>>> from string import punctuation
>>> punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

>>> para = '"Hi", there! How *are* you? All fine here.'
>>> para.translate(str.maketrans('', '', punctuation))
'Hi there How are you All fine here'

>>> chars_to_delete = ''.join(set(punctuation) - set('.!?'))
>>> para.translate(str.maketrans('', '', chars_to_delete))
'Hi there! How are you? All fine here.'

Video demo:

See also my 100 Page Python Intro ebook.

It's 2022 and we're (still) not deploying enough

smcleod.net — Tue, 23 Aug 2022 10:00:00 GMT

We’re (still) not deploying enough

It’s 2022 and not deploying frequently enough is still one of the most common causes of software failure.

The Moving AI Goalpost

jodavaho.io — Tue, 23 Aug 2022 03:00:00 GMT

This post is a work in progress…

We have enough AI to explain conscioiusness now, and therefore to create an AI that feels “General”.

AI can generate words from sounds
GPT is well known to generate narratives and conversations given rough priors
Something like Dall E to generate mental imagery around those narratives, when required
Constantly retrained by feeding what we hear (verbally) and what we see (after “object detection”) in a feedback loop, with all of our “story” as well.

The complication is the starting conditions. We are meant to believe that we are magic “conscious” creatures, even though centuries of contemplatives dispute this.

Science is searching for something that doesn’t exist, and so the expectations of AI are too high. The above is enough.

License

CC BY-SA 4.0

The Evolution of Marginalia's crawling

Weblog on marginalia.nu — Tue, 23 Aug 2022 03:00:00 GMT

In the primordial days of Marginalia Search, it used a dynamic approach to crawling the Internet.

It ran a number of crawler threads, 32 or 64 or some such, that fetched jobs from a director service, that grabbed them straight out of the URL database, these jobs were batches of 100 or so documents that needed to be crawled.

Crawling was not planned ahead of time, but rather decided through a combination of how much of a website had been visited, and the quality score of that website determined where to go next. It also promoted crawling websites adjacent to high quality websites.

What's the big deal about key-value databases like FoundationDB and RocksDB?

Notes on software development — Tue, 23 Aug 2022 03:00:00 GMT

Let's assume you're familiar with basic SQL databases like PostgreSQL and MySQL, and document databases like MongoDB and Elasticsearch. You probably know Redis too.

But you're hearing more and more about embedded key-value stores like RocksDB, LevelDB, PebbleDB, and so on. And you're hearing about distributed key-value databases like FoundationDB and TiKV.

What's the big deal? Aren't these just the equivalent of Redis or Java's ConcurrentHashMap?

Let's take a look.

Extensible databases

Over the last 10 years or so (at least), databases have become more extensible. MySQL has around 10 different open-source storage engines. More surely exist in the wild.

Mongo supports multiple storage engines. Relatively late, PostgreSQL version 12 added support for pluggable storage engines.

OrioleDB and Citus 10's columnar compression are two particularly interesting databases making use of PostgreSQL's pluggable storage engine. But since neither use an embedded key-value store, I won't talk about them more in this post.

And so on.

But why?

Swapping out storage engines allows you to tune the performance of your database. It can allow you to swap out row-oriented storage for column-oriented storage (useful for analytics workloads).

It can allow you to swap B-Trees (traditional choice) for LSM Trees (new hotness) as the underlying storage method (useful for optimizing write-heavy workloads).

And since some storage engines themselves are built on distributed consensus (like FoundationDB and TiKV), it may even allow you to turn a non-distributed database into a distributed database.

Mapping SQL to key-value storage

But how the heck do you turn SQL, row-oriented data, into key-value data?

CockroachDB is a SQL database built on RocksDB originally and now their own LevelDB-inspired database called PebbleDB.

The reason I mention that here is because they maintain a great doc about their method of encoding rows to key-value form.

To simplify that doc though you can imagine mapping each row to a key-value form like this:

${TABLE_IDENTIFIER}_${PRIMARY_KEY}_${ROW_IDENTIFIER}: ${ENCODED_VALUE}

Embedded key-value stores almost always support efficient scanning of rows by a key-prefix. This means that you can efficiently grab all rows within a table by prefix-scanning on the table identifier. If you also include a primary key value along with the table identifier prefix, you get efficient primary key lookup.

Even though the key space is flat.

For the encoded value you can pick any encoding scheme; as inefficient as JSON or as efficient as some binary scheme like Protocol Buffers or Parquet.

Thanks to Justin Jaffray for pointing me at the CockroachDB doc and confirming some of my thoughts on encoding strategies.

Tutorials

I've written a couple of tutorials on building a database. They build on top of embedded key-value stores. If you're interested in seeing minimal code walkthroughs of how this process can work, check these posts out:

Major aspects of key-value stores

Now that you understand how a database can map to a key-value store, let's take a look at the particular properties that distinguish all these key-value stores from systems like Redis and Memcached.

Reliable storage

Maybe the single most important thing a storage system does is actual store data reliably. You can't just open() and write(). To quote Dan Luu, files are hard.

Deferring storage correctness to a dedicated system means database developers can worry about other aspects of database development.

Embeddable

Along with reliable storage is the fact that the storage needs to run in process. Redis, for example, is not embeddable. There are many other things on top of the storage that need to happen in a high-level database and RPC calls between processes for storage is an unnecessary overhead.

Efficient prefix scans

As mentioned above, support for scans is pretty important for how indexes and namespaces (tables in SQL) get mapped to key-value queries.

You shouldn't need to look through all table rows in the flat key space to find the rows for one table.

Additional aspects

The above isn't a complete list. Different stores provide different useful aspects like improved performance on certain workloads/in certain environments, builtin transactions, and so on.

And sometimes it's helpful just to have an embedded store in your language rather than going through a foreign-function interface.

Survey of databases built on embedded key-value stores

Lastly, let's take a look at a few databases that build on top of embedded key-value stores.

Note that some of them are not the primary version of the database (e.g. MyRocks vs MySQL, MongoRocks vs Mongo). Some of them are the primary version (e.g. CockroachDB, YugabyteDB).

Document databases built on key-value stores

MongoRocks (Mongo on RocksDB)

SQL databases built on key-value stores

MyRocks (MySQL on RocksDB)
CockroachDB (RocksDB originally, now their own PebbleDB)
YugabyteDB (on DocDB on RocksDB)
Apache Ignite (Calcite on RocksDB)

Redis-compatible databases built on key-value stores

ZippyDB (Redis-compatible database on RocksDB)
Redis Enterprise Flash (Redis on RocksDB)

Other databases built on key-value stores

Rocksandra (Cassandra on RocksDB)

Missing a database? Let me know!

Separately, distributed key-value stores

There is a different kind of key-value store that is a standalone app designed for distributed data. This list includes Consul, etcd, likely FoundationDB, and likely ZippyDB. (There's a nice comparison table about some of these databases on the etcd page).

These systems are designed to be used sort of like Redis except for that they are persistant and reliable stores. They are designed to always be up and always correct. For that reason they form the data storage backbone of core infrastructure like Kubernetes.

It is possibly how Snowflake uses FoundationDB but I'm not 100% sure.

TiKV is not an embedded key-value database but it's not being used the same way etcd/Consul are as far as I can tell. It forms the backbone of TiDB, an HTAP (hybrid OLAP/OLTP) SQL database.

Maybe FoundationDB and TiKV deserve their own new category.

But in general these databases have an RPC API that you communicate with over TCP. They are not generally embedded. You manage their process(es) separately.

Conclusion

So in this post we saw that databases are extensible. Storage engines are often swappable. Dedicated embedded key-value stores allow database developers to hand off data storage to a dedicated library. Different key-value stores have different performance characteristics that help developers and operators tune a database for their workload.

Embedded key-value stores are a great foundation for all kinds of databases; SQL databases like CockroachDB, document databases like Mongo, wide-store databases like Cassandra, and caching databases like ZippyDB or Redis Enterprise Flash.

This is a complex topic with many, many variations of systems. Hopefully this was a useful introduction.

Overall if you're not a database developer and you're not running databases at a massive scale, you can probably ignore the details of the storage layer.

Did I get something wrong? Or miss something important? Let me know. :)

Corrections

An earlier version of this post suggested that FoundationDB was embedded. It is not. Thanks adaszko on Lobsters for correcting.
An earlier version of this post suggested that TiKV was embedded. It is not. Thanks eis on Hacker News.

What's the big deal about embedded key-value databases like FoundationDB ands RocksDB?

I wrote a new blog post that might give a little context. :)https://t.co/kNFM1hVGx6 pic.twitter.com/H4SouStZHk
— Phil Eaton (@phil_eaton) August 23, 2022

Cost of a integer cast in Go

Ben E. C. Boyter — Mon, 22 Aug 2022 03:00:00 GMT

Recently have been doing interviews at work for Go developers.

The filter for this is a simple review exercise. We present a small chunk of code and ask them to review it over 15 minutes pointing out any issues they see. The idea is to respect their and our time. It works pretty well and we can determine how much experience someone has by their ability to pick up the obvious vs subtle bugs.

The Surprisingly High Table Stakes of Modern Blogs

swyx's site RSS Feed — Sun, 21 Aug 2022 23:21:53 GMT

Bottom Line Up Front: You are probably underestimating how much goes into blogging technology these days.

How to write a Python Twitter Unfollow Script in 2022

swyx's site RSS Feed — Sun, 21 Aug 2022 07:28:59 GMT

The Twitter API has changed (from v1 to v2), and Python has gone from 2 to 3, and Google is still serving up loads of outdated results.

SQLite has pretty limited builtin functions

Notes on software development — Sun, 21 Aug 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

2022–08–21: Sound on Pinephone Pro

megi's PinePhone Development Log — Sun, 21 Aug 2022 01:00:00 GMT

Writing my first Security blogpost

swyx's site RSS Feed — Fri, 19 Aug 2022 05:51:52 GMT

Today's fun emergency at work was a first - writing a security postmortem for a breach of an open source user (aka not a breach of us, which seems the norm).

Marginaliacoin, and hidden forums

Weblog on marginalia.nu — Thu, 18 Aug 2022 03:00:00 GMT

I discovered someone has made a cryptocurrency called “Memex Marginalia Inu”. It appears to have been created February 23, which is around when the entry “I Have No Capslock And I Must Scream” went absurdly viral to the point where Elon Musk tweeted a link to it.

I Have No Capslock…

Mr Musk’s twitter orbit is exceptionally strange. The tweet was followed by a deluge of bizarre activity, strange emails with calls about stonk canine lunar expeditions, and apparently also a cryptocurrency land-grab of sorts. I can’t claim to understand why, but many of the emails got after the tweet were on the theme “what does this mean?”, almost as though Elon’s tweet was some sort of prophetic omen.

Vim prank: alias vim='vim -y'

learnbyexample — Wed, 17 Aug 2022 03:00:00 GMT

Poster created using Canva

While going through :h vim-arguments for my Vim Reference Guide ebook, I came across the -y option:

Easy mode. Implied for evim and eview. Starts with 'insertmode' set and behaves like a click-and-type editor. This sources the script $VIMRUNTIME/evim.vim. Mappings are set up to work like most click-and-type editors, see evim-keys. The GUI is started when available.

It was so weird to use. Copy and paste works with Ctrl+c and Ctrl+v respectively. Text can be selected with mouse and typing new text overwrites this selected portion. Esc key doesn't work (gasp!), so I couldn't quit until I used the window buttons. Later I tried and found that Ctrl+o works, which would then allow you to use :q as usual.

So, if you want to prank a Vim user:

alias vim='vim -y'

I didn't expect such a good response on /r/vim/ and twitter for this "easy" feature. So, decided to write this mini blog post as well. Also, I got to know a few more ways to escape this madness from the /r/vim/ sub:

One hint: If you want to go to Normal mode to be able to type a sequence of commands, use CTRL-L. https://vimhelp.org/starting.txt.html#evim-keys

Use <c-\><c-n> See :h CTRL-\_CTRL-N

Update

So, this post reached front page on Hacker News! Plenty of interesting comments and got to know about novim-mode plugin (which aims to make Vim behave more like a 'normal' editor).

I also found an old discussion on /r/vim/ discussing ways to trick a Vim user.

GNU BRE/ERE cheatsheet and differences between grep, sed and awk

learnbyexample — Wed, 17 Aug 2022 03:00:00 GMT

Poster created using Canva

This post covers Basic Regular Expressions (BRE) and Extended Regular Expressions (ERE) syntax supported by GNU grep, sed and awk. You'll also learn the differences between these tools — for example, awk doesn't support backreferences within regexp definition (i.e. the search portion).

BRE and ERE🔗

From GNU grep manual:

In basic regular expressions the meta-characters ?, +, {, |, (, and ) lose their special meaning; instead use the backslashed versions \?, \+, \{, \|, $, and $.

grep and sed support BRE by default and enables ERE when -E option is used. awk supports only ERE. Assume ERE for descriptions in this post unless otherwise mentioned.

This post is intended as a reference for BRE/ERE flavor of regular expressions. For a more detailed explanation with examples and exercises, see these chapters from my ebooks:

Anchors🔗

Pattern	Description
`^`	restricts the match to the start of the string
`$`	restricts the match to the end of the string
`\<`	restricts the match to the start of word
`\>`	restricts the match to the end of word

The -x cli option in grep is equivalent to ^pattern$.

Word characters include alphabets, digits and underscore. Here's some more alternate ways to specify word anchors:

Pattern	Description
`\b`	restricts the match to the start/end of words, applicable for `grep` and `sed`
`\y`	restricts the match to the start/end of words, applicable for `awk` (`\b` means backspace)
`\B`	matches wherever `\b` (or `\y`) doesn't match

grep also supports -w cli option. It is equivalent to (?<!\w)pattern(?!\w). The three different ways to specify word anchors are not exactly equivalent though, see Word boundary differences section from my book for details and examples.

Alternation and Grouping🔗

Pattern	Description
`pat1\|pat2\|pat3`	match `pat1` or `pat2` or `pat3`
	use `\\|` in BRE mode
`()`	group pattern(s), `a(b\|c)d` is same as `abd\|acd`
	use `` in BRE mode

The alternative patterns can have their own independent anchors. Alternative which matches earliest in the input gets precedence. Longest matching portion wins if multiple alternatives start from the same location (irrespective of the order of alternatives). In case of a tie with same lengths, leftmost alternative wins (see stackoverflow: Non greedy matching in sed for a practical use case).

Escaping metacharacters🔗

Pattern	Description
`\`	prefix metacharacters with `\` to match them literally
`\\`	to match `\` literally

With grep and sed, switching between ERE and BRE can reduce the number of escapes needed for some cases. For fixed string matching, grep has -F option and awk has string comparison operators (whole string) and the index function (partial string).
sed requires both ( and ) characters to be escaped (in ERE mode), whereas grep and awk don't require ) to be escaped.
sed requires { to be escaped (in ERE mode) even if it isn't part of a valid quantifier syntax, whereas grep and awk don't require escaping. For example, you'd need \{a} in sed whereas {a} is enough for the other two.
In BRE mode, grep and sed don't require ^ and $ to be escaped if they are used away from their customary positions.

Dot metacharacter and Quantifiers🔗

Pattern	Description
`.`	match any character, including the newline character
`?`	match `0` or `1` times
	use `\?` in BRE mode
`*`	match `0` or more times
`+`	match `1` or more times
	use `\+` in BRE mode
`{m,n}`	match `m` to `n` times
`{m,}`	match at least `m` times
`{,n}`	match up to `n` times (including `0` times)
`{n}`	match exactly `n` times
	use `\{\}` in BRE mode
`pat1.*pat2`	any number of characters between `pat1` and `pat2`
`pat1.pat2\|pat2.pat1`	match both `pat1` and `pat2` in any order

Precedence rule is longest match wins, which is mostly similar but not exactly same as greedy quantifiers. For example, with foo123312baz as input string, o[123]+(12baz)? will match o123312baz with these tools, whereas it will match o123312 with greedy quantifiers.

Character class🔗

Pattern	Description
`[set123]`	match any of these characters once
`[^set123]`	match except any of these characters once
`[3-7AM-X]`	range of characters from `3` to `7`, `A`, another range from `M` to `X`
`[.`	open collating symbol
`.]`	close collating symbol
`[=`	open equivalence class
`=]`	close equivalence class

Specific placement will help to match character class metacharacters literally.

Pattern	Description
`[a-z-]`	`-` should be first/last character to match literally
`[+^]`	`^` shouldn't be first character
`[]=]`	`]` should be first character (second if `^` is used to invert the set)

\ isn't special within character class in grep.
\ can be used to escape character class metacharacters in awk.

Some commonly used character sets have predefined escape sequences:

Pattern	Description
`\w`	similar to `[a-zA-Z0-9_]` for matching word characters
`\s`	similar to `[ \t\n\r\f\v]` for matching whitespace characters
`\W`	match non-word characters
`\S`	match non-whitespace characters

Undefined escape sequences will be treated as the character it escapes. For example, \e will match e (not \ and e).
- in addition, awk gives a "not a known regexp operator" warning.
The above escape sequences cannot be used inside character classes and behavior varies between the tools.
- For example, using [\w] will match \ or w characters in grep and sed whereas it will match only w in awk.
These escape sequences are also locale aware, for example αλεπού and \u2028 (line separator) will be considered as word and whitespace characters respectively in appropriate locales.
These tools do not support \d and \D, commonly featured in other regexp implementations for digits and non-digits.

Escape sequences🔗

This section is applicable only for sed and awk unless otherwise specified and can be used within character classes too. See also ASCII Codes Table Standard characters.

Escape sequence	Description
`\a`	alert
`\b`	backspace in `awk`, word boundary in `grep` and `sed`
	`\b` inside a character class in `sed` will act as a backspace
`\f`	formfeed
`\n`	newline
`\r`	carriage return
`\t`	horizontal tab
`\v`	vertical tab
`\cx`	CONTROL-x in `sed`

You can also represent ASCII characters using their codepoint values.

Escape sequence	Description
`\xNN`	hexadecimal digits
`\NNN`	octal digits in `awk`
`\oNNN`	octal digits in `sed`
`\dNNN`	decimal digits in `sed`

In search section, a metacharacter specified by escape sequences will still act as the metacharacter. For example, /\x5eco/ will match co only at the start of the string.
In replacement section,
- escape sequences in sed produces literal character. For example, s/.*/"\x26"/ will have "&" as the replacement value.
- escape sequences in awk is treated as metacharacter. For example, sub(/.*/, "[&]") and sub(/.*/, "[\x26]") are equivalent.

Ways to use escape sequences with grep:

ANSI-C Quoting — for example, $'a\tb' will match a and b with a tab in between.
-P option, see my chapter on Perl Compatible Regular Expressions for more details.

Named character sets🔗

The below table lists named sets and their equivalent character class in ASCII encoding. These can be used inside character classes only. For example, [[:digit:]] is same as [0-9] and [[:alnum:]_] is equivalent to \w.

Named set	Description
`[:digit:]`	`[0-9]`
`[:lower:]`	`[a-z]`
`[:upper:]`	`[A-Z]`
`[:alpha:]`	`[a-zA-Z]`
`[:alnum:]`	`[0-9a-zA-Z]`
`[:xdigit:]`	`[0-9a-fA-F]`
`[:cntrl:]`	control characters — first 32 ASCII characters and 127th (DEL)
`[:punct:]`	all the punctuation characters
`[:graph:]`	`[:alnum:]` and `[:punct:]`
`[:print:]`	`[:alnum:]`, `[:punct:]` and space
`[:blank:]`	space and tab characters
`[:space:]`	whitespace characters, same as `\s`

From grep manual:

Their interpretation depends on the LC_CTYPE locale; for example, [[:alnum:]] means the character class of numbers and letters in the current locale.

Backreferences🔗

Pattern	Description
`\N`	backreference, gives matched portion of Nth capture group
	possible values: `\1`, `\2` up to `\9`
`&`	represents entire matched string in the replacement section
`\0`	equivalent to `&` in `sed`

Notes for awk:

backreferences can be used only in replacement section, not allowed in search section.
sub and gsub functions allow only the & backreference.
gensub function allows \N form of backreference as well.
- but need to use \\0, \\1, \\2 etc since they are specified using string syntax.

sed flags🔗

This section discusses flags (also known as modifiers) that change the regexp behavior. When used with regexp addressing:

Flag	Description
`I`	match case insensitively

When used with substitution command:

Flag	Description
`i` or `I`	match case insensitively
`g`	replace all occurrences instead of just the first match
`N`	a number will cause only the Nth match to be replaced
`Ng`	replace from Nth match to the end
`m` or `M`	multiline mode
	`.` will not match the newline character
	`^` and `$` will match every line's start and end locations (line separator is `\n` by default and NUL when `-z` option is used)
\`	always match the start of string irrespective of multiline mode
`\'`	always match the end of string irrespective of multiline mode

Flags are not supported by grep or awk. But these equivalent/alternative options can be used:

-i cli option in grep and setting IGNORECASE to non-zero value in awk will match case insensitively.
tolower or toupper functions can be used in awk to convert input to single case.
you can also use character classes for small strings, for example [cC][aA][tT] will match cat case insensitively.
sub function in awk replaces only the first matching occurrence and gsub function is equivalent to using the g flag.
third argument of gensub function in awk supports replacing only the Nth match as well as the g flag.

The behavior of sed and awk differs for Nth match if the pattern can match empty string:

$ echo 'a,,c,d,,f' | sed 's/[^,]*/b/2'
a,b,c,d,,f
$ echo 'a,,c,d,,f' | sed 's/[^,]*/e/5'
a,,c,d,e,f

$ echo 'a,,c,d,,f' | awk '{print gensub(/[^,]*/, "b", 2)}'
ab,,c,d,,f
$ echo 'a,,c,d,,f' | awk '{print gensub(/[^,]*/, "e", 5)}'
a,,ce,d,,f

sed case conversion🔗

Escape sequence	Description
`\E`	indicates end of case conversion in replacement section
`\l`	convert next character to lowercase
`\u`	convert next character to uppercase
`\L`	convert following characters to lowercase, stops if `\U` or `\E` is found
`\U`	convert following characters to uppercase, stops if `\L` or `\E` is found

sed delimiters🔗

/ is idiomatically used as the delimiter.
Any character except \ and newline character can also be used. For example: s#/home/learnbyexample/#~/# is same as s/\/home\/learnbyexample\//~\//.
For regexp addressing, the first delimiter has to be escaped. For example: \;/foo/bar/;p is same as /foo\/bar\//p.

Vim tip 13: repeat last change

learnbyexample — Tue, 16 Aug 2022 03:00:00 GMT

It is way too easy to repeat the last change you made:

. the Normal mode dot command repeats the last change
- you can also use a number prefix to override the count of the last change

For example,

if the last change was 2dd (delete current line and the line below), dot key will repeat 2dd
- using 3. will mean 3dd and not 6dd, since the count prefix replaces the earlier number
if the last change was 5x (delete current character and four characters to the right), dot key will repeat 5x
if the last change was C123<Esc> and dot key is pressed, it will clear from the current character to the end of the line, insert 123 and go back to Normal mode

From :h 4.3:

The . command works for all changes you make, except for u (undo), CTRL-R (redo) and commands that start with a colon (:).

See :h repeat.txt for complex repeats, using Vim scripts, etc.

Video demo:

2022–08–14: USB recovery/hacking tool for Pinephone Pro

megi's PinePhone Development Log — Sun, 14 Aug 2022 01:00:00 GMT

Rest in Peace, Optane

Speculative Branches — Fri, 12 Aug 2022 03:00:00 GMT

Intel's Optane memory modules launched with a lot of fanfare in 2015, and were recently discontinued, in 2022, with similar fanfare. It was a sad day for me, a lover of abstraction-breaking technologies, but it was forseeable and understandable.

At the time of Optane's launch, a lot of us were excited about the idea of having a new storage tier, sitting between DRAM and flash. It was announced as having DRAM endurance and speed with the persistence and size of flash. It was a futuristic memory technology, but the technology of the future met the full force of Wright's Law.

Native-GUI distributed system in a tweet

metablog — Thu, 11 Aug 2022 04:55:16 GMT

If I've been a bit quiet recently it's not due to lack of progress, but rather the very opposite: so much progress in Objective-S land hat my head is spinning and I am having a hard time both processing it all and seeing where it goes.

But sometimes you need to pause, reflect, and show your work, in whatever intermediate state it currently is. So without further ado, here is the distributed system, with GUI, in a tweet:

#!env stui
scheme:s3 ← ref:http://defiant.local:2345/ asScheme
text ← #NSTextField{ #stringValue:'',#frame:(10@45 extent:180@24) }.
window ← #NSWindow{ #frame:(300@300 extent:200@105),#title:'S3', #views:#[text]}.
text → ref:s3:bucket1/msg.txt.
app runFromCLI:window.
— Marcel Weiher 🇪🇺 (@mpweiher) August 9, 2022

It pops up a window with a text field, and stores whatever the user enters in an S3 bucket. It continues to do this until the user closes the window, at which point the program exits.

Of course, it's not much of a distributed system, particularly because it doesn't actually include the code for the S3 simulator.

Anyway, despite fitting in a tweet, the Objective-S script is actually not code golf, although it may appear as such to someone not familiar with Objective-S.

Instead, it is a straightforward definition and composition of the elements required:

A storage combinator for interacting with data in S3.
A text field inside a window, defined as object literals.
A connection between the text field and a specific S3 bucket.

That's it, and it is no coincidence that the structure of the system maps directly onto the structure of the code. Let's look at the parts in detail.

S3 via Storage Combinator

The first line of the script sets up an S3 scheme handler so we can interact with the S3 buckets almost as if they were local variables. For example the following assignment statement stores the text 'Hello World!' in the "msg.txt" file of "bucket1":

   s3:bucket1/msg.txt ← 'Hello World!'

Retrieving it works similarly:

   stdout println: s3:bucket1/msg.txt

The URL of our S3 simulator is http://defiant.local:2345/, so running on host defiant in the local network, addressed by Bonjour and listening on port 2345. As Objective-S supports Polymorphic Identifiers (pdf), this URL is a directly evaluable identifier in the language. Alas, that directness poses a problem, because writing down an identifier in most programming languages yields the value of the variable the identifier identifies, and Objective-S is no exception. In the case of http://defiant.local:2345/, that value is the directory listing of the root of the S3 server, encoded as the following XML response:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner><ID>123</ID><DisplayName>FakeS3</DisplayName></Owner>
<Buckets>
<Bucket>
<Name>bucket1</Name>
<CreationDate>2022-08-10T15:18:32.000Z</CreationDate>
</Bucket>
</Buckets>
</ListAllMyBucketsResult>

That's not really what we want, we want to refer to the URL itself. The ref: allows us to do this by preventing evaluation and thus returning the reference itself, very similar to the & operator that creates pointers in C.

Except that an Objective-S reference (or more precisely, a binding) is much richer than a C pointer. One of its many capabilities is that it can be turned into a store by sending it the -asScheme message. This new store uses the reference it was created from as its base URL, all the references it receives are evaluated relative to this base reference.

The upshot is that with the s3: scheme handler defined and installed as described, the expression s3:bucket1/msg.txt evaluates to http://defiant.local:2345/bucket1/msg.txt.

This way of defining shorthands has proven extremely useful for making complex references usable and modular, and is an extremely common pattern in Objective-S code.

Declarative GUI with object literals

Next, we need to define the GUI: a window with a text field. With object literals, this is pretty trivial. Object literals are similar to dictionary literals, except that you get to define the class of the instance defined by the key/value pairs, instead of it always being a dictionary.

For example, the following literal defines a text field with certain dimensions and assigns it to the text local variable:

   text ← #NSTextField{ #stringValue:'',#frame:(10@45 extent:180@24) }.

And a window that contains the text field we just defined: window ← #NSWindow{ #frame:(300@300 extent:200@105),#title:'S3', #views:#[text]}. It would have been nice to define the text field inline in its window definition, but we currently still need a variable so we can connect the text field (see next section).

Connecting components

Now that we have a text field (in a window) and somewhere to store the data, we need to connect these two components. Typically, this would involve defining some procedure(s), callback(s) or some extra-linguistics mechanism to mediate or define that connection. In Objective-S, we just connect the components:

   text → ref:s3:bucket1/msg.txt.

That's it.

The right-arrow "→" is a polymorphic connection "operator". The complete connection is actually significantly more complex:

From a port of the source component
To a role of the mediating connector compatible with that source port
To a role of the mediating connector compatible with the target object's port
To that compatible port of the target component

If you want, you can actually specify all these intermediate steps, but most of the time you don't have to, as the machinery can figure out what ports and roles are compatible. In this case, even the actual connector was determined automatically.

If we didn't want a remote S3 bucket, we could also have stored the data in a local file, for example:

   text → ref:file:/tmp/msg.txt.

That treats the file like a variable, replacing the entire contents of the file with the text that was entered. Speaking of variables, we could of course also store the text in a local variable:

   text → ref:var:message.

In our simple example that doesn't make a lot of sense because the variable isn't visible anywhere and will disappear once the script terminates, but in a larger application it could then trigger further processing.

Alternatively, we could also append the individual messages to a stream, for example to stdout:

   text → stdout.

So every time the user hits return in the text field, the content of the text field is written to the console. Or appended to a file, by connecting to the stream associated with the file rather the file reference itself: text → ref:file:/tmp/msg.txt outputStream. This doesn't have to be a single stream sink, it can be a complex processing pipeline.

I hope this makes it clear, or at least strongly hints, that this is not the usual low-code/no-code trick of achieving compact code by creating super-specialised components and mechanisms that work well for a specific application, but immediately break down when pushed beyond the demo.

What it is instead is a new way of creating components, defining their interfaces and then gluing them together in a very straightforward fashion.

Eval/apply vs. connect and run

Having constructed our system by configuring and connecting components, what's left is running it. CLIApp is a subclass of NSApplication that knows how to run without an associated app wrapper or Info.plist file. It is actually instantiated by the stui script runner before the script is started, with the instance dropped into the app variable for the script.

This is where we leave our brave new world of connected components and return (or connect with) the call/return world, similar to the way Cocoa's auto-generated main with call to NSApplicationMain() works.

The difference between eval/apply (call/return) and connect/run is actually quite profound, but more on that in another post.

Of course, we didn't leave call/return behind, it is still present and useful for certain tasks, such as transforming an element into something slightly different. However, for constructing systems, having components that can be defined, configured and connected directly ("declaratively") is far superior to doing so procedurally, even than the fluent APIs that have recently popped up and that have been mislabeled as "declarative".

This project is turning out even better than I expected. I am stoked.

CLI tip 14: specify permissions during directory creation

learnbyexample — Tue, 09 Aug 2022 03:00:00 GMT

You can use mkdir -m instead of creating a directory with mkdir first and then changing the directory permissions with the chmod command. The argument to the -m (mode) option uses the same syntax as the chmod command.

# instead of this
$ mkdir back_up
$ chmod 750 back_up

# do this
$ mkdir -m 750 back_up
$ stat -c '%a %A' back_up
750 drwxr-x---

Here are some more examples:

$ mkdir -m =rx dummy_dir
$ stat -c '%a %A' dummy_dir
555 dr-xr-xr-x

$ mkdir -m go-rwx dot_files
$ stat -c '%a %A' dot_files
700 drwx------

Video demo:

See also my Linux Command Line Computing ebook.

Tools I use: Eclipse

bayindirh — Tue, 09 Aug 2022 03:00:00 GMT

This is first of a series of posts where I talk about the tools and technologies I prefer to use. They are not intended to be in-depth technical documents, but posts answering the questions like Why & How with the history behind it. As always, all comments are welcome.

Eclipse is an IDE framework. A complex scaffolding which can host a lot of functionality at the same time. This means an Eclipse installation can be almost anything, and many things at once. You want a C++ IDE? Check. Python? Check. Java? Go? Web development? Check, check & check.

This makes Eclipse decidedly different from other tools at the same category, and puts it somewhere between an old school IDE and a modern code-aware text editor with plugins. As a result, when a newcomer sees Eclipse first time, there's a big, unfamiliar terrain to navigate.

Moreover, Eclipse is old. Around 20 years old, to be precise. As a result, it doesn't follow the new conventions, styling, UI densities, and whatnot from current era. It's written in Java. In other words, It's a prehistoric artifact when evaluated from a technology perspective.

Or is it?

Personally, I'm using Eclipse since 2006. Before that, I have used various software on and off, but while I was developing my graduation project, I needed a Java IDE, and found Eclipse. It was acceptably fast, worked on both Windows and Linux (I was dual-booting at that time), and most importantly, it was portable. I installed Eclipse to a 256MB flash drive alongside my workspace, and used that USB flash drive to work on my code while I was at the university (laptops were luxury back then).

Eclipse provided great Java tools at that time (it was on par with greatest IDEs of the time for Java), and allowed me to develop and debug my graduation and master's projects. I've published papers from these research projects, too. Being able to develop, debug and oversee the whole process from a single point hooked me to Eclipse.

Then, I found out that Eclipse can do more than Java (and Subversion).

Being interested in C++ lead me to install C/C++ Development Tools (CDT), and Linux profiling integration tools (Linux Tools). Then, Eclipse published its Git integration. Installing these three things converted Eclipse to a modern, Linux native C/C++ development environment with a single click. Eclipse has a notion called "Perspectives", which are "task oriented views" in its essence. With perspectives, Eclipse can transform on the fly, and doesn't get cluttered.

While Eclipse had its dark periods of being a heavy, slow and crash-prone application, especially in the C++ department, the project found its footing in the last 7-8 years, and fixed all its problems, and changed to a quarterly release schedule. While history of Eclipse warrants a single and much longer post, let's get into how I use Eclipse.

Eclipse puts "Integrated" into IDE firmly and squarely.

Eclipse uses "Projects" to store everything related to a project, including its source, settings, and everything in between. A project can override any global setting, and these settings are carried with the project itself. While not favored broadly, by submitting these files into your Git repository, you can carry all your pipeline with the source itself. This allows me to carry and share a project with all its intricate details and settings without detailed documentation and scripts. All are there when I import my project back to Eclipse.

On top of that, Eclipse supports a great deal of programmability out of the box. This means, almost every path and every setting can contain variables. Using these variables allows a project to behave the same way between systems, operating systems and directory trees, making it more portable.

Another feature I love is "Profiles". A project can have many build and debug profiles, which can contain different compiler flags, toolchains, folder includes and excludes, and environment variables. As a result, you can target different architectures, feature/optimization levels, or in extreme cases, a different source tree. This last feature makes compiling test suites with Catch 2 a breeze. While I didn't use extensively, you can even tune which ELF parser to use for debugging and understanding your errors, per profile.

Eclipse has an integrated Git client for a long time. Arguably, it's the best Git client I have ever used (Git Tower comes a close second). It's completely integrated to all views, provides great visual aid for history, encapsulates all common operations to easy to understand flows, and has a useful, integrated diffing tool. Moreover, it guides you during conflict resolution, provides advanced push/pull/merge/rebase controls and even discourages you from doing wrong things with your repo.

On the language specific features provided by the "Toolkits", C/C++ Development Tools' indexer is something worth mentioning. It's called an indexer, but for historic reasons. It supports auto-completion (incl. templates), code formatting (with different profiles nonetheless), documentation access, declaration/instantiation location, advanced refactoring, code analysis, and probably some other things I didn't need or explore yet.

When combined with Linux Tools (e.g. Valgrind), you can run the tests and profiling from your Eclipse window and get your results directly into your window, with graphs and annotations on your source code directly. This is a true integration. Not "tool launch" support.

PyDev, which is the Python development environment for Eclipse, provides almost the same set of features, and some Python specific ones, as well.

The last part I want to touch in this category is synchronization, and export/import capabilities.

You can export all your configuration (installed items) and settings in Eclipse, with small XML files. This allows you or anyone to backup or share your Eclipse installation. You can import your configuration to any Eclipse installation to transform it to the Eclipse you got used to, and import your preferences on top of it to continue from where you left. If you want a more automatic way, you can login to your Eclipse account from any number of installations and keep them in sync via Oomph feature.

There are more Eclipse features I'm aware of, but don't use actively. Hence, I'll leave a concise list of them:

Language Server Protocol support.
Compile into / Run in Docker container.
Remote development support (via SSH or dedicated daemon).
Automatic Change Log generation.
True multi monitor / multi window support.
A broad official package ecosystem.
A board third party package ecosystem via official Marketplace.
...and many more.

As a result, today's Eclipse is neither slow, nor heavy (it uses less RAM than a certain, popular editor of today), nor left behind. With its quarterly updates, diverse package and tool integration ecosystem, it's as capable and flexible as any tool used by today's developers.

On top of that, Eclipse provides a true, single point of command and control for all the software development pipeline, from foundation to testing to deployment/publishing. It's a true marvel and powerhouse.

Python tip 14: sequence unpacking

learnbyexample — Mon, 08 Aug 2022 03:00:00 GMT

You can assign the individual elements of a sequence to multiple variables. This is known as sequence unpacking and it is handy in many situations.

>>> details = ['2018-10-25', 'car', 2346]

>>> purchase_date, vehicle, qty = details
>>> purchase_date
'2018-10-25'
>>> vehicle
'car'
>>> qty
2346

Here's how you can easily assign and swap multiple variables.

# multiple assignments
>>> num1, num2, num3 = 3.14, 42, -100

# swapping values
>>> num1, num2, num3 = num3, num1, num2

>>> print(f'{num1 = }\n{num2 = }\n{num3 = }')
num1 = -100
num2 = 3.14
num3 = 42

Unpacking isn't limited to mapping every element of the sequence. You can use a * prefix to catch all the remaining values (if any is left) in a list variable.

>>> values = ('first', 100, 200, 300, 'last')

>>> x, *y = values
>>> x
'first'
>>> y
[100, 200, 300, 'last']

>>> s1, *nums, s2 = values
>>> s1
'first'
>>> nums
[100, 200, 300]
>>> s2
'last'

See Unpacking with starred assignments for more examples and explanations.

Video demo:

See also my 100 Page Python Intro ebook.

Breaking Barbarian

swyx's site RSS Feed — Mon, 08 Aug 2022 01:51:07 GMT

This week in a Svelte Radio recording, @rich-harris commented that something I said was "uniquely swyx": an offhand observation that "we are all professional streamers now" [^1]. I responded that I've been calling this behavior "barbarianism" - you can listen to the full explanation when the episode drops in future.

2022–08–06: Pinephone keyboard bugs and new findings

megi's PinePhone Development Log — Sat, 06 Aug 2022 01:00:00 GMT

Botspam Apocalypse

Weblog on marginalia.nu — Wed, 03 Aug 2022 03:00:00 GMT

Bots are absolutely crippling the Internet ecosystem.

The “future” in the film Terminator 2 is set in the 2020s. If you apply its predictions to the running of a website, it’s honestly very accurate.

Modern bot traffic is virtually indistinguishable from human traffic, and can pummel any self-hosted service into the ground, flood any form with comment spam, and is a chronic headache for almost any small scale web service operator.

How good a Data Scientist is GPT-3?

Bytepawn - Marton Trencseni — Sun, 31 Jul 2022 01:00:00 GMT

Recently I have been playing around with OpenAI's GPT-3 and I am very impressed by it. It reminds of the famous Arthur C. Clarke quote, “Any sufficiently advanced technology is indistinguishable from magic.” Here I will show a "conversation" with GPT-3 to gauge how good a Data Scientist — or an illusion of a Data Scientist — it is.

The Great Unzippening

swyx's site RSS Feed — Sat, 30 Jul 2022 06:26:29 GMT

Society is splintering in an unacceptable way and I have a metaphor for it.

Waste Spammers Time to Kill Their Return on Investment

Brightball Articles — Sat, 30 Jul 2022 03:00:00 GMT

Continuing our series from 2012 where I accidentally ended up combating phishing and fraud for a year, we move onto the spam issue. Everything that happened that year was an exercise in triage. Problems were everywhere on the system and in the marketplace. The site I was working on was the leader in a niche space but it wasn't just the phish who tried to capitalize on the chaos, it was our competitors too. Spam takes a time investment and every time investment is a business decision. If you can't stop it completely, you can at least dramatically increase their costs...and have fun doing it.

On composability

bayindirh — Sat, 30 Jul 2022 03:00:00 GMT

Lately, I’m working on a new tool which will help me to automate the tasks I manually do using my desktop computer. While these tasks are mundane, they increase life quality of me and many individuals. As I mentioned before, I won’t have a Linux desktop system, and will move to a Mac laptop.

Naturally, I wanted to move these workflows to macOS.

Moving these workflows to macOS is indeed possible. However, I wanted to automate them to save time and create consistent outputs while at it, and this proved to be a different story.

macOS has automation capabilities, namely AppleScript and Automator. First one is in maintenance mode for a long time, and second one relies on endpoints published by applications themselves. Moreover, Apple doesn’t want you to enable these third party automations much, and the software I have doesn’t expose the functionality I need.

However, Automator has one more trick: ability to run .sh scripts. That would be useful, if the software I had CLI counterparts (like Tower, BBEdit and Monodraw), but no, they are GUI first apps.

Hence, I can’t automate the tasks the way I need on macOS. Also, I’m not a fan of Homebrew and other package managers for macOS, hence I’m stuck.

Then, I remembered having no desktop computer doesn't mean I can't run Linux.

I took out a Raspberry Pi 3, installed a 64 bit OS, and started to play around, then decided to design the tool I mentioned at the beginning of this post. While designing the system I wanted to create, something made itself apparent.

The magic of Linux is rooted in its composability.

To create the system I want, I can use a lot of available packages, plug them together, and lastly write the little tool which binds them (yes, like The Ring), and create the bigger system I have dreamt of.

Today's developers use and exploit these principles a lot, and build amazing things with them. Since they didn't live in a world where these things are absent, they take these features as granted and sometimes do not apply the same principles, or underestimate the power of them.

It's important to understand that the UNIX principles and the GPL protecting the tools which makes this composability possible underpins and enables almost everything we do with Linux systems today. Dismissing them as useless, old-fashioned or backwards is equal to cutting the branch we're sitting on.

Until next time,

Be kind.

Please stop citing TIOBE

Krishna's blog — Thu, 28 Jul 2022 03:00:00 GMT

This article critiques the TIOBE Programming Community index and provides better ways to evaluate programming languages for projects and as a developer.

Use One Big Server

Speculative Branches — Wed, 27 Jul 2022 03:00:00 GMT

A lot of ink is spent on the "monoliths vs. microservices" debate, but the real issue behind this debate is about whether distributed system architecture is worth the developer time and cost overheads. By thinking about the real operational considerations of our systems, we can get some insight into whether we actually need distributed systems for most things.

We have all gotten so familiar with virtualization and abstractions between our software and the servers that run it. These days, "serverless" computing is all the rage, and even "bare metal" is a class of virtual machine. However, every piece of software runs on a server. Since we now live in a world of virtualization, most of these servers are a lot bigger and a lot cheaper than we actually think.

The Poor American Narrative

jodavaho.io — Tue, 26 Jul 2022 03:00:00 GMT

I recently read a wonderful piece by the New York Times titled “What happened when 7 Trump voters and 6 Biden voters tried to find common ground”. What struck me was how quickly even the most middling of issues quickly diverged along party lines, often with classic party narratives appearing. It is absolutely worth a read.

These folks are living profoundly different American experiences, and yet both feel that the future is bleak. Today, one person can blithely say that an entire political party, or an entire race, or an entire economic group, or even the entire country is to blame for what they believe is the fundamental problem in the country. “Systemic racism”, “Illegal immegrants”, “White people”, “Corporations”, “Billionaires”, “Religion”. At least one participant all but admitted this, saying the country needs to be torn down and rebuilt.

The entire nation, it seems, has embraced a kind of catastrophic negative self-talk.

Why?
At the risk of sounding trite, I think the stories we tell ourselves matter. We force ourselves, or are forced by others, to focus on the negative things happening to our group, especially as perptrated by another group. Our internal narrative tells us the bad people are everywhere, they have no concern for our wellbeing or in fact want to hurt us, they are taking power from us for their own selfish desires. They are unfeeling, uncaring, malicious, and irredeemably so. They are not part of the solution, they are a maniacal distraction from progress towards a better world.

If a person were generating these narratives internally, they would have legitimate reason to seek professional help with their mental health.

The three main narratives of depression can be summarized as: “I’m bad. The world is bad. There’s nothing anyone can do about it” 1 Whatever the troubles of the world, a laser focus on the negative and how it reinforces a fundamentally dire internal narrative is a big part of the definition of clinical depression. Believing that problems are so big that there is nothing one can undertake to change the narrative is another. Words like “systemic” are excellent for this! How does one non-violently dismantle a whole system! All that’s missing is the sense that oneself is somehow fundamentally flawed, and if you don’t think that already, someone outside your political persuasion is happy to tell you this.

And yet, because we allow others to generate this narrative, and reinforce it in us, we don’t wonder at it. We don’t seek help getting rid of it. It’s not unhealthy! I’m awake when everyone else is asleep!

I’m worried we’re letting bad stories, even if they have a kernel of truth, dominate our attention. I’m worried that we’re unaware that obsessive negative self-talk is fundamentally changing the way we see eachother, our institutions, and our country, in a way that blinds us to what is positive and what is possible. I’m worried this causes us to “write off” whole groups of people simply because the story says they are fundamentally bad. I’m concerned that we have personally allowed these stories to become all-consuming, so that even unrelated negative events, even minor unrelated events, are getting vacuumed up in a false causal relationship.

Research into depression and mental illness has identified this pattern repeatedly, and has a worthwhile answer.

The answer is trite, but I believe important. It is something each person can do. We must positively reframe every event, every news story, and every political cycle by default. We have to learn to see the good in day-to-day events. We need to learn to look plainly at societal problems believing that a resolution is possible and those who disagree with you are part of the solution, not the cause of the problem. And we need to mention this positive reframing to others, to set an example about a more healthy way of thinking and problem solving.

There are not concise, key causes to all the worlds evils. Down that road is madness.

License

CC BY-SA 4.0

Vim tip 12: save and restore sessions

learnbyexample — Tue, 26 Jul 2022 03:00:00 GMT

You can save and restore Vim sessions to continue working with the same setup before you had to quit Vim for reasons like switching off the machine, switching to another project, etc.

:mksession proj.vim save the current Vim session with details like cursor position, file list, layout, etc
- you can customize things to be saved using the sessionoptions setting
- for example, :set sessionoptions+=resize will save resized window information as well
:mksession! proj.vim overwrite existing session
:source proj.vim restore Vim session from proj.vim file
- vim -S proj.vim restore a session from the command line when launching Vim

See :h 21.4, :h views-sessions and :h 'sessionoptions' for more details.

See stackoverflow: How to save and restore multiple different sessions in Vim? for custom settings to automate the save and restore process and other tips and tricks. See also Learn-Vim: Views, Sessions, and Viminfo.

Video demo:

Enterprise Challenges with DMARC Deployment

Brightball Articles — Mon, 25 Jul 2022 03:00:00 GMT

DMARC deployment projects in larger organizations come with their own variety of challenges. A great many more people are involved, so there will be more communication, more approvals and more politics. Others will object on the basis of size. "Our company is simply too large!" some will say.In the final section of our DMARC guide, we will discuss these common concerns and how to address the challenges. If 74% of the US Federal goverment did this in about a year, you can too.

Container scheduling strategies for integration testing 14 different databases in Github Actions

Notes on software development — Mon, 25 Jul 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

What to do when iCloud is stuck on uploading items

swyx's site RSS Feed — Sun, 24 Jul 2022 21:52:42 GMT

I use iCloud as my syncing engine for my Obsidian Second Brain, and twice now I've seen iCloud get corrupted into a really bad state. I also back up everything to github, so I dont really experience much data loss, but it is annoying to see iCloud be so unreliable at even the simple task of syncing mostly text files.

Deploying DMARC Without Breaking Everything

Brightball Articles — Sat, 23 Jul 2022 03:00:00 GMT

Too scary? Messing with the configuration on your domain email is scary, especially if you're already sending a lot of it. You have to worry that you're going to screw something up and break all of the email communications for the entire company. That's what I was worried when I first rolled this out and had no idea what I was doing. One of the reasons I'm such a big advocate for DMARC today is that it was painless, easy and involve no risk at all.

The Elements of Typographic Style

Bytepawn - Marton Trencseni — Sat, 23 Jul 2022 01:00:00 GMT

The Elements of Typographic Style by Robert Bringhurst is the most beautiful book I've ever held in my hand. This stunning piece of readable art shows Bringhurst's love for the craft of design, typography and writing, and his mastery of these subject, a result of his life-long devotion to them. I am not a professional typographer, but I enjoyed glancing at, reading and appreciating every page of this book.

The Culture Map

Bytepawn - Marton Trencseni — Fri, 22 Jul 2022 01:00:00 GMT

The Culture Map by Erin Meyer is a system of 8 scales which can be used to determine how cultures vary along a spectrum. The scales can be used to analyse one culture relative to another and decode how culture influences your international collaborations. I find the considerations in the book helpful irrespective of cultural background; the 8 scales can be applied to individuals as well, irrespective of where they are from.

If you succeed, you will fail

Ben E. C. Boyter — Thu, 21 Jul 2022 03:00:00 GMT

If you succeed, you will fail.

Something said at a client engagement I was at some time ago. I figure enough time has passed that it is worth sharing this story.

One of the early engagements was to deliver for a very large company an iOS Application powered by a RESTful API. This was problematic due to what is common for every large organisation, a collection of older legacy backends. Some with their own API’s, and bizzare authentication processes. Part of the deliverable was also working with push notifications for things like billing updates, documenting the systems (since few people knew how everything worked) and load testing.

CLI tip 13: join lines of two files based on the first field

learnbyexample — Wed, 20 Jul 2022 03:00:00 GMT

By default, join combines two files based on the first field content (also referred as key). Only the lines with common keys will be part of the output. The key field will be displayed first in the output (this distinction will come into play if the first field isn't the key). Rest of the line will have the remaining fields from the first and second files, in that order. One or more blanks (space or tab) will be considered as the input field separator and a single space will be used as the output field separator. If present, blank characters at the start of the input lines will be ignored.

# sample sorted input files
$ cat jan.txt
apple   10
banana  20
soap    3
tshirt  3
$ cat feb.txt
banana  15
fig     100
pen     2
soap    1

# combine common lines based on the first field
$ join jan.txt feb.txt
banana 20 15
soap 3 1

Here's an awk version to do the same. Helpful if you want to do some additional processing that won't be possible with the join command. Another advantage is that this solution will work even if the input files are not sorted.

$ awk 'NR==FNR{a[$1]=$2; next} $1 in a{print $1, a[$1], $2}' jan.txt feb.txt
banana 20 15
soap 3 1

Video demo:

See join chapter from my Command line text processing with GNU Coreutils ebook for more details and examples.

The Trouble With Many Skeptics

Beetle Space — Tue, 19 Jul 2022 10:00:00 GMT

Back when I was at university, a friend mentioned to me that he wanted to read The Demon Haunted World by Carl Sagan. I had been a fan of Sagan since my teenage years, but had put off reading that book. Asking him if he’d read any of Sagan …

Word generation in Iron Arachne

Ben Overmyer's Site — Tue, 19 Jul 2022 03:00:00 GMT

When I first rewrote Iron Arachne as a static JavaScript website, I quickly hacked together several name generators using different methods. Most assembled pieces from several different chunks into a single word.

That quickly became insufficient. The results were predictable and small in number. After awhile, I wrote a simple module called "invented" which was meant to create words for invented languages, or conlangs. It wasn't a full language generator like previous iterations of Iron Arachne had. That still doesn't exist for the JavaScript version. However, it did have a simple pattern matching algorithm; if you passed it an array of strings, it would pick a random item from that array, and then go through that string character-by-character. Each character was a symbol representing a category of sounds - phonemes - that it would then pick from at random and return that. Most characters were lowercase letters. If a character was uppercase or not a letter, it would be returned unmodified - so a pattern like KONG would always return "kong" and not something with random phonemes.

The result was an invented word generator that worked well enough. I started using it in almost every part of Iron Arachne that needed made-up words.

Recently, though, things changed just a little bit. I was running out of letters of the alphabet to use as symbols representing phoneme categories. I was using two of them - u and d - to represent a doubled vowel and consonant, respectively. I decided to reclaim those letters for other things. To do so, I needed to figure out how to change the system so I could still generate doubled letters but not linearly add new symbols for them.

The solution was to add the first modifier symbol to the invented library. Instead of being a simple iterative lookup, the generate function would now store the last generated phoneme, and if the next symbol was a + it would return that last phoneme instead of looking up a different one.

That, in turn, made me think about a common scenario where I wanted to have a particular type of word use a random list of sounds instead of one of the pre-existing categories of phonemes. How would I do that, though? My existing module only checked each character in turn. Rather than try and continue down that route, I decided it was finally time to build an actual parser.

I'd never written a parser before. Not even in CS, way back in high school and college. I pivoted to a history degree before taking the otherwise-inevitable compilers class. Lexers and parsers were something I used, not something I wrote. Even now, I'm cheating a bit and using regular expressions to handle most of the heavy lifting.

The idea is that I'll be able to write patterns like this:

(K,G)LvNGON

And the generator will spit out words like this:

Klangon
Glingon
Glengon
Klungon

So anytime the parser encounters a series of characters in parenthesis, it returns a random unmodified item from that list instead of a random phoneme from a lookup table. I want the system to be flexible enough that I can add additional workflows later. Perhaps I might want to add a way to return a random phoneme that rhymes with a pattern, for example.

I haven't completed the new parser as I'm writing this. However, I imagine it'll be done sometime in the next few weeks.

Combating Phishing with DMARC

Brightball Articles — Mon, 18 Jul 2022 03:00:00 GMT

Email shouldn't feel like a dark art, but to a lot of people it does. Everyone should have DMARC setup by this point, but they don't. Here's the first piece of a 3 part guide covering why it works and how to set it up.Since writing about how to reverse account takeovers last week I've decided to write a security series covering all the weird things I encountered back in 2012, when I accidentally ended up combating phishing and fraud for a year. In the last article, the first recommendation was to setup DMARC. So let's take a deeper look at why, how and what's involved in long term management once it's setup.

AWS Abusing Search Engine Gets Abused

Ben E. C. Boyter — Sun, 17 Jul 2022 03:00:00 GMT

Some time ago I wrote about building an Australian search engine by abusing how AWS Lambdas work. You can view it at bonzamate.com.au, and read the post about it here.

Recently I noticed that Bonzamate.com.au was in turn being abused by a series of bots repeatedly making queries against it.

Now normally I don’t worry about bots. There is little you can do to stamp them out entirely, and its always nice to see someone trying to exploit the things you work on. However should the bots either result in degraded service or in this case increased personal costs to me I will actively try to work against them.

How to Manifest a Meetup

swyx's site RSS Feed — Sat, 16 Jul 2022 01:48:48 GMT

2024 update: for those who work in devrel/organize less-social-more-technical meetups, I have now written How to Organize Meetups Good on Dx.tips

Checklist For Evaluating Cloud Backup Services

Beetle Space — Thu, 14 Jul 2022 10:00:00 GMT

If I delete a file and sync, can I recover the deleted file?
- Is there a deadline (e.g. only within 90 days)? Or will they keep it forever?
If I made changes to the file and synced, can I access prior versions?
- Is there a limit to how many …

On Prescriptive Descriptions

Weblog on marginalia.nu — Thu, 14 Jul 2022 03:00:00 GMT

I’d like to discuss a mental somersault that I’ve found has caused me a lot of grief in the past, which is prescriptive descriptions. Let’s break this down a bit:

A descriptive statement is a statement about how something is.
A prescriptive statement is a statement about how something must be, a rule or a law.

If I stay up a bit late, do most of my work in the evenings, wake up tired and just sort of putter about until noon, I might describe myself as a night person because of this.

Python tip 13: formatting numbers with underscore separation

learnbyexample — Wed, 13 Jul 2022 03:00:00 GMT

For readability purposes, you can use underscores while declaring large numbers. For example:

>>> 1_000_000_000
1000000000

>>> 0b1000_1111
143

Did you know that you can also format numbers with underscore separation?

>>> n = 14310023

# underscore separation
>>> f'{n:_}'
'14_310_023'

# you can also use comma separation for integers
>>> f'{n:,}'
'14,310,023'

Here are some examples for displaying numbers in binary, octal and hexadecimal formats:

>>> n = 14310023

>>> f'{n:_b}'
'1101_1010_0101_1010_1000_0111'
>>> f'{n:#_b}'
'0b1101_1010_0101_1010_1000_0111'

>>> f'{n:#_x}'
'0xda_5a87'

>>> f'{n:#_o}'
'0o6645_5207'

And here's an example with zero filling:

>>> for n in (3, 20, 28):
...     print(f'{n:09_b}')
... 
0000_0011
0001_0100
0001_1100

See docs.python: Formatted string literals for documentation and other examples.

Video demo:

See also my 100 Page Python Intro ebook.

Automatically Reversing Account Takeovers

Brightball Articles — Mon, 11 Jul 2022 03:00:00 GMT

Today, Brian Krebs reported on account takeovers happening at Experian, one of the 3 major credit agencies. The first step after getting account access is to lock out the account owner, usually by swapping the email address. 10 years ago I dealt with this problem extensively, so I'd like to share how to solve it.

Overstretching

bayindirh — Sun, 10 Jul 2022 03:00:00 GMT

Recently, I have been following a lot of events that made me uneasy about the future of both GNU/Linux and Free Software in general. Normally I have been an observer in these events, but I found that I have formed pretty coherent set of ideas, feelings and preferences for the future evolution of these things.

As a result, I've found myself inside many discussions about these topics, and I came out pretty unimpressed and disappointed about the current state of things. In a sense I have been disillusioned and found out that many fundamental aspects I support and defend is either misunderstood or dismissed as useless.

Experiencing this, and trying to put my ideas forward has drained me beyond my comfort zone, and more importantly somewhat impaired my capacity to function in areas I need to function.

This got me thinking. How can I continue to support the things I care without burning myself out and impairing my daily life? The answer I found is to dive deeper into these subjects, learn more, and talk less, but in a richer manner. Because, I found out that I'm not knowledgeable enough to discuss the things at the depth I want to discuss, and counter the arguments I receive with the clarity I crave for.

On the other hand, I'll commit more into the things I care. I'll be more strict about licensing my software, and where I host and serve said software. I may write more about these subjects in this blog, and voice my objections more frequently, but in a more kinder and easier to understand language.

More importantly, I've found out that these turn of events have created the spark I have been missing for a long time. Settling into a comfortable software stack and unintentionally locking myself into a bubble has isolated me from the events which were taking place around me. I feel like I'm awaken from a long slumber. I think I can use this newfound energy and motivation for good things.

Hope to see you around,

Be kind.

Implementing a simple jq clone in Go, and basics of Go memory profiling

Notes on software development — Sun, 10 Jul 2022 03:00:00 GMT

In this post we'll build a basic jq clone in Go. It will only be able to pull a single path out of each object it reads. It won't be able to do filters, mapping, etc.

$ cat large-file.json | head -n2 | ./jqgo '.repo.url'
"https://api.github.com/repos/petroav/6.828"
"https://api.github.com/repos/rspt/rspt-theme"

We'll start by building a "control" implementation that uses Go's builtin JSON library with a JSON path tool on top.

Then we'll implement a basic path-aware JSON parser in 600 lines of Go. It's going to use a technique (that may have a better name but) I call "partial parsing" or "fuzzy parsing" where we fully parse what we care about and only sort of parse the rest.

Why partial parsing? There are two general reasons. One is to use less memory than parsers that must always turn all of a text into an object in your language. The other is for when the language has complexities you don't want or need to deal with. We'll basically have to deal with all the complexities of JSON so this post is about the former reason: using less memory. I've written about a case for the second reason though in building a simple, fast SCSS implementation.

This partial parser is more complex than a typical handwritten parser. If you are unfamiliar with handwritten JSON parsers, you may want to take a look at previous articles I've written about parsing JSON.

Once we get this partial parser working we'll turn to Go's builtin profiler to find what we can do to make it faster.

All code for this post is available on Github.

Machine specs, versions

Since we're going to be doing some rudimentary comparisons of performance, here are my details. I am running everything on a dedicated server, OVH Rise-1.

RAM: 64 GB DDR4 ECC 2,133 MHz
Disk: 2x450 GB SSD NVMe in Soft RAID
Processor: Intel Xeon E3-1230v6 - 4c/8t - 3.5 GHz/3.9 GHz

And relevant versions:

$ jq --version
jq-1.6
$ go version
go version go1.18 linux/amd64
$ uname -a
Linux phil 5.18.10-100.fc35.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 7 17:41:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Now buckle up!

jq using Go's builtin JSON library

This is a very simple program. We just parse JSON data from stdin in a loop. And after parsing each time we'll call a extractValueAtPath function to grab the value at the path the user asks for.

To keep our path "parser" very simple we'll treat array access the same as object access. So we'll look for x.0 instead of x[0], unlike jq.

package main

import (
    "encoding/json"
    "io"
    "log"
    "os"
    "strconv"
    "strings"
)

func extractValueAtPath(a map[string]any, path []string) (any, error) {
    // TODO
}

func main() {
    path := strings.Split(os.Args[1], ".")
    if path[0] == "" {
        path = path[1:]
    }

    dec := json.NewDecoder(os.Stdin)
    var a map[string]any

    enc := json.NewEncoder(os.Stdout)

    for {
        err := dec.Decode(&a)
        if err == io.EOF {
            break
        }

        if err != nil {
            log.Fatal(err)
        }

        v, err := extractValueAtPath(a, path)
        if err != nil {
            log.Fatal(err)
        }

        err = enc.Encode(v)
        if err != nil {
            log.Fatal(err)
        }
    }
}

Then we implement the extractValueAtPath function itself, entering into JSON arrays and objects until we reach the end of the path.

func extractValueAtPath(a map[string]any, path []string) (any, error) {
    if len(path) == 0 {
        return nil, nil
    }

    var v any = a

    for _, part := range path {
        if arr, ok := v.([]any); ok {
            n, err := strconv.Atoi(part)
            if err != nil {
                return nil, err
            }

            v = arr[n]
            continue
        }

        m, ok := v.(map[string]any)
        if !ok {
            // Path into a non-map
            return nil, nil
        }

        v, ok = m[part]
        if !ok {
            // Path does not exist
            return nil, nil
        }
    }

    return v, nil
}

Alright, let's give it a go module and build and run it!

$ go mod init control
$ go mod tidy
$ go build
# Grab a test file
$ curl https://raw.githubusercontent.com/json-iterator/test-data/master/large-file.json | jq -c '.[]' > large-file.json
$ cat large-file.json | head -n2 | ./control '.repo.url'
"https://api.github.com/repos/petroav/6.828"
"https://api.github.com/repos/rspt/rspt-theme"

Sweet. Now let's make sure it produces the same thing as jq.

$ cat large-file.json | ./control '.repo.url' > control.test
$ cat large-file.json | jq '.repo.url' > jq.test
$ diff jq.test control.test
$ echo $?
0

Great! It's working for a basic query. Let's see how it performs.

$ hyperfine \
  "cat large-file.json | ./control '.repo.url' > control.test" \
  "cat large-file.json | jq '.repo.url' > jq.test"
Benchmark 1: cat large-file.json | ./control '.repo.url' > control.test
  Time (mean ± σ):     310.0 ms ±  14.4 ms    [User: 296.2 ms, System: 49.3 ms]
  Range (min … max):   296.1 ms … 344.9 ms    10 runs

Benchmark 2: cat large-file.json | jq '.repo.url' > jq.test
  Time (mean ± σ):     355.8 ms ±   1.1 ms    [User: 348.8 ms, System: 27.7 ms]
  Range (min … max):   354.8 ms … 358.5 ms    10 runs

Summary
  'cat large-file.json | ./control '.repo.url' > control.test' ran
    1.15 ± 0.05 times faster than 'cat large-file.json | jq '.repo.url' > jq.test'

Now that's surprising! This naive implementation in Go is a bit faster than standard jq. But our implementation supports a heck of a lot less than jq. So this benchmark on its own isn't incredibly meaningful.

However, it's a good base for comparing to our next implementation.

Astute readers may notice that this version doesn't use a buffered reader from stdin, while the next version will. I tried this version with and without wrapping stdin in a buffered reader but it didn't make a meaningful difference. It might be because Go's JSON decoder does its own buffering. I'm not sure.

Let's do the fun implementation.

Partial parsing

Unlike a typical handwritten parser this partial parser is going to contain almost two parsers. One parser will care exactly about the structure of JSON. The other parser will only care about reading past the current value (whether it be a number or string or array or object, etc.) The path we pass to the parser will be used to decide whether each value should be fully parsed or partially parsed.

I'll reiterate: this partial parser is more complex than a typical handwritten parser. If you are unfamiliar with handwritten JSON parsers, you may want to take a look at previous articles I've written about parsing JSON.

The shell of this partial parser is going to look similar to the shell of the first parser.

package main

import (
    "bufio"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "os"
    "strconv"
    "strings"
)

type jsonReader struct {
    read []byte
}

... TO IMPLEMENT ...

func main() {
    path := strings.Split(os.Args[1], ".")
    if path[0] == "" {
        path = path[1:]
    }

    b := bufio.NewReader(os.Stdin)
    enc := json.NewEncoder(os.Stdout)

    var jr jsonReader
    var val any
    var err error

    for {
        jr.reset()

        val, err = jr.extractDataFromJsonPath(b, path)
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Println("Read", string(jr.read))
            log.Fatalln(err)
        }

        err = enc.Encode(val)
        if err != nil {
            log.Fatalln(err)
        }
    }
}

Except instead of using the builtin JSON parser we'll call our own extractDataFromJsonPath function that handles parsing and extraction all at once.

Before doing that we'll add a few helper functions. The first one grabs a byte from a reader and stores the read byte locally (so we can print out all read bytes if the program fails).

type jsonReader struct {
    read []byte
}

func (jr *jsonReader) readByte(r *bufio.Reader) (byte, error) {
    c, err := r.ReadByte()
    if err != nil {
        return byte(0), err
    }

    jr.read = append(jr.read, c)

    return c, nil
}

The reset member zeroes out the read bytes and gets called before each object is parsed in the main main loop.

func (jr *jsonReader) reset() {
    jr.read = nil
}

Now let's get into extractDataFromJsonPath.

extractDataFromJsonPath

This is the real parser. It expects a JSON object and fully parses the object, almost.

func (jr *jsonReader) extractDataFromJsonPath(r *bufio.Reader, path []string) (any, error) {
    if len(path) == 0 {
        return nil, nil
    }

    err := jr.eatWhitespace(r)
    if err != nil {
        return nil, err
    }

    b, err := jr.readByte(r)
    if err != nil {
        return nil, err
    }

    // Make sure we're actually going into an object
    if b != '{' {
        return nil, fmt.Errorf("Expected opening curly brace, got: '%s'", string(b))
    }

    i := -1

    var result any
    for {
        i++

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

        bs, err := r.Peek(1)
        if err != nil {
            return nil, err
        }
        b := bs[0]

        // We found the end of the object
        if b == '}' {
            r.Discard(1)
            break
        }

        // Key-value pairs must be separated by commas
        if i > 0 {
            if b == ',' {
                r.Discard(1)
            } else {
                return nil, fmt.Errorf("Expected comma between key-value pairs, got: '%s'", string(b))
            }
        }

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

        // Grab the key
        s, err := jr.expectString(r)
        if err != nil {
            return nil, err
        }

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

        // Find a colon separating key from value
        b, err = jr.readByte(r)
        if err != nil {
            return nil, err
        }

        if b != ':' {
            return nil, fmt.Errorf("Expected colon, got: '%s'", string(b))
        }

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

Up to this point it looks like any old handwritten parser. There are a few helpers in there (eatWhitespace, expectString) we'll implement shortly.

But once we see each key and are ready to look for a value we can decide if we need to fully parse the value (if the path goes into this key) or if we can partially parse the value (because the path does not go into this key).

        // If the key is not the start of this path, skip past this value
        if path[0] != s {
            err = jr.eatValue(r)
            if err != nil {
                return nil, err
            }

            continue
        }

        // Otherwise this is a path we want, grab the value
        result, err = jr.expectValue(r, path[1:])
        if err != nil {
            return nil, err
        }
    }

    return result, nil
}

And that's it! The core parsing loop is done. The meat now becomes 1) the eatValue function that partially parses JSON and 2) the expectValue function that either encounters a scalar value and returns it or recursively calls extractDataFromJsonPath to enter some new object.

Notes on helper naming

There are three main kinds of helpers you'll see. expectX helpers like expectString will return early with an error if they fail to find what they're looking for. eatX helpers like eatWhitespace will not return any value and will only move the read cursor forward. And tryX helpers like tryNumber will do the same thing as expectString but return an additional boolean argument. So the caller can decide whether or not to make other attempts at parsing.

But first let's fill in the two helpers we skipped. First off, eatWhitespace.

eatWhitespace

This function peeks and reads bytes while the bytes are whitespace.

func (jr *jsonReader) eatWhitespace(r *bufio.Reader) error {
    for {
        bs, err := r.Peek(1)
        if err != nil {
            return err
        }
        b := bs[0]

        isWhitespace := b == ' ' ||
            b == '\n' ||
            b == '\t' ||
            b == '\r'
        if !isWhitespace {
            return nil
        }

        r.Discard(1)
    }
}

That's it! Next we need to fill in expectString.

expectString

This is a standard handwritten parser helper that looks for a double quote and keeps collecting bytes until it finds an ending double quote that is not escaped.

func (jr *jsonReader) expectString(r *bufio.Reader) (string, error) {
    var s []byte

    err := jr.eatWhitespace(r)
    if err != nil {
        return "", err
    }

    // Look for opening quote
    b, err := jr.readByte(r)
    if err != nil {
        return "", err
    }

    if b != '"' {
        return "", fmt.Errorf("Expected double quote to start string, got: '%s'", string(b))
    }

    var prev byte
    for {
        b, err := jr.readByte(r)
        if err != nil {
            return "", err
        }

        if b == '\\' && prev == '\\' {
            // Just skip
            prev = byte(0)
            continue
        } else if b == '"' {
            // Overwrite the escaped double quote
            if prev == '\\' {
                s[len(s)-1] = '"'
            } else {
                // Otherwise it's the actual end
                break
            }
        }

        s = append(s, b)
        prev = b
    }

    return string(s), nil
}

Standard stuff! Now let's get back to those meaty functions we introduced before, starting with expectValue.

expectValue

This function is called by extractDataFromJsonPath when it wants to fully parse a value.

If we see a left curly brace, we call extractDataFromJsonPath with it.

func (jr *jsonReader) expectValue(r *bufio.Reader, path []string) (any, error) {
    bs, err := r.Peek(1)
    if err != nil {
        return nil, err
    }
    c := bs[0]

    if c == '{' {
        return jr.extractDataFromJsonPath(r, path)

Otherwise if we see a left bracket we call a new helper extractArrayDataFromJsonPath which will be almost identical to extractDataFromJsonPath but for parsing array syntax.

    } else if c == '[' {
        return jr.extractArrayDataFromJsonPath(r, path)
    }

If the value we're trying to parse isn't an array or object and there's more of a path then we have to return null because we can't enter into a scalar value.

    // Can't go any further into a path

    if len(path) != 0 {
        // Reached the end of this object but more of
        // the path remains. So this object doesn't
        // contain this path.
        return nil, nil
    }

Then we try to parse a scalar (numbers, strings, booleans, null) and ultimately return an error if nothing worked.

    ok, val, err := jr.tryScalar(r)
    if err != nil {
        return nil, err
    }
    if !ok {
        return nil, fmt.Errorf("Expected scalar, got: '%s'", string(c))
    }

    return val, err
}

Let's implement tryScalar and its dependencies now. And we'll come back to extractArrayDataFromJsonPath afterward.

tryScalar

The tryScalar is similar to expectValue. It's called tryScalar because it's allowed to fail.

We peek at the first byte and switch on a dedicated parsing helper based on it.

func (jr *jsonReader) tryScalar(r *bufio.Reader) (bool, any, error) {
    bs, err := r.Peek(1)
    if err != nil {
        return false, nil, err
    }
    c := bs[0]

    if c == '"' {
        val, err := jr.expectString(r)
        return true, string(val), err
    } else if c == 't' {
        val, err := jr.expectIdentifier(r, "true", true)
        return true, val, err
    } else if c == 'f' {
        val, err := jr.expectIdentifier(r, "false", false)
        return true, val, err
    } else if c == 'n' {
        val, err := jr.expectIdentifier(r, "null", nil)
        return true, val, err
    }

    return jr.tryNumber(r)
}

This passes control flow to two new functions, expectIdentifier and tryNumber. Let's do expectIdentifier next.

expectIdentifier

This function tries to match the reader on a string passed to it.

func (jr *jsonReader) expectIdentifier(r *bufio.Reader, ident string, value any) (any, error) {
    var s []byte

    for i := 0; i < len(ident); i++ {
        b, err := r.ReadByte()
        if err != nil {
            return nil, err
        }

        s = append(s, b)
    }

    if string(s) == ident {
        return value, nil
    }

    return nil, fmt.Errorf("Unknown value: '%s'", string(s))
}

Thanks Michael Lynch for pointing out in an earlier version that expectIdentifier does not need to Peek/Discard but can just ReadByte instead.

tryNumber

This function tries to parse a number. We'll do a very lazy number parser that will most likely allow all valid numbers. Internally we'll call json.Unmarshal on the bytes we build up to do the conversion itself.

func (jr *jsonReader) tryNumber(r *bufio.Reader) (bool, any, error) {
    var number []byte

    // Loop trying to find all number-like characters in a row
    for {
        bs, err := r.Peek(1)
        if err != nil {
            return false, nil, err
        }
        c := bs[0]

        isNumberCharacter := (c >= '0' && c <= '9') || c == 'e' || c == '-'
        if !isNumberCharacter {
            break
        }

        number = append(number, c)

        r.Discard(1)
    }

    if len(number) == 0 {
        return false, nil, nil
    }

    var n float64
    err := json.Unmarshal(number, &n)
    return true, n, err
}

If we can't find a number, that's ok. We'll just say so in the first argument by returning false.

Outstanding functions

Ok we've come a while building out helper functions. The last two remaining helpers are extractArrayDataFromJsonPath and eatValue. Let's finish up these real parser functions before getting to eatValue, the primary partial parsing function.

extractArrayDataFromJsonPath

This function is almost identical to extractDataFromJsonPath but rather than parsing key-value pairs inside curly braces it parses values inside brackets.

func (jr *jsonReader) extractArrayDataFromJsonPath(r *bufio.Reader, path []string) (any, error) {
    // Path inside an array must be an integer
    n, err := strconv.Atoi(string(path[0]))
    if err != nil {
        return nil, err
    }

    // Look for opening bracket. Make sure we're in an array
    b, err := jr.readByte(r)
    if err != nil {
        return nil, err
    }

    if b != '[' {
        return nil, fmt.Errorf("Expected opening bracket, got: '%s'", string(b))
    }

    var result any
    i := -1
    for {
        i++

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

        bs, err := r.Peek(1)
        if err != nil {
            return nil, err
        }
        b := bs[0]

        // Found closing bracket, exit the array
        if b == ']' {
            r.Discard(1)
            break
        }

        // Array values must be separated by a comma
        if i > 0 {
            if b == ',' {
                r.Discard(1)
            } else {
                return nil, fmt.Errorf("Expected comma between key-value pairs, got: '%s'", string(b))
            }
        }

        err = jr.eatWhitespace(r)
        if err != nil {
            return nil, err
        }

Just like extractDataFromJsonPath it either calls eatValue or expectValue depending on whether the current index matches the requested path.

        // If the key is not the start of this path, skip past this value
        if i != n {
            err = jr.eatValue(r)
            if err != nil {
                return nil, err
            }

            continue
        }

        result, err = jr.expectValue(r, path[1:])
        if err != nil {
            return nil, err
        }
    }

    return result, nil
}

That's it for full parser functions! Let's do the partial parser, eatValue.

eatValue

This function is simpler than the full parser functions we wrote before.

First off it looks for the simple case where the value is a scalar.

func (jr *jsonReader) eatValue(r *bufio.Reader) error {
    var stack []byte

    inString := false
    var prev byte

    err := jr.eatWhitespace(r)
    if err != nil {
        return err
    }

    ok, _, err := jr.tryScalar(r)
    if err != nil {
        return err
    }

    // It was a scalar, we're done!
    if ok {
        return nil
    }

All it does is read until the value ends.

If the value is not a scalar though we need to read past complete JSON arrays and/or objects.

To do this we'll simply read through bytes, monitoring a stack of open and close braces and brackets. If we enter a string we'll skip all bytes inside the string until the string ends.

    // Otherwise it's an array or object
    first := true

    for first || len(stack) > 0 {
        first = false

        bs, err := r.Peek(1)
        if err != nil {
            return err
        }
        b := bs[0]

        if inString {
            if b == '"' && prev != '\\' {
                inString = false
            }

            // Two \\-es cancel eachother out
            if b == '\\' && prev == '\\' {
                prev = byte(0)
            } else {
                prev = b
            }

            r.Discard(1)
            continue
        }

        switch b {
        case '[':
            stack = append(stack, b)
        case ']':
            c := stack[len(stack)-1]
            stack = stack[:len(stack)-1]
            if c != '[' {
                return fmt.Errorf("Unexpected end of array: '%s'", string(c))
            }
        case '{':
            stack = append(stack, b)
        case '}':
            c := stack[len(stack)-1]
            stack = stack[:len(stack)-1]
            if c != '{' {
                return fmt.Errorf("Unexpected end of object: '%s'", string(c))
            }
        case '"':
            inString = true
            // Closing quote case handled elsewhere, above
        }

        r.Discard(1)
        prev = b
    }

    return nil
}

And we're finally done the first pass of the path-aware jq implementation.

Build, test, benchmark

Let's give it a go module, build and test it.

$ go mod init jqgo
$ go build
$ curl https://raw.githubusercontent.com/json-iterator/test-data/master/large-file.json | jq -c '.[]' > large-file.json
$ cat large-file.json | ./jqgo '.repo.url' > jqgo.test
$ cat large-file.json | jq '.repo.url' > jq.test
$ diff jq.test jqgo.test
$ echo $?
0

Great! :) Let's benchmark it against jq and the control implementation.

$ hyperfine --warmup 2 \
  "cat large-file.json | ./control/control '.repo.url' > control.test" \
  "cat large-file.json | ./jqgo '.repo.url' > jqgo.test" \
  "cat large-file.json | jq '.repo.url' > jq.test"
Benchmark 1: cat large-file.json | ./control/control '.repo.url' > control.test
  Time (mean ± σ):     302.0 ms ±   3.4 ms    [User: 283.7 ms, System: 53.1 ms]
  Range (min … max):   297.4 ms … 309.0 ms    10 runs

Benchmark 2: cat large-file.json | ./jqgo '.repo.url' > jqgo.test
  Time (mean ± σ):     258.8 ms ±   2.2 ms    [User: 230.3 ms, System: 47.6 ms]
  Range (min … max):   256.3 ms … 262.6 ms    11 runs

Benchmark 3: cat large-file.json | jq '.repo.url' > jq.test
  Time (mean ± σ):     357.6 ms ±   2.9 ms    [User: 350.0 ms, System: 28.3 ms]
  Range (min … max):   355.0 ms … 362.9 ms    10 runs

Summary
  'cat large-file.json | ./jqgo '.repo.url' > jqgo.test' ran
    1.17 ± 0.02 times faster than 'cat large-file.json | ./control/control '.repo.url' > control.test'
    1.38 ± 0.02 times faster than 'cat large-file.json | jq '.repo.url' > jq.test'

Now to my surprise we're already beating the non-path-aware control implementation! When I first wrote the path-aware version, it was slower than the control. So I had to start performance profiling. For this blog post I tried to remake the slowest variation I could remember but I couldn't get it slower than this.

That said, the best version was faster than this so I can demonstrate the process of profiling to improve performance.

Let's dig in. :)

Profiling in Go

There are various ways to enable profiling in Go. One way some people recommend is through the builtin benchmark support in go test. I don't really like this method though. I prefer to use pkg/profile manually in main.go.

@@ -9,6 +9,8 @@
        "os"
        "strconv"
        "strings"
+
+       "github.com/pkg/profile"
 )

 type jsonReader struct {
@@ -450,6 +452,7 @@
 }

 func main() {
+       defer profile.Start().Stop()
        path := strings.Split(os.Args[1], ".")
        if path[0] == "" {
                path = path[1:]

Build and run:

$ go mod tidy
$ go build
$ cat large-file.json | ./jqgo '.repo.url' > /dev/null
2022/07/11 02:38:57 profile: cpu profiling enabled, /tmp/profile3691177944/cpu.pprof
2022/07/11 02:38:58 profile: cpu profiling disabled, /tmp/profile3691177944/cpu.pprof

Go can run a web server to visualize the pprof results but I find (after literally a few years of trying to figure it out) the CLI makes more sense to me.

$ go tool pprof /tmp/profile3691177944/cpu.pprof
File: jqgo
Type: cpu
Time: Jul 11, 2022 at 2:38am (UTC)
Duration: 401.63ms, Total samples = 270ms (67.23%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

Now we run top10 to see where we spend the bulk of time.

(pprof) top10
Showing nodes accounting for 260ms, 100% of 260ms total
Showing top 10 nodes out of 31
      flat  flat%   sum%        cum   cum%
      90ms 34.62% 34.62%      230ms 88.46%  main.(*jsonReader).eatValue
      60ms 23.08% 57.69%       70ms 26.92%  bufio.(*Reader).Peek
      50ms 19.23% 76.92%       60ms 23.08%  bufio.(*Reader).Discard
      20ms  7.69% 84.62%       20ms  7.69%  syscall.Syscall
      10ms  3.85% 88.46%       10ms  3.85%  bufio.(*Reader).Buffered (inline)
      10ms  3.85% 92.31%       10ms  3.85%  main.(*jsonReader).readByte
      10ms  3.85% 96.15%       10ms  3.85%  runtime.slicebytetostring
      10ms  3.85%   100%       10ms  3.85%  runtime.stkbucket
         0     0%   100%       10ms  3.85%  bufio.(*Reader).fill
         0     0%   100%       10ms  3.85%  encoding/json.(*Encoder).Encode

Now this is weird. Why are Peek and Discard so expensive? And why are we spending so much time in syscall.Syscall? The entire point of buffered I/O is to avoid hitting syscalls too frequently.

But since 88% of time is spent in eatValue, let's verify where in eatValue we are spending that time.

Within the pprof REPL we can enter list X where X is a regexp of a function name.

(pprof) list eatValue
Total: 260ms
ROUTINE ======================== main.(*jsonReader).eatValue in /home/phil/tmp/jqgo/mainprof.go
      90ms      230ms (flat, cum) 88.46% of Total
         .          .    159:   err := jr.eatWhitespace(r)
         .          .    160:   if err != nil {
         .          .    161:           return err
         .          .    162:   }
         .          .    163:
         .       20ms    164:   ok, _, err := jr.tryScalar(r)
         .          .    165:   if err != nil {
         .          .    166:           return err
         .          .    167:   }
         .          .    168:
         .          .    169:   // It was a scalar, we're done!
         .          .    170:   if ok {
         .          .    171:           return nil
         .          .    172:   }
         .          .    173:
         .          .    174:   // Otherwise it's an array or object
         .          .    175:   first := true
         .          .    176:
         .          .    177:   for first || len(stack) > 0 {
         .          .    178:           first = false
         .          .    179:
         .       60ms    180:           bs, err := r.Peek(1)
         .          .    181:           if err != nil {
         .          .    182:                   return err
         .          .    183:           }
      10ms       10ms    184:           b := bs[0]
         .          .    185:
      10ms       10ms    186:           if inString {
         .          .    187:                   if b == '"' && prev != '\\' {
         .          .    188:                           inString = false
         .          .    189:                   }
         .          .    190:
         .          .    191:                   // Two \\-es cancel eachother out
      10ms       10ms    192:                   if b == '\\' && prev == '\\' {
         .          .    193:                           prev = byte(0)
         .          .    194:                   } else {
         .          .    195:                           prev = b
         .          .    196:                   }
         .          .    197:
         .       60ms    198:                   r.Discard(1)
         .          .    199:                   continue
         .          .    200:           }
         .          .    201:
         .          .    202:           switch b {
         .          .    203:           case '[':
         .          .    204:                   stack = append(stack, b)
      10ms       10ms    205:           case ']':
         .          .    206:                   c := stack[len(stack)-1]
         .          .    207:                   stack = stack[:len(stack)-1]
         .          .    208:                   if c != '[' {
         .          .    209:                           return fmt.Errorf("Unexpected end of array: '%s'", string(c))
         .          .    210:                   }
         .          .    211:           case '{':
         .          .    212:                   stack = append(stack, b)
         .          .    213:           case '}':
      50ms       50ms    214:                   c := stack[len(stack)-1]
         .          .    215:                   stack = stack[:len(stack)-1]
         .          .    216:                   if c != '{' {
         .          .    217:                           return fmt.Errorf("Unexpected end of object: '%s'", string(c))
         .          .    218:                   }
         .          .    219:           case '"':

So by rank we can see we do spend the most time in Peek and Discard. Then in pulling the last item out of the stack??? That's weird. Let's ignore that.

Peek and Discard

Let's look at Peek in the pprof REPL:

(pprof) list Peek
Total: 260ms
ROUTINE ======================== bufio.(*Reader).Peek in /usr/local/go/src/bufio/bufio.go
      60ms       70ms (flat, cum) 26.92% of Total
         .          .    130:// also returns an error explaining why the read is short. The error is
         .          .    131:// ErrBufferFull if n is larger than b's buffer size.
         .          .    132://
         .          .    133:// Calling Peek prevents a UnreadByte or UnreadRune call from succeeding
         .          .    134:// until the next read operation.
      10ms       10ms    135:func (b *Reader) Peek(n int) ([]byte, error) {
         .          .    136:   if n < 0 {
         .          .    137:           return nil, ErrNegativeCount
         .          .    138:   }
         .          .    139:
      10ms       10ms    140:   b.lastByte = -1
         .          .    141:   b.lastRuneSize = -1
         .          .    142:
      20ms       20ms    143:   for b.w-b.r < n && b.w-b.r < len(b.buf) && b.err == nil {
         .       10ms    144:           b.fill() // b.w-b.r < len(b.buf) => buffer is not full
         .          .    145:   }
         .          .    146:
         .          .    147:   if n > len(b.buf) {
         .          .    148:           return b.buf[b.r:b.w], ErrBufferFull
         .          .    149:   }
         .          .    150:
         .          .    151:   // 0 <= n <= len(b.buf)
         .          .    152:   var err error
         .          .    153:   if avail := b.w - b.r; avail < n {
         .          .    154:           // not enough data in buffer
         .          .    155:           n = avail
         .          .    156:           err = b.readErr()
         .          .    157:           if err == nil {
      10ms       10ms    158:                   err = ErrBufferFull
         .          .    159:           }
         .          .    160:   }
      10ms       10ms    161:   return b.buf[b.r : b.r+n], err
         .          .    162:}
         .          .    163:
         .          .    164:// Discard skips the next n bytes, returning the number of bytes discarded.
         .          .    165://
         .          .    166:// If Discard skips fewer than n bytes, it also returns an error.

The bulk of time here is spent in refilling the buffer (the fill method). So it seems like while bufio.Reader buffers reads it basically seems to not buffer peeks.

But hey, we were peeking and discarding one at a time anyway. Peeking and discarding were the same cost in eatValue. So let's ignore peeking for a second and think about discarding.

We could avoid doing so many discards if we just keep track of how much we are peeking at in the loop and only discard once at the end of the loop. (As an implementation detail, since there's a max internal buffer size we'll need to actually periodically discard when we try to peek and get a "buffer full" error.)

And based on that top10 result above, we need to do this in eatValue.

@@ -170,16 +170,31 @@
        }

        // Otherwise it's an array or object
+       length := 0
        first := true
-
+       var bs []byte
        for first || len(stack) > 0 {
+               length++
                first = false

-               bs, err := r.Peek(1)
-               if err != nil {
-                       return err
+               for {
+                       bs, err = r.Peek(length)
+                       if err == bufio.ErrBufferFull {
+                               _, err = r.Discard(length - 1)
+                               if err != nil {
+                                       return err
+                               }
+
+                               length = 1
+                               continue
+                       }
+                       if err != nil {
+                               return err
+                       }
+
+                       break
                }
-               b := bs[0]
+               b := bs[length-1]

                if inString {
                        if b == '"' && prev != '\\' {
@@ -193,7 +208,6 @@
                                prev = b
                        }

-                       r.Discard(1)
                        continue
                }

@@ -219,11 +233,11 @@
                        // Closing quote case handled elsewhere, above
                }

-               r.Discard(1)
                prev = b
        }

-       return nil
+       _, err = r.Discard(length)
+       return err
 }

 func (jr *jsonReader) tryScalar(r *bufio.Reader) (bool, any, error) {

Comment out the pkg/profile bits (profiling slows the whole thing down), rebuild, and rerun:

$ hyperfine --warmup 2 \
  "cat large-file.json | ./control/control '.repo.url' > control.test" \
  "cat large-file.json | ./jqgo '.repo.url' > jqgo.test" \
  "cat large-file.json | jq '.repo.url' > jq.test"
Benchmark 1: cat large-file.json | ./control/control '.repo.url' > control.test
  Time (mean ± σ):     302.0 ms ±   4.2 ms    [User: 287.7 ms, System: 49.7 ms]  Range (min … max):   296.6 ms … 308.2 ms    10 runs

Benchmark 2: cat large-file.json | ./jqgo '.repo.url' > jqgo.test
  Time (mean ± σ):     215.0 ms ±   1.6 ms    [User: 189.1 ms, System: 46.9 ms]
  Range (min … max):   213.5 ms … 218.7 ms    13 runs

Benchmark 3: cat large-file.json | jq '.repo.url' > jq.test
  Time (mean ± σ):     355.7 ms ±   1.4 ms    [User: 349.9 ms, System: 26.4 ms]
  Range (min … max):   354.3 ms … 359.1 ms    10 runs

Summary
  'cat large-file.json | ./jqgo '.repo.url' > jqgo.test' ran
    1.40 ± 0.02 times faster than 'cat large-file.json | ./control/control '.repo.url' > control.test'
    1.65 ± 0.01 times faster than 'cat large-file.json | jq '.repo.url' > jq.test'

Great! We've shaved off another 40ms. Let's enable profiling, re-run the program and go back into the pprof REPL.

$ cat large-file.json | ./jqgo '.repo.url' > /dev/null
2022/07/11 03:12:07 profile: cpu profiling enabled, /tmp/profile2229743747/cpu.pprof
2022/07/11 03:12:07 profile: cpu profiling disabled, /tmp/profile2229743747/cpu.pprof
$ go tool pprof /tmp/profile2229743747/cpu.pprof
File: jqgo
Type: cpu
Time: Jul 11, 2022 at 3:12am (UTC)
Duration: 401.33ms, Total samples = 210ms (52.33%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 210ms, 100% of 210ms total
Showing top 10 nodes out of 20
      flat  flat%   sum%        cum   cum%
     100ms 47.62% 47.62%      180ms 85.71%  main.(*jsonReader).eatValue
      70ms 33.33% 80.95%       70ms 33.33%  bufio.(*Reader).Peek
      10ms  4.76% 85.71%       10ms  4.76%  encoding/json.(*encodeState).string
      10ms  4.76% 90.48%       20ms  9.52%  main.(*jsonReader).expectString
      10ms  4.76% 95.24%       10ms  4.76%  main.(*jsonReader).readByte
      10ms  4.76%   100%       10ms  4.76%  reflect.Value.Type
         0     0%   100%       10ms  4.76%  encoding/json.(*Encoder).Encode
         0     0%   100%       10ms  4.76%  encoding/json.(*decodeState).literalStore
         0     0%   100%       10ms  4.76%  encoding/json.(*decodeState).unmarshal
         0     0%   100%       10ms  4.76%  encoding/json.(*decodeState).value

Nice, syscall.Syscall is no longer in the top 10. But eatValue is and we're still spending a bunch of time in Peek. We didn't try to stop calling Peek so much, we just cut down on calling Discard.

List eatValue.

(pprof) list eatValue
Total: 210ms
ROUTINE ======================== main.(*jsonReader).eatValue in /home/phil/tmp/jqgo/mainpeek.go
     100ms      180ms (flat, cum) 85.71% of Total
         .          .    159:   err := jr.eatWhitespace(r)
         .          .    160:   if err != nil {
         .          .    161:           return err
         .          .    162:   }
         .          .    163:
         .       20ms    164:   ok, _, err := jr.tryScalar(r)
         .          .    165:   if err != nil {
         .          .    166:           return err
         .          .    167:   }
         .          .    168:
         .          .    169:   // It was a scalar, we're done!
         .          .    170:   if ok {
         .          .    171:           return nil
         .          .    172:   }
         .          .    173:
         .          .    174:   // Otherwise it's an array or object
         .          .    175:   length := 0
         .          .    176:   first := true
         .          .    177:   var bs []byte
      20ms       20ms    178:   for first || len(stack) > 0 {
      10ms       10ms    179:           length++
         .          .    180:           first = false
         .          .    181:
         .          .    182:           for {
      20ms       80ms    183:                   bs, err = r.Peek(length)
         .          .    184:                   if err == bufio.ErrBufferFull {
         .          .    185:                           _, err = r.Discard(length - 1)
         .          .    186:                           if err != nil {
         .          .    187:                                   return err
         .          .    188:                           }
         .          .    189:
         .          .    190:                           length = 1
         .          .    191:                           continue
         .          .    192:                   }
         .          .    193:                   if err != nil {
         .          .    194:                           return err
         .          .    195:                   }
         .          .    196:
         .          .    197:                   break
         .          .    198:           }
      10ms       10ms    199:           b := bs[length-1]
         .          .    200:
      10ms       10ms    201:           if inString {
      10ms       10ms    202:                   if b == '"' && prev != '\\' {
                         203:                           inString = false
         .          .    204:                   }
         .          .    205:
         .          .    206:                   // Two \\-es cancel eachother out
      20ms       20ms    207:                   if b == '\\' && prev == '\\' {
         .          .    208:                           prev = byte(0)
         .          .    209:                   } else {
         .          .    210:                           prev = b
         .          .    211:                   }
         .

The bulk of time is spent in Peek. Let's list Peek again.

(pprof) list Peek
Total: 210ms
ROUTINE ======================== bufio.(*Reader).Peek in /usr/local/go/src/bufio/bufio.go
      70ms       70ms (flat, cum) 33.33% of Total
         .          .    130:// also returns an error explaining why the read is short. The error is
         .          .    131:// ErrBufferFull if n is larger than b's buffer size.
         .          .    132://
         .          .    133:// Calling Peek prevents a UnreadByte or UnreadRune call from succeeding
         .          .    134:// until the next read operation.
      10ms       10ms    135:func (b *Reader) Peek(n int) ([]byte, error) {
         .          .    136:   if n < 0 {
         .          .    137:           return nil, ErrNegativeCount
         .          .    138:   }
         .          .    139:
         .          .    140:   b.lastByte = -1
         .          .    141:   b.lastRuneSize = -1
         .          .    142:
      10ms       10ms    143:   for b.w-b.r < n && b.w-b.r < len(b.buf) && b.err == nil {
         .          .    144:           b.fill() // b.w-b.r < len(b.buf) => buffer is not full
         .          .    145:   }
         .          .    146:
         .          .    147:   if n > len(b.buf) {
         .          .    148:           return b.buf[b.r:b.w], ErrBufferFull
         .          .    149:   }
         .          .    150:
         .          .    151:   // 0 <= n <= len(b.buf)
         .          .    152:   var err error
         .          .    153:   if avail := b.w - b.r; avail < n {
         .          .    154:           // not enough data in buffer
         .          .    155:           n = avail
         .          .    156:           err = b.readErr()
         .          .    157:           if err == nil {
         .          .    158:                   err = ErrBufferFull
         .          .    159:           }
         .          .    160:   }
      50ms       50ms    161:   return b.buf[b.r : b.r+n], err
         .          .    162:}
         .          .    163:
         .          .    164:// Discard skips the next n bytes, returning the number of bytes discarded.
         .          .    165://
         .          .    166:// If Discard skips fewer than n bytes, it also returns an error.

Well it's not really clear to me from this why we spend so much time slicing here.

We might be able to use Peek much less if we kept our own FIFO queue of peeked-at bytes. But I don't feel like writing a correct, efficient FIFO queue (a ring buffer, basically) and maybe there are other aspects of this program we can look at. So let's give this train of thought a break.

Memory profiling

Let's change tactics entirely. Memory allocation tends to be expensive. Allocating in a loop is generally a bad idea. And this entire program is a loop. So let's try doing a memory profile instead of a CPU profile.

Instead of defer profile.Start().Stop() we'll set defer profile.Start(profile.MemProfile).Stop().

Build, rerun and enter pprof with the -alloc_space flag. We want to see where memory is being allocated.

$ go build
$ cat large-file.json | ./jqgo '.repo.url' > /dev/null
2022/07/11 03:24:55 profile: memory profiling enabled (rate 4096), /tmp/profile1407859643/mem.pprof
2022/07/11 03:24:56 profile: memory profiling disabled, /tmp/profile1407859643/mem.pprof
$ go tool pprof -alloc_objects /tmp/profile1407859643/mem.pprof
File: jqgo
Type: alloc_objects
Time: Jul 11, 2022 at 3:24am (UTC)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 365899, 99.95% of 366086 total
Dropped 24 nodes (cum <= 1830)
Showing top 10 nodes out of 14
      flat  flat%   sum%        cum   cum%
    227585 62.17% 62.17%     262708 71.76%  main.(*jsonReader).expectString
     40945 11.18% 73.35%      40945 11.18%  main.(*jsonReader).readByte
     39500 10.79% 84.14%     252585 69.00%  main.(*jsonReader).tryScalar
     30009  8.20% 92.34%      41924 11.45%  main.(*jsonReader).tryNumber
     12055  3.29% 95.63%     215416 58.84%  main.(*jsonReader).eatValue
      7555  2.06% 97.70%      11915  3.25%  encoding/json.Unmarshal
      4360  1.19% 98.89%       4360  1.19%  encoding/json.(*decodeState).literalStore
      3847  1.05% 99.94%       3847  1.05%  main.(*jsonReader).expectIdentifier
        43 0.012% 99.95%     365931   100%  runtime.main
         0     0% 99.95%       4360  1.19%  encoding/json.(*decodeState).unmarshal

And just like in the CPU profile we can list functions to see where the allocations happen in code. Let's list the biggest memory user here, expectString.

(pprof) list expectString
Total: 366086
ROUTINE ======================== main.(*jsonReader).expectString in /home/phil/tmp/jqgo/mainpeek.go
    227585     262708 (flat, cum) 71.76% of Total
         .          .     58:   err := jr.eatWhitespace(r)
         .          .     59:   if err != nil {
         .          .     60:           return "", err
         .          .     61:   }
         .          .     62:
         .       4941     63:   b, err := jr.readByte(r)
         .          .     64:   if err != nil {
         .          .     65:           return "", err
         .          .     66:   }
         .          .     67:
         .          .     68:   if b != '"' {
         .          .     69:           return "", fmt.Errorf("Expected double quote to start string, got: '%s'", string(b))
         .          .     70:   }
         .          .     71:
         .          .     72:   var prev byte
         .          .     73:   for {
         .      30182     74:           b, err := jr.readByte(r)
         .          .     75:           if err != nil {
         .          .     76:                   return "", err
         .          .     77:           }
         .          .     78:
         .          .     79:           if b == '\\' && prev == '\\' {
         .          .     80:                   // Just skip
         .          .     81:                   prev = byte(0)
         .          .     82:                   continue
         .          .     83:           } else if b == '"' {
         .          .     84:                   // Overwrite the escaped double quote
         .          .     85:                   if prev == '\\' {
         .          .     86:                           s[len(s)-1] = '"'
         .          .     87:                   } else {
         .          .     88:                           // Otherwise it's the actual end
         .          .     89:                           break
         .          .     90:                   }
         .          .     91:           }
         .          .     92:
    146302     146302     93:           s = append(s, b)
         .          .     94:           prev = b
         .          .     95:   }
         .          .     96:
     81283      81283     97:   return string(s), nil
         .          .     98:}
         .          .     99:
         .          .    100:func (jr *jsonReader) expectIdentifier(r *bufio.Reader, ident string, value any) (any, error) {
         .          .    101:   var s []byte
         .          .    102:

And the biggest offender is growing the string! The good thing is that growing this string can be amortized because we can share the underlying string memory across calls on the jsonResponse struct. This way, expectString only needs to grow the string when it actually sees a bigger string than we've already seen.

The builtin bytes.Buffer type does exactly this. We can put a bytes.Buffer on the jsonResponse struct because this code isn't multithreaded and because expectString doesn't call itself.

@@ -2,6 +2,7 @@

 import (
        "bufio"
+       "bytes"
        "encoding/json"
        "fmt"
        "io"
@@ -13,6 +14,8 @@

 type jsonReader struct {
        read []byte
+
+       expectString_buffer bytes.Buffer
 }

 func (jr *jsonReader) reset() {
@@ -51,7 +54,7 @@
 }

 func (jr *jsonReader) expectString(r *bufio.Reader) (string, error) {
-       var s []byte
+       jr.expectString_buffer.Reset()

        err := jr.eatWhitespace(r)
        if err != nil {
@@ -81,18 +84,18 @@
                } else if b == '"' {
                        // Overwrite the escaped double quote
                        if prev == '\\' {
-                               s[len(s)-1] = '"'
+                               jr.expectString_buffer.Bytes()[jr.expectString_buffer.Len()-1] = '"'
                        } else {
                                // Otherwise it's the actual end
                                break
                        }
                }

-               s = append(s, b)
+               jr.expectString_buffer.WriteByte(b)
                prev = b
        }

-       return string(s), nil
+       return jr.expectString_buffer.String(), nil
 }

Or instead of sharing memory on the struct, maybe this would be a good place to use sync.Pool?

Disable pkg/profile, build and rerun with hyperfine.

$ hyperfine --warmup 2 \
  "cat large-file.json | ./control/control '.repo.url' > control.test" \
  "cat large-file.json | ./jqgo '.repo.url' > jqgo.test" \
  "cat large-file.json | jq '.repo.url' > jq.test"
Benchmark 1: cat large-file.json | ./control/control '.repo.url' > control.test
  Time (mean ± σ):     307.2 ms ±  10.8 ms    [User: 292.8 ms, System: 49.4 ms]
  Range (min … max):   296.5 ms … 326.2 ms    10 runs

Benchmark 2: cat large-file.json | ./jqgo '.repo.url' > jqgo.test
  Time (mean ± σ):     210.8 ms ±   2.2 ms    [User: 185.4 ms, System: 44.9 ms]
  Range (min … max):   209.1 ms … 216.8 ms    14 runs

Benchmark 3: cat large-file.json | jq '.repo.url' > jq.test
  Time (mean ± σ):     356.1 ms ±   2.6 ms    [User: 349.1 ms, System: 26.9 ms]
  Range (min … max):   354.1 ms … 362.9 ms    10 runs

Summary
  'cat large-file.json | ./jqgo '.repo.url' > jqgo.test' ran
    1.46 ± 0.05 times faster than 'cat large-file.json | ./control/control '.repo.url' > control.test'
    1.69 ± 0.02 times faster than 'cat large-file.json | jq '.repo.url' > jq.test'

And we've shaved another 20ms off. That's not bad!

Coming to a close

There is more we could do but this is a long post already.

For example, in the project repo I also built a generic vector type with a pop operation that is used for the stack in the eatValue function. It is shared on the jsonReader instance like the expectString buffer. This ended up shaving another 20ms. And I also got rid of most conversions from []byte to string (which is an expensive allocation you may notice listed as bytes.String() in the top10 of -alloc_objects if you run the profiler again now.)

But hopefully you're getting the gist of how you might investigate CPU and memory usage. For me it's still a lot of poking around and trying different things. But after a few years of trying to get better at profiling Go programs I think I'm starting to get the hang of it.

I wrote a new blog post on implementing a simple jq clone from scratch in Go. This post explores partial/fuzzy parsing again and finishes with my approach to debugging memory/CPU usage in Go programs. It's a bit of a long post but hopefully worthwhile! :)https://t.co/DxilIVaUBa pic.twitter.com/as3Sr5I2G0
— Phil Eaton (@phil_eaton) July 11, 2022

Working Backwards

Bytepawn - Marton Trencseni — Sun, 10 Jul 2022 01:00:00 GMT

Amazon established a set of principles and mechanisms, enabling the company to grow from a single founder to several hundred thousand employees while remaining stubbornly true to its mission of obsessing over customers to create long-term shareholder value.

DALL·E 2 - Selected works

smcleod.net — Fri, 08 Jul 2022 10:00:00 GMT

Over the past month I’ve been playing with OpenAI’s DALL·E 2, below are some of the interesting images I’ve generated.

“DALL·E 2 is a new AI system that can create realistic images and art from a description in natural language.”

“A renaissance oil painting of two developers arguing over which javascript framework is the worst”

“A Ralph Steadman painting of 7 cats sitting on a fence, and one raven”

HBR Guide to Better Business Writing

Bytepawn - Marton Trencseni — Fri, 08 Jul 2022 01:00:00 GMT

If your writing is sloppy and artless people will think you are the same. They won’t care about your message, they won’t do business with you. It’s not true that only ideas matter. Good writing gets ideas noticed.

Vim tip 11: replace characters in Normal mode

learnbyexample — Wed, 06 Jul 2022 03:00:00 GMT

Often, you just need to change one character. For example, changing i to j, 2 to 4, ' to " and so on.

rj replace the character under the cursor with j
ry replace the character under the cursor with y
3ra replace the character under cursor as well as the two characters to the right with aaa
- no changes will be made if there aren't sufficient characters to match

To replace multiple characters with different characters, use R.

Rlion followed by Esc replace the character under cursor and three characters to the right with lion
- Esc key marks the completion of R command
- Backspace key will act as an undo command to give back the character that was replaced
- if you are replacing at the end of a line, the line will be automatically extended if needed

The advantage of r and R commands is that you remain in the Normal mode, without needing to switch to Insert mode and back.

Video demo:

Moving to Source Hut

bayindirh — Wed, 06 Jul 2022 03:00:00 GMT

I have been following the whole GitHub Copilot issue for a while, and I was not happy about Microsoft using open-source code hosted in GitHub without consent for training Copilot.

Consent is only one part of a bigger issue though. As everybody knows, Microsoft has used open-source code on GitHub to train Copilot, disregarding the licenses. This means many repositories containing GPL licensed code have been used as well.

Using GPL-licensed code in this manner is problematic on two fronts. The first is legal. GPL family of licenses doesn't allow source code to be closed. If you lift a function from a GPL-licensed codebase and integrate it into your code, you have to share your code under the same GPL license. The same is true for derivations of the code lifted from a GPL licensed codebase. However, Copilot disregards licenses on both ends while deriving code from its training set. It can start from a GPL function or mix GPL licensed code to a function licensed any other way. Since copilot is available to anyone, there's a serious risk of injecting GPL-licensed code into any codebase, regardless of license compatibility.

This is a clear breach of the GPL license and is inexcusable.

The second issue is ethical. Microsoft is not transparently communicating how they trained Copilot. They give vague answers and try to downplay the issue while denying all responsibility. See the second paragraph of this document, which is quoted below:

You are responsible for ensuring the security and quality of your code. We recommend you take the same precautions when using code generated by GitHub Copilot that you would when using any code you didn't write yourself. These precautions include rigorous testing, IP scanning, and tracking for security vulnerabilities. GitHub provides a number of features to help you monitor and improve code quality, such as GitHub Actions, Dependabot, CodeQL and code scanning. All these features are free to use in public repositories. For more information, see "Understanding GitHub Actions" and "GitHub security features."

However the issue is pretty serious to be downplayed. Copilot is a paid product, marketed to all developers writing all sorts of software. Using open-source code regardless of its license and giving it to anyone to do anything is exploiting the developers who explicitly stated that they want their code to stay open and developed in a certain way.

Moreover, GitHub contains all sorts of code, from leaked source code to codebases with wildly incompatible licenses. Not all of these codebases can be used while developing your product.

This is embrace, extend, extinguish at its finest, with a minefield attached to it.

In the light of these events, I decided to stop using GitHub, and move to Source Hut. Source Hut is Drew DeVault's brainchild, a Free Software advocate. It's built with different priorities in mind, and without a social component deliberately. The software can be self-hosted if desired, however, I decided to pay for the service and support its development further.

While I have started my move to Source Hut, there won't be much activity visible at first. My private research repositories (which will be eventually opened when they mature enough) are going first, then I'll archive my public archives and other public repositories, step by step.

This slow migration will allow me to ponder on the plans for each project and make sure that I move them with utmost care and deliberation.

Meanwhile, many people will inevitably wonder about their code's visibility and discoverability if they move from the most crowded open-source hub to somewhere relatively obscure and has no social components.

I thought about this and decided that software freedom, ethics, self-respect, and doing the things you praise are much more important than being discoverable on a particular website. My homepage has a link to my Source Hut page, and if people click that, it's great.

Hope you at least take a look at Source Hut or, other alternatives like Codeberg and Gitea.

Hope to see you over here.

The Most Useful Statistical Test You Didn't Learn in School

Speculative Branches — Mon, 04 Jul 2022 03:00:00 GMT

In performance work, you will often find many distributions that are weirdly shaped: fat-tailed distributions, distributions with a hard lower bound at a non-zero number, and distributions that are just plain odd. Particularly when you look at latency distributions, it is extremely common for the 99th percentile to be a lot further from the mean than the 1st percentile. These sorts of asymmetric fat-tailed distributions come with the business.

Often times, when performance engineers need to be scientific about their work, they will take samples of these distributions, and put them into into a $t$-test to get a $p$-value for the significance of their improvements. That is what you learned in a basic statistics or lab science class, so why not? Unfortunately, the world of computers is more complicated than the beer quality experiments for which the $t$-test was invented, and violates one of its core assumptions: that the sample means are normally distributed. When you have a lot of samples, this can hold, but it often doesn't.

We need to do a hard thing

--Mike-- — Sun, 03 Jul 2022 07:49:54 GMT

We do need to do a hard thing, we need to reconfigure our world to cope with the end of the one time greatest economic bonanza of all times... the fossil fuel pulse.

If you think I'm about to go on about global warming/climate change/etc... NOPE

We've tapped most of the easy to reach oil in the world. Now we're fracking to get at things, and those wells have decay rates of 40%/year, instead of per decade. If we stopped all drilling, we'd have 40% less oil next year from all the new wells. (Or so I've been lead to believe. It's my opinion that we've got 20 years, at the outside to complete this project)

The new shockingly high fuel costs? They're here to stay, no matter who "runs the country". They stopped building new refineries in the 1970s, because they knew this day was coming, and it didn't make sense to build more capacity for an industry they knew was heading to a peak. The only surprise was that some things broke faster than planned.

Our civilization depends on a resource in decline. All the people at the top have zero clue as to how to actively manage the situation. We, the people, have to start from the grass roots and figure this out ourselves.

Abortion isn't all about ending life - saying it is, is evil!

--Mike-- — Fri, 01 Jul 2022 20:10:41 GMT

I've lost multiple nights of sleep, and simply have no coping skills because there exist in this world a large number of people who just love calling abortion "Baby Murder"

These people obviously know the difference between a dead child lying on the ground in a classroom in Texas, and a few cells removed in the first half of a pregnancy... yet they persist in this evil game.

If the upcoming laws in my Red State had been in effect 2 decades ago, I'd have no wife, she would have been killed, rather than have life saving procedure after we lost a more than welcome pregnancy. Thus she wouldn't have been with me to bring our lovely child into the world later.

How many men will lose their wives, sisters, daughters... how many children will lose their mothers because of this evil befalling the land?

How the f*ck can I get some sleep at night, knowing of all the evil indifference these people carry in their hearts?

QDX Revision 3 Build Begins in REAL TIME

MiscDotGeek — Fri, 01 Jul 2022 05:16:16 GMT

Hey everyone- been a busy few months lately. Between bad health, travelling, and then working hard just take care of Live Stuff, I’ve hardly had time for the Ye Olde Blog! I’ll try to write more regularly. A lot of my attention has been going to YouTube and in this series, starting with the video …

The post QDX Revision 3 Build Begins in REAL TIME appeared first on MiscDotGeek.

CLI tip 12: squeeze empty lines

learnbyexample — Wed, 29 Jun 2022 03:00:00 GMT

awk has a builtin feature to process input content paragraph wise (by setting RS to an empty string). But, did you know that cat, less and grep can also be used to squeeze empty lines?

cat -s (and less -s) will squeeze multiple empty lines in the input to a single empty line in the output. Here's an example:

$ cat ip.txt
hello




world

apple
banana
cherry


tea coffee
chocolate
$ cat -s ip.txt
hello

world

apple
banana
cherry

tea coffee
chocolate

Here's an example with empty lines at the start/end of the input:

$ printf '\n\n\ndragon\n\n\nunicorn\n\n\n'



dragon


unicorn


$ printf '\n\n\ndragon\n\n\nunicorn\n\n\n' | cat -s

dragon

unicorn

And here's a solution with awk. Unlike the -s option, this will completely remove empty lines at the start/end of the input.

$ awk -v RS= '{print s $0; s="\n"}' ip.txt
hello

world

apple
banana
cherry

tea coffee
chocolate

$ printf '\n\n\ndragon\n\n\nunicorn\n\n\n' | awk -v RS= '{print s $0; s="\n"}'
dragon

unicorn

The awk solution would be easier to extend, given its programmable features. For example, two empty lines between the groups:

$ awk -v RS= '{print s $0; s="\n\n"}' ip.txt
hello


world


apple
banana
cherry


tea coffee
chocolate

And here's a surprising GNU grep solution, with a customizable group separator:

# single empty line
$ grep --group-separator= -A0 '.' ip.txt
hello

world

apple
banana
cherry

tea coffee
chocolate

# double empty line
# empty lines at the start/end of the input are removed too
$ printf '\n\n\ndragon\n\n\nunicorn\n\n\n' | grep --group-separator=$'\n' -A0 '.'
dragon


unicorn

Video demo:

2022–06–28: Pinephone Pro Type-C support is now complete

megi's PinePhone Development Log — Tue, 28 Jun 2022 01:00:00 GMT

Sharding Yourself

swyx's site RSS Feed — Mon, 27 Jun 2022 08:26:37 GMT

An advanced tip for high-volume writers.

Search index implementations

Ben E. C. Boyter — Sun, 26 Jun 2022 03:00:00 GMT

A living document where I will be putting information about search index implementations as I learn about them.

So there are a few talked about ways to search across content or build an index that I am aware of. Lets discuss each in term with its advantages and disadvantages

Brute Force

Brute Force, note that this isn’t an index strategy per say, but worth discussing anyway. Assuming you can get the entire corpus you are search into RAM it is possible to brute force search. I’m including it here so I have the advantages and disadvantages. While brute force isnt a solution to every problem, a lot of problems are pretty brute forcible given enough CPU and RAM.

On The Importance of 15-5 Updates

swyx's site RSS Feed — Sat, 25 Jun 2022 23:09:00 GMT

We had a delightful discussion on the importance of writing weekly updates in this week's Coding Career Community meetup. I rarely get so excited about an idea I immediately know I need to start doing it, so I'm choosing to write it up to commit to it, and to share it with you.

How to add a tachometer to a Triumph Street Twin (2016-2019)

Subclassed — Thu, 23 Jun 2022 11:48:10 GMT

The Problem

I recently gave in to family tradition: I'm originally from Bologna, home of Ducati and heart of the Italian "Motor Valley", so sooner or later I had to buy a motorbike (or two, but nevermind).

As the main workhorse I went for a 2018 Triumph Street Twin, because it's practical but still fun, fairly cheap, and simply beautiful. I love it to bits, but one thing I really missed: a tachometer ("rev counter"). While trying their hardest to segment the Bonneville range, Triumph clearly thought that people would pay a few grand more and get a Speed Twin just so that they could see RPMs. It took them about 4 years to understand the silliness of this position (or to produce enough Bonneville variations that nobody cares about that bit anymore), so post-2020 Street Twins display RPMs on the digital display - in a pretty small way, but at least it's there.

The Solution

So, how could we go about adding a tachometer to pre-2020 models? Various web searches produced a few different methods, with a number of pitfalls. There are a few electrical schematics floating around, but I'm not good at that sort of thing and comments seemed to be that they are easy to get wrong.

A much easier method is to attach an OBDII ("OBD2") reader to the related port, which is available under the seat.

OBDII is a standard interface for debugging systems in vehicles; it's recently been mandated by the European Union, so it's available in pretty much any vehicle produced in the last decade or so. Once you connect a reader, you can see a wealth of information about the vehicle, from fuel consumption to engine load to pretty much any sensor available... including RPMs! So that's how we can get a tachometer without effectively modifying anything in the bike itself.

There are two ways to show the extracted information: wired, and wireless. Each comes with pros and cons.

Wireless (Bluetooth)

Wireless is less intrusive but fiddlier: a compact reader stays entirely under the seat, and communicates with a phone or tablet over Bluetooth (as far as I know, there are no dedicated wireless displays). It's described in this Youtube video, and the dongles are easy to find on Amazon. As a permanent solution, it suffers from lag and the requirement that the phone screen stays on all the time, set on the specific app. (Side note: I found the best iOS app to be FourStroke - most of them are focused on cars and might not even work with cheap aftermarket readers.) I have an old phone I mount on handlebars, but I typically want to use it to display GPS stuff and play music, so that just didn't work for me. You also have to manually open the app and connect every time, after you start the bike, which is annoying.

Wired

A wired solution, on the other hand, requires a bit more setup, but you do it once and that's it, you don't have to worry about it ever again; readings are a bit less laggy (although still not as quick as hardware ones), and leave your phone free to do other stuff. They typically also feature hardware controls, which are easier to use with gloves than touchscreens (yes yes, I have modern gloves - in practice, they are still too big to be precise).

There are a few OBDII readers on the market, largely meant for the car market. I grabbed this cheap thing from AliExpress, but there are equivalents on Amazon and so on. Setup is trivial: you connect it to the port, turn on the bike, and it just works, letting you choose what to display. You turn the bike off and the display shuts down automatically, so your battery is not drained. Nice!

Finding a location for the wire is basically the same as for the USB you also find under the seat; if you've wired it up to a handlebar mount for phones, as I have, you can basically do the same - it's actually easier here, since you can push it further back with the other cables.

Unresolved issues

There are a few annoyances though, due to the fact that this is fundamentally a product for cars:

The display can be a bit hard to see in daylight, particularly when the sun shines directly on it.
You'll need to hack together a mounting system; the stand that comes with the product is meant for car dashboards. You can see my solution in the pic of the display above - not the most beautiful, but it will do for now; I plan to revisit it at some point.
It's not waterproof. In the rainy North of England, this is a significant problem. I'm trying to think about solutions; if you have any suggestion, feel free to drop a comment.

All these issues could be solved if manufacturers started making readers dedicated to the motorbike market. I hope that happens at some point, because it would be awesome - it would free consumers from the tiranny of OEM displays.

Hot Engine is Hot

Last bit of advice: while you're testing, be careful with your dangling cables! As you can see from the pic below, I mistakenly left mine over the exhaust for a few seconds and... well. Amazingly, it still works!

Fun with Anchor Text Keywords

Weblog on marginalia.nu — Thu, 23 Jun 2022 03:00:00 GMT

Anchor texts are a very useful source of keywords for a search engine, and in an older version of the search engine, it used the text of such hyperlinks as a supplemental source for keywords, but due to a few redesigns, this feature has fallen off.

Last few days has been spent working on trying to re-implement it in a new and more powerful fashion. This has largely been enabled by a crawler re-design from a few months ago, which offers the crawled data in a lot more useful fashion and allows a lot more flexible post-processing.

2022–06–23: Further Pinephone Pro camera development

megi's PinePhone Development Log — Thu, 23 Jun 2022 01:00:00 GMT

Python tip 12: negate a regex grouping

learnbyexample — Wed, 22 Jun 2022 03:00:00 GMT

You might be familiar with negating a character class, for example:

>>> import re

# remove first two columns
>>> re.sub(r'\A([^:]+:){2}', '', 'apple:42:banana:1000:cherry:512')
'banana:1000:cherry:512'

# filter all elements not ending with `r` or `t`
>>> words = ['surrender', 'unicorn', 'newer', 'door', 'empty', 'eel', 'pest']
>>> [w for w in words if re.search(r'[^rt]\Z', w)]
['unicorn', 'empty', 'eel']

But do you know how to match characters based on a negated group? You can use a combination of negative lookahead and quantifiers as shown in the examples below:

>>> pets = 'fox,cat,dog,parrot'

# match if 'do' is not present between 'at' and 'par'
>>> bool(re.search(r'at((?!do).)*par', pets))
False

# match if 'go' is not present between 'at' and 'par'
>>> bool(re.search(r'at((?!go).)*par', pets))
True

# easier to understand by looking at the matched portions
>>> re.search(r'at((?!go).)*par', pets)[0]
'at,dog,par'
>>> re.search(r'\A((?!par).)*', pets)[0]
'fox,cat,dog,'

The . in ((?!go).)* will match a character only if the sequence of current and next characters are not go. Similarly, the . in ((?!par).)* matches a character only if the current and next two characters are not par. The * quantifier is applied on the outer group to match zero or more characters satisfying the given condition.

The outer group in the above examples are capturing groups, though it wasn't required. Just makes the pattern concise. However, capturing groups affect the behavior of functions like re.split and re.findall. You can use non-capturing groups in such cases:

# capture group affects the behavior of 're.findall'
>>> re.findall(r'\b((?!42)\w)+\b', 'a422b good bad42 nice100')
['d', '0']

# so, use a non-capturing group here
>>> re.findall(r'\b(?:(?!42)\w)+\b', 'a422b good bad42 nice100')
['good', 'nice100']

Test your understanding by solving this exercise. Construct a regex solution that works for all three sample transformations shown below:

Power(x,2) should be replaced with (x)*(x)

Power(Power(x,2) + x,2) should be changed to ((x)*(x) + x)*((x)*(x) + x)

Power(x + Power(x,2),2) should be changed to (x + (x)*(x))*(x + (x)*(x))

If that was easy, make it work for general powers instead of just 2:

Power(Power(x,2),3) translates to ((x)*(x))*((x)*(x))*((x)*(x))

The above exercise is based on this stackoverflow Q&A.

Video demo:

See also my Understanding Python re(gex)? ebook.

CLI tip 7: limiting number of filtered lines

learnbyexample — Mon, 20 Jun 2022 03:00:00 GMT

grep supports -m option to specify the maximum number of matching lines in the output.

# all input lines containing 'a'
$ printf 'goal\nrate\neat\npit\n' | grep 'a'
goal
rate
eat

# maximum of 2 matching lines
$ printf 'goal\nrate\neat\npit\n' | grep -m2 'a'
goal
rate
$ printf 'goal\nrate\neat\npit\n' | grep -m2 'pi'
pit

# example with -v option
$ printf 'goal\nrate\neat\npit\n' | grep -v 'e'
goal
pit
$ printf 'goal\nrate\neat\npit\n' | grep -v -m1 'e'
goal

With multiple file input, the restriction is applied for each file separately.

$ cat table.txt 
brown bread mat cake 42
blue cake mug shirt -7
yellow banana window shoes 3.14
$ printf 'goal\nrate\neat\npit\n' > ip.txt

$ grep -m1 'i' table.txt ip.txt
table.txt:blue cake mug shirt -7
ip.txt:pit

# use 'cat' if you want to operate on combined input
$ cat table.txt ip.txt | grep -m1 'i'
blue cake mug shirt -7
$ cat table.txt ip.txt | grep -m1 'go'
goal

Video demo:

See my CLI text processing with GNU grep and ripgrep ebook if you are interested in learning about GNU grep and ripgrep commands in more detail.

Python tip 7: creating a deepcopy of collections

learnbyexample — Mon, 20 Jun 2022 03:00:00 GMT

From copy built-in module:

Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without changing the other.

The shared binding is helpful for cases like in-place modification of lists within a user defined function:

>>> def rotate(ip):
...     ip.insert(0, ip.pop())
... 
>>> nums = [321, 1, 1, 0, 5.3, 2]
>>> rotate(nums)
>>> nums
[2, 321, 1, 1, 0, 5.3]

You can use the copy.deepcopy() method if you wish to recursively create new copies of all the elements of a mutable object:

>>> import copy

>>> nums_2d = [[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d_deepcopy = copy.deepcopy(nums_2d)

>>> nums_2d_deepcopy[0][0] = 'yay'

>>> nums_2d_deepcopy
[['yay', 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]
>>> nums_2d
[[1, 3, 2, 10], [1.2, -0.2, 0, 2], [100, 200]]

Video demo:

See Mutability chapter from my 100 Page Python Intro ebook for more details on this topic.

Data outlasts Code, yet Code keeps winning

swyx's site RSS Feed — Sat, 18 Jun 2022 21:56:56 GMT

My recent End of Localhost piece on Hacker News came with the usual dash of HN criticism devolving into blaming beginners for not knowing the same parts of the stack that they consider mandatory:

Culture Docs: Facebook, Netflix and Valve

Bytepawn - Marton Trencseni — Sat, 18 Jun 2022 01:00:00 GMT

Many companies have some sort of "Culture Doc", a booklet or similar, which explains to new joiners what the company is about. I received Facebook's "Little Red Booklet" when I joined in 2016 February, and I was amazed how good it was. Recently I was researching other companies' Culture Docs, and found a version of Netflix's and Valve's online. It's interesting to compare and contrast what these different companies choose to put in their Culture Doc. Facebook's Culture Doc is very mission and execution oriented and serious, Netflix is analytical and HR-focused, and Valve's is a lighthearted explanation of how the company works.

CLI tip 6: filtering lines based on multiple conditions

learnbyexample — Fri, 17 Jun 2022 03:00:00 GMT

Usually, grep is used for filtering lines based on a regexp pattern. Matching multiple patterns as a conditional OR is easy to apply using alternation. For example, grep -E 'bread|banana' matches lines containing either bread or banana.

However, conditional AND requires multiple grep commands or the use of lookarounds if PCRE feature is available. A simpler alternative is to use awk.

$ cat table.txt
brown bread mat cake 42
blue cake mug shirt -7
yellow banana window shoes 3.14

# line containing 'cake' but not 'at'
# same as: grep 'cake' table.txt | grep -v 'at'
# with PCRE: grep -P '^(?!.*at).*cake' table.txt
$ awk '/cake/ && !/at/' table.txt
blue cake mug shirt -7

# first field containing 'low' or the last field is less than 0
# not easy to construct a grep solution here due to field/numeric comparison
$ awk '$1 ~ /low/ || $NF<0' table.txt
blue cake mug shirt -7
yellow banana window shoes 3.14

Video demo:

See my CLI text processing with GNU awk ebook if you are interested in learning about the awk command in more detail.

Python tip 6: inplace file editing

learnbyexample — Fri, 17 Jun 2022 03:00:00 GMT

The built-in fileinput module has nice features for file processing, especially handy for multiple files and inplace editing. Here's an example:

# inplace.py
import fileinput

with fileinput.input(inplace=True) as f:
    for line in f:
        # do some processing
        op = line.replace('search', 'replace')
        # print() the text you want to write back to the input files
        print(op, end='')

By default, files passed as CLI arguments will be processed:

$ python3 inplace.py *.md

If you already know which inputs have to be processed, use the files argument. Use the backup argument if you want to make copies of original files in case something goes wrong. See my blog post In-place file editing with fileinput module for more details and examples.

Video demo:

See my 100 Page Python Intro ebook for a short, introductory guide for the Python programming language.

CLI tip 5: aligning columns

learnbyexample — Fri, 17 Jun 2022 03:00:00 GMT

The column command is a nifty tool to align input data column wise. By default, whitespace is used as the input delimiter. Space character is used to align the output columns, so whitespace characters like tab will get converted to spaces.

$ printf 'one two three\nfour five six\n'
one two three
four five six

$ printf 'one two three\nfour five six\n' | column -t
one   two   three
four  five  six

You can use the -s option to customize the input delimiter. Note that the output delimiter will still be made up of spaces only.

$ cat scores.csv
Name,Maths,Physics,Chemistry
Ith,100,100,100
Cy,97,98,95
Lin,78,83,80

$ column -s, -t scores.csv
Name  Maths  Physics  Chemistry
Ith   100    100      100
Cy    97     98       95
Lin   78     83       80

$ printf '1:-:2:-:3\napple:-:banana:-:cherry\n' | column -s:-: -t
1      2       3
apple  banana  cherry

Input should have a newline at the end, otherwise you'll get an error:

$ printf '1 2 3\na   b   c' | column -t
column: line too long
1  2  3

Video demo:

See my Linux Command Line Computing ebook and man column for more details.

On automation

bayindirh — Thu, 16 Jun 2022 03:00:00 GMT

Exploring minimalism is a multifaceted journey. As I travel down the path; different ideas, problems, ways and possibilities appear. These branches lead to different interpretations of minimalism, but some of them brings the ultimate vision of more time and inner peace nearer. One of such paths is workflows.

Every human being performs repetitive tasks. The tasks are small in size, relatively easy and doesn't require much brain power, hence we don't think about them. However, they require time, as every task. Moreover, they like to appear in inconvenient times or pile up when we decide to skip them a couple of times, and this adds to the background stress of our lives.

These factors create incentives to reduce the background stress the task at hand creates, and there are possible ways to tackle this problem. If possible, we can eliminate it. If not, we can transfer the task to someone, and that someone doesn't have to be another human.

Enter automation.

Automation has similar promises with minimalism. Less mental load and more time. By creating workflows which can handle all the steps needed to accomplish a task, a 5-minute task can become a 5-second one. While this is a nice and enticing promise, it sounds too good to be true.

Because it is.

Many tasks we do intuitively can't be understood by computers well. This is why we have artificial intelligence and machine learning, but there are tasks we can automate. This automation can be done with dedicated tools or we can write ours, and program our woes away we think. However, wise men have threaded this path before and left us clues.

The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.
-- Tom Cargill, Bell Labs.

There's another, visual explanation by venerable Randall Munroe of XKCD.

In reality, the path which looked exciting at first becomes thorny as we start to explore the depths, hence we have to thread carefully, and be aware of the traps ahead.

However, there's no need to abandon all hope.

With little preparation, and by being mindful of what we are trying to accomplish, we can navigate this maze and create tools which can take the loads from our shoulders and bear it without any complaints. The key is remembering two things: keeping it simple and iterating.

The first thing is more important one of the bunch, because as a passionate developer, I tend to over engineer. I need to avoid that, hence I plan for the minimum feature set which the tool needs to get the job done. The second one allows me to improve the tool, without crippling my life and the workflow. These iterations add emerging requirements or the features I thought it'd be useful (hint: I abandon at least half of these ideas since they bring no benefit).

When a tool deploys to production and starts to do its job, the mental space it creates is priceless. Because a job always takes more time than it looks. You mentally prepare for it, think about it until the right time comes and then perform. Offloading this to an impeccable agent is a nice thing.

Lastly, watching the machinery I built run makes me happy and creates a satisfaction which I can't describe.

One such tool I created is Magazine Renamer, which allows me to organize my IEEE magazines when they arrive to my inbox. Another, a more ambitious tool is in the works, but it'll take some time to get ready. Will share that one too, but it needs to be useable first.

I think everyone should automate some parts of their life without getting overzealous about it, because sometimes doing things manually are better, however discussion of this a matter of another time.

The 8 Jobs of Management

swyx's site RSS Feed — Wed, 15 Jun 2022 23:12:25 GMT

Recently, John Cutler tweeted a chart on the view of management:

Vim tip 10: Undo and Redo

learnbyexample — Wed, 15 Jun 2022 03:00:00 GMT

In Normal mode, you can undo and redo changes using the following commands:

u undo last change
- press u again for further undos
U undo latest changes on last edited line
Ctrl+r redo a change undone by u
U redo changes undone by U

See :h 32.3 for details on g- and g+ commands that you can use to undo branches.

Video demo:

Python tip 5: random choice and sample

learnbyexample — Tue, 14 Jun 2022 03:00:00 GMT

Here are a couple of commonly used methods for the built-in random module:

choice() method helps you get a random element
sample() method helps you get a list of a specific count of random elements

>>> import random
>>> nums = [1, 4, 5, 2, 51, 3, 6, 22]

>>> random.choice(nums)
3

>>> random.sample(nums, k=4)
[51, 2, 3, 1]

>>> random.sample(range(1000), k=5)
[490, 26, 9, 745, 919]

Both these methods will work on any sequence object. The sample() method also accepts a set object, but that will be deprecated.

Video demo:

See my 100 Page Python Intro ebook for a short, introductory guide for the Python programming language.

CLI tip 4: serialize file contents to a single line

learnbyexample — Tue, 14 Jun 2022 03:00:00 GMT

The -s option is one of the useful, but lesser known feature of the paste command. It helps you to serialize input file contents to a single output line.

$ cat colors.txt
blue
white
orange

$ paste -sd, colors.txt
blue,white,orange

If multiple files are passed, serialization of each file is displayed on separate output lines.

$ paste -sd: <(seq 3) <(seq 5 9)
1:2:3
5:6:7:8:9

The advantage of using paste instead of other options like tr, awk, etc is that you do not have to worry about trailing delimiters, newlines, etc. For example:

# issue 1: trailing comma
# issue 2: no newline at the end
$ <colors.txt tr '\n' ','
blue,white,orange,

# correcting the above two issues
$ <colors.txt tr '\n' ',' | sed 's/,$/\n/'
blue,white,orange

Here's an equivalent awk solution for single file input. While slower and complicated compared to the paste solution, you get more flexibility since awk is a programming language. For example, it is pretty easy to use multicharacter output delimiter.

$ awk -v ORS= 'NR>1{print ","} 1; END{print "\n"}' colors.txt
blue,white,orange

$ awk -v ORS= 'NR>1{print " : "} 1; END{print "\n"}' colors.txt
blue : white : orange

Video demo:

See paste command chapter from my Command line text processing with GNU Coreutils ebook for more details.

See my CLI text processing with GNU awk ebook if you are interested in learning about the awk command.

CLI tip 3: place backups in another directory with GNU sed

learnbyexample — Tue, 14 Jun 2022 03:00:00 GMT

You can use * to place backups of original files in another directory when using the -i option with GNU sed. Consider these two sample input files in the current directory:

$ cat f1.txt
good morning
that was good, just too good!
$ cat f2.txt 
goodie goodbye

Create a backups directory and use * under this directory as a placeholder for the filenames passed to the sed command.

$ mkdir backups
$ sed -i'backups/*' 's/good/nice/' f1.txt f2.txt
$ ls backups/
f1.txt  f2.txt

# modified content
$ cat f1.txt
nice morning
that was nice, just too good!
$ cat f2.txt
niceie goodbye

# backed-up original content
$ cat backups/f1.txt 
good morning
that was good, just too good!
$ cat backups/f2.txt
goodie goodbye

Since * expands to the name of the input files, you can also use this feature when you need to add a prefix for the backups.

$ sed -i'bkp.*' 's/green/yellow/' colors.txt

$ ls *colors*
bkp.colors.txt  colors.txt

The * trick works with Perl as well, see In-place file editing chapter from my Perl One-Liners Guide ebook for examples.

Video demo:

See my CLI text processing with GNU sed ebook if you are interested in learning about the GNU sed command in more detail.

Beginner Harmonica Practice Exercises

smcleod.net — Mon, 13 Jun 2022 16:00:00 GMT

Beginner Harmonica Practice Exercises

Way back in around 2008 I wrote up a doc with a list of beginner harp exercises and practice riffs, at the time it was uploaded to a few forums and a shared Google doc - both have since disappeared. To this day I still get regular emails asking for a copy of it so I thought I’d copy it to a post here.

Note: I haven’t updated these since around 2011.

A chance to act, let's close the Boyfriend Loophole - the right way!

--Mike-- — Mon, 13 Jun 2022 04:31:53 GMT

Closing the "Boyfriend Loophole" could do a great deal to stop future mass shootings. Someone who can't handle their feelings around people close to them enough to avoid violence, shouldn't be allowed to own firearms.

This is THE BEST CHANCE we have at the moment, likely in the next year or two, for actually fixing some of the biggest problems facing our nation. The huge overlap between those who commit domestic violence, and mass shooters, is well proven.

I hope that we can push our representatives to do the right thing on this. Pretty much EVERYONE agrees that this legislation is needed, let's not let them f*ck it up.

Here's Beau explaining why the details matter.

The Hard Problem of Rendering Tweets

swyx's site RSS Feed — Sun, 12 Jun 2022 20:47:39 GMT

I've been unhappy with my tweet rendering strategy for a while - Twitter encourages you to use their heavy JS script to render tweets, which undoubtedly heaps all sorts of tracking on the reader, docks your lighthouse performance score by ~17 points, adds ~4 seconds to Time to Interactive, occasionally gets adblocked (so nothing renders!)

Python tip 4: comparison chaining

learnbyexample — Fri, 10 Jun 2022 03:00:00 GMT

You can chain comparison operators arbitrarily. Apart from terser code, this also has the advantage of having to evaluate the middle expression only once.

>>> from math import factorial

# factorial function gets called twice for this example
>>> factorial(3) > 2 and factorial(3) < 10
True

# function needs to be called only once here
>>> 2 < factorial(3) < 10
True

# another example
>>> 'bat' < 'cat' < 'cater'
True

Video demo:

See my 100 Page Python Intro ebook for a short, introductory guide for the Python programming language.

One year as a solo dev building open-source data tools without funding

Notes on software development — Fri, 10 Jun 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

The End of Localhost

swyx's site RSS Feed — Wed, 08 Jun 2022 20:26:56 GMT

This post was originally published on my new dedicated DX site: https://dx.tips/the-end-of-localhost

CLI tip 11: longest line length

learnbyexample — Wed, 08 Jun 2022 03:00:00 GMT

You can use wc -L to report the length of the longest line in the input (excluding the newline character of a line).

$ echo 'apple' | wc -L
5

# last line not ending with newline won't be a problem
$ printf 'apple\nbanana' | wc -L
6

$ cat greeting.txt
hi there
have a nice day
$ wc -L <greeting.txt
15

If multiple files are passed, the last line summary will show the maximum length among the given inputs.

$ wc -L greeting.txt sample.txt para.txt
 15 greeting.txt
 26 sample.txt
 11 para.txt
 26 total

-L won't count non-printable characters and tabs are converted to equivalent spaces. You can use awk if these are not acceptable.

# tab characters can occupy up to 8 columns
$ printf '\t' | wc -L
8
$ printf '(\t)' | wc -L
9
$ printf '(\t)' | awk '{print length()}'
3

# non-printable characters aren't counted
$ printf '(\34)' | wc -L
2
$ printf '(\34)' | awk '{print length()}'
3

Note that the awk command in the above illustration is similar to wc -L only for single line inputs. For multiple lines, you can use the following command:

awk '{len = length(); if(len > max) max = len} END{print max}'

Multibyte characters and grapheme clusters will each be counted as 1, assuming the current locale is set appropriately:

# multibyte characters are counted as 1 each in supported locales
$ printf 'αλεπού' | wc -L
6

# grapheme cluster example
$ printf 'cag̈e' | wc -L
4

# non-supported locales can cause them to be treated as non-printable
$ printf 'αλεπού' | LC_ALL=C wc -L
0

Video demo:

Iron Arachne Dungeon Generator and future plans

Ben Overmyer's Site — Tue, 07 Jun 2022 03:00:00 GMT

This weekend past, I launched the dungeon generator for Iron Arachne. I've been working on it off and on for a few weeks and I finally got it to the point where it was usable and useful. However, it's by no means complete, and a lot of features were left unfinished.

The above is an example of the maps it currently generates. In terms of pathing, there is a pretty linear flow to the dungeon layout - a higher-number room is deeper into the dungeon. This makes it easy to determine where to put goals like boss encounters. However, it also means the dungeon doesn't encourage exploration or reward players who enjoy that activity, and really it gets a little boring.

So, one of the first changes is going to be the addition of more loops throughout the layout. This will result in more connections between rooms. Corridors will need to be addressed. Right now, there's no logic for creating even straight corridors between rooms. Angled corridors will require more effort.

Another early change will be the addition of non-rectangular rooms. This can include organic-looking caverns, but also different regular rooms, like circular chambers.

I plan on adding something else to the terrain, too - "lakes." In this case, lakes can refer to bodies of water, but also pools of lava and bottomless chasms. These will have be carefully generated so that they add character without cutting off sections of the dungeon.

Creatures and Characters

Mobs are divided into two types - creatures and characters. Anything that thinks requires creating two resources in the system - a set of naming rules, and a species. Non-sentient things are classified as creatures, and only require the addition of a single resource. The trouble is that most encounters are comprised of characters. For the initial release of the dungeon generator, I added orcs, goblins, and trolls. As far as the underlying system goes, they're equal to PC races like humans and elves. I intend to keep it that way - they're not "monsters," they're just a different kind of thinking being. This makes any added species useful in a wide variety of ways. They're not just for dungeon encounters. They can be used in any system that uses characters.

I plan on adding all of the common "monster" species in the near future. This includes, but is not limited to, the following:

Bugbears
Centaurs
Dark elves
Deep dwarves (duergar)
Deep gnomes
Dragons (this one's difficult)
Dryads
Fairies
Firbolgs
Gnolls
Gorgons
Half-celestials (aasimar)
Half-elementals (genasi)
Hobgoblins
Kobolds
Lizardfolk
Mind flayers
Minotaurs
Nymphs
Ogres
Pixies
Satyrs
Sea elves
Serpentfolk (based on Yuan-ti)
Troglodytes

In addition, there are currently a few "templates" that can be applied to both creatures and characters. Right now the list only consists of zombies, vampires, and skeletons. However, I plan on adding the following:

Ghost
Lich
Werebear
Weretiger
Werewolf

Also, the list of creatures is going to expand too. Eventually there will be a great many monsters from a great many sources.

Locks, Keys, and Treasure

At the moment, keys are only used for doors. I plan on making keys operate only on locks, and locks be present in doors, chests, and other things. Like the existing keys, they will be populated in places that don't make them unobtainable.

Treasure is limited to coins only at the moment. The system that generates treasure needs a rewrite in order to support mixed treasure piles. This won't be too difficult to implement. I already have gemstone support written, even though it's not used yet. Art objects will be fairly straightforward. Magic items, on the other hand, are going to be more difficult to implement. I plan on leaning on the existing magic weapon generator for this. I also will need to build other types of magic item generators.

All of these, incidentally, will be as system-neutral as possible. Like the dungeon generator itself though, they'll be heavily informed by old school systems.

Closing thoughts and other things

There is a lot of work that could be done that other generators have not yet. I won't promise anything beyond what I've outlined above, but there are some interesting ideas that I want to explore. I'm borrowing from the dungeon generation and population algorithms of roguelikes for some of this. After I finish my entry for /r/rpg_generators' June 2022 challenge, I'll come back to the dungeon generator and iterate more. Thanks for reading.

Bash compound commands and redirection

learnbyexample — Sat, 04 Jun 2022 03:00:00 GMT

I've been using Linux for about 15 years. There are a lot of features I don't know and some that I've used but not often enough or to the full extent of possibilities.

Recently, I had written a bash function, which required saving the output of a for loop to a file. I knew that compound commands support redirection, but it didn't strike me at that time as I haven't had to use them often.

Here's a simplified version of the function I wrote first:

pf()
{
    > input.txt
    for f in "$@" ; do echo "$f $f.bkp" >> input.txt ; done
    cmd input.txt > output.txt
}

Having to empty the file using > input.txt got me thinking that perhaps I was missing some obvious solution. Few days later, I realized that instead of using >> during every iteration of the loop, I should have just applied > to the loop itself.

pf()
{
    for f in "$@" ; do echo "$f $f.bkp" ; done > input.txt
    cmd input.txt > output.txt
}

echo and cmd in the above examples are just placeholders for illustration purposes. I needed both input.txt and output.txt after calling the function, which is why I didn't use | or process substitution.

Why I Moved Off Dev.to

swyx's site RSS Feed — Wed, 01 Jun 2022 18:30:26 GMT

2 years ago I moved all my blogging to Dev.to. Today my main blog is on Github Issues and I've just launched DXTips on Hashnode.

Python tip 11: capture external command output

learnbyexample — Wed, 01 Jun 2022 03:00:00 GMT

The subprocess module provides plethora of features to execute external commands, capturing output being one of them. There are two ways to do so:

passing capture_output=True to subprocess.run()
subprocess.check_output() if you only want stdout

By default, results are provided as bytes data type. You can change that by passing text=True.

>>> import subprocess
>>> cmd = ('date', '-u', '+%A')

>>> p = subprocess.run(cmd, capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
                 stdout='Wednesday\n', stderr='')
>>> p.stdout
'Wednesday\n'

>>> subprocess.check_output(cmd, text=True)
'Wednesday\n'

With check_output(), you'll get an exception if something goes wrong with the command being executed. With run(), you'll get that information from stderr and returncode as part of the CompletedProcess object.

>>> cmd = ('ls', 'xyz.txt')

>>> subprocess.run(cmd, capture_output=True, text=True)
CompletedProcess(args=('ls', 'xyz.txt'), returncode=2, stdout='',
         stderr="ls: cannot access 'xyz.txt': No such file or directory\n")

>>> subprocess.check_output(cmd, text=True)
ls: cannot access 'xyz.txt': No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('ls', 'xyz.txt')' returned
                               non-zero exit status 2.

You can also use legacy methods subprocess.getstatusoutput() and subprocess.getoutput() but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.

Video demo:

See also my 100 Page Python Intro ebook.

What Happened with FPGA Acceleration?

Speculative Branches — Wed, 01 Jun 2022 03:00:00 GMT

In 2018, I took the jump from being primarily an FPGA hardware engineer to being primarily a software engineer. At the time, things were looking great for FPGA acceleration, with AWS and later Azure bringing in VMs with FPGAs and the two big FPGA vendors setting their sights on application acceleration. Almost 5 years later, I am working on another project with FPGAs, this time a cloud-oriented one. That has inspired me to write a retrospective on the last 5 years of what we thought would be an FPGA acceleration boom.

Memex Rewind

--Mike-- — Tue, 31 May 2022 16:25:33 GMT

In 1945, the person in charge of research and development, who oversaw most of the science that won WWII with the help of Logistics, wrote an article about the most pressing task facing mankind, the need to organize and access the ever growing flood of information created by all of this activity.

His vision was expressed in an article titled "As We May Think", which was eventually republished by Time Magazine. Many people credit it for the vision that eventually became the world wide web. I also held this view for a long time.

I no longer hold that view.

The web as we use it is about as useful as a picture of a box of tools, instead of the tools themselves. There was a vague notion expressed that you could gather up a huge canvas of inter-related thoughts and ideas, and freely associate them, and share the whole of that work effortlessly. We don't have it. I'm trying to use my time and effort to change that by making something that gets a bit more of that toolset into my hands, and hopefully into everyone's eventually.

Wish me luck.

The Radiating Circles of DX Architecture

swyx's site RSS Feed — Tue, 31 May 2022 00:56:12 GMT

This post was originally published on my new dedicated DX site: https://dx.tips/circles

On OneStream

Subclassed — Mon, 30 May 2022 14:49:39 GMT

If you follow me on LinkedIn, you might have noticed that, about two years ago, I joined OneStream.

I've since refrained from writing about it, for a number of reasons: the product is massive, so it took a while to get to grasp with it; my new role kinda constrained what I could talk about; and I thought I wasn't particularly well-qualified yet to speak about the subject.

I recently attended the Splash conference for the first time. One of the things I brought home from San Antonio (together with a certain virus most people thought defeated) was the belief that, by now, I actually know a few things about OneStream - and there is a big hunger for that knowledge among clients and partners. The leadership is aware of this, so it was easy to get the greenlight on a few related projects.

This means that I'll be writing a bunch of posts in the next couple of months, on OneStream-related subjects. They might not be published here, but wherever they end up, I'll make sure to link them from here. I'm a geek, not a marketer, so they will be technical posts about getting stuff done; there is already plenty of material on why you should use OneStream for your planning and financial consolidation needs - what people need is to learn how you can do that, and that's where I'm going to help.

In the meantime, if you'd like me to cover a particular topic, feel free to reach out (here, on LinkedIn, or at my OneStreamSoftware.com address (glacava@).

Lessons in Competitive Comms from the Plaid-Stripe Kerfuffle

swyx's site RSS Feed — Sun, 29 May 2022 18:39:38 GMT

Now that the dust has settled on the Plaid-Stripe thing it's time to recap lessons learned.

2022–05–29: Pinephone Pro camera pipeline testing app

megi's PinePhone Development Log — Sun, 29 May 2022 01:00:00 GMT

More Data Scientists should learn SQL

Bytepawn - Marton Trencseni — Sun, 29 May 2022 01:00:00 GMT

In my experience, many Data Scientists struggle to write SQL queries in interviews.

Running Docker without Docker Desktop

swyx's site RSS Feed — Fri, 27 May 2022 21:31:21 GMT

Docker is great. Docker Desktop sucks. Here's my fix.

marginalia.nu goes open source

Weblog on marginalia.nu — Fri, 27 May 2022 03:00:00 GMT

After a bit of soul searching with regards to the future of the website, I’ve decided to open source the code for marginalia.nu, all of its services, including the search engine, encyclopedia, memex, etc.

A motivating factor is the search engine has sort of grown to a scale where it’s becoming increasingly difficult to productively work on as a personal solo project. It needs more structure. What’s kept me from open sourcing it so far has also been the need for more structure. The needs of the marginalia project, and the needs of an open source project have effectively aligned.

CLI tip 2: counting number of matches

learnbyexample — Fri, 27 May 2022 03:00:00 GMT

Use grep -c to count the number of input lines containing a given pattern.

# number of input lines containing 'a'
$ printf 'goal\nrate\neat\npit' | grep -c 'a'
3

# number of input lines containing all the vowels
$ grep -icP '^(?=.*a)(?=.*e)(?=.*i)(?=.*o).*u' /usr/share/dict/words
640

# number of input lines NOT containing 'at'
$ printf 'goal\nrate\neat\npit' | grep -vc 'at'
2

With multiple file input, count is displayed for each file separately. Use cat if you need a combined count.

# separate count for each input file
$ grep -c 'a' names.txt purchases.txt 
names.txt:2
purchases.txt:6

# total count for all the input files
$ cat names.txt purchases.txt | grep -c 'a'
8

If total number of matches is required, use the -o option to display only the matching portions (one per line) and then use wc to get the count.

# -c gives count of matching lines only
$ printf 'goal\nrate\neat\npit' | grep -c '[aeiou]'
4

# use -o to get each match on a separate line
$ printf 'goal\nrate\neat\npit' | grep -o '[aeiou]' | wc -l
7

Note that if you use ripgrep, you can simply use -co or --count-matches instead of piping to the wc command.
# this behavior is different compared to GNU grep
$ printf 'goal\nrate\neat\npit' | rg -co '[aeiou]'
7

Video demo:

See my CLI text processing with GNU grep and ripgrep ebook if you are interested in learning about GNU grep and ripgrep commands in more detail.

Vim tip 9: named registers

learnbyexample — Tue, 24 May 2022 03:00:00 GMT

In Normal mode, you can use lowercase alphabets a-z to save some content for future use. You can also append some more content to those registers by using the corresponding uppercase alphabets A-Z at a later stage.

"ayy copy the current line to the "a register
"bdip delete the current paragraph, contents will also be saved to the "b register
"Ayj append the current line and the line below to the "a register
- "ayy followed by "Ayj will result in total three lines in the "a register
"ap paste content from the "a register
"eyiw copy word under the cursor to the "e register

You can use :reg (short for :registers) to view the contents of the registers. Specifying one or more characters (next to each other as a single string) will display contents only for those registers.

The named registers are also used for saving macros. You can record an empty macro to clear the contents, for example qbq clears the "b register.

Video demo:

My 2022 New Mac Setup

swyx's site RSS Feed — Mon, 23 May 2022 01:03:12 GMT

I set up a new Mac for work today. Here's everything I use on a Mac for fullstack web development.

2022–05–22: Pinephone Pro camera improvements

megi's PinePhone Development Log — Sun, 22 May 2022 01:00:00 GMT

2022–05–22: Pinephone Pro cameras kernel support

megi's PinePhone Development Log — Sun, 22 May 2022 01:00:00 GMT

Useful Python decorators for Data Scientists

Bytepawn - Marton Trencseni — Sun, 22 May 2022 01:00:00 GMT

I show toy implementations of Python decorator patterns that may be useful for Data Scientists.

Teach Your Kids Bridge

Speculative Branches — Sat, 21 May 2022 03:00:00 GMT

A post recently made the rounds on hacker news claiming that you should teach your kids poker, not chess. The comments on that post go through a lot of the reasons why poker is a bad game to teach your children, but I felt that I was well suited to opine on this topic, and explain why duplicate bridge is the best game for practicing the life skills involved in business and programming, compared to all of the alternatives.

Traveling light

bayindirh — Thu, 19 May 2022 03:00:00 GMT

I had to travel recently for a conference, and I decided to apply what I've learned about minimalism so far.

Traveling as a tech person means carrying more electronics and peripherals than most people, and things can get complicated and heavy quickly. This time, I tried to prevent that, and I succeeded.

Since it's a work-related conference, my office laptop came with me, which is small and can be charged with a 30W USB-C charger. Also, a portable 4G modem, a power bank, an e-book reader, and a phone took the ride with me.

That meant a lot of chargers and cables before.

To my surprise, all the peripherals needed for this arsenal of devices can be reduced to a 2-port charger and two cables. That was unimaginable until recently. That removes a lot of volume and weight from the backpack, making it much easier to haul around both on the plane and at the destination. While we still have 15 competing standards for these things, the devices use a small subset of these. Convergence is good.

It's worth noting that the cables and charger are aftermarket items. While this adds another cost item, their long life, daily use, and the convenience they bring are incomparable to this cost. As I've noted before, minimalism and frugality need not be the same thing but, these one-time costs become negligible if the item has a meaningful lifetime.

In my opinion, peace of mind is priceless.

On the luggage side, I took a similar approach. Reducing what's packed to bare essentials and adding extras for critical items only saved a lot of space and weight. That, in turn, enabled me to fly without checked luggage and saved me a lot of time at the airport. I'm not a person who carries everything and the kitchen sink everywhere, but making small changes had a much greater impact than I anticipated.

Another nice side-effect is the time I've saved while packing. Fewer items, less thinking, less hauling, and importantly, much less stress and anxiety.

That is precisely the thing that I wanted to achieve when I started exploring minimalism.

I'm starting to see the results I hoped to achieve with minimalism, but I'm at the beginning of my journey. I'm sure that as I go deeper, I'm going to meet with issues I've never thought about before, but that part comes later for now.

Unfinished Business with Postgres

CRAIG KERSTIENS — Wed, 18 May 2022 19:52:56 GMT

7 years ago I left Heroku. Heroku has had a lot of discussion over the past weeks about its demise, whether it was a success or failure, and the current state. Much of this was prompted by the recent, and on-going security incident, but as others have pointed out the product has been frozen in time for some years now.

I’m not here to rehash the many debates of what is the next Heroku, or whether it was a success or failure, or how it could have been different. Heroku is still a gold standard of developer experience and often used in pitches as Heroku for X. There were many that tried to imitate Heroku for years and failed. Heroku generates sizable revenue to this day. Without Heroku we’d all be in a worse place from a developer experience perspective.

But I don’t want to talk about Heroku the PaaS. Instead I want share a little of my story and some of the story of Heroku Postgres (DoD - Department of Data as we were internally known). I was at Heroku in a lot of product roles over the course of 5 yrs, but most of my time was with that DoD team. When I left Heroku it was a team of about 8 engineers running and managing over 1.5m Postgres databases–a one in a million problem was once a week, we engineered a system that allowed us to scale without requiring a 50 person ops team just for databases

This will be a bit of a personal journey, but also hopefully give some insights into what the vision was and hopefully a bit of what possibilities are for Postgres services in the future.

I wasn’t originally hired to work on anything related to Postgres. As an early PM I first worked on billing, then later on core languages and launching the Python support for Heroku. It was a few months in when I found myself having conversations with many of the internal engineers about Postgres. “Why aren’t you using hstore?”, “Transactional DDL to rollback transactions is absolutely huge!”, “Concurrent index creation runs in the background while not holding a lock, this should always be how you add an index.” Now we had some great engineers, but it was the typical engineer that interacted through ActiveRecord and didn’t want to think about the database.

As I found myself evangelizing Postgres, suddenly I was being recruited by the lead of the Postgres team to come and do marketing, I didn’t know anything about marketing and thought they were joking. A couple months later found myself doing PM and marketing for DoD.

Why did Heroku pick Postgres?

But, I’m getting a little bit ahead of myself. How did Heroku even start doing Postgres or why? Running a PaaS (platform as a service) is a lot of work, running a database is a lot of work. In some sense doing both is splitting your focus. And I’m increasingly coming to the conclusion that platform companies will do best to focus on their platform and data companies will do best to focus on the data. Well, way back in the day we had all these Rails developers asking for a database and we thought how hard could it be? (It was more work than we expected). So we’re gonna run a database, the question becomes which one? Most folks didn’t have strong opinion, but one of our security/ops engineers chimed in “Postgres has always had a good record of being safe and reliable for data, I’d go with it.”

And with that, we were building and launching Heroku Postgres.

The first version of Heroku Postgres, no automation, no self service provisioning, you opened a ticket and we’d correspond and ask when you wanted us to set it up for you.

Before Heroku Postgres was Postgres

The very first version before we committed to Heroku Postgres was internally known as Shen. The model was much more akin to shared hosting that was common for that time and place. Within a single instance we’d pack in a lot of Postgres databases, simply running createdb, creating a user for you, then giving you access to that DB. This worked fine for those just kicking the tires and building small apps, but despite telling them to not use it for production people continued to.

While Heroku had Postgres before Heroku Postgres it became a project and its own orchestration layer for databases as a more first class citizen around 2010. The initial codename for the project was “bifrost”.

The original design

Heroku Postgres was designed to have a central FSM (finite state machine) that would orchestrate the databases. This design pattern to my knowledge came from @PvH’s work and appreciation of video game system design. It felt like a novel approach to the software being built at this time. The fact that it is a more common design patten now shows what a great design it was, and how ahead of its time the level that team was building at.

It was a basic Ruby application that would go out and check the state databases and go through the needed steps when interacting with either Postgres or AWS APIs. AWS APIs back then were not what they are now and this allowed to build in necessary retries, redundancies, and quality of service.

Sometimes you’re good, sometimes you’re lucky

Sometimes you’re good, sometimes you’re lucky, sometimes it’s both. Over time we built out more reliable provisioning and monitoring of databases. In early 2011 we felt a need to continue improving this. It was the early days of EC2 and reliability wasn’t the strongest spot, instances could go offline.

Per @danfarina “We were thinking about working on replication but skipping over archiving (by more carefully managing state between servers, e.g. by directly moving things through rsync, which was/is still pretty normed postgres stuff predating pg_basebackup) but then one of the shared databases (shen) had a near miss when a disk was lost that caused a rather horrific amount of effort and some nailbiting in restoring from pgbackups.”

“We then decided to take previouly rejected approach of building everything up on the archives. the first versions were s3cmd based, and were something of a prototype, upon request from PvH to ship something more raw, but more quickly. We had just got early versions into production when the major disk apocalypse hit in April 2011, though it was in something of an evaluation period and it was not yet a well-exercised & monitored program, so we crossed our fingers and, thankfully, it worked on every database, once AWS had capacity to spare.”

At the end of the event it was communicated eventually that if we wanted AWS could give us all of the EBS disks back, but it could be all of them were corrupted. As someone we can’t recall adequately described it: “It’s like getting a gallon of ice cream, but it may have a turd in it”.

More from @danfarina: “Were it up to me at the time, I probably would have moved onto converting it to boto (one of the most mature AWS client bindings at the time, by far) before stopping to deploy the s3cmd near-prototype on every instance, but that would have been a disaster.”

Our applications were resilient because we could leverage multiple dynos and the routing layer available to our system. Databases are a little different, but how could we at scale give the features you most needed for a database:

No data loss
Improved availability

Number one was always a core charter for the team and we would prioritize this over features, and over uptime. Uptime mattered, features mattered, but as a data provider if we lost data we’d lose trust. Thus that quick prototype became wal-e, which then went on to power many of the future features of Heroku Postgres, and be used for many years, though now has been deemed obsolete in favor of more modern tooling such as pgBackRest. But for it’s time and place it was some good execution and some luck on timing.

Thinking about the entire experience

As Heroku sat at a central point of app deployment we actively tried to think about the experience end to end. This manifested in some of the small things we actively campaigned for and collaborated with the community members who could make these happen. A few key examples come to mind for this.

The first was DATABASE_URL. Some of this originated from the 12factor concepts, others in that having 5 environment variables to define what you were connecting too felt verbose and cumbersome. Why couldn’t Rails just use DATABASE_URL if it were defined. I don’t recall the specifics here, but suspect this was something we nudged @hone02 to help with.

The second was around some features of Postgres. At the time Postgres was going well, but most of the core community focused on performance or enterprise-y features. We were coming at it from a different angle with an audience of Rails developers. We were intentional and engaged with some of the Postgres consultancies that employed committers, with a general theme of how can we help contribute, while also advancing Postgres based on the knowledge we have from users. A few highlights here included:

Not just the DATABASE_URL on the Rails side, but also on the native libpq wire protocol. While we didn’t do the work here, we were spent notable time advocating and engaging around it.
pg_stat_statements in my opinion now the most valuable Postgres extension existed before, but was effectively unusable for most applications. Funding this work was foundational to make Postgres have more usable insights natively.
JSON/JSONB collaboration

Of note we later hired several contributors to largely focus their time on upstream Postgres itself.

Dataclips vs. the team

Throughout the history of Heroku Postgres various individuals made a series of bets. First it was @PvH pushing for Wal-E to get out the door as an MVP, which was absolutely the right call in retrospect. Perhaps the one that is most exciting to me and people least associate with Heroku is dataclips. Matt Soldo, had this idea of GitHub Gists for your database. But in the early days of Heroku as a PM, like many places as PM you didn’t have an ease to mandate engineers try a thing. You had to campaign and convince for a thing.

Soldo didn’t build up enough buy in from the team, but had done plenty of awesome things that it was worth letting him run with this. We were all wrong. Dataclips was built by an external consulting company as a separate standalone Heroku app. It was only live for weeks and suddenly it was powering all of our internal dashboards. A live query you could embed into google sheets, suddenly our internal KPI rails app was replaced by dataclips and a google sheet. We didn’t need looker or tableau or other fancy BI tools and this lasted into 100m in revenue for insights into the business. To this day dataclips is one of my favorite features of Heroku, and I look forward to making an experience like that but even better, thanks Soldo for not listening to the rest of us back then.

Names matter

For a long time databases haven’t been known for being user friendly. We wanted to pull from other paradigms as we were making key database tenets available to people. We looked heavily to git for inspiration around forking/following. Database terms were common as master/slave, but we knew we could do better. Ee wanted to give the user a sense of what they could do with it. Archiving the WAL every 16 MB or 60 seconds (whichever was less) became continuous protection. Forking, was a snapshot as of some point in time. Follower, something that followers a leader node (a read-replica). I still recall an hour long analyst call with Redmonk with @monkchips and @sogrady–it was mostly wind them up and let them go (for the record @monkchips didn’t love fork/follow, I think he may have come around now).

This started even earlier than my time with being intentional about Postgres vs. PostgreSQL. But I’ve covered that one before, and you can even see it in some of the other lobbying around libpq.

Peacetime vs. Wartime

Things were rolling along well, we were shipping new features. We’d added dataclips, fork/follow were in existence for a while–and then we got a note from Amazon they were launching Postgres support for RDS at the next ReInvent. I was in person at that ReInvent and I’d never seen a roaring standing applause like that at a tech conference before when the moment was announced. In private channels we heard notes that this was because of us, the excitement and demand for Postgres became too clear for them to ignore and they had to add support.

We felt vindicated in our choice of Postgres and in what we’d built. But we also knew that now we had competition, running a database as a service on another infrastructure provider how can you compete? Well from some years of experience now I can say there are definitely ways and am confident that sharp narrow focus allows you to build amazing products which can be harder to do inside a large giant corporation. It was at one of allballs (UTC 00:00:00-when the data team would do happy hour on Fridays) that we were drinking beers and discussing how now we really had to focus. @PvH and @danfarina were discussing me personally as a leader–apparently I’m okay in peacetime when things are good but in wartime they were willing to bet on me.

Two weeks later I walked into the exec team meeting with a 2-pager assessment of what may happen, how we could compete, including 3 potential acquisition targets that could allow us to have a more differentiated offering. Within a few months we made one of those acquisitions which later became Heroku Connect. It made a lot of sense for many reasons, including Adam Gross was an angel investor in Heroku and knew it well and had helped build Salesforce Platform in early years. That wasn’t the end, but was just the beginning of how we could actively compete vs. simply being “hosted Postgres” vs. a more fully managed experience.

Metrics and Monitoring that almost was

The next vision and goal for Heroku Postgres was to continue to give better ease of use and insights into your database. Postgres itself already has a ton of awesome data inside it about how it’s been used. The catalog tables and extensions like pg_stat_statements have a wealth of information, but querying it looks like 200 lines of black magic SQL. @leinweber was perhaps one of the first, and the best at the team and quickly making something usable for people. The first step on our metrics journey was him making these Postgres insights trivially accessible via pg-extras.

Continuing on the journey internal foundational systems were built, in fact we just spent 3 months @crunchydata building similar systems that were spiritually aligned, to focus on collection of various meta data from systems and the ability to notify and communicate. I’m blanking on some of the systems, though some were obvious–observatory (I’m not sure if still in use or not) would observe databases.

These systems started to house a lot of information that then powered pieces within your Heroku Postgres dashboard. Things like slow queries and high IO or CPU load would give good insights when you logged in. The eventual goal was to connect the dots through proactive notifications. It’s one thing to get an alert from pagerduty that things are off, login to a dashboard and fix it. But what if in the early signs of things starting to go south we emailed you that you’ve got an increasing number of sequential scans that are starting to put you at risk of IOPS saturation, and because we understand Postgres you can add an index and resolve this with a single command. We could even give you an easy button in your email to add the index. All the foundations were in place, if you’ve run on Heroku you’ve gotten the notifications about database maintenance, that was powered entirely from the underlying notification system built with this goal in mind.

Postgres can still be better for developers

But we never made it there. Some of us shuffled to different teams, some of us moved on to new challenges, and many folks came on after to continue to run and power a great database service. Some of those original engineers Daniel Farina and Will Leinweber (along with PvH) understand the design and why as well as anyone. The goal from early on was that we could do better for developers.

Two years ago when people asked why I came to Crunchy Data I told them I had unfinished business. After the success at Citus tackling scalability problems where the average customer being 40TB in data, I attracted to the idea of returning to the vision we had back at the DoD of bringing a better Postgres experience to developers. Despite the rapid growth of successful DBaaS offerings, there was still something missing - that initial idea of DoD that we still wanted to create.

Postgres is an amazing database, can handle hundreds of thousands of transactions per second often without batting an eye. Has internal data that you can easily look at to assess and improve performance. Has a rich set of datatypes, indexing, and functionality. The extension ecosystem is vast. But as a developer you don’t have time to become an expert on Postgres.

What if we told you when a N+1 query snuck into your Rails app?
Connections don’t have to be a limitation on Postgres when you have pgBouncer right there.
Have excessive indexes from the early stage of building your app? What if we told you about them and with a button click you could drop them.

One of our @crunchydata customers described it better than I ever could. When working with some of our experts on some deep dives into their database they said “you should take all his Postgres expertise and just bottle it up and send it in email or slack reports to me.” I want that expertise as a product.

My product strategy isn’t to go and change the world of databases. Postgres is a great database with a community that is making it better daily. I want to help make open source Postgres better and give back to it along the way. My product strategy is to distribute deep Postgres expertise in a consumable form to every customer of ours in the coming years. Oh we’ll ship some cool things along the way too.

Fixed Point Arithmetic

Speculative Branches — Wed, 18 May 2022 03:00:00 GMT

When we think of how to represent fractional numbers in code, we reach for double and float, and almost never reach for anything else. There are several alternatives, including constructive real numbers that are used in calculators, and rational numbers. One alternative predates all of these, including floating point, and actually allows you to compute faster than when you use floating point numbers. That alternative is fixed point: a primitive form of decimal that does not offer any of the conveniences of float, but allows you to do decimal computations more quickly and efficiently. Fixed point still has usage in some situations today, and it can be a potent tool in your arsenal as a programmer if you find yourself working with math at high speed.

Firefox Addons for 2022

smcleod.net — Tue, 17 May 2022 11:00:00 GMT

Firefox Addons - 2022 Edition

My list of must-have Firefox addons - 2022 edition

Updated: 2022-11-07

Privacy and Security

Firstly - you should have Firefox’s Enhanced Tracking Protection enabled.

Don’t Track Me Google
LocalCDN
UTM Tracking Token Stripper
- Note: You can accomplish some of what this does by setting up the removeparam uBlock origin rules I’ve listed below.
Multi-Account Containers
- Useful for setting sites such as Amazon, eBay, Twitter, LinkedIn, Banking etc… each to always open in their own isolated container.
The official addon for whatever Password Manager you use.
Firefox Translations

uBlock Origin

Probably the single most important addon.

Linux compared to Window 11 in terms of suckage

--Mike-- — Tue, 17 May 2022 08:09:42 GMT

I came across this comparison video, and it makes a comparison of features between Pop Linux and Windows 11 For the most part, they're equivalent.

Let's build a distributed Postgres proof of concept

Notes on software development — Tue, 17 May 2022 03:00:00 GMT

What is CockroachDB under the hood? Take a look at its go.mod and notice a number of dependencies that do a lot of work: a PostgreSQL wire protocol implementation, a storage layer, a Raft implementation for distributed consensus. And not part of go.mod but still building on 3rd party code, PostgreSQL's grammar definition.

To be absurdly reductionist, CockroachDB is just the glue around these libraries. With that reductionist mindset, let's try building a distributed Postgres proof of concept ourselves! We'll use only four major external libraries: for parsing SQL, handling Postgres's wire protocol, handling Raft, and handling the storage of table metadata and rows themselves.

For a not-reductionist understanding of the CockroachDB internals, I recommend following the excellent Cockroach Engineering blog and Jordan Lewis's Hacking CockroachDB Twitch stream.

By the end of this post, in around 600 lines of code, we'll have a distributed "Postgres implementation" that will accept writes (CREATE TABLE, INSERT) on the leader and accept reads (SELECT) on any node. All nodes will contain the same data.

Here is a sample interaction against the leader:

$ psql -h localhost -p 6000
psql (13.4, server 0.0.0)
Type "help" for help.

phil=> create table x (age int, name text);
CREATE ok
phil=> insert into x values(14, 'garry'), (20, 'ted');
could not interpret result from server: INSERT ok
INSERT ok
phil=> select name, age from x;
  name   | age 
---------+-----
 "garry" |  14
 "ted"   |  20
(2 rows)

And against a follower (note the different port):

$ psql -h 127.0.0.1 -p 6001
psql (13.4, server 0.0.0)
Type "help" for help.

phil=> select age, name from x;
 age |  name
-----+---------
  20 | "ted"
  14 | "garry"
(2 rows)

All code for this post is available on Github in the fondly named WaterbugDB repo.

Plan of attack

Influenced by Philip O'Toole's talk on rqlite at Hacker Nights we'll have a Postgres wire protocol server in front. As it receives queries it will respond immediately to SELECTs. Otherwise for CREATE TABLEs and INSERTs it will send the entire query string to the Raft cluster. Each process that is part of the Raft cluster will implement the appropriate functions for handling Raft messages. In this case the messages will just be to create a table or insert data.

So every running process will run a Postgres wire protocol server, a Raft server, and an HTTP server that you'll see is an implementation detail about how processes join to the same Raft cluster.

Every running process will have its own directory for storing data.

Raft

There is likely a difference between Raft, the paper, and Raft, the implementations. When I refer to Raft in the rest of this post I'm going to be referring to an implementation.

And although CockroachDB use's etcd's Raft implementation, I didn't realize that when I started building this project. I used Hashicorp's Raft implementation.

Raft allows us to reliably keep multiple nodes in sync with a log of messages. Each node in the Raft cluster implements a finite state machine (FSM) with three operations: apply, snapshot, and restore. Our finite state machine will embed a postgres engine we'll build out after this to handle query execution.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net"
    "net/http"
    "os"
    "path"
    "strings"
    "time"

    "github.com/google/uuid"
    "github.com/hashicorp/raft"
    "github.com/hashicorp/raft-boltdb"
    "github.com/jackc/pgproto3/v2"
    pgquery "github.com/pganalyze/pg_query_go/v2"
    bolt "go.etcd.io/bbolt"
)

type pgFsm struct {
    pe *pgEngine
}

From what I understand, the snapshot operation allows Raft to truncate logs. It is used in conjuction with restoring. On startup if there is a snapshot, restore is called so you can load the snapshot. Then afterwards all logs not yet snapshotted are replayed through the apply operation.

To keep this implementation simple we'll just fail all snapshots so restore will never be called and all logs will be replayed every time on startup through the apply operation. This is of course inefficient but it keeps the code simpler.

When we write the startup code we'll need to delete the database so that these apply calls happen fresh.

type snapshotNoop struct{}

func (sn snapshotNoop) Persist(sink raft.SnapshotSink) error {
    return sink.Cancel()
}

func (sn snapshotNoop) Release() {}

func (pf *pgFsm) Snapshot() (raft.FSMSnapshot, error) {
    return snapshotNoop{}, nil
}

func (pf *pgFsm) Restore(rc io.ReadCloser) error {
    return fmt.Errorf("Nothing to restore")
}

Finally, applying is receiving a single message and applying it for the node. In this project the message will be a CREATE TABLE or INSERT query. So we'll parse the query and pass it to the postgres engine for execution.

func (pf *pgFsm) Apply(log *raft.Log) interface{} {
    switch log.Type {
    case raft.LogCommand:
        ast, err := pgquery.Parse(string(log.Data))
        if err != nil {
            panic(fmt.Errorf("Could not parse payload: %s", err))
        }

        err = pf.pe.execute(ast)
        if err != nil {
            panic(err)
        }
    default:
        panic(fmt.Errorf("Unknown raft log type: %#v", log.Type))
    }

    return nil
}

Panic-ing here is actually the advised behavior.

Raft server

Now we can set up the actual Raft server and pass an instance of this FSM. This is a bunch of boilerplate that would matter in production installs but for us basically we just need to tell Raft where to run and how to store its own internal data, including its all-important message log.

func setupRaft(dir, nodeId, raftAddress string, pf *pgFsm) (*raft.Raft, error) {
    os.MkdirAll(dir, os.ModePerm)

    store, err := raftboltdb.NewBoltStore(path.Join(dir, "bolt"))
    if err != nil {
        return nil, fmt.Errorf("Could not create bolt store: %s", err)
    }

    snapshots, err := raft.NewFileSnapshotStore(path.Join(dir, "snapshot"), 2, os.Stderr)
    if err != nil {
        return nil, fmt.Errorf("Could not create snapshot store: %s", err)
    }

    tcpAddr, err := net.ResolveTCPAddr("tcp", raftAddress)
    if err != nil {
        return nil, fmt.Errorf("Could not resolve address: %s", err)
    }

    transport, err := raft.NewTCPTransport(raftAddress, tcpAddr, 10, time.Second*10, os.Stderr)
    if err != nil {
        return nil, fmt.Errorf("Could not create tcp transport: %s", err)
    }

    raftCfg := raft.DefaultConfig()
    raftCfg.LocalID = raft.ServerID(nodeId)

    r, err := raft.NewRaft(raftCfg, pf, store, store, snapshots, transport)
    if err != nil {
        return nil, fmt.Errorf("Could not create raft instance: %s", err)
    }

    // Cluster consists of unjoined leaders. Picking a leader and
    // creating a real cluster is done manually after startup.
    r.BootstrapCluster(raft.Configuration{
        Servers: []raft.Server{
            {
                ID:      raft.ServerID(nodeId),
                Address: transport.LocalAddr(),
            },
        },
    })

    return r, nil
}

Every instance of this process will run this and will start off as a leader in a new cluster. We'll expose an HTTP server that allows a leader to talk to other leaders to tell them to stop leading and follow it. This HTTP endpoint in the HTTP server is how we'll get from N process with N leaders and N clusters to N processes with 1 leader and 1 cluster.

That's basically it for the core Raft bits. So let's build out that HTTP server and follow endpoint.

HTTP follow endpoint

Our HTTP server will have just one endpoint that tells the process (a) to contact another process (b) so that process (b) joins the process (a) cluster.

The HTTP server will need to have the process (a)'s Raft instance to be able to start this join action. And in order for Raft to know how to contact the process (b) we'll need to tell it both the process (b)'s unique Raft node id (we'll give it a unique id ourselves when we start the process) and the process (b)'s Raft server port.

type httpServer struct {
    r *raft.Raft
}

func (hs httpServer) addFollowerHandler(w http.ResponseWriter, r *http.Request) {
    followerId := r.URL.Query().Get("id")
    followerAddr := r.URL.Query().Get("addr")

    if hs.r.State() != raft.Leader {
        json.NewEncoder(w).Encode(struct {
            Error string `json:"error"`
        }{
            "Not the leader",
        })
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    err := hs.r.AddVoter(raft.ServerID(followerId), raft.ServerAddress(followerAddr), 0, 0).Error()
    if err != nil {
        log.Printf("Failed to add follower: %s", err)
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    w.WriteHeader(http.StatusOK)
}

That's it! Let's move on to the query engine.

Query engine

The query engine is a wrapper around a storage layer. We'll bring in bbolt.

I originally built this with Cockroach's pebble but pebble has a transitive dependency on a C library that has function names that conflict with function names in the C library that pg_query_go wraps.

type pgEngine struct {
    db         *bolt.DB
    bucketName []byte
}

func newPgEngine(db *bolt.DB) *pgEngine {
    return &pgEngine{db, []byte("data")}
}

bbolt organizes data into buckets. Buckets might be a natural way to store table rows (one bucket per table) but to keep the implementation simple we'll put all table metadata and row data into a single `data` bucket.

The entrypoint we called in the Raft apply implementation above was execute. It took a parsed list of statements. We'll iterate over the statements, figuring out the kind of each statement, and call out to a dedicated helper for each kind.

func (pe *pgEngine) execute(tree *pgquery.ParseResult) error {
    for _, stmt := range tree.GetStmts() {
        n := stmt.GetStmt()
        if c := n.GetCreateStmt(); c != nil {
            return pe.executeCreate(c)
        }

        if c := n.GetInsertStmt(); c != nil {
            return pe.executeInsert(c)
        }

        if c := n.GetSelectStmt(); c != nil {
            _, err := pe.executeSelect(c)
            return err
        }

        return fmt.Errorf("Unknown statement type: %s", stmt)
    }

    return nil
}

The pg_query_go docs are not super helpful. I had to build a separate AST explorer program to make it easier to understand this parser.

Let's start with creating a table.

Create table

When a table is created, we'll need to store its metadata.

type tableDefinition struct {
    Name        string
    ColumnNames []string
    ColumnTypes []string
}

First we pull that metadata out of the AST.

func (pe *pgEngine) executeCreate(stmt *pgquery.CreateStmt) error {
    tbl := tableDefinition{}
    tbl.Name = stmt.Relation.Relname

    for _, c := range stmt.TableElts {
        cd := c.GetColumnDef()

        tbl.ColumnNames = append(tbl.ColumnNames, cd.Colname)

        // Names is namespaced. So `INT` is pg_catalog.int4. `BIGINT` is pg_catalog.int8.
        var columnType string
        for _, n := range cd.TypeName.Names {
            if columnType != "" {
                columnType += "."
            }
            columnType += n.GetString_().Str
        }
        tbl.ColumnTypes = append(tbl.ColumnTypes, columnType)
    }

Now we need to store this in the storage layer. The easiest/dumbest way to do this is to serialize the metadata to JSON and store it with key: tables_${tableName}.

    tableBytes, err := json.Marshal(tbl)
    if err != nil {
        return fmt.Errorf("Could not marshal table: %s", err)
    }

    err = pe.db.Update(func(tx *bolt.Tx) error {
        bkt, err := tx.CreateBucketIfNotExists(pe.bucketName)
        if err != nil {
            return err
        }

        return bkt.Put([]byte("tables_"+tbl.Name), tableBytes)
    })

    if err != nil {
        return fmt.Errorf("Could not set key-value: %s", err)
    }

    return nil
}

Next we'll build a helper to reverse that operation, pulling out table metadata from the storage layer by the table name:

func (pe *pgEngine) getTableDefinition(name string) (*tableDefinition, error) {
    var tbl tableDefinition

    err := pe.db.View(func(tx *bolt.Tx) error {
        bkt := tx.Bucket(pe.bucketName)
        if bkt == nil {
            return fmt.Errorf("Table does not exist")
        }

        valBytes := bkt.Get([]byte("tables_" + name))
        err := json.Unmarshal(valBytes, &tbl)
        if err != nil {
            return fmt.Errorf("Could not unmarshal table: %s", err)
        }

        return nil
    })

    return &tbl, err
}

That's it for our basic CREATE TABLE support! Let's do INSERT next.

Insert row

Our support for insert will only support literal/constant VALUES.

func (pe *pgEngine) executeInsert(stmt *pgquery.InsertStmt) error {
    tblName := stmt.Relation.Relname

    slct := stmt.GetSelectStmt().GetSelectStmt()
    for _, values := range slct.ValuesLists {
        var rowData []any
        for _, value := range values.GetList().Items {
            if c := value.GetAConst(); c != nil {
                if s := c.Val.GetString_(); s != nil {
                    rowData = append(rowData, s.Str)
                    continue
                }

                if i := c.Val.GetInteger(); i != nil {
                    rowData = append(rowData, i.Ival)
                    continue
                }
            }

            return fmt.Errorf("Unknown value type: %s", value)
        }

It would be better to abstract this VALUES code into a helper so it could be used by SELECTs too but out of laziness we'll just keep this here.

Next we need to write the row to the storage layer. We'll serialize the row data to JSON (inefficient because we know the row structure, but JSON is easy). We'll store the row with a prefix including the table name and we'll give its key a unique UUID. When we're iterating over rows in the table we'll be able to do a prefix scan that will recover just the rows in this table.

        rowBytes, err := json.Marshal(rowData)
        if err != nil {
            return fmt.Errorf("Could not marshal row: %s", err)
        }

        id := uuid.New().String()
        err = pe.db.Update(func(tx *bolt.Tx) error {
            bkt, err := tx.CreateBucketIfNotExists(pe.bucketName)
            if err != nil {
                return err
            }

            return bkt.Put([]byte("rows_"+tblName+"_"+id), rowBytes)
        })
        if err != nil {
            return fmt.Errorf("Could not store row: %s", err)
        }
    }

    return nil
}

Finally we can move on to support SELECT!

Select rows

Unlike CREATE TABLE and INSERT, SELECT will need to return rows, column names, and because the Postgres wire protocol wants it, column types.

type pgResult struct {
    fieldNames []string
    fieldTypes []string
    rows       [][]any
}

First we pull out the table name and the fields selected, looking up field types in the table metadata.

func (pe *pgEngine) executeSelect(stmt *pgquery.SelectStmt) (*pgResult, error) {
    tblName := stmt.FromClause[0].GetRangeVar().Relname
    tbl, err := pe.getTableDefinition(tblName)
    if err != nil {
        return nil, err
    }

    results := &pgResult{}
    for _, c := range stmt.TargetList {
        fieldName := c.GetResTarget().Val.GetColumnRef().Fields[0].GetString_().Str
        results.fieldNames = append(results.fieldNames, fieldName)

        fieldType := ""
        for i, cn := range tbl.ColumnNames {
            if cn == fieldName {
                fieldType = tbl.ColumnTypes[i]
            }
        }

        if fieldType == "" {
            return nil, fmt.Errorf("Unknown field: %s", fieldName)
        }

        results.fieldTypes = append(results.fieldTypes, fieldType)
    }

Finally, we do a prefix scan to grab all rows in the table from the storage layer.

    prefix := []byte("rows_" + tblName + "_")
    pe.db.View(func(tx *bolt.Tx) error {
        c := tx.Bucket(pe.bucketName).Cursor()

        for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
            var row []any
            err = json.Unmarshal(v, &row)
            if err != nil {
                return fmt.Errorf("Unable to unmarshal row: %s", err)
            }

            var targetRow []any
            for _, target := range results.fieldNames {
                for i, field := range tbl.ColumnNames {
                    if target == field {
                        targetRow = append(targetRow, row[i])
                    }
                }
            }

            results.rows = append(results.rows, targetRow)
        }

        return nil
    })

    return results, nil
}

That's it for SELECT! The last function we'll implement is a helper for deleting all data in the storage layer. This will be called on startup before Raft logs are applied so the database always ends up in a consistent state.

func (pe *pgEngine) delete() error {
    return pe.db.Update(func(tx *bolt.Tx) error {
        bkt := tx.Bucket(pe.bucketName)
        if bkt != nil {
            return tx.DeleteBucket(pe.bucketName)
        }

        return nil
    })
}

And we're ready to move on to the final layer, the Postgres wire protocol.

Postgres wire protocol server

jackc/pgproto3 is an implementation of the Postgres wire protocol for Go. It allows us to implement a server that can respond to requests by Postgres clients like psql.

It works by wrapping a TCP connection. So we'll start by building a function that does the TCP serving loop.

func runPgServer(port string, db *bolt.DB, r *raft.Raft) {
    ln, err := net.Listen("tcp", "localhost:"+port)
    if err != nil {
        log.Fatal(err)
    }

    for {
        conn, err := ln.Accept()
        if err != nil {
            log.Fatal(err)
        }

        pc := pgConn{conn, db, r}
        go pc.handle()
    }
}

The pgConn instance needs access to the database directly so it can respond to SELECTs. And it needs the Raft instance for all other queries.

type pgConn struct {
    conn net.Conn
    db   *bolt.DB
    r    *raft.Raft
}

The handle function we called above will grab the current message via the pgproto3 package and handle startup messages and regular messages.

func (pc pgConn) handle() {
    pgc := pgproto3.NewBackend(pgproto3.NewChunkReader(pc.conn), pc.conn)
    defer pc.conn.Close()

    err := pc.handleStartupMessage(pgc)
    if err != nil {
        log.Println(err)
        return
    }

    for {
        err := pc.handleMessage(pgc)
        if err != nil {
            log.Println(err)
            return
        }
    }
}

Startup messages include authorization and SSL checks. We'll allow anything in the former and respond "no" to the latter.

func (pc pgConn) handleStartupMessage(pgconn *pgproto3.Backend) error {
    startupMessage, err := pgconn.ReceiveStartupMessage()
    if err != nil {
        return fmt.Errorf("Error receiving startup message: %s", err)
    }

    switch startupMessage.(type) {
    case *pgproto3.StartupMessage:
        buf := (&pgproto3.AuthenticationOk{}).Encode(nil)
        buf = (&pgproto3.ReadyForQuery{TxStatus: 'I'}).Encode(buf)
        _, err = pc.conn.Write(buf)
        if err != nil {
            return fmt.Errorf("Error sending ready for query: %s", err)
        }

        return nil
    case *pgproto3.SSLRequest:
        _, err = pc.conn.Write([]byte("N"))
        if err != nil {
            return fmt.Errorf("Error sending deny SSL request: %s", err)
        }

        return pc.handleStartupMessage(pgconn)
    default:
        return fmt.Errorf("Unknown startup message: %#v", startupMessage)
    }
}

Within the main handleMessage logic we'll check the type of message.

func (pc pgConn) handleMessage(pgc *pgproto3.Backend) error {
    msg, err := pgc.Receive()
    if err != nil {
        return fmt.Errorf("Error receiving message: %s", err)
    }

    switch t := msg.(type) {
    case *pgproto3.Query:
                // TODO
    case *pgproto3.Terminate:
        return nil
    default:
        return fmt.Errorf("Received message other than Query from client: %s", msg)
    }

    return nil
}

If the message is a query we'll parse it and respond immediately to SELECTs.

    switch t := msg.(type) {
    case *pgproto3.Query:
        stmts, err := pgquery.Parse(t.String)
        if err != nil {
            return fmt.Errorf("Error parsing query: %s", err)
        }

        if len(stmts.GetStmts()) > 1 {
            return fmt.Errorf("Only make one request at a time.")
        }

        stmt := stmts.GetStmts()[0]

        // Handle SELECTs here
        s := stmt.GetStmt().GetSelectStmt()
        if s != nil {
            pe := newPgEngine(pc.db)
            res, err := pe.executeSelect(s)
            if err != nil {
                return err
            }

            pc.writePgResult(res)
            return nil
        }

(We'll implement that writePgResult helper shortly below.) Otherwise we'll add the query to the Raft log and return a basic response.

        // Otherwise it's DDL/DML, raftify
        future := pc.r.Apply([]byte(t.String), 500*time.Millisecond)
        if err := future.Error(); err != nil {
            return fmt.Errorf("Could not apply: %s", err)
        }

        e := future.Response()
        if e != nil {
            return fmt.Errorf("Could not apply (internal): %s", e)
        }

        pc.done(nil, strings.ToUpper(strings.Split(t.String, " ")[0])+" ok")
    case *pgproto3.Terminate:
        return nil
    default:
        return fmt.Errorf("Received message other than Query from client: %s", msg)
    }

    return nil
}

done is an important helper that tells the Postgres connection that the query is complete and the server is ready to receive another query. Without this response psql just hangs.

func (pc pgConn) done(buf []byte, msg string) {
    buf = (&pgproto3.CommandComplete{CommandTag: []byte(msg)}).Encode(buf)
    buf = (&pgproto3.ReadyForQuery{TxStatus: 'I'}).Encode(buf)
    _, err := pc.conn.Write(buf)
    if err != nil {
        log.Printf("Failed to write query response: %s", err)
    }
}

And now let's implement the writePgResult helper. This function needs to translate from our pgResult struct to the format require by pgproto3.

var dataTypeOIDMap = map[string]uint32{
    "text":            25,
    "pg_catalog.int4": 23,
}

func (pc pgConn) writePgResult(res *pgResult) {
    rd := &pgproto3.RowDescription{}
    for i, field := range res.fieldNames {
        rd.Fields = append(rd.Fields, pgproto3.FieldDescription{
            Name:        []byte(field),
            DataTypeOID: dataTypeOIDMap[res.fieldTypes[i]],
        })
    }
    buf := rd.Encode(nil)
    for _, row := range res.rows {
        dr := &pgproto3.DataRow{}
        for _, value := range row {
            bs, err := json.Marshal(value)
            if err != nil {
                log.Printf("Failed to marshal cell: %s\n", err)
                return
            }

            dr.Values = append(dr.Values, bs)
        }

        buf = dr.Encode(buf)
    }

    pc.done(buf, fmt.Sprintf("SELECT %d", len(res.rows)))
}

And we're done with everything but func main()!

Main

On startup, each process must be assigned (by the parent process) a unique node id (any unique string is ok) and ports for the Raft server, Postgres server, and HTTP server. We'll build a short getConfig helper to grab these from arguments.

type config struct {
    id       string
    httpPort string
    raftPort string
    pgPort   string
}

func getConfig() config {
    cfg := config{}
    for i, arg := range os.Args[1:] {
        if arg == "--node-id" {
            cfg.id = os.Args[i+2]
            i++
            continue
        }

        if arg == "--http-port" {
            cfg.httpPort = os.Args[i+2]
            i++
            continue
        }

        if arg == "--raft-port" {
            cfg.raftPort = os.Args[i+2]
            i++
            continue
        }

        if arg == "--pg-port" {
            cfg.pgPort = os.Args[i+2]
            i++
            continue
        }
    }

    if cfg.id == "" {
        log.Fatal("Missing required parameter: --node-id")
    }

    if cfg.raftPort == "" {
        log.Fatal("Missing required parameter: --raft-port")
    }

    if cfg.httpPort == "" {
        log.Fatal("Missing required parameter: --http-port")
    }

    if cfg.pgPort == "" {
        log.Fatal("Missing required parameter: --pg-port")
    }

    return cfg
}

Now in main we'll grab the config and set up this process's database. All processes will put their data in a top-level data directory to make managing the directories easier. But within that directory each process will have their own unique directories for data storage based on the unique node id.

func main() {
    cfg := getConfig()

    dataDir := "data"
    err := os.MkdirAll(dataDir, os.ModePerm)
    if err != nil {
        log.Fatalf("Could not create data directory: %s", err)
    }

    db, err := bolt.Open(path.Join(dataDir, "/data"+cfg.id), 0600, nil)
    if err != nil {
        log.Fatalf("Could not open bolt db: %s", err)
    }
    defer db.Close()

We need to clean up the database.

    pe := newPgEngine(db)
    // Start off in clean state
    pe.delete()

Set up the Raft server.

    pf := &pgFsm{pe}
    r, err := setupRaft(path.Join(dataDir, "raft"+cfg.id), cfg.id, "localhost:"+cfg.raftPort, pf)
    if err != nil {
        log.Fatal(err)
    }

Set up the HTTP server.

    hs := httpServer{r}
    http.HandleFunc("/add-follower", hs.addFollowerHandler)
    go func() {
        err := http.ListenAndServe(":"+cfg.httpPort, nil)
        if err != nil {
            log.Fatal(err)
        }
    }()

And finally, kick off the Postgres server.

    runPgServer(cfg.pgPort, db, r)
}

Finally. Finally. Finally done. Let's give it a go. :)

What hath god wrought

First, initialize the go module and then build the app.

$ go mod init waterbugdb
$ go mod tidy
$ go build

Now in terminal 1 start an instance of the database,

$ ./waterbugdb --node-id node1 --raft-port 2222 --http-port 8222 --pg-port 6000

Then in terminal 2 start another instance.

$ ./waterbugdb --node-id node2 --raft-port 2223 --http-port 8223 --pg-port 6001

And in terminal 3, tell node1 to have node2 follow it.

$ curl 'localhost:8222/add-follower?addr=localhost:2223&id=node2'

And then open psql against port 6000, the leader.

$ psql -h localhost -p 6000
psql -h 127.0.0.1 -p 6000
psql (13.4, server 0.0.0)
Type "help" for help.

phil=> create table x (age int, name text);
CREATE ok
phil=> insert into x values(14, 'garry'), (20, 'ted');
could not interpret result from server: INSERT ok
INSERT ok
phil=> select name, age from x;
  name   | age 
---------+-----
 "garry" |  14
 "ted"   |  20
(2 rows)

Now kill node1 in terminal 1. Then start it up again. node2 will now be the leader. So exit psql in terminal 3 and enter it again pointed at node2, port 6001. Add new data.

$ psql -h 127.0.0.1 -p 6001
psql (13.4, server 0.0.0)
Type "help" for help.

phil=> insert into x values(19, 'ava'), (18, 'ming');
could not interpret result from server: INSERT ok
phil=> select age, name from x;
 age |  name
-----+---------
  20 | "ted"
  14 | "garry"
  18 | "ming"
  19 | "ava"

Exit psql in terminal 3 and start it up again against node1 again, port 6000.

$ psql -h 127.0.0.1 -p 6000
psql (13.4, server 0.0.0)
Type "help" for help.

phil=> select age, name from x;
 age |  name
-----+---------
  20 | "ted"
  14 | "garry"
  18 | "ming"
  19 | "ava"
(2 rows)

Nifty stuff.

Summary

So on the one hand this was a more complex post than my usual. Each process needed three servers running. Two of those servers we managed directly and the Raft server was managed by the Raft library.

On the other hand, we did this all in a really small amount of code. Yes many edge cases were unhandled and massive amount of SQL was unhandled. And yes there are tons of inefficiencies like using JSON, an unstructured format when every table has fixed structure. But hopefully now you have an idea of how a project like this could be structured. And there's the beginnings of a framework for filling in syntax/edge cases over time.

Additionally, the only problem we solved with consensus was replication, not sharding. This, and it's more complicated cousin (cross-shard transactions), is truly the special sauce Cockroach brings.

Read more about building an intuition for sharding, replication, and distributed consensus [here](https://notes.eatonphil.com/2024-02-08-an-intuition-for-distributed-consensus-in-oltp-systems.html.

New blog post is up :) Let's build a distributed postgres proof of concept.https://t.co/Z8BDzF1bUw pic.twitter.com/aSkOjr9Yrh
— Phil Eaton (@phil_eaton) May 17, 2022

The Law of Conservation of Risk

swyx's site RSS Feed — Mon, 16 May 2022 05:15:58 GMT

Risk cannot be created or destroyed, it can only be redistributed

Python tip 3: expression and result with f-string

learnbyexample — Mon, 16 May 2022 03:00:00 GMT

In case you haven't yet discovered this awesome f-string feature, you can add = after an expression to get both the expression and the result in the output.

>>> num1 = 42
>>> num2 = 7

>>> f'{num1 + num2 = }'
'num1 + num2 = 49'
>>> f'{num1 + (num2 * 10) = }'
'num1 + (num2 * 10) = 112'

I use it often to quickly test a function:

>>> def isodd(n):
...     return bool(n % 2)
... 
>>> print(f'{isodd(42) = }')
isodd(42) = False
>>> print(f'{isodd(123) = }')
isodd(123) = True

See docs.python: Format String Syntax, docs.python: Formatted string literals and fstring.help for documentation and examples.

Video demo:

See my 100 Page Python Intro ebook for a short, introductory guide for the Python programming language.

Python tip 2: membership operator

learnbyexample — Mon, 16 May 2022 03:00:00 GMT

The in membership operator checks if a given value is part of a collection of values. Here's an example with range() function:

>>> num = 5

# checks if num is present among the integers 3 or 4 or 5
>>> num in range(3, 6)
True

Instead of a series of == comparisons combined with the or boolean operator, you can utilize the in operator.

>>> pet = 'cat'

# instead of doing this
>>> pet == 'bat' or pet == 'cat' or pet == 'dog'
True

# use the membership operator
>>> pet in ('bat', 'cat', 'dog')
True

When applied to strings, the in operator performs substring comparison.

>>> fruit = 'mango'

>>> 'an' in fruit
True
>>> 'at' in fruit
False

To invert the membership test, use the not in operator.

>>> pet = 'parrot'

>>> pet in ('bat', 'cat', 'dog')
False
>>> pet not in ('bat', 'cat', 'dog')
True

Video demo:

See docs.python: Membership test operations for documentation. See also my 100 Page Python Intro ebook.

Debug woes 3: matching uppercase letters

learnbyexample — Fri, 13 May 2022 03:00:00 GMT

So, I was going through GNU bash manual: Shell Parameter Expansion and trying out examples to check if I was understanding the features well.

When it came to case conversion, it was a bit confusing to know that you can only use a single character length glob. Here are the examples for lowercase to uppercase conversion that I used:

$ fruit='apple'

# all characters to uppercase
$ echo "${fruit^^}"
APPLE

# convert any character that matches [g-z] to uppercase
$ echo "${fruit^^[g-z]}"
aPPLe

# this won't work since 'sky-' is not a single character
$ c='sky-rose'
$ echo "${c^^*-}"
sky-rose

To convert uppercase to lowercase, you just need to use , instead of ^. Sounds simple right? It really is. But, I got stuck while trying to modify the above examples:

$ fruit='APPLE'

# worked as expected
$ echo "${fruit,,}"
apple

# expected 'ApplE' but got 'APPLE'
# can you spot the mistake?
$ echo "${fruit,,[g-z]}"
APPLE

I usually go through documentation and stackexchange sites when I'm stuck. After going through some threads, I came across this unix.stackexchange example:

$ str="HELLO"
$ printf '%s\n' "${str,,[HEO]}"
heLLo

Okay, I thought, this seems similar to what I wanted. Need to check out if this works on my machine. Before I even finished typing the example, my brain's light bulb turned on. I should have used G-Z instead of lowercase range.

$ echo "${fruit,,[G-Z]}"
ApplE

CLI tip 10: version sort

learnbyexample — Fri, 13 May 2022 03:00:00 GMT

You can use sort -V for sorting numerical input that is mixed with other characters. It also helps when you want to treat digits after a decimal point as whole numbers, for example if 1.10 should be greater than 1.2.

$ printf '1.5\n1.10\n1.2' | sort -n
1.10
1.2
1.5
$ printf '1.5\n1.10\n1.2' | sort -V
1.2
1.5
1.10

$ cat versions.txt
file2
cmd5.2
file10
cmd1.6
file5
cmd5.10
$ sort -V versions.txt
cmd1.6
cmd5.2
cmd5.10
file2
file5
file10

Here's an example of dealing with numbers reported by the time command (assuming all the entries have the same format).

$ cat timings.txt
5m35.363s
3m20.058s
4m11.130s
3m42.833s
4m3.083s

$ sort -V timings.txt
3m20.058s
3m42.833s
4m3.083s
4m11.130s
5m35.363s

See GNU coreutils manual: Version sort ordering for more details. Also, note that the ls command uses lowercase -v for this task.

Video demo:

See sort command chapter from my Command line text processing with GNU Coreutils ebook for more details.

Regexp gotcha 1: grouping common portions

learnbyexample — Fri, 13 May 2022 03:00:00 GMT

Similar to a(b+c)d = abd+acd in maths, you get a(b|c)d = abd|acd in regular expressions. However, you'll have to be careful if quantifiers are involved.

For example, (a*|b*) isn't the same as (a|b)*. Can you reason out why? Here's a railroad diagram to help you out:

Credit: debuggex.com

The difference is that (a*|b*) only matches same letter sequences like a, bb, aaaaaa, etc. But (a|b)* can match mixed sequences like ababbba too. You can also simplify (a|b)* to [ab]* since it is just single character alternation in this particular example.

Here's an illustration using Python:

>>> import re

>>> test = ['aa', 'abbaba', 'aaabbb', 'bbbbb', 'abc']

>>> [s for s in test if re.fullmatch(r'(a*|b*)', s)]
['aa', 'bbbbb']

>>> [s for s in test if re.fullmatch(r'(a|b)*', s)]
['aa', 'abbaba', 'aaabbb', 'bbbbb']

Want to learn regular expressions from the basics with plenty of examples and exercises? I've written regexp ebooks for Python, JavaScript, Ruby and CLI tools.

SQLite in Go, with and without cgo

Notes on software development — Thu, 12 May 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Building a toy Python @dataclass decorator

Bytepawn - Marton Trencseni — Thu, 12 May 2022 01:00:00 GMT

I write a toy implementation of Python's @dataclass decorator to improve my Python fu and learn more about decorators and metaprogramming.

Finished Shadowbringers

Ben Overmyer's Site — Wed, 11 May 2022 03:00:00 GMT

A few days ago I finished the content for Shadowbringers (the Final Fantasy XIV expansion). Now I'm working on the content that's between that and Endwalker.

Well, that's the vague plan, but I'm also doing other types of content completely unrelated to the main story quest. Like fishing. I enjoy fishing in FFXIV, just like fishing was my primary activity in Animal Crossing when I still played that. I'm not sure why. I don't really like fishing in the real world - it squicks me out.

Auto Pet Battle released a new expansion yesterday, so I'm playing that too. I'm starting to think I should try my hand at creating a single-player video game in the vein of Auto Pet Battle and Monster Train, where order and combos matter.

Python tip 10: removeprefix and removesuffix string methods

learnbyexample — Wed, 11 May 2022 03:00:00 GMT

Python supports plenty of string methods that reduces the need for regular expressions. The removeprefix() and removesuffix() string methods were added in the Python 3.9 version. See PEP 616 for more details.

These methods help to delete an exact substring from the start and end of the input string respectively. Here are some examples:

# remove 'sp' if it matches at the start of the input string
>>> 'spare'.removeprefix('sp')
'are'
# 'par' is present in the input, but not at the start
>>> 'spare'.removeprefix('par')
'spare'

# remove 'me' if it matches at the end of the input string
# only one occurrence of the match will be removed
>>> 'this meme'.removesuffix('me')
'this me'
# characters have to be matched exactly in the same order
>>> 'this meme'.removesuffix('em')
'this meme'

These remove methods will delete the given substring only once from the start or end of the string. On the other hand, the strip methods treat the argument as a set of characters to be matched any number of times in any order until a non-matching character is found. Here are some examples:

>>> 'these memes'.removesuffix('esm')
'these memes'
>>> 'these memes'.rstrip('esm')
'these '

>>> 'effective'.removeprefix('ef')
'fective'
>>> 'effective'.lstrip('ef')
'ctive'

Video demo:

See also my 100 Page Python Intro ebook.

Seeing is understanding

bayindirh — Tue, 10 May 2022 03:00:00 GMT

This post builds upon "I decided to trust cloud storage". If you didn't read it, please spare a few minutes for going through it for a better context.

As I started to move my files to cloud, a lot has happened. I've decided on a single provider for my files, bought extended file revision membership on said provider, and more importantly, started to move my files over there. The backup plan is not completely realized yet, but progressing slowly. I know I'm late to the cloud party, but being able to access your files anywhere is life changing and nice.

However, it is not all rainbows and unicorns.

I can confidently say that extended file revision membership paid itself off. I use a couple of tools to interface with my cloud storage account. One of them is capable of "out of tree synchronization", which is a fancy name for synchronizing a folder on the cloud anywhere on your computer. This allows syncing large folders to secondary drives and accessing them without filing the primary drive. Due to a complex case of application miscommunication and user error, I deleted a folder containing over 40,000 files. However, the files were restorable and nothing is lost. Having a year for recovering files is assuring if something goes unnoticed. Lastly, the storage provider sends an email in the lines of "You deleted a lot of files. If you didn't do that intentionally, here's a list. You have a year to restore them", which is neat.

Another revealing thing about cloud storage is the visibility it provides for your files. As a person who has a similar history to a nice internet stranger who documented his 25+ year journey; I have a lot files, on a lot of drives, organized in a plethora of ways, over a very long time. When combined with the density of modern era storage mediums, a lot files can be stored in a deceptively small volume of space, making them real-world bags of holding.

Previously, I have started to consolidate these drives to reduce the drive count, however, as it turned out, copying files and drinking tea while waiting for tasks to finish is a poor way for gauging the depth of the task at hand. Digging into these drives, moving the files to the cloud and organizing them was revealing.

Much more revealing than I ever imagined, to be honest.

In the era of multimedia, large files are the norm. A video recorded with a nice camcorder or a mirrorless camera can use gigabytes of space without much effort, and working with these files distorts your perception after a while. In reality, a great deal of information can be stored in small files. PDFs, text files, documents, and similar files can pack a lot of information into a minuscule space. Tucking these into a folder make them disappear. When opened for organization, the collection greets with all of its might.

Then, I understood the height of the mountain I need to scale.

As I move to the files to the cloud or put them aside to consider later, I started to grasp the size of accumulation happened over time, how I tucked things away with anticipation of returning back to them to handle them properly, but conned by the technology and by myself. Growing drives allowed hiding more into a smaller space, faster transfer speeds and more convenient interfaces allowed following the path of least resistance without thinking at all.

However, when the day has come to organize all this accumulation and separate what matters from the cruft, all the debt has to be paid in full. It's possible to use the features which enabled all this accumulation for an advantage too. Advanced tools can find duplicates faster, bigger drives can give a bigger scratch space for organization, and faster interfaces & network can allow quicker transfer across drives or the planet.

Now, the cloud storage has more folders and files than it ever has, but all the files are useful, meaningful and are the things I want to access. Seeing them reminds their existence, slows the accumulation and allows much greater reuse. The journey is in its opening legs, but it'll be much more fruitful than I ever anticipated. I can see that.

Lastly, a 64 GB USB flash drive can carry a world, and then some. Keep that in mind.

Python decorator patterns

Bytepawn - Marton Trencseni — Sun, 08 May 2022 01:00:00 GMT

I show toy implementations of Python decorator patterns such as @measure, @repeat, @trace, @count, @singleton, and @app.route (made famous by Flask).

Talk Notes: Third Age of JavaScript - Three Years In

swyx's site RSS Feed — Sat, 07 May 2022 20:25:45 GMT

Slides and show notes for my updated Talk at Reactathon

I don't know how to build software

Weblog on marginalia.nu — Fri, 06 May 2022 03:00:00 GMT

There are a lot of ways of building software, there are many languages you could choose to build it with, many libraries to rely on, many frameworks to leverage, many architectural approaches, many platforms to choose, many paradigms of daily operations to follow.

It takes years to get in-depth experience with just one permutation of these options.

I’ve been programming for over twenty years, only half the time professionally, but that is how long I’ve been building software. I’ve built about twelve applications in my twenty years of development, of varying size and complexity.

On minimalism and frugality

bayindirh — Thu, 05 May 2022 03:00:00 GMT

For more background information about this post, and to see where I'm coming from, see previous post on this subject, Meditations on minimalism.

As I ponder about minimalism, its advantages and what it brings to one's life; other promises start to show themselves. One of the biggest secondary promises is saving money.

Saving money is a desirable action all in itself. It enables one to have a bigger buffer for darker days, allows greater freedom to try more things, or provide better opportunities for your youngsters while they're growing up. However, does being minimal automatically enable these things? I'm not sure.

There is another mindset which is focused on saving money: Frugality. Its primary concern is saving money, and reducing influx of goods or expenditure of money is both the method and the target itself. This is where it contrasts with minimalism, which is concerned with reduction of goods, but not the expenditure itself directly. While frugality has a secondary promise of less belongings to deal with, this promise is a subject for another day.

During my quest for reducing physical goods and minimizing the number of belongings, a pattern has emerged. I was replacing these physical goods with their digital equivalents. Books are replaced with ebooks, bulky storage devices are consolidated and pushed to cloud with backup plans. This migration also required a couple of new software purchases and service subscriptions. As a result, this process is costing me both time and money.

While the cost looks grim at first, there's an important catch. I'm saving future time by going through this migration and transformation. Reducing the number of computers, storage systems and services I use has a net positive effect on my free time and efficiency, even visible today. Normally, writing this blog wouldn't be possible if I was continuing my old ways of doing things.

Today, I can say that frugality is not a given secondary effect of minimalism, at least in the initial stages of transformation. However, this subject needs further evaluation as the transformation progresses, things settle and new ways become the new norms. At that point, a logical next step may be another service level consolidation to eliminate unnecessary subscriptions. But, it's important to ensure that the elimination won't destabilize the established ways and shouldn't create a slippery slope towards extreme frugality, which affects one's quality of life for the sake of saving money.

Building a toy Python Enum class - Part II

Bytepawn - Marton Trencseni — Thu, 05 May 2022 01:00:00 GMT

I extend my previous toy implementation of Python's Enum class to add more features.

Vim tip 8: join lines

learnbyexample — Wed, 04 May 2022 03:00:00 GMT

In Normal mode, you can join lines using J and gJ commands. These differ in how the end-of-line character and indentation at the start of lines being joined are handled.

J joins the current line and the next line
- the deleted <EOL> character is replaced with a space (unless there are trailing spaces or the next line starts with a ) character)
- indentation from the lines being joined are removed, except the current line
3J joins the current line and next two lines with one space in between the lines
gJ joins the current line and the next line
- <EOL> character is deleted (space character won't be added)
- indentation won't be removed

joinspaces, cpoptions and formatoptions settings will affect the behavior of these commands. See :h J and scroll down for more details.

Video demo:

You (Probably) Shouldn't use a Lookup Table

Speculative Branches — Wed, 04 May 2022 03:00:00 GMT

I have been working on another post recently, also related to division, but I wanted to address a comment I got from several people on the previous division article. This comment invariably follows a lot of articles on using math to do things with chars and shorts. It is: "why are you doing all of this when you can just use a lookup table?"

Even worse, a stubborn and clever commenter may show you a benchmark where your carefully-crafted algorithm performs worse than their hamfisted lookup table. Surely you have made a mistake and you should just use a lookup table. Just look at the benchmark!

How the Democrats kicked out actual liberals

--Mike-- — Tue, 03 May 2022 03:29:51 GMT

Krystal Ball lays out exactly how the Clintons sold out the liberal agenda.

CLI tip 1: remove metadata from images

learnbyexample — Tue, 03 May 2022 03:00:00 GMT

Want to remove metadata (DateTime, Model, Orientation, ShutterSpeedValue, etc) from your images? You can use mogrify or convert tools provided by ImageMagick.

GUI image viewer applications will usually allow you to see some of the image metadata. You can also use the identify command line tool to get all the metadata:

# remove 'head' to get the entire list of metadata
$ identify -verbose insect.jpg | grep 'exif' | head
    exif:ApertureValue: 113/32
    exif:ColorSpace: 1
    exif:ComponentsConfiguration: 1, 2, 3, 0
    exif:CompressedBitsPerPixel: 3/1
    exif:CustomRendered: 0
    exif:DateTime: 2016:12:03 11:26:10
    exif:DateTimeDigitized: 2016:12:03 11:26:10
    exif:DateTimeOriginal: 2016:12:03 11:26:10
    exif:DigitalZoomRatio: 4000/4000
    exif:ExifOffset: 240

And here's how you can remove such metadata from images:

# to create a new image with metadata removed
$ convert -strip insect.jpg op.jpg

# to modify the input image itself
$ mogrify -strip insect.jpg

You can also pass multiple images to mogrify:

$ mogrify -strip *.jpg

Note that the image size after metadata removal may vary because of recompression.

Video demo:

Further Reading:

ImageMagick — create, edit, compose, or convert digital images
How can I read and remove meta (exif) data from my photos using the command line?
How to strip metadata from image files
What does mogrify mean?
wikipedia: Exif

Building a toy Python Enum class - Part I

Bytepawn - Marton Trencseni — Tue, 03 May 2022 01:00:00 GMT

I write a toy implementation of Python's Enum class to learn about Python metaclasses.

Thoughts on writing

--Mike-- — Tue, 03 May 2022 00:38:01 GMT

Nobody writes well written prose.

Everyone writes, then does a better or worse job of editing. Like code, prose tends to crystalize into smaller and higher quality pieces over time.

--- Process ---

Write all of your thoughts in a stream of consciousness flow, don't worry about how it looks, just get your thoughts out of your head, and into the storage medium of your choice. This frees up stack space in your brain.

Next - Iterate. Strategies you can use to help include:

Read it out loud to yourself. You'll immediately notice typos, grammatical and flow issues.
Walk away from it to gain some distance in time and space. When you come back you'll notice gaps or repetitions.

Repeat until you're happy.

QRP Labs QDX Rev 3 is here… all 2000 of them!

MiscDotGeek — Sun, 01 May 2022 11:28:23 GMT

Yep, get your wallet ready: The QDX Rev 3 is here. And there’s more. There are shirts, a hat, a mug, and more. Check it out in the video below:

The post QRP Labs QDX Rev 3 is here… all 2000 of them! appeared first on MiscDotGeek.

No Man's Sky

Ben Overmyer's Site — Sat, 30 Apr 2022 03:00:00 GMT

I've been playing a lot of No Man's Sky lately. It tends to go in waves - I'll play a lot of it for a short period of time, then I'll stop playing for months or even years. Then the cycle repeats.

This time, I'm doing a lot more combat than I used to and not because I seek it out. The recent pirate expansion has increased the number and strength of pirate ships attacking me in space. The addition of a hotkey to automatically follow an enemy has made this a lot less annoying than before - combat is really easy in No Man's Sky, so having a feature to make it take less time is very welcome. Also, I had to do a quest chain that culminated in me fighting a very large army of Sentinels at one of my bases. That was harder, and I almost died a couple times trying to kill the swarm while also whittling away at very large mecha. That this was essentially a required quest in order to unlock certain (non-combat) features of my base was irritating, but eh, it was a one-time deal.

With the help of the Internet, I discovered a couple tricks to getting resources. The first is that you can multiply your carbon pretty easily and make a ton of money off of it. You'll need a medium or large refiner, and a lot of oxygen. Then you just refine the carbon with the oxygen, and refine the result with the oxygen too, and just keep doing that until you have a ridiculous amount of carbon products. A quick way to get oxygen is to just buy it off of pilots in space station landing bays. The other trick is getting large quantities of other resources by equipping your ship with a positron ejector and then blasting the bejeesus out of the terrain at slow speed and low altitude. You'll get thousands of units of ferrite dust and other things this way.

I discovered a perfect paradise planet. No inclement weather, no Sentinels, no hostile wildlife, a beautiful blue sky with two suns and a ringed planet off in the distance, and a generally lovely aesthetic. I named it Asgard, even if that's rather tired at this point. I'm working on building my largest base there now, though my primary base is still on a bit of a hellworld several star systems away.

We'll see how long No Man's Sky keeps my interest. I'm still playing Final Fantasy XIV, and Galactic Civilizations IV launched a couple days ago, so that also has my attention.

"Over 70% of all Porsche vehicles ever built are still on the road today"

Bytepawn - Marton Trencseni — Sat, 30 Apr 2022 01:00:00 GMT

Porsche proudly advertises "Over 70% of all Porsche vehicles ever built are still on the road today". Is this a testament of the quality and longevity of Porsche cars, or simply a result of the brand switching from niche sportscar manufacturing to mass production around the year 1999?

Python tip 1: tuple argument for startswith/endswith methods

learnbyexample — Fri, 29 Apr 2022 03:00:00 GMT

You'd probably know about the startswith() and endswith() string methods.

>>> sentence = 'This is a sample string'

>>> sentence.startswith('This')
True
>>> sentence.startswith('is')
False

>>> sentence.endswith('ing')
True
>>> sentence.endswith('ly')
False

But did you know that you can also pass a tuple of strings?

>>> words = ['refuse', 'impossible', 'fire', 'present', 'read', 'shim']
>>> prefix = ('im', 're', 'use')

>>> [w for w in words if w.startswith(prefix)]
['refuse', 'impossible', 'read']

>>> [w for w in words if w.endswith(prefix)]
['refuse', 'fire', 'shim']

Video demo:

See also my 100 Page Python Intro ebook.

I decided to trust cloud storage

bayindirh — Fri, 29 Apr 2022 03:00:00 GMT

For more background information and to read where I'm coming from, see my previous post, meditations on minimalism.

I'm an old school person. For years, I always accomplished my things the old school way. Using a desktop system, preferring local-first file storage, organizing and curating these files by hand, sometimes painstakingly. This never meant that I denounce newer technologies, but I always put them to the back of the queue, or limited their use to the bare minimum. Over the years, this created a centralized file silo at home, with backups. I've built systems, and an ecosystem of applications to handle these files, their organization and other marginalia.

Life was good.

However, due to changing life circumstances, I have to change this storage system, for good. I won't have a place to put my silo for some unknown time, won't have the same time frame to organize it, and would need to access these files from more places to get things done.

Enter, cloud storage.

Cloud storage is nothing new, and I'm no stranger to it. To be honest, I'm using it for quite some time via Dropbox and Google Drive. While their support on the platform I prefer (Linux) is not first class, they're perfectly usable, and works reliably, at least. As a result, some of my files are living there, to enable mobility, collaboration and sharing. However, as I've said, the number of files living in these spaces were minimal.

The reasons for this were numerous. First, trusting a remote server you don’t control with your files was hard. Then, having almost no bandwidth for uploading files made the service not practical. Lastly, but most importantly, I always had access to my silo, and the time to work there everyday.

But, I missed the signs.

First, I was using the tech already. I had accounts with two leading providers, and was paying one of them a good amount of money for ample storage and other features. Some merchants I was using were delivering my files there directly, automatically organizing them during the process.

Second, the problems I mentioned was disappearing slowly. Mobile internet became affordable, and I was able to afford the top tier service. Home network was steadily becoming faster, incl. upload speeds. I was backing-up that space every day automatically, and never lost a file in any of these providers.

Lastly, my workflows were integrating to these services. Some devices I got had their own cloud service, and many apps were seamlessly integrating over it. I started collaborating over cloud. Moreover, I was using these services more and more to integrate different systems and platforms, and make them work together.

In short, I migrated to cloud without even knowing it, but still using my old ways hid that truth from myself.

As a result, in the light of the recent changes in my life, I decided to move to and trust cloud even more extensively and knowingly, but with more robust backup procedures.

At the same time, I realized that moving to a single place which can be accessed everywhere makes the data much more visible, easier to organize and pare down. This allows me to build systems around that storage system which automates other tasks, making my life much easier.

While the cloud is "someone else's computer", trusting that someone while being prepared brings many advantages, at first blush. I'll see how it feels as I integrate and use it more.

Let's see what moving to cloud going to bring next.

Uncertain Future For Marginalia Search

Weblog on marginalia.nu — Thu, 28 Apr 2022 03:00:00 GMT

I found myself effectively without a job on short notice.

I’m not at all worried about finding another one, I have savings, and I have experience, and I have demonstrable skill. What I am concerned about is finding a source of income that’s compatible with putting some time on my personal projects.

Last bunch of years, I’ve been working 32 hour weeks, which is a pretty sweet deal especially combined with the zero hour commute you get working from home during the pandemic. Not every employer is fine with that, and while I do have options, I’m in a worse bargaining position than I have been before.

CLI tip 9: awk paragraph mode

learnbyexample — Wed, 27 Apr 2022 03:00:00 GMT

awk provides a handy shortcut to process input content paragraph wise. When RS is set to empty string, one or more consecutive empty lines is used as the input record separator. Consider the below sample file:

$ cat para.txt
hi there
how are you

2 apples
12 bananas


blue sky
yellow sun
brown earth

Here are some simple examples to filter paragraphs based on some criteria:

# paragraphs containing 'sun'
$ awk -v RS= '/sun/' para.txt
blue sky
yellow sun
brown earth

# paragraphs containing any digit character
$ awk -v RS= '/[0-9]/' para.txt
2 apples
12 bananas

# print the first paragraph
$ awk -v RS= 'NR==1' para.txt
hi there
how are you

See Paragraph mode section from my GNU awk ebook for more examples and corner cases.

Video demo:

See my CLI text processing with GNU awk ebook if you are interested in learning about the GNU awk command in more detail.

HTML event handler attributes: down the rabbit hole

Notes on software development — Tue, 26 Apr 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

"The company is all hot air"

Bytepawn - Marton Trencseni — Tue, 26 Apr 2022 01:00:00 GMT

Theranos, WeWork and the startup hustle.

Testing and Running Go API GW Lambda's Locally

Ben E. C. Boyter — Mon, 25 Apr 2022 03:00:00 GMT

I have been working with AWS API Gateway and Lambda using Go a lot recently. One of the more annoying things about API Gateway and Lambda is the inability to run things locally. As a result I tend to write a lot of unit tests to compensate. This works up till you start doing things like SQL queries as mocking away the database is a less than ideal situation if you are working with raw SQL, which I tend to do since I don’t like most ORM’s. These limitations of unit tests is why I prefer integration tests.

Upgrading the Iron Arachne planet generator

Ben Overmyer's Site — Fri, 22 Apr 2022 03:00:00 GMT

Today I upgraded the graphics for the planet generator on Iron Arachne.

It was not a small task, as I taught myself how to do domain warping and rewrote most of the GLSL shaders that are used to create these graphics.

But first, the only one that did NOT get changed, because I like it as is:

I might end up tweaking the gas giant a bit, but not today.

The arid planet is a little more colorful, but still looks bleak. The new cloud shader is a huge improvement.

It's less obvious in this particular shot, but the barren planet gained a lot more detail in its surface texture. I still don't care for how I did the vertex displacement, but eh, that can wait for another day.

The garden world was the hardest. I'm still not happy with it, but it's better than it was.

The jungle world looks much better. It definitely has more of a forested feel.

The ice world is much more colorful than it used to be. Originally I was going to go for a color scheme similar to the original, but I liked this version too much and kept it.

The ocean world now feels a lot less plastic and has more detail to it. The new clouds help.

The new swamp world looks just as unfriendly as the original, but perhaps more colorful now. I might tone down the brighter hues in the future.

The toxic world is less cartoony now. It no longer has the neon green oceans and generally looks like an awful place to visit.

The new volcanic world is much more volcano-y. Instead of just oceans of lava, there are now rivers too, and the glow is less pronounced.

All in all, I think it's a pretty impressive upgrade, though there's still a lot of work to be done. At some point I really want to figure out atmospheric scattering.

Meditations on minimalism

bayindirh — Fri, 22 Apr 2022 03:00:00 GMT

I've been pondering about minimalism, and trying to implement it into my life for quite some time. However, the idea proved itself to be elusive due to various reasons. While this elusiveness can be partly attributed to the nature of the beast, other parts can be attributed to me.

The idea minimalism puts forward is an enticing one, which can be simplified as "having no excess, but what you need", which brings many promises and possible outcomes with it. Out of these promises, there are two which attract me most:

Reduced mental load.
Having more time.

Both of these promises stem from the argument that, if you have less possessions, you'll need less energy and time to manage them, and that's correct.

However, while the idea is neat, it involves humans.

I'm a curious, sometimes overly-curious, human being, and lack acquisition filters, partly because even if I don't have time, I have ample storage space to store and organization systems to remember what I want to come back to later. Material, or digital, it doesn't matter. As a result, while I'm a very organized person, I'm a crowded one at the same time.

If I have the space, and the occasional time, what's the problem? Considering digital items don't take any physical space?

The problems are numerous, and they're affecting much more than it looks.

The first problem is the background pressure created by all the items in the back of my head. I already have a lengthy to do list which continues to grow steadily, and I'm not sure I can complete them all in a lifetime.

The second problem is the crowdedness it creates in every living space. Physical and digital. Physical storage space becomes sparse as new gadgets and tools gets acquired, and every drive you open is full of folders. Putting them inside boxes or folders and labeling them "Organize this" doesn't help in any way. It's the same thing with pushing traumas to subconscious mind. They'll take up space in your closet, in your drive and in your mind, with the bonus of brewing at the background like a beer vat without sufficient pressure relief. Since the things you have put in these boxes are now harder to reach, it's possible that you'll re-acquire the same things again, which will further increase the space you use, with no added benefit.

This visual and mental overload will take your energy, time and sanity away, because you'll spend more and more time to maintain something wobbly, trying to keep it upright (people who played World of Goo will understand).

Remedying this requires fundamental shifts in one's mentality of storing, organizing and keeping things.

First off, acquisition filters have the utmost importance, because if you can't stop the influx, reducing what you have will be sustaining the status quo at best. On the other hand, having no influx is not realistic, so preventing the unnecessary from entering is the best way.

Secondly, you need to reduce what you have, right? This is one of the hardest parts of the whole process, because as humans, we tend to bond to things. Giving possessions away or deleting files feels like giving or throwing away parts of ourselves most of the time. As a result, this creates a lot of pain.

However, all is not lost. For myself, I've found ways that work better than others. The simplest term summarizing it is "integration".

Integration means opening the folders and boxes labeled "Organize these", and putting everything where it should be. If you feel that the file or the item shouldn't be there, it means you won't need it most probably. Put it somewhere visible, or the place where it should be nevertheless, and give it away or delete when you feel for it. It's not carefree, but makes the process easier.

For material possessions which are perfectly functioning, throwing away is a huge betrayal for the item, the environment and the society in my view. I prefer donating them. Giving it away to someone I won't be seeing again. When I don't see the item in question, I forget the item after some time.

I'm going slowly, but making progress. As my life changes, space for my belongings is shrinking. Both material and digital. This change is urging me gently, but I'm not panicking.

I'll document my journey and progress here, but I can't promise any regularity.

Python types for Data Scientists - Part III

Bytepawn - Marton Trencseni — Fri, 22 Apr 2022 01:00:00 GMT

I show slightly more advanced aspects of type checking in Python for Data Scientists.

Interview on Will Larson's Infrastructure Engineering

swyx's site RSS Feed — Wed, 20 Apr 2022 10:36:02 GMT

a reprint of my interview on Will Larson's InfraEng project.

React Miami: Temporal - React for the Backend

swyx's site RSS Feed — Tue, 19 Apr 2022 11:59:59 GMT

These are the raw notes of my talk prep for my React Miami 2022 talk - Temporal - React for the Backend. Includes links and initial draft at the bottom.

Interview With Phil of DataStation

Notes on software development — Sun, 17 Apr 2022 03:00:00 GMT

This is an external interview. Click here if you are not redirected.

Python types for Data Scientists - Part II

Bytepawn - Marton Trencseni — Sun, 17 Apr 2022 01:00:00 GMT

I show slightly more advanced uses of type checking in Python.

Python tip 9: applying set-like operations for dictionaries

learnbyexample — Sat, 16 Apr 2022 03:00:00 GMT

You can merge two dictionaries using the | operator (similar to union of sets). If a key is found in both the dictionaries, the insertion order of the first dictionary will be maintained, but the value of the second dictionary will be used. In other words, keys are updated to the new value during the merge.

>>> marks_1 = {'Rahul': 86, 'Ravi': 92, 'Rohit': 75}
>>> marks_2 = {'Jo': 89, 'Rohit': 78, 'Joe': 75, 'Ravi': 100}

# use unpacking, i.e. {**d1, **d2} for Python 3.8 and below versions
>>> marks_1 | marks_2
{'Rahul': 86, 'Ravi': 100, 'Rohit': 78, 'Jo': 89, 'Joe': 75}

Use update() method if you want to modify instead of getting a new dictionary.

>>> marks_1.update(marks_2)
>>> marks_1
{'Rahul': 86, 'Ravi': 100, 'Rohit': 78, 'Jo': 89, 'Joe': 75}

The keys() and values() dictionary methods return set-like objects, but with insertion order maintained. You get a set object as output when you apply set operators on these objects.

>>> marks_1 = {'Rahul': 86, 'Ravi': 92, 'Rohit': 75}
>>> marks_2 = {'Jo': 89, 'Rohit': 78, 'Joe': 75, 'Ravi': 100}

# union of keys
>>> marks_1.keys() | marks_2.keys()
{'Rohit', 'Rahul', 'Ravi', 'Jo', 'Joe'}

# common keys
>>> marks_1.keys() & marks_2.keys()
{'Ravi', 'Rohit'}

# difference: keys not present in the other dict
>>> marks_1.keys() - marks_2.keys()
{'Rahul'}
>>> marks_2.keys() - marks_1.keys()
{'Jo', 'Joe'}

# symmetric difference: union of above two differences
>>> marks_1.keys() ^ marks_2.keys()
{'Jo', 'Joe', 'Rahul'}

Video demo:

See also my 100 Page Python Intro ebook.

Vim tip 7: changing case in Normal mode

learnbyexample — Tue, 12 Apr 2022 03:00:00 GMT

You can use the following commands to change the case of characters:

~ invert the case of the character under the cursor (i.e. lowercase becomes UPPERCASE and vice versa)
g~ followed by motion inverts the case of those characters
- for example: g~e, g~$, g~iw, etc
gu followed by motion changes those characters to lowercase
- for example: gue, gu$, guiw, etc
gU followed by motion changes those characters to UPPERCASE
- for example: gUe, gU$, gUiw, etc

You can also provide a count prefix to these commands. For example, 3~ will invert the case of the current character and two characters to the right.

Video demo:

Being powerless as a customer

bayindirh — Tue, 12 Apr 2022 03:00:00 GMT

Web ecosystem has changed a lot in the recent years, esp. in the area of e-commerce, with the rise of cloud computing and subscription economy.

With the evolving technology, many parts of an e-commerce system, or commerce parts of any website became increasingly commoditized. You need a payment processor? Paddle, Stripe, or any other alternative is there for you. Need to send files? SendOwl or just Amazon S3 with private, long links will do. Download limits? Add some transparent rules to these S3 links, or use a more sophisticated solution.

These solutions are simple, yet incomplete and is not conductive to many things customers get accustomed to. Let's see some examples:

Need your files again for any reason? You have to dig your emails to find the links. Unsurprisingly, at least one of them would be expired. So, you need to contact customer support, which may or may not provide the links for the files you've paid for. How about a dashboard to get a list of files and transactions for managing them yourself? Sorry. that's not provided by our platform.
A mystery charge on your card? No worries, a receipt will arrive. It didn't? So, contact customer service, which may or may not give you answers about that charge. If you want to monitor them via a dashboard, you won't have any, because you should have received a receipt with relevant information and links in the first place, but you didn't.

These examples are real experiences I have. All in all, if somehow the perfect workflow breaks for whatever reason, you have no power to reach anything, anyone or understand what's happening, and that's really crippling you as a customer.

These problems are compounded when these events collide with a bigger company which is mostly enterprise oriented, but you're tech savvy enough to use their personal plans. You get lower tier support while paying for expensive subscriptions and get canned responses to your support requests. It's really making life harder.

So, yes. The shiny product facades of integration companies have a dark side too. The seller gets a simple interface and a distributed tech stack which allows him/her to sleep better at night, but the customers served by that stack are in a worse state, and can't access to what they've bought and can't see why they paid this money again to the company.

At that point the seller needs to help the customer, but it's not a given, and being left in the cold is a very bad feeling for a customer.

If this trend continues to get adopted, looks like it'd be much more harder to cancel subscriptions, get our files back or just get support, and this means many unhappy customers, which will hopefully lead a change in the industry for the better.

However, I'm not holding my breath, yet.

2022–04–12: Pinephone keyboard keymaps

megi's PinePhone Development Log — Tue, 12 Apr 2022 01:00:00 GMT

Lexicon Architectural Rubberducking

Weblog on marginalia.nu — Mon, 11 Apr 2022 03:00:00 GMT

I’m going to think out loud for a moment about a problem I’m considering.

RAM is a precious resource on any server. Look at VPS servers, and you’ll be hard pressed to find one with much more than 32 Gb. Look at leasing a dedicated server, and it’s the RAM that really drives up the price. My server has 128 Gb, and it it’s so full it needs to unbutton its pants to sit down comfortably. Anything I can offload to disk is great.

Surveying SQL parser libraries in a few high-level languages

Notes on software development — Mon, 11 Apr 2022 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Python types for Data Scientists - Part I

Bytepawn - Marton Trencseni — Fri, 08 Apr 2022 01:00:00 GMT

I show how to use basic type hints and get type checking working in ipython notebooks.

The Bargain Bin B-Tree

Weblog on marginalia.nu — Thu, 07 Apr 2022 03:00:00 GMT

I’ve been working lately on a bit of an overhaul of how the search engine does indexing. How it indexes its indices. “Index” is a bit of an overloaded term here, and it’s not the first that will crop up.

Let’s start from the beginning and build up and examine the problem of searching for a number in a list of numbers. You have a long list of numbers, let’s sort them because why not.

Shakedown cruise on the Baltic Sea

Henri Bergius — Thu, 07 Apr 2022 03:00:00 GMT

Just in time for a new cruising season to start, the story of our 2021 Baltic shakedown cruise is now online.

This was a 666NM trip that we did on our new-to-us Amigo 40 cruising boat in August-September 2021. Apart from engine trouble in the beginning, this was a very enjoyable little adventure on the coasts of Sweden and Bornholm.

The trip even earned us the first prize in the cruising log contest of our sailing club:

Read the story now.

CLI tip 8: extract from start of file until matching line

learnbyexample — Wed, 06 Apr 2022 03:00:00 GMT

The GNU sed command has a couple of handy commands to extract text from the start of input until a matching line is found. The q and Q commands are similar, except how they process the matching line.

The q command will exit sed immediately, after printing the current pattern space if applicable.

# quit after a line containing 'st' is found
$ printf 'apple\nsea\neast\ndust' | sed '/st/q'
apple
sea
east

The Q command is similar to q but won't print the matching line.

# matching line won't be printed in this case
$ printf 'apple\nsea\neast\ndust' | sed '/st/Q'
apple
sea

tac+sed+tac will help you get lines starting from the last occurrence of the search string till the end of the input.

$ printf 'apple\nsea\neast\ndust\n' | tac | sed '/ea/q' | tac
east
dust

Be careful if you want to use q or Q commands with multiple files, as sed will stop even if there are other files left to be processed. You can use mixed address ranges as a workaround. See also unix.stackexchange: applying q to multiple files.

Video demo:

See my CLI text processing with GNU sed ebook if you are interested in learning about the GNU sed command in more detail.

Is There A Better Hard Drive Metaphor?

Weblog on marginalia.nu — Sun, 03 Apr 2022 03:00:00 GMT

This is mostly a post to complain about something that chafes. I wish there was a programming language (ideally several) that acknowledged that computers have hard drives, not just a processor, RAM and other_devices[].

Something that has struck me when I’ve been working with the search engine is how unfinished the metaphor for accessing physical disks is in most programming languages. It feels like an after-thought, half left to the operating system to figure out, a byzantine relic of the days when computers had tape drives and not SSDs.

A Conversational Bill of Rights

jodavaho.io — Sat, 02 Apr 2022 03:00:00 GMT

Feel free to issue a pull request or file issues at github.com/jodavaho/conversational_rights

Preamble

The ability of humans to exchange information quickly and efficiently is our super power. Our other limited abilities and general fragility could not have made us the dominant species of the entire planet, and enabled us to do the things we do. More specifically, the written and verbal expression of culture has enabled the alignment of the individuals of our species toward long term goals of incredible complexity. So pervasive is our reliance on verbal and written communication that I believe it deserves to be considered a fundamental right. Moreover, it is imperative to establish norms in the kinds of exchanges and how they evolve, and what ideas are allowed or what cross examinations are considered polite and welcome, if only to increase the quality of discourse for exchangers and observers alike.

Rights in the exchange of information and culture

A clear understanding of the issue and main points is critical to make thoughtful and useful contributions to the issue at hand. Thus, in any conversation, be it written or spoken, You have the right to remain silent. You have the right to withhold judgement and contributions, and to have your act of remaining silent remain free of any perceived agenda – not condoning or condemning any part of any point of any argument in any conversation or exchange of information. The silent party has no obligation to state why they are remaining silent or withholding contributions.
You have the right to have your contributions interpreted in the most favorable light. It is not the responsibility of the contributor to specify the full context of any contribution. It is the responsibility of the reader / listener to request clarification whenever ambiguity presents various interpretations which are both favorable and unfavorable, either in the receivers’ own mind(s) or on behalf of possible misunderstandings by other observers.
Learning and maturation is inevitable and good in any person or organization. Thus, you have the right to express a change in past contributions, and to make updated contributions that overrule your past contributions, whenever those past contributions are referenced as part of this change.
Even if your contribution’s thesis has not changed, you have the right to clarify past contributions, and to have the most recent clarification override the past contributions – to the extent that they do not overtly contradict those past contributions.
You have the right to request clarification on any point made by any contributor. The requestee is not bound by any special constraints, and retains all the rights herein. Furthermore, you have the right to request supporting information, context, or background information of any opinion or statement. Such requests shall not in themselves be viewed as a malicious act.
You have the right to pronounce judgement and opinion for or against any contribution or argument by any party as part of your contribution. No party is under any obligation to acknowledge or respond to any pronunciation thereof.
In order to allow context and clear train of thought by a contributor, you have the right to request uninterrupted time or space to present an argument, and to withhold any contributions or information unless that time is given. Similarly, you have the right to request a limit to the uninterrupted time or space given to any party, and are under no obligation to consider discourse that continues outside of this time or space, should they agree to those limits.
You have the right to specify the audience for your arguments. That is, you have the right to make any statement or express an opinion without the intent to convince the listener, for example to make it part of the conversation record or be it part of posterity.
You have the right to request and /or specify a structure for the exchange of information. These “rules of order” shall be agreed to by all parties or not used, and shall apply to all parties equally, with the exception of an optional “moderator” who directs the conversation according to the rules. If used, the moderator shall act impartially and shall be precluded from contributing to the exchange. Specifically, you have the right to request a moderator in any exchange of information.
You have the right to request limited or expanded audience. You can, at any time, request that an exchange of information or dialog be conducted in the party of any observer(s) (or in private), and to withhold your contributions if this is not agreed to. The observer is under no obligation to keep their observations confidential unless agreed upon by all parties.

License

CC BY-SA 4.0

2022–04–02: Pinephone keyboard power manager

megi's PinePhone Development Log — Sat, 02 Apr 2022 01:00:00 GMT

April 2022 update on my RPG projects

Ben Overmyer's Site — Fri, 01 Apr 2022 03:00:00 GMT

It's been quite some time since I last wrote about the tabletop RPG projects I have in the works.

For the most part, there have been very few updates over the last year. However, I have been occasionally plinking away at the Ingenium Second Edition manuscript, and lately I've been planning on how to bring a few others closer to completion.

Updates on existing projects

Ingenium Second Edition has seen the bulk of the work. Species were renamed to bloodlines. An entire continent was removed from the setting. A lot of things were reworked for the 2d6 core mechanic. A lot of Talents and spells were added. Professions got organizations that you have a relationship with starting out. A languages index was added. All of the remaining Talents were planned out. Most were fleshed out. Wizard marks were made a core part of the magic system instead of a Talent.

Mysteries of a Broken World was... just converted to LaTeX. Nothing substantial changed here.

Vox Draconis: Kingdoms of Stone and Fire similarly had little done to it. A few cosmetic changes and typo fixes.

Plans for the future

I'm going to continue to bring Ingenium Second Edition closer to completion. It remains my primary focus. I plan on doing all of the work myself, including layout and illustration. For the immediate future I'm going to focus on writing all of the spells, miracles, spirit tricks, and Talents.

Mysteries of a Broken World is going to get the most dramatic change of the three. I'm going to rework it to be purely a solo RPG. The survival theme will get stronger, since there will no longer be other players to worry about.

Vox Draconis: Kingdoms of Stone and Fire will probably see the least work this year. It's a tight concept and really just needs fleshing out rather than any system development. From time to time I'll add to it, but Ingenium will take up more of my attention.

Lastly, I plan on formally adding Ingenium Second Edition to this website. The trademark and copyright are mine now instead of Silver Gryphon Games'. I'll make a bigger post about this later.

Who Controls a DAO?

Speculative Branches — Fri, 01 Apr 2022 03:00:00 GMT

In honor of April Fools' Day, I decided to write about a blockchain topic. The crypto economy is in the process of speedrunning their way from zero to a modern economy, and when you move that fast, a few things have to break along the way. One of those things is corporate governance.

Matt Levine's "Money Stuff" is a financial newsletter that I can't recommend enough. If you are at all interested in finance, stocks, and markets, it is funny and informative read. One of the recurring topics of Money Stuff is "who controls a company?" Quoting a bit of the newsletter:

Vim Reference Guide: two week sales report

learnbyexample — Thu, 31 Mar 2022 03:00:00 GMT

I've previously written about events and strategies that led to increased ebook sales during the last quarter of 2021.

Very pleased to inform that I continue to see more than expected sales. I had released my 12th ebook Vim Reference Guide on March 15th. Here's how the sales looked on Gumroad during the first two weeks:

I used to offer my ebooks for free on release. For the past couple of releases, I have also added heavily discounted ebook bundles which seems to be the major factor in increased paid sales I'm seeing. Luck certainly plays a role too, reaching front page of Hacker News and top of subreddits cannot be always counted upon. Here are some of the ways I promoted my latest ebook:

Announcement post on Gumroad and sending an email to existing readers
Show HN post on Hacker News, got lucky to be placed in top 10
Pinned tweet
Posting on /r/commandline/ and /r/linux/
- You might wonder why not /r/vim? Somebody else posted before I could, and unfortunately it got downvoted. I'll probably make my own post after I release the next version
Promo video on youtube
Mentioned in two of my learnbyexample weekly newsletter issues
And of course, I wrote a release post on this blog and also mentioned it on my GitHub Readme

Apart from Gumroad, 500+ readers downloaded the guide from Leanpub and I got a few paid sales as well. I wrote about pros and cons of Gumroad/Leanpub here.

PS: Make sure to read the rules and be a regular user before self-promoting your content on the social media platforms mentioned above.

2022–03–31: Keyboard light

megi's PinePhone Development Log — Thu, 31 Mar 2022 01:00:00 GMT

Python tip 8: dict.fromkeys() method

learnbyexample — Wed, 30 Mar 2022 03:00:00 GMT

A lesser known way to create a dictionary is to use the fromkeys() method that accepts an iterable and an optional value (default is None). The same value will be assigned to all the keys, so be careful if you want to use a mutable object.

>>> colors = ('red', 'blue', 'green')

>>> dict.fromkeys(colors)
{'red': None, 'blue': None, 'green': None}

>>> dict.fromkeys(colors, 255)
{'red': 255, 'blue': 255, 'green': 255}

When you iterate over a dictionary object, you'll get only the keys. For example:

>>> fruits = {'banana': 12, 'papaya': 5, 'mango': 10}

>>> for fruit in fruits:
...     print(fruit)
... 
banana
papaya
mango

Recent Python versions ensure that the insertion order is maintained for a dictionary. So, you can remove duplicate items from a list while maintaining the order by building a dictionary using the fromkeys() method and converting it back to a list.

>>> nums = [1, 4, 6, 22, 3, 5, 4, 3, 6, 2, 1, 51, 3, 1]

# remove duplicates, if you don't care about the element order
>>> list(set(nums))
[1, 2, 3, 4, 5, 6, 51, 22]

# remove duplicates, if you want to maintain the element order
>>> list(dict.fromkeys(nums))
[1, 4, 6, 22, 3, 5, 2, 51]

Video demo:

See also my 100 Page Python Intro ebook.

Writing a document database from scratch in Go: Lucene-like filters and indexes

Notes on software development — Mon, 28 Mar 2022 03:00:00 GMT

In this post we'll write a rudimentary document database from scratch in Go. In less than 500 lines of code we'll be able to support the following interactions, inspired by Elasticsearch:

$ curl -X POST -H 'Content-Type: application/json' -d '{"name": "Kevin", "age": "45"}' http://localhost:8080/docs
{"body":{"id":"5ac64e74-58f9-4ba4-909e-1d5bf4ddcaa1"},"status":"ok"}

$ curl --get http://localhost:8080/docs --data-urlencode 'q=name:"Kevin"' | jq
{
  "body": {
    "count": 1,
    "documents": [
      {
        "body": {
          "age": "45",
          "name": "Kevin"
        },
        "id": "5ac64e74-58f9-4ba4-909e-1d5bf4ddcaa1"
      }
    ]
  },
  "status": "ok"
}

$ curl --get http://localhost:8080/docs --data-urlencode 'q=age:<50' | jq
{
  "body": {
    "count": 1,
    "documents": [
      {
        "body": {
          "age": "45",
          "name": "Kevin"
        },
        "id": "5ac64e74-58f9-4ba4-909e-1d5bf4ddcaa1"
      }
    ]
  },
  "status": "ok"
}

The latter query, being a range query, will do a full table scan. But the first query, an exact match, will use an index and be much faster.

Document databases in general may be able to support indexes on ranges but our rudimentary one won't.

Furthermore, this post will not implement full text search.

All code for this project is available on Github. Let's get started.

Server basics

Run go mod init and set up main.go with Julien Schmidt's httprouter. We'll create three routes: one for inserting a document, one for retrieving a document by its id, and one for searching for documents.

package main

import (
    "encoding/json"
    "log"
    "net/http"

    "github.com/julienschmidt/httprouter"
)

type server struct {
    port string
}

func main() {
    s server{"8080"}
    router := httprouter.New()
    router.POST("/docs", s.addDocument)
    router.GET("/docs", s.searchDocuments)
    router.GET("/docs/:id", s.getDocument)

    log.Println("Listening on " + s.port)
    log.Fatal(http.ListenAndServe(":"+s.port, router))
}

Now add the routes:

func (s server) addDocument(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
    panic("Unimplemented")
}

func (s server) searchDocuments(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
    panic("Unimplemented")
}

func (s server) getDocument(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
    panic("Unimplemented")
}

That's good enough for now! Let's think about storage.

Storage

If you wanted to do this project fully from scratch you could handle storage by just writing JSON blobs to disk. Nothing in this project will be much more complex than just writing JSON to disk and the equivalent of using ls on the filesystem. I mention this because I said this project is "from scratch" but I'm going to bring in a storage engine. My point is that you could easily follow this post and just read/write directly to disk if you felt strongly.

Because there were so many folks misconstruing this paragraph, I've ported this blog post without Pebble as proof :D. You can find the diff here. Took me an hour for the +40/-40 diff that is still <500 lines of code. You may notice the code basically looks identical. That's because the storage engine isn't the interesting part. :)

Any storage engine would be fine: direct read/write, SQLite, PostgreSQL. But we're going to grab a key-value storage engine. I've used Badger before so I'm going to try out Cockroach Lab's Pebble this time instead.

Add "github.com/cockroachdb/pebble" to the list of imports. Then upgrade the server struct to store an instance of a Pebble database.

type server struct {
    db      *pebble.DB
    port    string
}

func newServer(database string, port string) (*server, error) {
    s := server{db: nil, port: port}
    var err error
    s.db, err = pebble.Open(database, &pebble.Options{})
    return &s, err
}

And upgrade main:

func main() {
    s, err := newServer("docdb.data", "8080")
    if err != nil {
        log.Fatal(err)
    }
    defer s.db.Close()

    router := httprouter.New()
    router.POST("/docs", s.addDocument)
    router.GET("/docs", s.searchDocuments)
    router.GET("/docs/:id", s.getDocument)

    log.Println("Listening on " + s.port)
    log.Fatal(http.ListenAndServe(":"+s.port, router))
}

In the future these server settings could be user-configurable. For now they're hard-coded.

Storing data

When the user sends a JSON document we need to give it a unique ID and store the ID and document in the database. Since we're using a key-value storage engine we'll just use the ID as the key and the JSON document as the value.

To generate the ID we'll use Google's UUID package. So make sure to import "github.com/google/uuid".

func (s server) addDocument(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    dec := json.NewDecoder(r.Body)
    var document map[string]any
    err := dec.Decode(&document)
    if err != nil {
        jsonResponse(w, nil, err)
        return
    }

    // New unique id for the document
    id := uuid.New().String()

    bs, err := json.Marshal(document)
    if err != nil {
        jsonResponse(w, nil, err)
        return
    }
    err = s.db.Set([]byte(id), bs, pebble.Sync)
    if err != nil {
        jsonResponse(w, nil, err)
        return
    }

    jsonResponse(w, map[string]any{
        "id": id,
    }, nil)
}

Nothing special: just accept a JSON POST body and store it in the database, return the generated document id.

I'm not sure that using UUIDs here is a good idea but it is easier than keeping track of the number of rows in the database.

The jsonResponse helper can be defined as:

func jsonResponse(w http.ResponseWriter, body map[string]any, err error) {
    data := map[string]any{
        "body":   body,
        "status": "ok",
    }

    if err == nil {
        w.WriteHeader(http.StatusOK)
    } else {
        data["status"] = "error"
        data["error"] = err.Error()
        w.WriteHeader(http.StatusBadRequest)
    }
    w.Header().Set("Content-Type", "application/json")

    enc := json.NewEncoder(w)
    err = enc.Encode(data)
    if err != nil {
        // TODO: set up panic handler?
        panic(err)
    }
}

It's a basic wrapper so that all responses are structured JSON.

Retrieving by ID

Before we try to test out inserts, let's get retrieval hooked up. Inserts return an ID in the HTTP reponse. GETs will grab a document by ID.

func (s server) getDocumentById(id []byte) (map[string]any, error) {
    valBytes, closer, err := s.db.Get(id)
    if err != nil {
        return nil, err
    }
    defer closer.Close()

    var document map[string]any
    err = json.Unmarshal(valBytes, &document)
    return document, err
}

func (s server) getDocument(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    id := ps.ByName("id")

    document, err := s.getDocumentById([]byte(id))
    if err != nil {
        jsonResponse(w, nil, err)
        return
    }

    jsonResponse(w, map[string]any{
        "document": document,
    }, nil)
}

We've now got enough in place to test out these basics!

$ go mod init docdb
$ go mod tidy
$ go build
$ ./docdb
2022/03/28 19:28:19 Listening on 8080

Now, in another terminal, insert a document:

$ curl -X POST -H 'Content-Type: application/json' -d '{"name": "Kevin", "age": "45"}' http://localhost:8080/docs
{"body":{"id":"c458a3ce-9faf-4431-a058-d9ae2a1651e1"},"status":"ok"}

$ curl http://localhost:8080/docs/c458a3ce-9faf-4431-a058-d9ae2a1651e1
{"body":{"document":{"age":"45","name":"Kevin"}},"status":"ok"}

Perfect! Now let's implement search.

A filter language

First off we need to pick a filter language. Using a JSON data structure would be fine. We could require the user POSTs against a search endpoint so that the POST body contains the JSON filter.

But Lucene is a pretty simple language and we can implement enough parts of it easily. The result is more fun.

In our simplification of Lucene there will only be key-value matches. Field names and field values can be quoted. They must be quoted if they contain spaces or colons, among other things. Key-value matches are separated by whitespace. They can only be AND-ed together and that is done implicitly.

The following are some valid filters in our implementation:

a:1
b:fifteen a:<3
a.b:12
title:"Which way?"
" a key 2":tenant
" flubber ":"blubber "

Nested paths are specified using JSON path syntax (i.e. a.b would retrieve 4 in {"a": {"b": 4, "d": 100}, "c": 8}).

Lexing strings

Both keys and values are lexed as strings. If they start with a quote, we keep on accumulating all characters until the ending quote. Otherwise we accumulate until we stop seeing a digit, letter, or period.

// Handles either quoted strings or unquoted strings of only contiguous digits and letters
func lexString(input []rune, index int) (string, int, error) {
    if index >= len(input) {
        return "", index, nil
    }
    if input[index] == '"' {
        index++
        foundEnd := false

        var s []rune
        // TODO: handle nested quotes
        for index < len(input) {
            if input[index] == '"' {
                foundEnd = true
                break
            }

            s = append(s, input[index])
            index++
        }

        if !foundEnd {
            return "", index, fmt.Errorf("Expected end of quoted string")
        }

        return string(s), index + 1, nil
    }

    // If unquoted, read as much contiguous digits/letters as there are
    var s []rune
    var c rune
    // TODO: someone needs to validate there's not ...
    for index < len(input) {
        c = input[index]
        if !(unicode.IsLetter(c) || unicode.IsDigit(c) || c == '.') {
            break
        }
        s = append(s, c)
        index++
    }

    if len(s) == 0 {
        return "", index, fmt.Errorf("No string found")
    }

    return string(s), index, nil
}

This is not something you get right without unit tests. I wrote unit tests for it while building this project. Always unit test tricky code where you're likely to have off-by-one errors! I had a bunch.

Query parser

Now we can write the query parser. It first lexes a string for the key. Then it looks for the operator which can be one of : (meaning equality), :> (meaning greater than), or :< (meaning less than). It accumulates each key-value pair into an overall list of AND-ed arguments that make up the query.

type queryComparison struct {
    key   []string
    value string
    op    string
}

type query struct {
    ands []queryComparison
}

// E.g. q=a.b:12
func parseQuery(q string) (*query, error) {
    if q == "" {
        return &query{}, nil
    }

    i := 0
    var parsed query
    var qRune = []rune(q)
    for i < len(qRune) {
        // Eat whitespace
        for unicode.IsSpace(qRune[i]) {
            i++
        }

        key, nextIndex, err := lexString(qRune, i)
        if err != nil {
            return nil, fmt.Errorf("Expected valid key, got [%s]: `%s`", err, q[nextIndex:])
        }

        // Expect some operator
        if q[nextIndex] != ':' {
            return nil, fmt.Errorf("Expected colon at %d, got: `%s`", nextIndex, q[nextIndex:])
        }
        i = nextIndex + 1

        op := "="
        if q[i] == '>' || q[i] == '<' {
            op = string(q[i])
        i++
        }

        value, nextIndex, err := lexString(qRune, i)
        if err != nil {
            return nil, fmt.Errorf("Expected valid value, got [%s]: `%s`", err, q[nextIndex:])
        }
        i = nextIndex

        argument := queryComparison{key: strings.Split(key, "."), value: value, op: op}
        parsed.ands = append(parsed.ands, argument)
    }

    return &parsed, nil
}

Since we're already writing a real lexer we could do better than strings.Split(key, ".") when it comes to find key path parts. But it isn't a huge deal at this stage. So we keep it simple.

Query matching

Now that we've got the query parser we need to implement an evaluator for the search endpoint. We need to be able to check that given a document, it meets the filter or not.

So we iterate over each argument and do the indicated comparison: equality, greater than or less than. If at any point the comparison fails, return false immediately. Otherwise if we got through all arguments and didn't return, there was a match!

func (q query) match(doc map[string]any) bool {
    for _, argument := range q.ands {
        value, ok := getPath(doc, argument.key)
        if !ok {
            return false
        }

        // Handle equality
        if argument.op == "=" {
            match := fmt.Sprintf("%v", value) == argument.value
            if !match {
                return false
            }

            continue
        }

        // Handle <, >
        right, err := strconv.ParseFloat(argument.value, 64)
        if err != nil {
            return false
        }

        var left float64
        switch t := value.(type) {
        case float64:
            left = t
        case float32:
            left = float64(t)
        case uint:
            left = float64(t)
        case uint8:
            left = float64(t)
        case uint16:
            left = float64(t)
        case uint32:
            left = float64(t)
        case uint64:
            left = float64(t)
        case int:
            left = float64(t)
        case int8:
            left = float64(t)
        case int16:
            left = float64(t)
        case int32:
            left = float64(t)
        case int64:
            left = float64(t)
        case string:
            left, err = strconv.ParseFloat(t, 64)
            if err != nil {
                return false
            }
        default:
            return false
        }

        if argument.op == ">" {
            if left <= right {
                return false
            }

            continue
        }

        if left >= right {
            return false
        }
    }

    return true
}

This bit of Go that requires separate case statements for every possible numeric so I can convert it to float is really annoying.

The only additional part to call out in there is getPath. We need to be able to grab any path within an object since the user could have made a filter like a.b:12. So let's keep things simple (but less safe) and implement getPath recursively.

func getPath(doc map[string]any, parts []string) (any, bool) {
        var docSegment any = doc
        for _, part := range parts {
                m, ok := docSegment.(map[string]any)
                if !ok {
                        return nil, false
                }

                if docSegment, ok = m[part]; !ok {
                        return nil, false
                }
        }

        return docSegment, true
}

A critical thing to point out is that filtering on arrays is not supported. Any filter that tries to enter an array will fail or return no results.

Search

Now that we've got all the tools in place we can implement the search endpoint. We'll just iterate over all documents in the database and return all documents that match the filter.

func (s server) searchDocuments(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    q, err := parseQuery(r.URL.Query().Get("q"))
    if err != nil {
        jsonResponse(w, nil, err)
        return
    }

    var documents []map[string]any

    iter := s.db.NewIter(nil)
    defer iter.Close()
    for iter.First(); iter.Valid(); iter.Next() {
        var document map[string]any
        err = json.Unmarshal(iter.Value(), &document)
        if err != nil {
            jsonResponse(w, nil, err)
            return
        }

        if q.match(document) {
            documents = append(documents, map[string]any{
                "id":   string(iter.Key()),
                "body": document,
            })
        }
    }

    jsonResponse(w, map[string]any{"documents": documents, "count": len(documents)}, nil)
}

Not bad! Let's try it out:

$ go build
$ ./docdb

And in another terminal, try out the search endpoint with no filter:

$ curl http://localhost:8080/docs | jq
{
  "body": {
    "count": 1,
    "documents": [
      {
        "body": {
          "age": "45",
          "name": "Kevin"
        },
        "id": "c458a3ce-9faf-4431-a058-d9ae2a1651e1"
      }
    ]
  },
  "status": "ok"
}

With an equality filter:

$ curl --get http://localhost:8080/docs --data-urlencode 'q=name:Mel' | jq
{
  "body": {
    "count": 0,
    "documents": null
  },
  "status": "ok"
}

$ curl --get http://localhost:8080/docs --data-urlencode 'q=name:Kevin' | jq
{
  "body": {
    "count": 1,
    "documents": [
      {
        "body": {
          "age": "45",
          "name": "Kevin"
        },
        "id": "c458a3ce-9faf-4431-a058-d9ae2a1651e1"
      }
    ]
  },
  "status": "ok"
}

And with greater than/less than filters:

$ curl --get http://localhost:8080/docs --data-urlencode 'q=age:<12' | jq
{
  "body": {
    "count": 0,
    "documents": null
  },
  "status": "ok"
}

$ curl --get http://localhost:8080/docs --data-urlencode 'q=age:<200' | jq
{
  "body": {
    "count": 1,
    "documents": [
      {
        "body": {
          "age": "45",
          "name": "Kevin"
        },
        "id": "c458a3ce-9faf-4431-a058-d9ae2a1651e1"
      }
    ]
  },
  "status": "ok"
}

Sweet.

Benchmarking

Now let's try inserting a few hundred thousand rows of real-world data. Grab movies.json from the Wikipedia Movie Data repo. This dataset only has 28,000 rows. But we can insert it multiple times. If we filter by movie name and movie year we'll be looking at only a small subset of the data but enough that we can get a sense about performance.

Here's a basic script to ingest that data a bunch of times once you've downloaded the file.

#!/usr/bin/env bash

set -e

count=50
for run in {1..50}; do
    jq -c '.[]' "$1" | while read data; do
        curl -X POST -H 'Content-Type: application/json' -d "$data" http://localhost:8080/docs
    done
done

Start it up and wait as long as you can. :)

$ chmod +x scripts/load_array.sh
$ ./scripts/load_array.sh movies.json

You can check how many items are in the database like so:

$ curl http://localhost:8080/docs | jq '.body.count'
12649

Once you have a few hundred thousand documents you'll start to notice exact equality queries start to take longer:

$ time curl -s --get http://localhost:8080/docs --data-urlencode 'q="year":1918' | jq '.body.count'
1152
curl -s --get http://localhost:8080/docs --data-urlencode 'q="year":1918'  0.00s user 0.00s system 0% cpu 0.992 total

And you think: although there are hundreds of thousands of documents, if I'm just asking for documents with a certain value such that there are only 1000 documents that match that value, shouldn't it be possible to grab them more quickly than in one whole second? Or, better than a time that grows with the number of documents in the database?

Yes. Yes it is possible.

Indexes

Document databases often index everything. We're going to do that. For every path in a document (that isn't a path within an array) we're going to store the path and the value of the document at that path.

First we'll open a second database that we'll use to store all of these path-value pairs.

type server struct {
        db      *pebble.DB // Primary data
        indexDb *pebble.DB // Index data
        port    string
}

func newServer(database string, port string) (*server, error) {
        s := server{db: nil, port: port}
        var err error
        s.db, err = pebble.Open(database, &pebble.Options{})
        if err != nil {
                return nil, err
        }

        s.indexDb, err = pebble.Open(database+".index", &pebble.Options{})
        return &s, err
}

Then when we insert, we'll call an index function to generate all path-value pairs and store them in this second database.

The index database will store the path-value pair as keys. And values will be the comma separated list of document IDs that have that path-value pair.

func (s server) index(id string, document map[string]any) {
        pv := getPathValues(document, "")

        for _, pathValue := range pv {
                idsString, closer, err := s.indexDb.Get([]byte(pathValue))
                if err != nil && err != pebble.ErrNotFound {
                        log.Printf("Could not look up pathvalue [%#v]: %s", document, err)
                }

                if len(idsString) == 0 {
                        idsString = []byte(id)
                } else {
                        ids := strings.Split(string(idsString), ",")

                        found := false
                        for _, existingId := range ids {
                                if id == existingId {
                                        found = true
                                }
                        }

                        if !found {
                                idsString = append(idsString, []byte(","+id)...)
                        }
                }

                if closer != nil {
                        err = closer.Close()
                        if err != nil {
                                log.Printf("Could not close: %s", err)
                        }
                }
                err = s.indexDb.Set([]byte(pathValue), idsString, pebble.Sync)
                if err != nil {
                        log.Printf("Could not update index: %s", err)
                }
        }
}

Keeping things simple we'll also implement this getPathValues helper recursively:

func getPathValues(obj map[string]any, prefix string) []string {
        var pvs []string
        for key, val := range obj {
                switch t := val.(type) {
                case map[string]any:
                        pvs = append(pvs, getPathValues(t, key)...)
                        continue
                case []interface{}:
                        // Can't handle arrays
                        continue
                }

                if prefix != "" {
                        key = prefix + "." + key
                }

                pvs = append(pvs, fmt.Sprintf("%s=%v", key, val))
        }

        return pvs
}

We'll update one line in s.addDocument to call this index function.

func (s server) addDocument(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
        dec := json.NewDecoder(r.Body)
        var document map[string]any
        err := dec.Decode(&document)
        if err != nil {
                jsonResponse(w, nil, err)
                return
        }

        // New unique id for the document
        id := uuid.New().String()

        s.index(id, document)

        bs, err := json.Marshal(document)
        if err != nil {
                jsonResponse(w, nil, err)
                return
        }
        err = s.db.Set([]byte(id), bs, pebble.Sync)
        if err != nil {
                jsonResponse(w, nil, err)
                return
        }

        jsonResponse(w, map[string]any{
                "id": id,
        }, nil)
}

And we'll add a reindex function to be called in main to handle any documents that were ingested and not indexed (i.e. all the ones we already inserted).

func (s server) reindex() {
        iter := s.db.NewIter(nil)
        defer iter.Close()
        for iter.First(); iter.Valid(); iter.Next() {
                var document map[string]any
                err := json.Unmarshal(iter.Value(), &document)
                if err != nil {
                        log.Printf("Unable to parse bad document, %s: %s", string(iter.Key()), err)
                }
                s.index(string(iter.Key()), document)
        }
}

func main() {
        s, err := newServer("docdb.data", "8080")
        if err != nil {
                log.Fatal(err)
        }
        defer s.db.Close()

        s.reindex()

        router := httprouter.New()
        router.POST("/docs", s.addDocument)
        router.GET("/docs", s.searchDocuments)
        router.GET("/docs/:id", s.getDocument)

        log.Println("Listening on " + s.port)
        log.Fatal(http.ListenAndServe(":"+s.port, router))
}

Using the index

When there is an equality filter we can look the equality filter up in the index database. Our filter language only supports AND-ed arguments. So the results matching the overall filter must be the set intersection of ids that match each individual equality filter. Greater than and less than filters will be filtered out after fetching all possible ids that match equality filters.

If no ids are found in the index database meeting all equality filters then we'll fall back to the full table scan we already have.

func (s server) searchDocuments(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
        q, err := parseQuery(r.URL.Query().Get("q"))
        if err != nil {
                jsonResponse(w, nil, err)
                return
        }

        isRange := false
        idsArgumentCount := map[string]int{}
        nonRangeArguments := 0
        for _, argument := range q.ands {
                if argument.op == "=" {
                        nonRangeArguments++

                        ids, err := s.lookup(fmt.Sprintf("%s=%v", strings.Join(argument.key, "."), argument.value))
                        if err != nil {
                                jsonResponse(w, nil, err)
                                return
                        }

                        for _, id := range ids {
                                _, ok := idsArgumentCount[id]
                                if !ok {
                                        idsArgumentCount[id] = 0
                                }

                                idsArgumentCount[id]++
                        }
                } else {
                        isRange = true
                }
        }

        var idsInAll []string
        for id, count := range idsArgumentCount {
                if count == nonRangeArguments {
                        idsInAll = append(idsInAll, id)
                }
        }

        var documents []any
        if r.URL.Query().Get("skipIndex") == "true" {
                idsInAll = nil
        }
        if len(idsInAll) > 0 {
                for _, id := range idsInAll {
                        document, err := s.getDocumentById([]byte(id))
                        if err != nil {
                                jsonResponse(w, nil, err)
                                return
                        }

                        if !isRange || q.match(document) {
                                documents = append(documents, map[string]any{
                                        "id":   id,
                                        "body": document,
                                })
                        }
                }
        } else {
                iter := s.db.NewIter(nil)
                defer iter.Close()
                for iter.First(); iter.Valid(); iter.Next() {
                        var document map[string]any
                        err = json.Unmarshal(iter.Value(), &document)
                        if err != nil {
                                jsonResponse(w, nil, err)
                                return
                        }

                        if q.match(document) {
                                documents = append(documents, map[string]any{
                                        "id":   string(iter.Key()),
                                        "body": document,
                                })
                        }
                }
        }

        jsonResponse(w, map[string]any{"documents": documents, "count": len(documents)}, nil)
}

The last unimplemented part is the lookup helper. Given a path-value pair it checks the database for IDs that match that pair.

func (s server) lookup(pathValue string) ([]string, error) {
        idsString, closer, err := s.indexDb.Get([]byte(pathValue))
        if err != nil && err != pebble.ErrNotFound {
                return nil, fmt.Errorf("Could not look up pathvalue [%#v]: %s", pathValue, err)
        }
        if closer != nil {
                defer closer.Close()
        }

        if len(idsString) == 0 {
                return nil, nil
        }

        return strings.Split(string(idsString), ","), nil
}

We're done. Finally! Let's build it:

$ go build
$ ./docdb

(This is going to take a while; to reindex.)

Once the server is ready we can run:

$ time curl -s --get http://localhost:8080/docs --data-urlencode 'q="year":1918' | jq '.body.count'
1280
curl -s --get http://localhost:8080/docs --data-urlencode 'q="year":1918'  0.01s user 0.00s system 29% cpu 0.029 total

Hey that's not bad.

Hey here's a new blog post on writing a document database from scratch with support for Lucene-like queries and basic indexes in less than 500 lines of Gohttps://t.co/M3js6Pj9h0
— Phil Eaton (@phil_eaton) March 28, 2022

Solving 5 algorithmic interview questions

Bytepawn - Marton Trencseni — Sat, 26 Mar 2022 01:00:00 GMT

Recently I was considering whether to introduce some CS style algorithmic interview questions into our Data Science hiring loop, since having an understanding of algorithms and data structures can be useful for Data Scientists. Not having done this soft of interview for a few years I picked up my copy of Daily Coding Problem and starting solving a few problems to refresh my feeling for what it feels like as a candidate, and whether it would give us any useful signals.

Vim tip 6: search word nearest to the cursor

learnbyexample — Thu, 24 Mar 2022 02:00:00 GMT

Vim provides handy commands to match words under (or near to) the cursor. You can choose to match whole or part of a longer word. If a match is found, the cursor will move to the next match in the chosen direction.

* searches the word nearest to the cursor in the forward direction (matches only the whole word)
- Shift followed by left mouse click can also be used if mouse is enabled
g* searches the word nearest to the cursor in the forward direction (matches as part of another word as well)
- for example, if you apply this command on the word the, you'll also get matches for them, lather, etc
# searches the word nearest to the cursor in the backward direction (matches only the whole word)
g# searches the word nearest to the cursor in the backward direction (matches as part of another word as well)

From :h word:

A word consists of a sequence of letters, digits and underscores, or a sequence of other non-blank characters, separated with white space (spaces, tabs, <EOL>). This can be changed with the iskeyword option.

Sequence of non-blank characters will be used only at the end of the line. Otherwise, the above commands will use the next word found on that line which is made up of letters, digits and underscores.

You can also provide a count prefix to these commands. For example, 2* will take you to the second match in the forward direction.

Video demo:

Growing Pains

Weblog on marginalia.nu — Wed, 23 Mar 2022 02:00:00 GMT

The search engine index has grown quite considerably the last few weeks. It’s actually surpassed 50 million documents, which is quite some milestone. In February it was sitting at 27-28 million or so.

About 80% of this is side-loading all of stackoverflow and stackexchange, and part of it is additional crawling.

The crawler has to date fetched 91 million URLs, but only about a third of what is fetched actually qualifies for indexing for various reasons, some links may be dead, some may be redirects, some may just have too much javascript and cruft to qualify.

Fair coin from biased coin

Bytepawn - Marton Trencseni — Tue, 22 Mar 2022 01:00:00 GMT

Given a biased coin, construct a fair coin.

Read Browser File Inputs with Go WebAssembly

Donat Studios — Mon, 21 Mar 2022 05:11:42 GMT

I have been playing with WebAssembly since Go added support for it in 2017.

For a very long time now, I've been wanting to be able to accept files from an end user and process them with Go on their local machine. Basically I just want to read files from file inputs. I knew it had to be possible, but could not in all this time find a good walkthrough online. A couple nights ago I set my mind to it and figured it out.

Essentially we need to execute the following JavaScript, but in Go via the JS syscall API.

So after transliterating that JavaScript into Go syscall/js calls, we then using js.CopyBytesToGo to pass the given Uint8Array back to our WASM runtime where we can do with the bytes as we please.

Note, given the keyword any here, this example requires Go 1.18 - the syscall/js library is still considered experimental and expect it to break on major go releases.

This is as written for brevity and could be improved in code safety in a number of pretty obvious ways.

The iframe below runs the compiled WASM and spits out any file you upload to it, via Go.

Example

2022–03–21: Some finer points about Alt-DP support in mainline kernel

megi's PinePhone Development Log — Mon, 21 Mar 2022 01:00:00 GMT

The Static File Startup

Weblog on marginalia.nu — Fri, 18 Mar 2022 02:00:00 GMT

Note: This is satirical in nature. Slight CW if you are at a point in life where “Office Space” has unveiled itself as a disturbing existential horror movie. This taps into that the same darkness.

A tale of six brave Internet pioneers.

Senior Business Founder / Senior CEO – Zach Senior Tech Lead / Senior Architect / Senior CTO – Kevin Senior Backend dev Senior Frontend dev – Erin Two Senior UX engineers

A Manifesto for the Metaverse

Tilting at Windmills — Thu, 17 Mar 2022 02:00:00 GMT

What is the Metaverse? Wired tried to figure it out recently and the results were inconclusive. Where is the metaverse? How do we log in to…

An excellent response, that didn't answer the question.

--Mike-- — Mon, 14 Mar 2022 21:43:05 GMT

Ask HN: As you get older, do you see programming as merely a means to an end?

That is, you become more interested in what your app does rather than the programming involved to get it to do that?

--- My Reply --- at https://news.ycombinator.com/item?id=30674662

Programming has gone through a weird path of effort/reward. Initially, you had to invest time and a lot of money to build your own computer from a kit, debug any errors, and get to the point of having your own, personal computer. At that point, any working results were amazing, as there was a fair chance you were the first person to solve the given problem.

This lasted an amazingly long time. In the matter of less than a decade, the same amount of money, minus the time and effort of building and constructing, got you a computer, could be skipped. For the same amount of money, you got a computer with a display, floppy disks, and it just worked out of the box.

DOS, BASIC, Turbo Pascal, Sidekick, PKzip, and so many tools game flooding in to make those computers even more easier to program, and productive.

It was at THAT POINT, where I started really making money as a programmer.

A decade passes, Windows 95 / Macintosh come along, and we get Visual Basic / Delphi / Hypercard. WOW... the amount of effort goes up just a tiny bit, but you can do amazing things, and that code will run reliably for most of your users without much tweaking on the programmers part.

Then networking, the internet, SQL as a layer, Email, Microsoft Office, etc. all boost productivity for the users. The table stakes remain remarkably stable.

Then the iPhone comes along, and with a lot of effort, and the approval of Apple, you can build a program that people can carry around in their pockets. Apple decides to limit the retail price of apps, and bottleneck things through the App store. Programming gets harder, and more specialized. Ugh

Meanwhile, the web has grown popular, and the idea that a program should just work anywhere shatters the simplicity of a program running on a single computer, directly talking to the OS that controls the display, and takes input. Again, programming gets harder, and more specialized.

Now we're at the point, where you're expected to make a program, sell it for less than a few dollars, support that program on any random phone, and hope that the two app stores don't arbitrarily change the rules and cut you off. The incentives go way up with audience size, and down with the giant filter of the app store.

To route around the damage of the App store, you are expected to make your program available on a server on the web, deal with hackers, man in the middle attacks, wildly varying desktop and mobile web browsers, bandwidth and legal restrictions, to deliver results to a user that might be at a desktop with multiple 4K screens on a gigabit internet... all the way to a person using an phone with no keyboard, who pays for data transfer, and might lose connection at any moment, might rotate the screen, and wants to interact with video, multiple cameras, gps, and any number of other sources of input or output.

After all of that discourse to set perspective. Yes, I find it amazing that it can be done at all. I'm saddened that Delphi on Windows XP was probably the most productive GUI build environment I'll ever experience as a programmer.

It's hard not to see the table stakes now as a huge barrier. For my own personal programs, it brings me joy every time something works. Every time I overcome a series of challenges, and my will becomes a tool... it amazes and overjoys me. I love programming. It's frustrating as hell, but I love it. I fail to see how anyone could stick with it if they didn't, at some level.

A meditation on correctness in software

Weblog on marginalia.nu — Mon, 14 Mar 2022 02:00:00 GMT

Let’s define a simple mathematical function, the function will perform integer factoring. It will take an integer, and return two integers, the product of which is the first integer.

F(int32 n) = (int32 A, int32 B)

so that

A*B = n

This is fairly straight forward, mathematical, objective. Let’s examine some answers an implementation might give.

F 50 = (5, 10) on ARM F 50 = (10, 5) on Intel

This seems like a bug, so let’s add the requirement that A <= B for deterministic results.

Could We Not Have Ideologies?

I Am Not a Robot — Sun, 13 Mar 2022 22:32:58 GMT

Ideology: “a system of ideas and ideals, especially one which forms the basis of economic or political theory and policy.”

Detecting Badger2040 boards and automating uploads

Subclassed — Sat, 12 Mar 2022 03:21:18 GMT

I recently bought a bunch of Pimoroni Badger2040 boards, and they are a lot of fun.

The Badger is basically a small microcontroller (the Raspberry Pico) with an eInk display, in the size of a typical office badge. It has a few buttons you can interact with, when powered, but because of eInk it doesn't actually need to be powered all the time - you can just set it to the desired screen, turn off the battery, and the screen will stay as it was more or less forever.

The fun bit is that it can run MicroPython, so programming it is a breeze. You don't have to deal with all the scary vagaries of C/C++; just write your Python scripts, save them to the board, and run them. Sweet!

There is already a fairly comprehensive tutorial on how to get started with Badger2040, but (like most Pico-related documentation out there) it assumes you're happy to use Thonny, an editor focused on the micropython ecosystem, in order to move files to the board. With all due respect, Thonny is a very limited editor, and it gets recommended only because it's the most intuitive when it comes to managing files on the Pico. I'm much happier when I live in my beloved PyCharm, but its MicroPython plugin is somewhat limited and requires manual interaction, so I investigated a strategy to automate the basic stuff directly from Python on my laptop.

The first step is detecting the board. It appears to the operating system as a serial port, so we have to list the available ports and find the one that looks like our guy.

 # badgerutils.py
import serial.tools.list_ports as list_ports
from serial.tools.list_ports_common import ListPortInfo
  
def is_badger(port: ListPortInfo):
    """ decide if the port looks like a Badger2040 """
    # mac, but other systems will probably be similar,
    # just add other "if" blocks for windows etc
    if sys.platform.startswith('darwin'):
    	# you should be more thorough, 
        # might want to check VID etc, but this will do for dev
        if port.manufacturer and \
           	    port.manufacturer.lower().startswith('micropython'):
            return True
    return False
  
def get_badger():
    """ loop through all the ports and find our board """
    ports = list(list_ports.comports())
    for p in ports:
        if is_badger(p):
            return p

The next step is where things get a bit hairy. Interacting over the serial port is not everyone's idea of fun, so we better stand on the shoulder of geeky giants if possible. We could dig through Thonny's code, but it's long and complicated and meant to support a lot of scenarios we don't really care about. Instead, we can reuse a little utility called ampy, which is slightly old but fairly robust and (more importantly) self-contained and easy to understand.

Ampy includes a couple of modules to interact with a micropython board. You can have a look at the functions found in its cli module to figure how to wrap them, but here's a simple approach to start pushing files to the board - some of the code is lifted almost entirely from ampy.cli, but it's MIT-licensed, so you can do that (just mention the original copyright notice somewhere, if you publish it!).

# BadgerManager.py
  
from serial.tools.list_ports_common import ListPortInfo
from ampy.files import Files, DirectoryExistsError
from ampy.pyboard import Pyboard
  
class MyBadger(Pyboard):

    def __init__(self, port: ListPortInfo):
        super(MyBadger, self).__init__(port.device)
        self.files = Files(self)

    def upload(self, file_path: Path, dest_path: Path):
        """ upload file or directory to board """
        if file_path.is_dir():
            # Directory copy, create the directory and walk all children 
            # to copy over the files. 
            for parent, child_dirs, child_files in os.walk(file_path):
                # Create board filesystem absolute path to parent directory.
                remote_parent = posixpath.normpath(
                    posixpath.join(dest_path, os.path.relpath(parent, file_path))
                )
                try:
                    # Create remote parent directory.
                    self.files.mkdir(remote_parent)
                except DirectoryExistsError:
                    # Ignore errors for directories that already exist.
                    pass
                # Loop through all the files and put them on the board too.
                for filename in child_files:
                    with open(os.path.join(parent, filename), "rb") as infile:
                        remote_filename = posixpath.join(remote_parent,
                                                         filename)
                        self.files.put(remote_filename, infile.read())
        else:
            # File copy
            # check if in subfolder
            if len(dest_path.parents) > 1:
                # subfolder was specified
                # each parent has to be created individually,
                # because of ampy limitations
                for d in sorted(dest_path.parents)[1:]:  # first is /, discard
                    self.files.mkdir(d)

            # Put the file on the board.
            with open(file_path, "rb") as infile:
                self.files.put(dest_path.absolute(), infile.read())

    def ls(self, dirname='/', recurse=True):
        """ List files on board """
        dirpath = dirname if type(dirname) == Path else Path(dirname)
        return self.files.ls(dirpath.absolute(),
                             long_format=False, recursive=recurse)

Putting both things together we can interact very easily with the board like this:

from badgerutils import get_badger
from BadgerManager import MyBadger

# Note: in real life, remember to manage error conditions ! 
port = get_badger()
board = MyBadger(port)
board.upload("./something.txt", "/something.txt")
assert('/something.txt' in board.ls())

Happy hacking!

The german tank problem in World War II

Bytepawn - Marton Trencseni — Sat, 12 Mar 2022 01:00:00 GMT

I run Monte Carlo simulations to show the frequntist solution to the German tank problem.

Here's what a "No Fly Zone" over Ukraine really means

--Mike-- — Fri, 11 Mar 2022 19:53:14 GMT

a "no fly" zone involves acquiring and maintaining air superiority over Ukraine and neighboring territory. To do this we would have to destroy any surface to air radar and missile sites in or near Ukraine. We would also have to shoot down any enemy aircraft that violated that airspace.

In sum total, it would be a declaration of war against Belarus and Russia.

It would then be up to Putin to decide if, where and when he wants to deploy the worlds largest stockpile of ThermoNuclear weapons.

Likely it would take about 15 minutes to confirm any launch, after which there might be a few minutes left to let the public know to make peace with God.

Your personal outcome at that point, depends on preparation and luck:

A> You've felt the blast, and are going to die, maybe in seconds, maybe in days

B> You've seen the flash, and if you take cover soon enough, can survive the fallout, might survive and watch the total collapse of civilization.

The $100b Bull Case for Temporal

swyx's site RSS Feed — Mon, 07 Mar 2022 19:21:34 GMT

Why Temporal is worth >$1b now, why it will be worth >$10b, and how it could be worth $100b

Python is Like Assembly

Speculative Branches — Sun, 06 Mar 2022 02:00:00 GMT

Python and Assembly have one thing in common: as a professional software engineer, they are both languages that you probably should know how to read, but be terrified to write. These languages seem to be (and are) at opposite ends of the spectrum: One is almost machine code, and the other is almost a scripting language. One is beginner-friendly and the other is seen as hostile to experts. One is viciously versatile with tons of libraries and ports, and the other is ridiculously limited in its capabilities. However, when you are creating production software, both are the wrong tool for the job.

Do Not Argue With Strangers on the Internet

I Am Not a Robot — Sat, 05 Mar 2022 18:56:53 GMT

Everyone knows that there are two kinds of superpowers: the useful and the shitty.

Lessons From A Tech Job Search

Krishna's blog — Sat, 05 Mar 2022 02:00:00 GMT

I share my experience and lessons learned from a recent tech job search, including tips on preparing for interviews and negotiating job offers.

Speeding up Go's builtin JSON encoder up to 55% for large arrays of objects

Notes on software development — Thu, 03 Mar 2022 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Vim tip 5: jumping back and forth in Normal mode

learnbyexample — Wed, 02 Mar 2022 02:00:00 GMT

Find yourself working on a large file? Or perhaps handling multiple buffers? Vim makes it easy to navigate previous locations:

Ctrl+o navigate to the previous location in the jump list (think o as old)
Ctrl+i navigate to the next location in the jump list (i and o are usually next to each other)
g; go to the previous change location
g, go to the newer change location
gi place the cursor at the same position where it was left last time in the Insert mode

Use :jumps to view the jump list. See :h jump-motions for more details.

Video demo:

How to Download Twitter Spaces That Aren't Yours

swyx's site RSS Feed — Mon, 28 Feb 2022 22:14:56 GMT

Grabbing the audio

Marginalia Search: 1 year

Weblog on marginalia.nu — Fri, 25 Feb 2022 02:00:00 GMT

I’ve caught some bug and don’t have the energy to write more than a brief note.

I want to commemorate the fact that work on the Marginalia search engine started one year ago. The first commit was on February 26th 2021, and contained a sketch for a website crawler and some data models.

In many ways, the paint is barely dry, yet it feels like this project has been around for a long while.

Racing the Hardware: 8-bit Division

Speculative Branches — Tue, 22 Feb 2022 02:00:00 GMT

Occasionally, I like to peruse uops.info. It is a great resource for micro-optimization: benchmark every x86 instruction on every architecture, and compile the results. Every time I look at this table, there is one thing that sticks out to me: the DIV instruction. On a Coffee Lake CPU, an 8-bit DIV takes a long time: 25 cycles. Cannon Lake and Ice Lake do a lot better, and so does AMD. We know that divider architecture is different between architectures, and aggregating all of the performance numbers for an 8-bit DIV, we see:

2022–02–22: Adding LibreELEC.tv to an existing Pinephone multi-distro image

megi's PinePhone Development Log — Tue, 22 Feb 2022 01:00:00 GMT

2022–02–22: Pinephone Pro – Debugging Type C port issues

megi's PinePhone Development Log — Tue, 22 Feb 2022 01:00:00 GMT

I have no capslock and I must scream

Weblog on marginalia.nu — Mon, 21 Feb 2022 02:00:00 GMT

In a near future, a team of desktop computer designers are looking at the latest telemetry and updating the schematics of the hardware-as-a-service self-assembling nanohardware.

Steve: “Hmm, they don’t seem to be using the power button very often.”

Bob: “Compared to the other buttons, it’s only used 0.1% of the time”

Steve: “Remove it?”

Bob: “Remove it!”

Computers now instantly boot up when plugged into the wall, and run until the plug is pulled. No more start-up time, the cases are aesthetically cleaner, and manufacturing cost is down at least a fraction of a dollar.

SMTP protocol basics from scratch in Go: receiving email from Gmail

Notes on software development — Sun, 20 Feb 2022 02:00:00 GMT

I've never run my own mail server before. Before today I had no clue how email worked under the hood other than the very few times I've set up mail clients.

I've heard no few times how hard it is to send mail from a self-hosted server (because of spam filters). But how hard can it be to hook up DNS to my personal server and receive email to my domain sent from Gmail or another real-world client?

I knew it would be simpler to just send local mail to a local mail server with a local mail client but that didn't seem as real. If I could send email from my Gmail account and receive it in my server I'd be happy.

I spent the afternoon digging into this. All code is available on Github. The "live stream" is in the Multiprocess Discord's #hacking-networks channel.

DNS

First I bought a domain. (I needed to be able to mess around with records without blowing up anything important.)

I knew that MX records controlled where mail for a domain is sent. But I had to look up the specifics. You need to create an MX record that points to an A or AAAA record. So you need both an MX record and an A or AAAA record.

Done.

Firewall

The firewall on Fedora is aggressive. Gotta open up port 25.

$ sudo firewall-cmd --zone=dmz --add-port=25/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=25/tcp --permanent
$ sudo firewall-cmd --reload

I don't understand what zones are here.

What protocols?

I knew that you send email with SMTP and you read it with POP3 or IMAP. But it hadn't clicked before that the mail server has to speak SMTP and if you only ever read on the server (which is of course impractical in the real world) you don't need POP3 or IMAP.

So to meaningfully receive email from Gmail all I needed to do was implement SMTP.

SMTP

First I found the RFC for SMTP (or one of them anyway) and the wikipedia page for it.

First off I'd need to run a TCP server on port 25.

package main

import (
        "errors"
        "log"
        "net"
        "strconv"
        "strings"
)

func logError(err error) {
        log.Printf("[ERROR] %s\n", err)
}

func logInfo(msg string) {
        log.Printf("[INFO] %s\n", msg)
}

type message struct {
        clientDomain string
        smtpCommands  map[string]string
        atmHeaders   map[string]string
        body         string
        from         string
        date         string
        subject      string
        to           string
}

type connection struct {
        conn net.Conn
        id   int
        buf  []byte
}

// TODO

func (c *connection) handle() {
        // TODO
}

func main() {
        l, err := net.Listen("tcp", "0.0.0.0:25")
        if err != nil {
                panic(err)
        }
        defer l.Close()

        logInfo("Listening")

        id := 0
        for {
                conn, err := l.Accept()
                if err != nil {
                        logError(err)
                        continue
                }

                id += 1
                c := connection{conn, id, nil}
                go c.handle()
        }
}

Just a basic TCP server that passes off connections inside a goroutine.

Greeting

After starting a connection, the server must send a greeting. The successful greeting response code is 220. It can optionally be followed by additional text. Like most commands in SMTP it must be ended with CRLF (\r\n).

So we'll add a helper function for writing lines that end in CRLF:

func (c *connection) writeLine(msg string) error {
        msg += "\r\n"
        for len(msg) > 0 {
                n, err := c.conn.Write([]byte(msg))
                if err != nil {
                        return err
                }

                msg = msg[n:]
        }

        return nil
}

And then we'll send that 220 in the handle function.

func (c *connection) handle() {
        defer c.conn.Close()
        c.logInfo("Connection accepted")

        err := c.writeLine("220")
        if err != nil {
                c.logError(err)
                return
        }

        c.logInfo("Awaiting EHLO")

        // TODO

EHLO

Next we need to be able to read requests from the client. We'll write a helper that reads until the next CRLF. We'll keep a buffer of unread bytes in case we accidentally get bytes past the next CRLF. We'll store that buffer in the connection object.

func (c *connection) readLine() (string, error) {
        for {
                b := make([]byte, 1024)
                n, err := c.conn.Read(b)
                if err != nil {
                        return "", err
                }

                c.buf = append(c.buf, b[:n]...)
                for i, b := range c.buf {
                        // If end of line
                        if b == '\n' && i > 0 && c.buf[i-1] == '\r' {
                                // i-1 because drop the CRLF, no one cares after this
                                line := string(c.buf[:i-1])
                                c.buf = c.buf[i+1:]
                                return line, nil
                        }
                }
        }
}

Now back in the handle-er we can read a line from the client. From the RFC we can see it should be HELO or EHLO. Both sendmail locally and Gmail only send EHLO though so we'll just check for that.

So we'll validate the message sent is an EHLO and then we'll send back a 250 with a space after it. We can ignore the rest of that response grammar since we don't have additional keywords we want to send to the client.

        ...

        c.logInfo("Awaiting EHLO")

        line, err := c.readLine()
        if err != nil {
                c.logError(err)
                return
        }

        if !strings.HasPrefix(line, "EHLO") {
                c.logError(errors.New("Expected EHLO got: " + line))
                return
        }

        msg := message{
                smtpCommands: map[string]string{},
                atmHeaders:  map[string]string{},
        }
        msg.clientDomain = line[len("EHLO "):]

        c.logInfo("Received EHLO")

        err = c.writeLine("250 ")
        if err != nil {
                c.logError(err)
                return
        }

        c.logInfo("Done EHLO")

        // TODO

Additional commands

Next up there are a few commands we need to read before we get to the message body. These include the recipient and the sender address. These are formatted vaguely similar to HTTP headers. They have a key on the left side of a colon and a value on the right. They may have a required order too, I'm not sure.

In response to the commands we'll send a 250 OK, although I'm not sure where in the RFC that is suggested.

In our code we'll just keep looking for these commands until we find the DATA command. This indicates the body is to follow. And to this command we respond with a 354 instead of a 250 OK.

In code:

        ...

        c.logInfo("Done EHLO")

        for line != "" {
                line, err = c.readLine()
                if err != nil {
                        c.logError(err)
                        return
                }

                pieces := strings.SplitN(line, ":", 2)
                smtpCommand := strings.ToUpper(pieces[0])

                // Special command without a value
                if smtpCommand == "DATA" {
                        err = c.writeLine("354")
                        if err != nil {
                                c.logError(err)
                                return
                        }

                        break
                }

                smtpValue := pieces[1]
                msg.smtpCommands[smtpCommand] = smtpValue

                c.logInfo("Got command: " + line)

                err = c.writeLine("250 OK")
                if err != nil {
                        c.logError(err)
                        return
                }
        }

        c.logInfo("Done SMTP commands, reading ARPA text message headers")

        // TODO

Message body, headers

Now that we've seen the DATA command we are within a message body. Within this body we still have to read some additional headers.

Through trial-and-error I know to look for some headers like Subject. By searching the RFC I noticed a reference to RFC 822 where these headers are defined.

These are ARPA internet text message headers apparently. They also look like HTTP headers but unlike HTTP headers they can span multiple lines. This stumped me for a bit.

I decided to write a new readLine function that would specifically look for these possibly multi-line headers where a CRLF followed by whitespace isn't a line delimiter.

func (c *connection) readMultiLine() (string, error) {
        for {
                noMoreReads := false
                for i, b := range c.buf {
                        if i > 1 &&
                                b != ' ' &&
                                b != '\t' &&
                                c.buf[i-2] == '\r' &&
                                c.buf[i-1] == '\n' {
                                // i-2 because drop the CRLF, no one cares after this
                                line := string(c.buf[:i-2])
                                c.buf = c.buf[i:]
                                return line, nil
                        }

                        noMoreReads = c.isBodyClose(i)
                }

                if !noMoreReads {
                        b := make([]byte, 1024)
                        n, err := c.conn.Read(b)
                        if err != nil {
                                return "", err
                        }

                        c.buf = append(c.buf, b[:n]...)

                        // If this gets here more than once it's going to be an infinite loop
                }
        }
}

func (c *connection) isBodyClose(i int) bool {
        return i > 4 &&
                c.buf[i-4] == '\r' &&
                c.buf[i-3] == '\n' &&
                c.buf[i-2] == '.' &&
                c.buf[i-1] == '\r' &&
                c.buf[i-0] == '\n'
}

Now back in the handle function we can read through all of these headers. According to RFC 822, we're done when we see a double CRLF, which in our code will show up as an empty line.

        ...

        c.logInfo("Done SMTP headers, reading ARPA text message headers")

        for {
                line, err = c.readMultiLine()
                if err != nil {
                        c.logError(err)
                        return
                }

                if strings.TrimSpace(line) == "" {
                        break
                }

                pieces := strings.SplitN(line, ": ", 2)
                atmHeader := strings.ToUpper(pieces[0])
                atmValue := pieces[1]
                msg.atmHeaders[atmHeader] = atmValue

                if atmHeader == "SUBJECT" {
                        msg.subject = atmValue
                }
                if atmHeader == "TO" {
                        msg.to = atmValue
                }
                if atmHeader == "FROM" {
                        msg.from = atmValue
                }
                if atmHeader == "DATE" {
                        msg.date = atmValue
                }
        }

        c.logInfo("Done ARPA text message headers, reading body")

        // TODO

Body, for real

We're finally at the email body as the user typed it. According to the SMTP RFC the body ends with a CRLF followed by a dot (period) followed by a CRLF.

So we'll write another helper to read until this marker.

func (c *connection) readToEndOfBody() (string, error) {
        for {
                for i := range c.buf {
                        if c.isBodyClose(i) {
                                return string(c.buf[:i-4]), nil
                        }
                }

                b := make([]byte, 1024)
                n, err := c.conn.Read(b)
                if err != nil {
                        return "", err
                }

                c.buf = append(c.buf, b[:n]...)
        }
}

And we can finish up the handle function.

        c.logInfo("Done ARPA text message headers, reading body")

        msg.body, err = c.readToEndOfBody()
        if err != nil {
                c.logError(err)
                return
        }

        c.logInfo("Got body (%d bytes)", len(msg.body))

        err = c.writeLine("250 OK")
        if err != nil {
                c.logError(err)
                return
        }

        c.logInfo("Message:\n%s\n", msg.body)

        c.logInfo("Connection closed")
}

Compile, setcap, run, and send

$ go build
$ sudo setcap 'cap_net_bind_service=+ep' ./gomail
$ ./gomail

And send an email in Gmail! It can be to any user since we haven't implemented anything regarding users. I'll send What hath god wrought as the subject and message to morse@binutils.org.

And I see:

2022/02/21 02:17:19 [INFO] Listening
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Connection accepted
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Awaiting EHLO
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Received EHLO
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Done EHLO
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Got header: MAIL FROM:<me@eatonphil.com>
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Got header: RCPT TO:<morse@binutils.org>
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Done SMTP headers, reading ARPA text message headers
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Done ARPA text message headers, reading body
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Got body (256 bytes)
2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Message:
--000000000000c4758905d87ddb81
Content-Type: text/plain; charset="UTF-8"

What hath god wrought

--000000000000c4758905d87ddb81
Content-Type: text/html; charset="UTF-8"

<div dir="ltr">What hath god wrought</div>

--000000000000c4758905d87ddb81--

2022/02/21 02:19:13 [INFO] [1: 209.85.222.47:40695] Connection closed

Which is pretty sweet!

Multipart wut

Ok this body still clearly has some format. And if we dump the ARPA text message headers we notice that Gmail 1) sets a Content-Type header and 2) it's value is multipart/alternative. I don't know where Content-Type as a valid header is defined because it's not in RFC 822. Maybe it's some "new-fangled" adhoc standard or maybe it's just in an extension RFC.

In any case this looks like multipart bodies in HTTP. I don't want to deal with that so I'm just going to stop here.

But I am curious about text-only email systems. So I sudo dnf install php sendmail and write a quick PHP script (thanks to @Josh on Discord for the suggestion):

<?php
  mail("morse@binutils.org", "What hath god wrought", "What hath god wrought", "");
?>

And run it:

$ php test.php

And in my gomail window I see:

2022/02/21 02:24:17 [INFO] Listening
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Connection accepted
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Awaiting EHLO
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Received EHLO
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Done EHLO
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Got header: MAIL From:<phil@dev1.eatonphil.com>
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Got header: RCPT To:<morse@binutils.org>
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Done SMTP headers, reading ARPA text message headers
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Done ARPA text message headers, reading body
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Got body (21 bytes)
2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Message:
What hath god wrought

2022/02/21 02:24:18 [INFO] [1: 127.0.0.1:45102] Connection closed

And I'm happy to call it a night.

I wrote a new blog post on building an SMTP server from scratch in Go that is correctly enough hooked up you can receive emails sent from Gmail to it!

Good fun and some learning too.https://t.co/8pYkkAbFnI
— Phil Eaton (@phil_eaton) February 21, 2022

p.s. if you want to see more networking software/hardware internals check out /r/NetworkDevelopment.

Drive Failure

Weblog on marginalia.nu — Sat, 19 Feb 2022 02:00:00 GMT

Not what I had intended to do this Saturday, but a hard drive failed on the server this morning, or at least so it seemed. MariaDB server went down, dmesg was full of error messages for the nvme drive it’s running off. That’s a pretty important drive.

The drive itself may actually be okay, the working hypothesis is either the drive itself or the bus overheated and reset. After a reboot the system seems fine.

In response to Doc's story of the Story

--Mike-- — Fri, 18 Feb 2022 20:42:40 GMT

Doc Searls was at it again back in 2018 doing a Ted Talk. His story rhymes with his overall message, and the things he wrote back then. I think it's time to examine how events have evolved since then.

*This story is me revising and extending my reply on Facebook. (It's buried in the comments, don't bother)

Doc worried about the lost center in coverage of the news, and overall discussion. I too feel the loss. There's an old saying, "The Net treats Censorship as Damage, and routes around it". I think that it takes time in the world of atoms, but that John Gilmore was right.

Communications, Data Storage, and Compute infrastructure have grown exponentially with the commercialized internet. The physical costs of producing and sharing content have fallen to almost zero. It's not perfect, and there are some left out, but for the most part anyone can be online, and have a say in the world.

In response to the visible failure of advertising driven funding, crowd funding now is filling in the missing channel of support to keep those who wish to share on the internet going. A number of new journalists are arising. Some of them cover things traditionally called news. It's not perfect, and there are bits of censorship creeping in around the edges, but it is working.

For me, my most trusted news sources at the moment are spread across YouTube. It makes it possible for me to give authority to my choices, make them my authoritative sources. I'm filling in that lost center with the people who I think can best tell me what I need to know, and why others tell the stories they do.

Now, having a centrist view does make discussions with those entrained in the left/right thought bubbles frustrating at times, but who knows, maybe we'll all pull the center back together?

Here's how I approach news gathering in 2022.

I'd heard from the mainstream that leaks in via friends that there was a ruckus brewing over Ukraine. For this story, most trusted sources were Beau of the Fifth Column, Breaking Points with Krystal and Sagaar, and Robb Law.

To be really sure, I stuck with first hand imaging via a list of live webcams across Ukraine. It was fairly easy to be certain no invasion was feared by the locals as they went about skiing in Ivano-Frankivsk, walking across Sophia Square in Kiev, or Pysanka museum (also in Ivano-Frankivsk).

For practical advice, matters of instruction, the future is really here. You can learn all manner of technical facts and skills. There are some limitations I've noticed, for instance one channel on machining had a really great multi-part series on the intricacies of making a part of a fyre arm, which he was forced to remove. I have no desire to actually do that, but there were a number of creative methods used, like using a quill of a milling machine as a vertical broach, and this was an outstanding example of why you would do it. That knowledge is now walled off, and considered tainted by association.

I've long stated that our computers aren't safe because of a design flaw at their core. I now believe that this censorious tendency in the chain of payment is a similar danger.

It remains to be seen how we'll route around it.

You Think You Consume Information. What if the Opposite Is True?

I Am Not a Robot — Thu, 17 Feb 2022 00:18:46 GMT

In 1981, an Italian six-year-old boy named Alfredo Rampi fell into a well.

Unabridged Conclusion to the State of JS

swyx's site RSS Feed — Wed, 16 Feb 2022 14:04:45 GMT

the full text of my State of JS writeup!

Let's choose a better timeline

--Mike-- — Sun, 13 Feb 2022 18:56:39 GMT

I believe in this causal chain,

Computers are insecure
Which many bad actors find profit in exploiting
Which makes new web sites a risk
Which makes users prefer their known "safe" spaces
Which leads to walled gardens
Which then sell the "users" for profit to advertisers
Which incentivizes dark behavior of those walled gardens
Which then attract the rentier class
Which then leverage control for more power

I believe this causal chain can be broken by fixing computer security. The necessary research was done in the 1970s. The Bell-Lapadula model [1] in 1973 was one of the significant results.

The Principle of Least Privilege [2] was adopted in the Unix system in a weak form. The superuser (root) account was a special privilege, which administrators and code was supposed to use as little as possible.

There were (are???) implementations of a multi-level secure systems, which saw limited application in the military, and briefly elsewhere. However, for general use, the root/user separation was widely seen as good enough.

There are now efforts to fully extend operating systems so that they can provide tools so that the users can also use the Principle of Least Privilege. I believe that eventually it will be as easy to use these as more conventional systems.

In these systems, no default permissions are given when running a program, the allowed resources, also known as capabilities, must be specified. This is similar to deciding which bank notes you are going to hand to a cashier, instead of handing over your wallet. It is up to US to demand that it be as easy for any user to do so, in a transparent way.

It is my hope, that should this model be accepted, a new causal chain will arise

Computers will be made secure
Which users will grow to trust
Which will allow experimentation
Which allows new ways of communicating
Which don't require corporate sponsorship
Which doesn't require the rentier class
Which helps innovation
Which helps society

1 - https://web.archive.org/web/20060618092351/http://www.albany.edu/acc/courses/ia/classics/belllapadula1.pdf

2 - https://en.wikipedia.org/wiki/Principle_of_least_privilege

The Meaning of Speed

Speculative Branches — Sun, 13 Feb 2022 02:00:00 GMT

A lot of the time, when engineers think of performance work, we think about looking at benchmarks and making the numbers smaller. We anticipate that we are benchmarking the right pieces of code, and we take it for granted that reducing some of those numbers is a benefit, but also "the root of all evil" if done prematurely. If you are a performance-focused software engineer, or you are working with performance engineers, it can help to understand the value proposition of performance and when to work on it.

Delightfully cynical half-truths about organizations

Bytepawn - Marton Trencseni — Sat, 12 Feb 2022 01:00:00 GMT

Five delightfully cynical half-truths about organizations: the Peter principle, the Dilbert principle, the Gervais principle, Negative selection and the Dunning-Kruger effect.

A Job Pays Much More Than a Salary

I Am Not a Robot — Fri, 11 Feb 2022 20:52:38 GMT

When comparing two job offers, the first thought that comes to my mind is, what is the difference in pay?

Full-disk encryption on Slackware, the modern way

RyNo’s Site — Thu, 10 Feb 2022 10:00:00 GMT

The McClusky Curve - Viral vs Evergreen Content

swyx's site RSS Feed — Thu, 10 Feb 2022 08:31:50 GMT

An insightful discussion on how to trade off aiming for viral or permanent posts.

The Most Incompetent Person I Ever Worked With

Ben E. C. Boyter — Thu, 10 Feb 2022 02:00:00 GMT

Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual. I thought it was time branch out into the most incompetent person I ever worked with.

So a bit of back history. This was at lendlease where we were working on a system that massaged all employee data into a single place allowing accurate counts of employees, and getting titles mapped to real ones and such. This was important because lendlease being an umbrella style company had gobbled up lots of other ones and so nobody had a clue how many people actually worked for it.

Rust in 2024

Krishna's blog — Thu, 10 Feb 2022 02:00:00 GMT

I predict the state of Rust programming language in 2024, highlighting areas of improvement and making predictions on various aspects of the language.

Vim tip 4: reposition current line in Normal mode

learnbyexample — Wed, 09 Feb 2022 02:00:00 GMT

You're likely to be familiar with commands to scroll through the contents like Ctrl followed by d or f or u or b.

Did you know that Vim also has handy options to keep the cursor on the current line while moving the contents around?

zz reposition the current line to the middle of the visible window
- useful to see context around lines that are nearer to the top/bottom of the visible window
zt reposition the current line to the top of the visible window
zb reposition the current line to the bottom of the visible window

See :h 'scrolloff' option if you want to always show context around the current line.

Video demo:

The world of PostgreSQL wire compatibility

Notes on software development — Tue, 08 Feb 2022 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Slackware 15.0 on the Framework

RyNo’s Site — Mon, 07 Feb 2022 10:00:00 GMT

The Anatomy of Search Engine Spam

Weblog on marginalia.nu — Mon, 07 Feb 2022 02:00:00 GMT

Black hat SEO is endlessly fascinating phenomenon to study. This post is about some tactics they use to make their sites rank higher.

The goal of blackhat SEO is to boost the search engine ranking of a page nobody particularly wants to see, usually ePharma, escort services, online casinos, shitcoins, hotel bookings; the bermuda pentagon of shady websites.

The theory behind most modern search engines is that if you get links from a high ranking domain, then your domain gets a higher ranking as well, which increases the traffic. The reality is a little more complicated than that, but this is a sufficient mental model to understand the basic how-to.

Moving to a GitHub CMS

swyx's site RSS Feed — Sun, 06 Feb 2022 22:50:11 GMT

Discussing the 2022 migration of swyx.io to SvelteKit and GitHub Issues

Moving to Obsidian as a Public Second Brain

swyx's site RSS Feed — Sun, 06 Feb 2022 20:08:26 GMT

The case for having a -Public- Second Brain, and why I picked Obsidian.

Michael Keaton is Underappreciated

Tilting at Windmills — Sat, 05 Feb 2022 02:00:00 GMT

Two things. Michael Keaton starred in the film Birdman or (The Unexpected Virtue of Ignorance) in 2014. Three years later in 2017 he played…

How to add JSDoc Typechecking to SvelteKit

swyx's site RSS Feed — Fri, 04 Feb 2022 12:22:01 GMT

As I build out swyxkit, I am finding that I am no longer prototyping and that I need to be able to refactor with confidence.

Can we unfuck internet discoverability?

Weblog on marginalia.nu — Fri, 04 Feb 2022 02:00:00 GMT

I’ve been thinking a lot about how difficult it has become to discover quality content on the Internet, not because it isn’t there, but because the signal to noise ratio is really bad, and most venues of discovery don’t seem to be able to handle it.

Recommendation algorithms seem to work almost too well, to the point where it’s all kind of just showing you things you already like, rarely anything new that you might like. It’s an absolute tragedy both for small websites and for their potential audience.

Installing voice2json on Ubuntu

Ben E. C. Boyter — Thu, 03 Feb 2022 02:00:00 GMT

I have been playing around with offline non cloud tool replacements for doing things like speech to text and the like. One that I found which looked promising was voice2json which is a command line tool for turning speech intent into text. It can do generic conversion too, but a review of that is not the subject of this post.

Following the instructions from the website http://voice2json.org/install.html worked fine up till I tried to run it and got the following issue.

Reason Is a Sword, Not a Swiss Army Knife

I Am Not a Robot — Wed, 02 Feb 2022 02:52:40 GMT

Why smart people sometimes do dumb things

PyDev of the Week

learnbyexample — Wed, 02 Feb 2022 02:00:00 GMT

Last month I had the wonderful opportunity to be part of PyDev of the Week series organized by Michael Driscoll.

It was a pleasure to walk down the memory lane for this interview: https://www.blog.pythonlibrary.org/2022/01/31/pydev-of-the-week-sundeep-agarwal/

I got to discuss about my education, career, writing ebooks and more. Hope you enjoy the interview!

Resurrecting Bogan Ipsum

Ben E. C. Boyter — Tue, 01 Feb 2022 02:00:00 GMT

I remember hearing about Bogan Ipsum, which was an alternative to Lorem Ipsum for you know bogan’s back in 2011 through LifeHacker. However it since became defunct. I became aware of this fact from Darren at work given the following message on slack?

Dude

Boganipsum where?

Clearly this needed to be rectified! A quick search found this https://www.npmjs.com/package/boganipsum implementation which seemed to work well enough, so I ported it to basic JavaScript (think 2001 style JavaScript) and threw together a quick HTML page. I then registered https://boganipsum.com.au/ and the result?

2022–02–01: Pinephone Pro – Type C port support

megi's PinePhone Development Log — Tue, 01 Feb 2022 01:00:00 GMT

Performance Numbers Worth Knowing

Speculative Branches — Mon, 31 Jan 2022 02:00:00 GMT

When you design software to achieve a particular level of performance, it can be a good idea to be familiar with the general speed regimes you are working with: fundamental limitations like storage devices and networks can drive software architecture. Here are a set of common benchmark numbers that can help you anchor performance conversations and think about the components that your software will interact with. As with all guidelines, these numbers are all slightly wrong, but still useful.

How to recommend books, or, stop recommending SICP

Notes on software development — Mon, 31 Jan 2022 02:00:00 GMT

Many "must-read" books are not well-written. I try to read a lot, but I still have a low tolerance for bad writing and bad editing. I write this post both to discourage thoughtless recommendations and to encourage the receivers of bad recommendations.

For software developers, Structure and Interpretation of Computer Programs is a prime example. Written for freshman at MIT, it is ostensibly an entry-level text. But it requires such a level of competence in math and physics, and the prose itself is so dense and archaic, that I couldn't imagine suggesting it to anyone.

And yet it is one of the most recommended books for developers.

This is not to say that SICP is a bad book or that you should not read it. I just don't think it should ever be suggested to anyone.

Goal

The core goal of a book recommendation is for the reader to get enjoyment or education from it. If you can't continue or finish a book, you get nothing from it.

You, the recommender, diminish your impact if you can only recommend books that people won't continue or finish.

Non-goal

Some people have the capacity to read and love challenging books. If that is you, you are not the audience of this post. I don't think you'd disagree that most people are not like you.

Why

I have a few, not-mutually-exclusive guesses why "must-read" books are often poorly written.

One guess is intelligence signalling. That it is human nature for a person to suggest a book in an attempt make herself look smart rather than to best assist the person asking for a suggestion.

Another guess is that most people don't read enough to have a good feel for better or worse writing and editing.

And a final guess is that books that are worth reading might not always be well-written. This is the most unfortunate guess of all. I don't disagree that sometimes it is necessary to learn from poorly-written books. But I begrudge this because of how much joy I get from reading well-written books, fiction and non-fiction.

I have a feeling my guesses apply to recommendations in general: music, art, film, musicals, restaurants, etc.

Instead

My suggestion then to folks who are in the position of giving recommendations:

If you had a hard time reading a book or it took you too long to read it (yes, this threshold is different for everyone), don't recommend it
Don't be scared to recommend nothing, or to recommend against (rather than for) a certain book
Read more books

And definitely don't recommend books you haven't read.

Mea culpa

I've definitely done a bad job recommending books in the past, including recommending books I haven't read. I've been trying to do better in the last 5 years or so.

What do you think?

I wrote a new blog post: a bit of flame bait on how to recommend books and why so many must-read books are impossible to read.

Or: stop recommending SICP.

If you love challenging books, you are neither the norm nor the audience of this post. 😀https://t.co/ZU92kgr4Kf
— Phil Eaton (@phil_eaton) January 31, 2022

2022–01–31: Pinephone Pro battery charging

megi's PinePhone Development Log — Mon, 31 Jan 2022 01:00:00 GMT

Probabilistic spin glass - Conclusion

Bytepawn - Marton Trencseni — Mon, 31 Jan 2022 01:00:00 GMT

I summarize the 5 previous posts on probabilistic spin glasses.

CVE-2022-0329 and the problems with automated vulnerability management

Tom Forbes — Sat, 29 Jan 2022 19:25:09 GMT

Update: Github have responded and said they will stop sending notifications about this CVE. Yesterday Github started notifying tens of thousands of people about a critical remote code execution vulnerability in a package named loguru. Their reviewed advisory is here. It references CVE-2022-0329 whic...

Rust has a small standard library (and that's ok)

Krishna's blog — Sat, 29 Jan 2022 02:00:00 GMT

This article explores the reasons why Rust has a small standard library and why it's okay.

The Swyx.io 2022 Rewrite

swyx's site RSS Feed — Thu, 27 Jan 2022 09:53:45 GMT

Discussing the 2022 migration of swyx.io to SvelteKit and GitHub Issues

What if Global Warming Is Good for You?

I Am Not a Robot — Wed, 26 Jan 2022 22:49:26 GMT

No, We Are Not All in It Together

Bootloader basics

Notes on software development — Sun, 23 Jan 2022 02:00:00 GMT

I spent a few days playing around with bootloaders for the first time. This post builds up to a text editor with a few keyboard shortcuts. I'll be giving a virtual talk based on this work at Hacker Nights on Jan 27.

There are a definitely bugs. But it's hard to find intermediate resources for bootloader programming so maybe parts of this will be useful.

If you already know the basics and the intermediates and just want a fantastic intermediate+ tutorial, maybe try this. It is very good.

The code on this post is available on Github, but it's more of a mess than my usual project.

Motivation: Snake

You remember snake bootloader in a tweet from a few years ago?

Install qemu (on macOS or Linux), nasm, and copy the snake.asm source code to disk from that blog post.

$ cat snake.asm
          [bits 16]                    ; Pragma, tells the assembler that we
                                       ; are in 16 bit mode (which is the state
                                       ; of x86 when booting from a floppy).
          [org 0x7C00]                 ; Pragma, tell the assembler where the
                                       ; code will be loaded.

          mov bl, 1                    ; Starting direction for the worm.
          push 0xa000                  ; Load address of VRAM into es.
          pop es

restart_game:
          mov       si, 320*100+160    ; worm's starting position, center of
                                       ; screen

          ; Set video mode. Mode 13h is VGA (1 byte per pixel with the actual
          ; color stored in a palette), 320x200 total size. When restarting,
          ; this also clears the screen.
          mov       ax, 0x0013
          int       0x10

          ; Draw borders. We assume the default palette will work for us.
          ; We also assume that starting at the bottom and drawing 2176 pixels
          ; wraps around and ends up drawing the top + bottom borders.
          mov       di, 320*199
          mov       cx, 2176
          rep
draw_loop:
          stosb                        ; draw right border
          stosb                        ; draw left border
          add       di, 318
          jnc       draw_loop          ; notice the jump in the middle of the
                                       ; rep stosb instruction.

game_loop:
          ; We read the keyboard input from port 0x60. This also reads bytes from
          ; the mouse, so we need to only handle [up (0x48), left (0x4b),
          ; right (0x4d), down (0x50)]
          in        al, 0x60
          cmp       al, 0x48
          jb        kb_handle_end
          cmp       al, 0x50
          ja        kb_handle_end

          ; At the end bx contains offset displacement (+1, -1, +320, -320)
          ; based on pressed/released keypad key. I bet there are a few bytes
          ; to shave around here given the bounds check above.
          aaa
          cbw
          dec       ax
          dec       ax
          jc        kb_handle
          sub       al, 2
          imul      ax, ax, byte -0x50
kb_handle:
          mov       bx, ax

kb_handle_end:
          add       si, bx

          ; The original code used set pallete command (10h/0bh) to wait for
          ; the vertical retrace. Today's computers are however too fast, so
          ; we use int 15h 86h instead. This also shaves a few bytes.

          ; Note: you'll have to tweak cx+dx if you are running this on a virtual
          ; machine vs real hardware. Casual testing seems to show that virtual machines
          ; wait ~3-4x longer than physical hardware.
          mov       ah, 0x86
          mov       dh, 0xef
          int       0x15

          ; Draw worm and check for collision with parity
          ; (even parity=collision).
          mov ah, 0x45
          xor [es:si], ah

          ; Go back to the main game loop.
          jpo       game_loop

          ; We hit a wall or the worm. Restart the game.
          jmp       restart_game

TIMES 510 - ($ - $$) db 0              ; Fill the rest of sector with 0
dw 0xaa55                              ; Boot signature at the end of bootloader

Now run:

$ nasm -f bin snake.asm -o snake.bin
$ qemu-system-x86_64 -fda snake.bin

What a phenomenal hack.

I'm not going to get anywhere near that level of sophistication in this post but I think it's great motivation.

Hello world

Bootloaders are a mix of assembly programming and BIOS APIs for I/O. Since you're thinking about bootloaders you already know assembly basics. Now all you have to do is learn the APIs.

The hello world bootloader has been explained in detail (see here, here, and here) so I won't go into too much line-by-line depth.

In fact, let's just pull the code from the latter blog post.

$ cat hello.asm
bits 16 ; tell NASM this is 16 bit code
org 0x7c00 ; tell NASM to start outputting stuff at offset 0x7c00
boot:
    mov si,hello ; point si register to hello label memory location
    mov ah,0x0e ; 0x0e means 'Write Character in TTY mode'
.loop:
    lodsb
    or al,al ; is al == 0 ?
    jz halt  ; if (al == 0) jump to halt label
    int 0x10 ; runs BIOS interrupt 0x10 - Video Services
    jmp .loop
halt:
    cli ; clear interrupt flag
    hlt ; halt execution
hello: db "Hello world!",0

times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroes
dw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

The computer boots, prints "Hello world!" and hangs.

But aside from clerical settings (16-bit assembly, where the program exists in memory, padding to 512 bytes) the only real bootloader-y magic in there is int 0x10, a BIOS interrupt.

BIOS interrupts = API calls for I/O

BIOS interrupts are API calls. Just like syscalls in userland programs they have a specific register convention and number to call for the family of APIs.

When you write bootloader programs you'll spend most of your time at first trying to understand the behavior of the various BIOS APIs.

The two families we'll deal with in this post are the keyboard family (documentation here) and the display family (documentation here).

Run hello world

Anyway, back to the hello world. Assemble it with nasm and run it with qemu.

$ nasm -f bin hello.asm -o hello.bin
$ qemu-system-x86_64 -fda hello.bin

Getting the hang of it?

IO Loop

The specific function we called above to write a character to the display is INT 10,E. The 0x10 is the argument that you call the int keyword with (e.g. int 0x10). And the E is the specific function within the 0x10 family. The E is written into the AH register before calling int. The ASCII code to be written is placed in the AL register.

Now that output makes some sense, let's do input. In the keyboard services documentation you may notice that INT 16,0 provides a way to block for user input. According to that page the ASCII character will be in AL when the interrupt returns.

Clearing the screen

You may have noticed some text gets displayed before our program runs. We can use INT 0x10,0 to clear the screen.

        ;; Clear screen
        mov ah, 0x00
        mov al, 0x03
        int 0x10

All together

Since the display function reads from the same register the input function outputs to, we can just call both interrupts after each other. Wrap this in a loop and we have the world's worst editor.

$ cat ioloop.asm
bits 16
org 0x7c00

main:
        ;; Clear screen
        mov ah, 0x00
        mov al, 0x03
        int 0x10

.loop:
        ;; Read character
        mov ah, 0
        int 0x16

        ;; Print character
        mov ah, 0x0e
        int 0x10

        jmp .loop

times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroes
dw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

By the way, the main label here (like the boot label above in hello.asm) is only to help the reader. It is not something the BIOS uses.

Now that we've got the code, let's run it!

$ nasm -f bin ioloop.asm -o ioloop.bin
$ qemu-system-x86_64 -fda ioloop.bin

Digression on abstraction

There are two ways to build abstractions: assembly functions and nasm macros.

We could build a clear screen function like this:

clear_screen:
        ;; Clear screen
        mov ah, 0x00
        mov al, 0x03
        int 0x10
        ret

And then we can call this in the ioloop program like so:

bits 16
org 0x7c00

jmp main

clear_screen:
        ;; Clear screen
        mov ah, 0x00
        mov al, 0x03
        int 0x10
        ret

main:
        call clear_screen

.loop:
        ;; Read character
        mov ah, 0
        int 0x16

        ;; Print character
        mov ah, 0x0e
        int 0x10

        jmp .loop

times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroes
dw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

On the other hand if you do it in a macro:

bits 16
org 0x7c00

jmp main

%macro cls 0 ; Zero is the number of arguments
        mov ah, 0x00
        mov al, 0x03
        int 0x10
%endmacro

main:
        cls

.loop:
        ;; Read character
        mov ah, 0
        int 0x16

        ;; Print character
        mov ah, 0x0e
        int 0x10

        jmp .loop

times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroes
dw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

And nasm macros even have a way to write macro-safe labels by prefixing them with %% which is useful if you have conditions or loops within a macro.

The benefit of a macro I guess is that you're not using up the stack. The benefit of a function call is that you're not duplicating code every place you use a macro. The amount of generated code eventually becomes important in bootloaders because the code must fit into 512 bytes.

I lean more toward using macros in this code.

Complex input

Reading ASCII characters is not complicated as we saw above. But what if we want to build Readline style shortcuts like ctrl-a for jumping to the start of the line?

Using INT 16,0 as we do above is fine. But rather than solely reading from the result of that function call, there is a section of memory that contains both the character pressed and control characters pressed.

Based on documentation for this memory area (found here or here), we can build a macro for reading the pressed character:

%macro mov_read_character_into 1
  mov eax, [0x041a]
  add eax, 0x03fe   ; Offset from 0x0400 - sizeof(uint16) (since head points to next free slot, not last/current slot)
  and eax, 0xFFFF

  mov %1, [eax]
  and %1, 0xFF
%endmacro

And another macro for reading the pressed control character (if any):

%macro mov_read_ctrl_flag_into 1
  mov %1, [0x0417]
  and %1, 0x04    ; Grab 3rd bit: %1 & 0b0100
%endmacro

Cursor location

Lastly we'll use some cursor APIs that allow us to handle newlines, backspace on the first column of a line, and ctrl-a (jump to beginning of line).

%macro get_position 0
  mov ah, 0x03
  int 0x10
%endmacro

%macro set_position 0
  mov ah, 0x02
  int 0x10
%endmacro

But there's something buggy about my goto_end_of_line function. Sometimes it works and sometimes it just jumps all over the screen in an infinite loop. Part of the problem is that the editor memory is the video card. The cursor location is only stored there and not in some program state like you might do in a high-level environment/language.

goto_end_of_line:
;; Get current character
  mov ah, 0x08
  int 0x10

;; Iterate until the character is null
  cmp al, 0
  jz .done

  inc dl
  set_position
  jmp goto_end_of_line

.done:
  ret

Alright, let's put all these pieces together.

Editor with keyboard shortcuts

Start with the basics in editor.asm.

; -*- mode: nasm;-*-

bits 16
org 0x7c00

  jmp main

Then add a clear screen macro.

%macro cls 0
  mov ah, 0x00
  mov al, 0x03
  int 0x10
%endmacro

Add macros for reading and printing.

%macro read_character 0
  ;; Read character
  mov ah, 0
  int 0x16
%endmacro

%macro print_character 1
  mov ax, %1
  mov ah, 0x0e
  int 0x10
%endmacro

Add cursor utilities.

%macro get_position 0
  mov ah, 0x03
  int 0x10
%endmacro

%macro set_position 0
  mov ah, 0x02
  int 0x10
%endmacro

goto_end_of_line:
;; Get current character
  mov ah, 0x08
  int 0x10

;; Iterate until the character is null
  cmp al, 0
  jz .done

  inc dl
  set_position
  jmp goto_end_of_line

.done:
  ret

And keyboard utilities.

%macro mov_read_ctrl_flag_into 1
  mov %1, [0x0417]
  and %1, 0x04    ; Grab 3rd bit: %1 & 0b0100
%endmacro

%macro mov_read_character_into 1
  mov eax, [0x041a]
  add eax, 0x03fe   ; Offset from 0x0400 - sizeof(uint16) (since head points to next free slot, not last/current slot)
  and eax, 0xFFFF

  mov %1, [eax]
  and %1, 0xFF
%endmacro

Now we can start the editor loop where we wait for a keypress and handle it.

editor_action:
  read_character

Don't print ASCII garbage if the key pressed is an arrow key. Just do nothing. (This isn't good editor behavior in general but ours is a limited one.)

;; Ignore arrow keys
  cmp ah, 0x4b    ; Left
  jz .done
  cmp ah, 0x50    ; Down
  jz .done
  cmp ah, 0x4d    ; Right
  jz .done
  cmp ah, 0x48    ; Up
  jz .done

Next handle backspace.

;; Handle backspace
  cmp al, 0x08
  jz .is_backspace

  cmp al, 0x7F    ; For mac keyboards
  jnz .done_backspace

.is_backspace:

  get_position

If this key is pressed at the first line and the first column, do nothing.

;; Handle 0,0 coordinate (do nothing)
  mov al, dh
  add al, dl
  jz .overwrite_character

Otherwise if backspace is pressed not at the beginning of the line, just overwrite the last character with the ASCII 0 (the code 0 not the digit 0).

  cmp dl, 0
  jz .backspace_at_start_of_line
  dec dl      ; Decrement column
  set_position
  jmp .overwrite_character

Otherwise you're at the beginning of the line and you need to jump to the end of the previous line.

.backspace_at_start_of_line:
  dec dh      ; Decrement row
  set_position

  call goto_end_of_line

.overwrite_character:
  mov al, 0
  mov ah, 0x0a
  int 0x10

  jmp .done

.done_backspace:

Next we handle the Enter key. This should move the cursor onto the next line and set the column back to zero.

;; Handle enter
  mov_read_character_into ax
  cmp al, 0x0d
  jnz .done_enter

  get_position
  inc dh      ; Increment line
  mov dl, 0     ; Reset column
  set_position

  jmp .done

.done_enter:

Next we handle ctrl-a, jump to start of line.

;; Handle ctrl- shortcuts

;; Check ctrl key
  mov_read_ctrl_flag_into ax
    jz .ctrl_not_set

;; Handle ctrl-a shortcut
  mov_read_character_into ax
  cmp al, 1     ; For some reason with ctlr, these are offset from a-z
  jnz .not_ctrl_a

;; Reset column
    mov dl, 0
    set_position

    jmp .done

.not_ctrl_a:

For ctrl-e, jump to the end of the line.

;; Handle ctrl-e shortcut
  mov_read_character_into ax
  cmp al, 5
  jnz .not_ctrl_e

  call goto_end_of_line
  jmp .done

.not_ctrl_e:
  jmp .done

.ctrl_not_set:

Finally if none of these cases are met, just print the pressed character and return.

  mov_read_character_into ax
  print_character ax

.done:
  ret

Finally, create the main function that calls this editor code in a loop.

main:
  cls

.loop:
  call editor_action
  jmp .loop

times 510 - ($-$$) db 0 ; pad remaining 510 bytes with zeroes
dw 0xaa55 ; magic bootloader magic - marks this 512 byte sector bootable!

And we're done! Try it out:

$ nasm -f bin editor.asm -o editor.bin
$ qemu-system-x86_64 -fda editor.bin

Tedious and buggy! But I learned something, I think.

I wrote a new post on my first time exploring bootloader basics! Neat to discover the BIOS APIs and spend some time actually coding in assembly versus just generating or emulating it.https://t.co/7iP6Nib620 pic.twitter.com/xSyG1IXgEB
— Phil Eaton (@phil_eaton) January 23, 2022

Constant-time Fibonacci

Speculative Branches — Sat, 22 Jan 2022 02:00:00 GMT

This is the second part in a 2-part series on the "Fibonacci" interview problem. We are building off of a previous post, so take a look at Part I if you haven't seen it.

Previously, we examined the problem and constructed a logarithmic-time solution based on computing the power of a matrix. Now we will derive a constant time solution using some more linear algebra. If you had trouble with the linear algebra in part I, it may help to read up on matrices, matrix multiplicaiton, and special matrix operations (specifically determinants and inverses) before moving on.

2022–01–22: Pinephone Keyboard – kernel driver merged

megi's PinePhone Development Log — Sat, 22 Jan 2022 01:00:00 GMT

Optimal coverage for Wordle with Monte Carlo methods - Part III

Bytepawn - Marton Trencseni — Sat, 22 Jan 2022 01:00:00 GMT

I present a simple Monte Carlo solution which finds a 25-letter-unique Worlde wordlist in about 10 minutes.

Optimal coverage for Wordle with Monte Carlo methods - Part II

Bytepawn - Marton Trencseni — Fri, 21 Jan 2022 01:00:00 GMT

I improve on the previous brute-force Monte Carlo approach for attacking the Wordle coverage problem.

Science Is Not Something You Listen To

I Am Not a Robot — Wed, 19 Jan 2022 19:39:32 GMT

Over the past couple of years, “listen to science” has been present in countless tweets and media soundbites.

Optimal coverage for Wordle with Monte Carlo methods - Part I

Bytepawn - Marton Trencseni — Wed, 19 Jan 2022 01:00:00 GMT

I show a simple brute-force Monte Carlo approach for attacking the Wordle coverage problem.

Discovery and Design Considerations

Weblog on marginalia.nu — Tue, 18 Jan 2022 02:00:00 GMT

It’s been a productive several weeks. I’ve got the feature pulling updates from RSS working, as mentioned earlier.

I’ve spent the last weeks designing the search engine’s web design, and did the MEMEX too for good measure.

It needed to be done as the blog theme that previously made the foundation for the design off had several problems, including loading a bunch of unnecessary fonts, and not using the screen space of desktop browsers well at all.

Fix a Pothole

Tilting at Windmills — Tue, 18 Jan 2022 02:00:00 GMT

The goal of media is to make every problem, your problem. — Naval Ravikant Today we hear about every problem around the world as it happens…

Vim tip 3: autocomplete words and lines in Insert mode

learnbyexample — Tue, 18 Jan 2022 02:00:00 GMT

Autocomplete word

Ctrl+p autocomplete word based on matching words in the backward direction
Ctrl+n autocomplete word based on matching words in the forward direction

If more than one word matches, they are displayed using a popup menu. You can use ↑/↓ arrow keys or Ctrl+p/Ctrl+n to move through this list.

With multiple matches, you'll notice that the first match is automatically inserted and moving through the list doesn't change the text that was inserted. You'll have to press Ctrl+y or Enter key to choose a different completion text. If you were satisfied with the first match, typing any character will make the popup menu disappear and insert whatever character you had typed. Or press Esc to select the first match and go to Normal mode.

Autocomplete line

Ctrl+x followed by Ctrl+l autocomplete line based on matching lines in the backward direction

If more than one line matches, they are displayed using a popup menu. You can use ↑/↓ arrow keys or Ctrl+p/Ctrl+n to move through this list. You can also use Ctrl+l to move up the list.

Autocomplete assist

Ctrl+e cancels autocomplete
- you'll retain the text you had typed before invoking autocomplete
Ctrl+y or Enter change the autocompletion text to the currently selected item from the popup menu

See :h ins-completion for more details and other autocomplete features.

Video demo:

Beyond Faster JSON Support for iOS/macOS, Part 9: CSV and SQLite

metablog — Sun, 16 Jan 2022 17:43:41 GMT

When looking at the MPWPlistStreaming protocol that I've been using for my JSON parsing series, one thing that was probably noticeable is that it isn't particularly JSON-focused. In fact, it wasn't even initially designed for parsing, but for generating.

So could we use this for other de-serialization tasks? Glad you asked!

CSV parsing

One of the examples in my performance book involves parsing Comma Separated Values quickly, within the context of getting the time to convert a 139Mb GTFS file to something usable on the phone down from 20 minutes using using CoreData/SQLite to slightly less than a second using custom in-memory data structures that are also several orders of magnitude faster to query on-device.

The original project's CVS parser took around 18 seconds, which wasn't a significant part of the 20 minutes, but when the rest only took a couple of hundred milliseconds, it was time to make that part faster as well. The result, slightly generalized, is MPWDelimitedTable ( .h .m ).

The basic interface is block-based, with the block being called for every row in the table, called with a dictionary composed of the header row as keys and the contents of the row as values.


-(void)do:(void(^)(NSDictionary* theDict, int anIndex))block;

Adapting this to the MPWPlistStreaming protocol is straightforward:


-(void)writeOnBuilder:(id )builder
{
    [builder beginArray];
    [self do:^(NSDictionary* theDict, int anIndex){
        [builder beginDictionary];
        for (NSString *key in self.headerKeys) {
            [builder writeObject:theDict[key] forKey:key];
        }
        [builder endDictionary];
    }];
    [builder endArray];
}

This is a quick-and-dirty implementation based on the existing API that is clearly sub-optimal: the API we call first constructs a dictionary from the row and the header keys and then we iterate over it. However, it works with our existing set of builders and doesn't build an in-memory representation of the entire CSV.

It will also be relatively straightforward to invert this API usage, modifying the low-level API to use MPWPlistStreaming and then creating a higher-level block- and dictionay-based API on top of that, in a way that will also work with other MPWPlistStreaming clients.

SQLite

Another tabular data format is SQL data bases. On macOS/iOS, one very common database is SQLite, usually accessed via CoreData or the excellent and much more light-weight fmdb.

Having used fmdb myself before, and bing quite delighted with it, my first impulse was to write a MPWPlistStreaming adapter for it, but after looking at the code a bit more closely, it seemed that it was doing quite a bit that I would not need for MPWPlistStreaming.

I also think I saw the same trade-off between a convenient and slow convenience based on NSDictionary and a much more complex but potentially faster API based on pulling individual type values.

So Instead I decided to try and do something ultra simple that sits directly on top of the SQLite C-API, and the implementation is really quite simple and compact:


@interface MPWStreamQLite()

@property (nonatomic, strong) NSString *databasePath;

@end

@implementation MPWStreamQLite
{
    sqlite3 *db;
}

-(instancetype)initWithPath:(NSString*)newpath
{
    self=[super init];
    self.databasePath = newpath;
    return self;
}

-(int)exec:(NSString*)sql
{
    sqlite3_stmt *res;
    int rc = sqlite3_prepare_v2(db, [sql UTF8String], -1, &res, 0);
    @autoreleasepool {
        [self.builder beginArray];
        int step;
        int numCols=sqlite3_column_count(res);
        NSString* keys[numCols];
        for (int i=0;i < numCols; i++) {
            keys[i]=@(sqlite3_column_name(res, i));
        }
        while ( SQLITE_ROW == (step = sqlite3_step(res))) {
            @autoreleasepool {
                [self.builder beginDictionary];
                for (int i=0; i < numCols; i++) {
                    const char *text=(const char*)sqlite3_column_text(res, i);
                    if (text) {
                        [self.builder writeObject:@(text) forKey:keys[i]];
                    }
                }
                [self.builder endDictionary];
            }
        }
        sqlite3_finalize(res);
        [self.builder endArray];
    }
    return rc;
}

-(int)open
{
    return sqlite3_open([self.databasePath UTF8String], &db);
}

-(void)close
{
    if (db) {
        sqlite3_close(db);
        db=NULL;
    }
}

Of course, this doesn't do a lot, chiefly it only reads, no updates, inserts or deletes. However, the code is striking in its brevity and simplicity, while at the same time being both convenient and fast, though with still some room for improvement.

In my experience, you tend to not get all three of these properties at the same time: code that is simple and convenient tends to be slow, code that is convenient and fast tends to be rather tricky and code that's simple and fast tends to be inconvenient to use.

How easy to use is it? The following code turns a table into an array of dictionaries:


#import <MPWFoundation/MPWFoundation.h>

int main(int argc, char* argv[]) {
   MPWStreamQLite *db=[[MPWStreamQLite alloc] initWithPath:@"chinook.db"];
   db.builder = [MPWPListBuilder new];
   if( [db open] == 0 ) {
       [db exec:@"select * from artists;"];
       NSLog(@"results: %@",[db.builder result]);
       [db close];
   } else {
       NSLog(@"Can't open database: %s\n", [db error]);
   }
   return(0);
}

This is pretty good, but probably roughly par for the course for returning a generic data structure such as array of dictionaries, which is not going to be particularly efficient. (One of my first clues that CoreData's predecessor EOF wasn't particularly fast was when I read that fetching raw dictionaries was an optimization, much faster than fetching objects.)

What if we want to get objects instead? Easy, just replace the MPWPListBuilder with an MPWObjectBuilder, parametrized with the class to create. Well, and define the class, but presumably you already havee that if the task is to convert to objects of that class. And it cold obviously also be automated.



#import <MPWFoundation/MPWFoundation.h>

@interface Artist : NSObject { }

@property (assign) long ArtistId;
@property (nonatomic,strong) NSString *Name;

@end

@implementation Artist

-(NSString*)description
{
	return [NSString stringWithFormat:@"<%@:%p id: %ld name: %@>",[self class],self,self.ArtistId,self.Name];
}

@end

int main(int argc, char* argv[]) {
   MPWStreamQLite *db=[[MPWStreamQLite alloc] initWithPath:@"chinook.db"];
   db.builder = [[MPWObjectBuilder alloc] initWithClass:[Artist class]];
   if( [db open] == 0) {
       [db exec:@"select * from artists"];
       NSLog(@"results: %@",[db.builder result]);
       [db close];
   } else {
       NSLog(@"Can't open database: %s\n", [db error]);
   }
   return(0);
}

Note that this does not generate a plist representation as an intermediate step, it goes straight from database result sets to objects. The generic intermediate "format" is the MPWPlistStreaming protocol, which is a dematerialized representation, both plist and objects are peers.

Pseudonymous

Weblog on marginalia.nu — Sat, 15 Jan 2022 02:00:00 GMT

A person might think I’m illusive, writing and working under a pseudonym. It’s not that I’m hiding, if you send me an email, I’ll respond to you with an email address containing a decent chunk of my real name. It’s not out of shame I wear clothes.

Besides bringing utility, marginalia.nu is an experiment, a bit of an art project, a place to challenge conventions and see what is and isn’t necessary.

Less-than-linear Fibonacci

Speculative Branches — Fri, 14 Jan 2022 02:00:00 GMT

Few interview problems are as notorious as the "Fibonacci" interview question. At first glance, it seems good: Most people know something about the problem, and there are several clever ways to achieve a linear time solution. Usually, in interviews, the linear time solution is the expected solution. However, the Fibonacci problem is unique among interview problems in that the expected solution is not the optimal solution. There is an $O(1)$ solution, and to get there, we need a little bit of linear algebra.

dsq: Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

Notes on software development — Tue, 11 Jan 2022 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Analyzing large JSON files via partial JSON parsing

Notes on software development — Thu, 06 Jan 2022 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Probabilistic spin glass - Part V

Bytepawn - Marton Trencseni — Thu, 06 Jan 2022 01:00:00 GMT

I use Monte Carlo simulations to explore the dynamic behaviour of probabilistic spin glasses, specifically how saturation scales with $p$ and $N$.

The year in books: 11 to recommend in 2021

Notes on software development — Wed, 05 Jan 2022 02:00:00 GMT

Last year (2021) I finished 17 books, a five year low. But that's ok! 4 fiction and 13 non-fiction. Another 30 started but not finished.

Non-fiction

It seems I was pretty focused on business history books and history of tech. The 8 non-fiction books I liked the most:

Designing Data-Intensive Applications, a must-read for anyone interacting with a database
My Years with General Motors, the business school classic; truly a good read. But sad to know that shortly after written, GM succumbs to the Japanese and South Korean competition
No Rules Rules: Netflix and the Culture of Reinvention
Working Backwards: Insights, Stories, and Secrets from Inside Amazon
Working in Public: The Making and Maintenance of Open Source Software, my review here
The Intel Trinity, an early history of Intel
The HP Way
Play Nice But Win, the story of Dell computers
West with the Night, beautiful memoir recommended by Ernest Hemingway and written in a similar style. Much more enjoyable than the other more popular colonial-African memoir, Out of Africa.

The rest

Fiction

The 3 fiction books I liked the most:

A Very British Coup, hilarious and depressing. A great companion to the TV show "Yes, Minister"
Mort, Terry Pratchett is a very funny author
Selected Stories of Philip K Dick, depressing and dystopian but very well written. I would not read again because it's too depressing

The rest

There and NEVER, EVER BACK AGAIN: A Dark Lord's Diary, I was looking for more parodies like Bored of the Rings (which itself wasn't great). This was worse

2022

This year I'm interested in continuing to find good business books and good books on the history of tech. I'm also getting into more American history to make up for all the years of not paying attention in high school.

I'm continuing to try to find good memoirs and fiction by non-English authors.

Starting the blog-year off gently with my recap of 2021 in books.

I spent too much time watching TV and trying new video games to keep up with past years 😅https://t.co/5mfXbBnihk pic.twitter.com/ZHmPsUcr3g
— Phil Eaton (@phil_eaton) January 5, 2022

Dark

Weblog on marginalia.nu — Sun, 02 Jan 2022 02:00:00 GMT

As is often the case these dark winter seasons, I’ve fallen into a bit of a funk. Inspiration it seems is as rare as sunlight, and sunlight is scarce indeed in the winters of the north.

I do know what is missing, novelty. I’ve fallen into consuming “content”. Infinite scroll is the torture rack of the spirit. What is necessary doing new things and seeing new inspiring sights, exposing myself to new inspiring thoughts.

Looking ahead to 2022

Ben Overmyer's Site — Sat, 01 Jan 2022 02:00:00 GMT

2022 promises to be a year of big changes. We're moving to North Carolina sometime in the first quarter. I'm planning on leaning hard into biking and beach culture, which should have the side effect of helping me become fit again. I want to try surfing and snorkeling and other such things.

For Iron Arachne, I plan to work on enhancing existing generators over adding more. In particular, I want to make them more customizable and allow better export options. The culture, heraldry, and region generators will see the bulk of the changes here. I will probably try again to write a map generator to go with the region generator. Though I have yet to understand the math behind it, I keep trying. Eventually I'll get it.

For fitness, my strategy is fairly simple. I'll spend less of my free time in front of a computer screen and more on running around outside. I hope to make a habit of logging off after work and going for a run or something. On weekends I'll need a different pivot point; maybe before breakfast I'll go out. That I'm less sure on. My doctor continues to recommend cholesterol medication, so I might finally take that advice.

I want to rejoin the Society for Creative Anachronism in 2022. I'll be in a brand new area completely devoid of past drama. It'll be a good opportunity to reconnect with more physical hobbies I used to have. I already have a device in review at the society level; hopefully it's accepted, because it's perfect for a cook who likes the beach.

Final Fantasy XIV continues to hold my interest, which is rare for an MMO. My free company, the Mystic Exiles, is a great group of people. There are a couple individuals in particular that I have fun playing with. Maybe someday I'll get to meet them all in person. I got the FF14 cookbook for Christmas, and I plan on making several dishes out of it this year. Maybe I'll write about my experiences with it here. Maybe I'll write more about what I do in FF14 in general, actually. That could be fun.

For my 40th birthday this year we're going to Hawai'i. I'm really looking forward to that trip. It's only a couple months away now. It gives me another incentive to rock the fitness thing early and hard. It's also possible that we'll be going to Italy with my family this year. Sarah and I are learning Italian, though she's much better about the learning habit than I am. I need to devote more time to it.

Assuming we're able to rent a house with no shared walls or at least thick walls, I also want to get back into playing musical instruments. I have a dizi flute that I've only just started to learn how to play. If the stars align, maybe I'll get a piano and start playing that again too. For now though, I'm only committing to learning the dizi.

All of the above makes for a pretty action-packed 2022. Hopefully nothing crazy happens that throws all of this out the window. If it does, well, my main goal will still be improving fitness and health.

Hacker News Personal Blogs 2022 | Top 100 Blogs

Energy, Sustainability and Deployment Frequency

Deployment / Delivery Frequency

2022 Year in Review

40th birthday trip to Hawai'i

Moving to Wilmington, North Carolina

Iron Arachne

Updates to my personal website

Giving up alcohol

From individual contributor to manager to tech lead

AWS Re:invent 2022

End of my boycott of Blizzard

Silver Gryphon Games comes back

Summary

Ranking #1 on HN in the December Dead Zone

2022: year in perspective

Books published🔗

Workshops🔗

Blog posts🔗

Tools🔗

Youtube🔗

Book sales🔗

Newsletter🔗

Building Twitter audience🔗

Fictional reading🔗

Goals for 2023🔗

Trouble with TypeScript Timeouts: How to Fix It

Explore the fediverse, but use block like it's a machine gun in a zombie apocalypse

Python tip 21: sorting iterables based on a key

Creepy Website Similarity

Crypto I: On cryptocurrencies, explained using FTX Tokens

On Wizards and Sorcerers

Old School Baroque

Announcing learnbyexample weekly newsletter

The Indie Python Extravaganza

Bundle contents🔗

Authors🔗

How can you help?🔗

Automating Excel with Python - book review

Book details🔗

Review🔗

Table of Contents🔗

Feedback and Reviews🔗

2021 was a wild ride

Books published🔗

Workshops🔗

Blog posts🔗

Book sales🔗

Newsletter🔗

Building Twitter audience🔗

Fictional reading🔗

Goals for 2022🔗

Finding similar API functions between Pytorch and Tensorflow with Doc2Vec

Vim tip 19: working with buffers

Writing as a Programmer

Testing GPT-3 spatial reasoning and comprehension

Markdown Tables on the Fedivee

Make the world small again

Media Services

Search Engines and Discovery

Social Media

Summary: Solutions for Each

Fred Brooks' The Mythical Man-Month

Reintroducing Opel: Put All Your Pelican Posts in One Org File

Motivation

CLI tip 20: expand and unexpand

How good an astrophysicist is GPT-3?

Similar posts recommendation with Doc2Vec - Part III

Building TUIs with textual: first impressions

Installation and Tutorials🔗

Building Square Tic Tac Toe board game🔗

What I liked🔗

What gave me trouble🔗

Next steps🔗

Strong Opinions & Eroding Discourse

Rendering LaTeX Formulae in mastodon.el

Python tip 20: saving and loading json

Abstraction is Expensive

First Day in Dragonflight

Similar posts recommendation with Doc2Vec - Part II

Types (`types.zig`, 10 LoC)

Lexing (`lex.zig`, 308 LoC)

`lex`

Parsing (`parse.zig`, 407 LoC)

`SELECT`

`CREATE TABLE`

`INSERT INTO`

`AST`

Storage (`storage.zig`, 338 LoC)

Execute (`execute.zig`, 210 LoC)