Hacker News Personal Blogs 2021 | All Blog Posts

Big Tech, so bad that even Big Tech dislikes it

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 31 Dec 2021 19:00:00 GMT

We truly live in interesting times.

jumping through hoops

Gingerlime — Fri, 31 Dec 2021 14:00:10 GMT

One recent pet peeve of mine has been how companies make you jump through hoops just to report an issue to support. Trello is a recent one. I used to be able to just email support@trello.com … Not anymore… You’ve emailed support@trello.com, which is no longer available for support. That’s great! And the best part?…

The post jumping through hoops appeared first on Gingerlime.

Three Good Books I Read in 2021

Nicola Iarocci — Fri, 31 Dec 2021 08:05:25 GMT

This year I’ve read twenty-one books or 5903 pages. That’s fewer books than last year (28 / 8064), the year before (25 / 8394), and the one before that (30 / 8447). Heck, I must look back at 2014 to score a win in my very own yearly reading challenge. What surprises me is not much the number of books but the pages I read, which constitutes a more relevant metric. Over the previous three years, that number stayed firmly over 8k while dropping to 6k this year.

My macOS programs and setup

Max Leiter — Fri, 31 Dec 2021 02:00:00 GMT

In the off-chance you're reading this today, happy new year! If you're reading this later, I hope $CURRENT_YEAR is treating you well.

Before this post, I kept a list of my programs in my iCloud in case my computer broke or I was setting up a work laptop. After some recent events, I trust iCloud less than I once did, and figured the list can live here instead. Like the rest of my blog, this exists for me but hopefully you find it useful too. I plan to write an Arch Linux version of this soon.

I try to use open-source where I can. If you know of an alternative to the few closed applications I use, please let me know!

Firefox Nightly
- Chrome for testing sites
VisualStudio Code
iTerm 2
BetterTouchTool allows advanced customization of the touchbar ($9 after 45 days)
- GoldenChaos preset
Homebrew is a package manager
- Handles installing Xcode developer tools
Rectangle lets you resize and snap windows with your keyboard
Fish is user-friendly shell (seriously, I hate zsh/bash now)
- Fisher: Fish plugin manager
- fish-nvm: nvm wrapper
NextDNS is a secure DNS service (I downloaded the macOS client from the App Store)
LastPass is my password manager, but I'm currently looking to start self-hosting.
Insomnia is my favorite REST client
IINA is a nice media player
VIA Configurator configures my keyboard
Lagrange for viewing Gemini articles
yarn is my node package manager of choice
MySQL Workbench for managing my few MySQL servers
TeXMaker for the rare times I need to write LaTeX
I'm going to try and use the Slack and Zoom browser versions
Qemu: `brew install qemu`
thefuck: `brew install thefuck`, command line correction tool
- Add `thefuck --alias | source` to `~/.config/fish/config.fish`
imagemagick: `brew install imagemagick`
ffmpeg: `brew install ffmpeg`

2021 status report

arnorhs blog - arnorhs.dev — Fri, 31 Dec 2021 02:00:00 GMT

What excites me most coming into 2022. Solid.js, Astro, Next.js, Remix, Netlify, Cloudflare workers, trpc, magic.link, planetscale, my npm packages and more.

Probabilistic spin glass - Part IV

Bytepawn - Marton Trencseni — Fri, 31 Dec 2021 01:00:00 GMT

This is a continuation of the previous articles on probabilistic spin glasses. I run simulations to understand the scaling behaviour for large spin glasses.

Certain biometric troubles in India...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 30 Dec 2021 19:00:00 GMT

Are nothing new, really.

Taming My Soda Addiction: What Worked and What Didn’t

Beetle Space — Thu, 30 Dec 2021 10:00:00 GMT

Here I’ll discuss various ways I’ve tried to reduce my soda intake - primarily to avoid sugar. [1] [2]

If you want the quick summary of what worked (in no order):

Working from home (no free soda)
Switching to Spindrift and Izze (more of the former)
Reducing portion size …

Learnings from patching a bug in a modern project

Greg Herlein — Thu, 30 Dec 2021 09:00:00 GMT

I’m continuing to dive deep on modern typescript/node development. I fixed a bug in a new tool I am using (projen) and learned a ton along the way. This post is to share some of that.

The Rites of Passage

Godsip Club — Thu, 30 Dec 2021 02:00:00 GMT

Flea Markets = Source of Treasures #

A few weeks ago, at the local flea market, I was very lucky to find a copy of The Rites of Passage by Arnold Van Gennep. It is an Italian edition from 1960s and it belonged to the library of a nearby village. It shows the signs of time, but for the price (about 1€) I couldn’t really complain! I immediately started studying it and I was glad to find, since the first pages, references to The Golden Bough, which I finished previously!

It may be not in a perfect state, but for me it’s glorious!

I’ll try to (badly) recap some of the most intriguing theories exposed here. Let’s remember that the given examples are general, so similarities can be observed with both our modern society and remote past civilisations around the globe.

First, the life of a human being is a series of stages: childhood, puberty, married life , etc. Also, just as important, throughout life, there are communities which a person belongs to. They can be categorised. The most basic partition is sexual-based: men and women – it’s too well-known that they have never been put on the same level. The other clear separation in everyday actions is based on magical-religious aspects: sacred and profane. It may seem subtle, but the distinction is very used even nowadays.

Similarities can be observed with both our modern society and even remote past civilisations around the globe.

In addition to the previous distinctions, imagine that a civilisation is composed essentially by classes: based on professions, castes, religious associations,… We could find so many examples. Moreover, you can picture the sub-classes that we could find, for example, if we divide them geographically (e.g., local parishes).

IT. IS. A. MESS.

‘I Don’t Belong Here’ – T. Yorke #

Ok, it’s like Monopoly: before starting, we have set many pieces on the table, but it gets easier once the game begins. The golden rule is that an individual cannot be part of different classes¹. So, in order to change group, a rite (or ceremony) must take place. That’s the focal point and goal of the book: to understand the sequence and, when possible, the meaning of the rituals that are performed in such occasions.

Another powerful picture is given in the very first pages: societies are like buildings with many rooms, one for each “class”. In the primitive civilisations, the walls of the rooms were thicker, so the communication among them was more difficult. In modern society the walls are thinner, so the distinctions are less pronounced. In other words, less rites are required to change class. Progress, we could say. Very true indeed, if it weren’t for some very archaic remnants part of our inner culture.

As other anthropologists did before, Van Gennep pointed out that similarities could be found in ceremonies dedicated to completely different situations: for instance, weddings, funerals, and coronations of kings. What is a common point among them?

The veil was used for many types of celebrations, not just for weddings. It acted as a protection.

It was common, since Roman times, to cover the spouses with a veil. The modern bride still wears it while she walks towards the altar. Why, though? It is to protect her holiness from the profane world that she comes from. The wedding is a change of state to a higher level, in which the woman will be seen with more attributes. The same goes for a king, which wears a cloak during the enthroning ceremony. And the same principle is valid for death: corpses were often covered in sheets because the place where the soul is headed is sacred – its journey must not be impeded by our “filthy” world.

Separation - Transition - Incorporation #

So, if the rites of passage are the ones required to pass from a class to another (or even from a world to a distinct one), this three-step sequence is their key:

Separation rites (from the previous reality)
Transition rites
Incorporation rites (to the new reality)

It may seem absurd to force such classification for all the ceremonies, but the results of Van Gennep’s analysis are quite stunning.

One very important concept is the threshold. Our home’s door, for example: it protects us from the outside world. In the past, both Arabs and Jews stroked an effigy or a casket near the door when entering or exiting the house – an incorporation or separation gesture, depending on the case. Also, in some parts of the world, pregnant women can’t enter through the main door of their houses². Even bodies, in funeral rites, are not let pass through the main door because it is seen as a bad omen.

Another great example of this concept, often underrated, is the triumphal arch: in Roman times, armies and important personalities passed under the arch to enter symbolically and physically into the new reality and to “wash away” the contamination of the outside³. The simplest threshold can be seen as a barrier between worlds: among many civilisations, it is spread the belief that spirits cannot cross water courses, which in this case act like some other type of boundaries.

In ancient times, swimmers used to mock spirits that couldn’t follow them into the water.

‘Is He Dead?’ ‘No, It’s Just Puberty’ #

Other similarities between rituals related to different moments in life can be observed analysing the initiation rites for special societies or, simply, just the ones for puberty. In a few words, the initiated boy temporarily dies to exit his previous state and enter the new one (adulthood). The example is clearer in Congo, for instance, where the kid was exiled (separation) and painted white, which is a color often related to death (margin). Other than that, when he was re-admitted in his village, he had to act like a baby for weeks, or months: adults had to teach him again the basic tasks, and he also ate baby food (incorporation).

The book is intriguing because there are many other cases illustrated and explained. One for all: circumcision. Why is it performed, and why at different ages (some cultures do it after a few weeks from birth, others after ten years)? Simple answer: it is a way to show the membership to a special class, or society. Quoting the text:

The mutilated individual is removed from the common mass of humanity by a rite of separation (this is the idea behind cutting, piercing, etc.) which automatically incorporates him into a defined group.

“Circumcision of Christ” by Giovanni Bellini. We’re with you, little fella.

Finally, probably the most obvious rite to assign a newborn to a “category”, even unintentionally, is by giving him or her a name. It’s not about using a middle name, as nowadays. In the past, a child could change name several times: first, it was named according to the sex with a generic one; then, an ancestor’s name was chosen; later, depending on the “phase”, or his age, he could change it again. In totemic societies, at the time of initiation, the name could vary once more. And even when a man becomes father, it could be changed again with his son’s.

To conclude the overview of this amazing book, which is such dense of notions yet brief, it is a must for me to spread the word that it is available for free on the Internet Archive!

Of course there are exceptions. It is just a rule to simplify the understanding. ↩︎
Nursing mothers and pregnant women have always been treated as “special”, but not always in a good way. ↩︎
Interestingly, Norse people had a similar tradition and divided their world between inside (Innangard) and outside (Utangard), and the village walls (gardr) were very important. We’ll talk about that. ↩︎

Guidance for Scaling - Reversible vs. Irreversible Decisions

CRAIG KERSTIENS — Wed, 29 Dec 2021 23:30:56 GMT

Was having a conversation with a founder earlier today and the topic of hiring functional leaders came up. I offered one of my common pieces of advice which was don’t hold the reins too tightly once you hire them. It’s something I see happen over and over to first time founders. You hire a new VP of Product and then still continue to oversee so much of the product process yourself. It is understandable, it’s your baby, you’ve spent years building it to this point, they don’t love it the same way you do.

The likely outcome is your new VP of product won’t find success. They’ll feel they’re not able to execute on a vision of their own. They’ll feel micromanaged. Even the smallest decisions they’ll feel aren’t fully theirs and get second guessed.

Perhaps you could do it better, that’s not necessarily the question. What you’re focused on is growing and scaling, this is the reason you hired them. So how do you do this without the business careening off tracks? Well first, empower them to make decisions, but beyond that there are a few things you can do so you feel more comfortable entrusting them with their functional area.

Communication is key

First, clearly communicate your priorities and thought process. This doesn’t mean tell them what to do, but priorities… Bob Iger states it as “You have to convey your priorities clearly and repeatedly. In my experience, it’s what separates great managers from the rest”. I couldn’t immediately find the reference, but recall reading that he started each week with his execs communicating his top 5 priorities across the company. This gives you a line of sight into the leadership’s thinking. This could be as simple as:

I’m worried about our pipeline for next year
I’m worried about retention and how it affects long term growth
I’m worried that we have the tech stack to scale for next 3 years

Something similar to that last one came up in the conversation, which led us down a brief conversation of reversibles and irreversibles. In product management one of the most common mistakes I see of folks is not making a decision and heading in a single direction, rather they want all the info before they make a decision which can cause as much harm as good (but I’m getting off track).

Reversible vs. Irreversible Decisions

Communicating to new execs a framework around which decisions are critical to get right and which ones can be recovered from if wrong is huge. Not all decisions are critical to get right. If ship a typo in a blog post, you can easily fix it. If you send a wrong message to a single customer, it could create a bad experience for that customer but not impact all the rest. Knowing the scope and impact of a failure is important in a framework for decision making.

We discussed that architecture and engineering decisions it’s harder to reverse than say marketing. But I’m not sure that holds true, if you have a huge PR nightmare from bad marketing it can take years to recover and heavily set back the business just like a bad architecture decision. Even within engineering decisions you can use a framework of how costly is this to commit to and how easy is it to reverse.

One of the best engineers I’ve ever worked with, for going on 15 years now states it as not foreclosing future options. Leaving optionality open in engineering is one of those 10x things. It’s not elegant code or overengineering, it’s highly immeasurable (at least to me it is), but trying to grow this is huge to enable in your functional leadership as well as more senior individual contributors.

Which decisions can later be changed and the cost of that is a great framework for how involved you should be vs. not, and the more you can guide your team through that the smoother growing the team will be.

New US Act about chipmaking is important for EVERYBODY

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 29 Dec 2021 19:00:00 GMT

Meanwhile, Europe and China…

Tree Survey using Machine Learning

Fabrice Aneche — Wed, 29 Dec 2021 15:00:00 GMT

My buddy and I bought some lands (20ha/50acres) in the Quebec province. Creating a new problem It all started because as a new “forest owner”, I wanted to do a tree survey. There are some good mobile apps to recognize flowers and trees but nothing professionally graded I knew (except paying for human expert). I also needed a solution that would work offline since the mobile network coverage is not good on the field.

I'll be home for Christmas: Quarantine Piano Edition

Joshua Rogers — Wed, 29 Dec 2021 14:00:00 GMT

This year, despite generally taking precautions and having two shots of Moderna, Christmas found me in quarantine in my home office, hiding away from my family. As far as Christmas traditions go, “physical isolation” has to be one of the more unusual ways to celebrate a holiday whose traditions frequently involve getting together in larger groups. It was Christmas day and I was alone, but it didn't really feel that way: my wife, daughter, and I spent Christmas lunch (and supper) talking via FaceTime.

Less is More: How Degrowth Will Save the World

Apurva Shukla's RSS Feed — Wed, 29 Dec 2021 13:30:55 GMT

⭐ ⭐ ⭐ ⭐ ⭐ This is one of those books that changes your theory of being. I cannot recommend this enough, this book should be on everyone’s…

Book review - Foundations of Library and Information Science

River Writes - A MediaWiki Blog — Wed, 29 Dec 2021 10:20:41 GMT

A review of “Foundations of Library and Information Science”

Using Go on iOS in 2021

Fabrice Aneche — Wed, 29 Dec 2021 07:00:00 GMT

I’ve already posted about gomobile on this blog: “Using gomobile for real”. For a lot of people it seems crazy to run Go on iOS and Android for some others it’s just routine, Zenly & Tailscale to name a few. Since I’m reviving the idea for a geo database on iPhone here are some details. I’ve started to port spatialite with the required dependencies to realize most of them were LGPL and incompatible with iOS distribution, so I went back to a simple solution using Go to display points on a map from a local database.

VR, VFIO and how latency ruined everything

./techtipsy — Wed, 29 Dec 2021 06:00:00 GMT

Alternative title: "Honey, I completely changed my server setup again!"

The Lowest Common Denominator

JonPaulUritis.com — Wed, 29 Dec 2021 02:00:00 GMT

One of the dumbest things a person can ever say is: "People are dumb."

No doubt you've heard it out there in a wild. I used to hear it all the time from sales reps, managers, ops people, etc. Now I hear it from software devs, PMs, technical writers, etc.

They even have little cartoons for it, that they send around:

Everyone seems to think that these humans that they are working on behalf of are not capable... and that things need to be "dumbed down" for them. Brought down to a easier level.

To the idea of "lowest common denominator," that peope are dumb, I call bullshit.

People are lazy, tired, and distracted. They are drunk, high, tripping, over/under caffeinated. They are are bombarded with requests for their focus/attention, their money, and their time. They have family, debt, pets, jobs, kids, coworkers, broken appliances, neigbhors, trees, and priests. They have racoons getting into their trash at night keeping them up, and robbers stealing rims in their neighborhood. They have a baby that got COVID-19 and a husband/wife with a medical condition that doctors don't understand. Their boss is getting a divorce and their kid is getting bullied at school.

The thought that a "normal person" could be some sort of subject matter expert in whatever you're selling, or that they are able to operate in your world at the drop of a hat, no matter how "easy" it is, is laughable. People are actually pretty smart, they just have a lot of things in the way of them acting like it.

Now a decade ago I probably would have said the same thing as everyone else, but then something magical happened- I interacted with those "dumb" people that my coworkers talk about. Not just a handful either. From ~2009 to 2013 I talked to somewhere in the realm of ~13,000 to ~15,000 different people. And what I found: Most people are pretty smart. They aren't "Rhodes Scholar" smart (chances are neither are you) but they're smart enough.

So whats all this business about serving the "lowest common denominator," making sure you dumb things down??? In my book it's laziness.

Here's what I'd recommend: instead of bringing the subject down to a place where even a corpse could understand you should try to bring people up to your level, or at least a place where they can operate. It isn't easy, and it's definitely a goal... but the world will be better for it. Your customers will thank you, and your users, clients, audience, whoever will thank you for it.

I'm going to leave off with a quote from John Collison (cofounder of Stripe):

"That one of our principles when it comes to the marketing at Stripe is that we speak up to the reader. You’re not trying to dumb things down for someone who isn’t familiar with something like this. You are speaking to an intelligent person who is busy, but knows what they’re talking about, knows what they’re doing. And it’s your job to kind of help educate them on this. And so we certainly made that a big part of the culture early on. And I think, I mean, like any culture, it’s self perpetuating. I think people for whom that’s important tend to be drawn to it.

4000 years

JonPaulUritis.com — Wed, 29 Dec 2021 02:00:00 GMT

My favorite Professor at KU, John Younger, was at Duke for about 25 years before coming to KU. He was their expert in classical studies (Greece, Rome, etc) and was considered a world expert. He left when they gave him a little clock to say thank you for the 25 years. For whatever reason he moved to Kansas. One time I was hanging out during office hours and he explained to me the difference between Duke and Kansas (paraphrased in my own words):

"When I was at Duke everyone was very attentive, they were all on their way to be doctors and lawyers, etc they studied hard, but never stopped to look around at all the interesting things that they could learn... at Kansas I was sitting in my office during office hours and a random kid who grew up on a farm 200 miles away from any town I've ever heard of came in to ask me some questions about an ancient dialect of greek that hasn't been spoken in ~4000 years. The kid had taught himself how to read this language sitting on a farm in the middle of Kansas- making him one of maybe 100 people in the world that could read/write in this anchient language... I could have sat at Duke for 4000 years waiting for something like that to happen and it never would have. These things happen here..."

I think about that interaction every once in a while... there's a lot in there.

Go in Web Applications with WebAssembly

Thoughts by Thomas Derflinger — Wed, 29 Dec 2021 02:00:00 GMT

With the advent of the new technology of WebAssembly, it is possible to leverage other languages than JavaScript in the browser. Go is one popular language that you can call from your JavaScript web application. This article shows how to do that with TinyGo.

Generative Art: Pen Plotting an Old Family Photo

Cameron Sun — Wed, 29 Dec 2021 02:00:00 GMT

Turning family history into art.

Un petit retour sur l'AOC 2021

ache: Blog personnel — Wed, 29 Dec 2021 02:00:00 GMT

L'Advent of Code 2021, le concept est simple. Chaque jour du calendrier de l'avent vous devez résoudre un exercice de programmation (dit des puzzles) qui comporte deux niveaux de difficulté. Tout ceci est mis en contexte puisque vous aidez les lutins du père Noël (ici des elfes, mais on France on parle de lutin !) à récupérer les clés du traîneau ... Oui le traîneau du père Noël a des clés, et un lutin les a fait tomber dans l'océan. Ne vous arrêtez pas déjà à la narration car ça va devenir encore plus improbable !

Vim tip 2: indent/unindent lines

learnbyexample — Wed, 29 Dec 2021 02:00:00 GMT

Normal mode

>> indent the current line
3>> indent the current line and two lines below (same as 2>j)
>k indent the current line and the line above (same as 1>k or >1k)
<< unindent the current line
5<< unindent the current line and four lines below (same as 4<j or <4j)
2<k unindent the current line and two lines above (same as <2k)
= auto indent code, use motion commands to indicate the portion to be indented
- =4j auto indents the current line and four lines below
- =ip auto indents the current paragraph

You can use any motion command with > and <. For example, >} indents till the end of the paragraph.

Visual mode

> indent the visually selected lines once
3> indent the visually selected lines three times
< unindent the visually selected lines once
= auto indent code

Consider the following unindented code:

for(i=1; i<5; i++)
{
for(j=i; j<10; j++)
{
statements
}
statements
}

Here's the result after applying vip= (you can also use =ip if you prefer Normal mode).

for(i=1; i<5; i++)
{
    for(j=i; j<10; j++)
    {
        statements
    }
    statements
}

Indentation depends on the shiftwidth setting.

See :h shift-left-right, :h = and :h 'shiftwidth' for documentation.

Video demo:

Chat Server Protocol

Home on mfashby.net — Wed, 29 Dec 2021 00:22:01 GMT

My favourite books of 2021

Gaganpreet Arora — Tue, 28 Dec 2021 23:58:45 GMT

This year I read more non-fiction than I usually do and I read 32 books this year. I also wrote a similar list for 2020.

Self-hosted email MUST remain "practical"

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 28 Dec 2021 19:00:00 GMT

You want your very own email server. Trust me on this.

Devlog #6: async activities

Dom Williams — Tue, 28 Dec 2021 14:00:00 GMT

An overview of many months of churning the entity activity system to use Rust's async/await support, for a much nicer design.

Announcing rp2040_hal 1.0

Jeremy Grosser — Tue, 28 Dec 2021 10:00:00 GMT

rp2040_hal is a library of drivers for the Raspberry Pi RP2040 microcontroller written in Ada.

Static HTTP share with S3 and Haskell

Anardil — Tue, 28 Dec 2021 10:00:00 GMT

Creating HTML index pages for a public S3 bucket with Haskell

Book Review: Consider the Lobster

Nicola Iarocci — Tue, 28 Dec 2021 08:05:25 GMT

I found a Consider the Lobster review on Goodreads that almost precisely matches my thoughts on DFW and the book. Hence, given the lazy Christmas-break mood I am in right now, I am conceding myself the right to copy-paste and edit David’s review right away.

I know of nobody else who writes as thoughtfully and intelligently as DFW. That he manages to write so informatively, with humor and genuine wit, on almost any subject under the sun is mind-blowing – it’s also why I am willing to forgive his occasional stylistic excesses. (Can you spell ‘footnote’?) You may not have a strong interest in lobsters or pornography, but the essays in question are terrific. The reporting on Ziegler and McCain is outstanding, heartbreakingly so, because it makes the relative shallowness of most reporting painfully evident.

useMousetrap()

Max Leiter — Tue, 28 Dec 2021 02:00:00 GMT

I've been messing around with a React implementation of The Lounge lately (for no real reason except that I can) and wanted a functional way to use Mousetrap, which TL already uses. The below hook let me replace the Vue use of Mousetrap (which was pretty much vanilla JS) in a React-like fashion. For bonus typing, add `@types/mousetrap` to your project.

Usage is pretty simple:

useMousetrap("escape", (evt, combo) => {
	evt.preventDefault
}, "keypress")

import mousetrap from "mousetrap"
import { useEffect, useRef } from "react"

const useMousetrap = (
	handlerKey: string | string[],
	handlerCallback: (evt: KeyboardEvent, combo: string) => void,
	evtType?: "keypress" | "keydown" | "keyup",
) => {
	let actionRef = useRef(handlerCallback)

	useEffect(() => {
		mousetrap.bind(
			handlerKey,
			(evt: KeyboardEvent, combo: string) => {
				typeof actionRef.current === "function" && actionRef.current(evt, combo)
			},
			evtType,
		)
		return () => {
			mousetrap.unbind(handlerKey)
		}
	}, [evtType, handlerKey])
}

export default useMousetrap

2021 Year in Review

Ben Overmyer's Site — Tue, 28 Dec 2021 02:00:00 GMT

2021 was an eventful year. The global pandemic was still a major problem, though vaccines finally showed up, and treatments started to become available at the end of the year. I quit my job at Solution Design Group after a golden opportunity presented itself, and took a 100% remote job with Apiture. Thus far, it has proven to be a fantastic job with a great group of people. The benefits are amazing.

I think I finally gave up on the idea of resurrecting Silver Gryphon Games this year. Kevin doesn't need that distraction and my tendency to change focus for months at a time would make a company with that level of activity difficult to maintain on my own. It's possible that in the future I will consider starting a company with a smaller focus, but that's not likely to happen in 2022.

Iron Arachne saw less development in the latter half of the year, but the first half was quite productive. I learned more about how concepts should be structured. In July, I rewrote the entire project in Typescript with Svelte. That was enlightening. I still don't care for Javascript generally, but Typescript at least makes it tolerable. I wish I didn't have to write the site in JS. However, the benefits of having a static site outweigh the drawbacks of writing in JS. I did end up building that planet generator I wrote about at the beginning of the year. I taught myself how to write GLSL shaders in the process.

One of the biggest events of 2021 was that I got Lasik surgery in late September. While it has been fantastic to be able to see well without glasses, I'm still experiencing issues due either to healing or to side effects. If I focus on something close up for more than a few seconds, then I'll be unable to re-focus on things farther away for about half a minute. Also, when I'm tired, my distance vision is blurry enough that it's difficult to read signs. Still, absolutely worth it.

We abandoned our idea of moving to Estonia for a variety of reasons. At first it was due to the difficulty involved in moving there with two cats. Over the course of the year, however, it became obvious that Russia's aggression in the area makes Estonia unsafe. Our new plan is much less disruptive: we're moving to North Carolina in the next couple months, and then in a few years, we'll move to Maine.

I'm in worse shape physically at the end of 2021 than I was at the beginning, I think. I'm not sure because I stopped logging progress a couple years ago. That's changing now. I got an Apple TV for Christmas, and I intend to expand my usage of Fitness+ from occasional workouts using my iPhone to daily workouts using the TV. I'll talk more about that in my "looking forward" post in a few days.

In the end, 2021 was a relatively positive year full of changes. The threat of Covid-19 still looms over everything, but I think there is light at the end of the tunnel now.

Using {Blocks} in Rust & Go for Fun & Profit

taylor.town — Tue, 28 Dec 2021 02:00:00 GMT

Did you know you can just whip out curly-braces all over the place in Rust and Go?

a := 1
a = 2

{ b := 1 }
b := 2

let mut a = 1;
a = 2;

{ let b = 1; };
let b = 2;

let c = {
  let mut d = 1;
  d = 2;
  d
};

But why on Earth would you even want to do this?

Creating "Construction-Zones"

When you're creating a variable that demands multiple initialization steps, it's tider to tuck the details into a block:

// before
let person = Person(id);
person.authenticate(token).await;
let raw_email = person.fetchEmail().await;
let email = Email.parse(email);

// after
let email = {
  let person = Person(id);
  person.authenticate(token).await;
  let raw_email = person.fetchEmail().await;
  Email.parse(email)
}

In this example, it's obvious that the "end-goal" is email. At a quick-glance, you can see which variables are important, and which are intermediate ephemera.

Containing Side-Effects and Error-Handling

Every golang programmer knows the pain of juggling result and err variables. With blocks, you can tidily handle each err and other variables without worrying about weird assignment problems with := and =:

id := 123
firstName := "John"
lastName := "Smith"

{
  result, err := db.Exec(
    `UPDATE usr SET first_name = $2 WHERE id = $1`,
    id,
    firstName,
  )
  if result.RowsAffected() = 0 || err != nil {
    return err
  }
}

{
  result, err := db.Exec(
    `UPDATE usr SET last_name = $2 WHERE id = $1`,
    id,
    lastName,
  )
  if result.RowsAffected() = 0 || err != nil {
    return err
  }
}

Restricting Variable Lifetimes

I like creating tidy "workspaces" for each group of intermediate variables:

// define vars
// do stuff with vars
{
  // define intermediate vars
  // manipulate top-level vars
}
// do more stuff with vars
{
  // define intermediate vars
  // manipulate top-level vars
}
// do even more stuff with vars

For example:

email := "j.smith@example.com"

usr := Usr{}
{
  var usrId int
  {
    err := db.QueryRow(
      `SELECT usr_id FROM usr_email WHERE email = $1`,
      email,
    ).Scan(&usrId)
    if err != nil {
      return err
    }
  }

  {
    err := db.QueryRow(
      `SELECT * FROM usr WHERE id = $1`,
      usrId,
    ).Scan(&usr)
    if err != nil {
      return err
    }
  }
}

In this very contrived example, you can easily see the purpose of everything just by glancing at the general structure:

email is at the beginning with an explicit value, so it's a given input
usr is declared empty, so the block directly beneath it will define it
usrId is declared empty, so the block directly beneath it will define it
usr and email are the only variables that we need to care about after this section

Conclusion

Use {} blocks to prevent your variables from leaking everywhere.

Your friends and coworkers will thank you.

Why JavaScript's Secret Comma Operator is So Darn Useful

taylor.town — Tue, 28 Dec 2021 02:00:00 GMT

Did you know you could do this in JS?

let n = (0, 1);

console.log(n); // 1

And this?

let n = 0;

n = (n++, n++, n);

console.log(n); // 2

And this?

// 😒
if (x) {
  foo();
  return bar();
} else {
  return "baz";
}

// 😍
return x ? (foo(), bar()) : "baz";

And this?

// 😒
const f = x => { x[0]=42; return x; };

// 😍
const g = x => (x[0]=42, x);

At first glance, this seems pretty useless outside of code golf. But I personally find it helpful for a few cases.

Quick Logging

Sometimes I want to log something deep in the code without disturbing the general structure:

// before
function f(x) {
  return x ? "left" : "right";
}

// after
function f(x) {
  return x ? (console.log("left", x), "left") : "right";
}

Without the comma-operator, we would've had to completely disassemble the ternary-statement into an equivalent if-statement.

Making Returnable If-Statements

// before
let x;
if (n > 0) {
  f();
  x = "left";
} else {
  x = "right";
}

// after
const x = n > 0 
  ? (f(), "left") 
  : "right";

It's nice that the code is smaller, but the real benefit of this example is that we don't need a mutable x variable polluting our scope! I personally find the ternary much safer and easier to read here.

A year in review - 123dev #52

Justin Garrison's Homepage — Tue, 28 Dec 2021 02:00:00 GMT

It was not an easy year for many reasons. I have hopes 2022 will be better or at

V8 Heap pwn and /dev/memes - WebOS Root LPE

David Buchanan's Blog — Tue, 28 Dec 2021 02:00:00 GMT

Writing a minimal Lua implementation with a virtual machine from scratch in Rust

Notes on software development — Tue, 28 Dec 2021 02:00:00 GMT

By the end of this guide we'll have a minimal, working implementation of a small part of Lua from scratch. It will be able to run the following program (among others):

function fib(n)
   if n < 2 then
      return n;
   end

   local n1 = fib(n-1);
   local n2 = fib(n-2);
   return n1 + n2;
end

print(fib(30));

This is my second project in Rust and only the third time I've invented an instruction set so don't take my style as gospel. However, I have found some Rust parsing tutorials overly complex so I'm hoping you'll find this one simpler.

All source code is available on Github.

Entrypoint

Running cargo init will give the boilerplate necessary. In src/main.rs we'll accept a file name from the command line, perform lexical analysis to retrieve all tokens from the file, perform grammar analysis on the tokens to retrieve a tree structure, compile the tree to a linear set of virtual machine instructions, and finally interpret the virtual machine instructions.

mod eval;
mod lex;
mod parse;

use std::env;
use std::fs;

fn main() {
    let args: Vec<String> = env::args().collect();
    let contents = fs::read_to_string(&args[1]).expect("Could not read file");

    let raw: Vec<char> = contents.chars().collect();

    let tokens = match lex::lex(&raw) {
        Ok(tokens) => tokens,
        Err(msg) => panic!("{}", msg),
    };

    let ast = match parse::parse(&raw, tokens) {
        Ok(ast) => ast,
        Err(msg) => panic!("{}", msg),
    };

    let pgrm = eval::compile(&raw, ast);

    eval::eval(pgrm);
}

Easy peasy. Now let's implement lex.

Lexical analysis

Lexical analysis drops whitespace (Lua is not whitespace sensitive) and chunks all source code characters into their smallest possible meaningful pieces like commas, numbers, identifiers, keywords, etc.

In order to have useful error messages, we'll keep track of state in the file with a Location struct that implements increment and debug.

This goes in src/lex.rs.

#[derive(Copy, Clone, Debug)]
pub struct Location {
    col: i32,
    line: i32,
    index: usize,
}

The increment function will update line and column numbers as well as the current index in the file.

impl Location {
    fn increment(&self, newline: bool) -> Location {
        if newline {
            Location {
                index: self.index + 1,
                col: 0,
                line: self.line + 1,
            }
        } else {
            Location {
                index: self.index + 1,
                col: self.col + 1,
                line: self.line,
            }
        }
    }

And the debug function will dump the current line with a pointer in text to the current column along with a message.

    pub fn debug<S: Into<String>>(&self, raw: &[char], msg: S) -> String {
        let mut line = 0;
        let mut line_str = String::new();
        // Find the whole line of original source
        for c in raw {
            if *c == '\n' {
                line += 1;

                // Done discovering line in question
                if !line_str.is_empty() {
                    break;
                }

                continue;
            }

            if self.line == line {
                line_str.push_str(&c.to_string());
            }
        }

        let space = " ".repeat(self.col as usize);
        format!("{}\n\n{}\n{}^ Near here", msg.into(), line_str, space)
    }
}

The smallest individual unit after lexical analysis is a token which is either a keyword, number, identifier, operator, or syntax. (This implementation is clearly skipping lots of real Lua syntax like strings.)

#[derive(Debug, PartialEq, Eq, Clone)]
pub enum TokenKind {
    Identifier,
    Syntax,
    Keyword,
    Number,
    Operator,
}

#[derive(Debug, Clone)]
pub struct Token {
    pub value: String,
    pub kind: TokenKind,
    pub loc: Location,
}

The top-level lex function will iterate over the file and call a lex helper for each kind of token, returning an array of all tokens on success. In between lexing it will "eat whitespace".

pub fn lex(s: &[char]) -> Result<Vec<Token>, String> {
    let mut loc = Location {
        col: 0,
        index: 0,
        line: 0,
    };
    let size = s.len();
    let mut tokens: Vec<Token> = vec![];

    let lexers = [
        lex_keyword,
        lex_identifier,
        lex_number,
        lex_syntax,
        lex_operator,
    ];
    'outer: while loc.index < size {
        loc = eat_whitespace(s, loc);
        if loc.index == size {
            break;
        }

        for lexer in lexers {
            let res = lexer(s, loc);
            if let Some((t, next_loc)) = res {
                loc = next_loc;
                tokens.push(t);
                continue 'outer;
            }
        }

        return Err(loc.debug(s, "Unrecognized character while lexing:"));
    }

    Ok(tokens)
}

Whitespace

Eating whitespace is just incrementing the location while we see a space, tab, newline, etc.

fn eat_whitespace(raw: &[char], initial_loc: Location) -> Location {
    let mut c = raw[initial_loc.index];
    let mut next_loc = initial_loc;
    while [' ', '\n', '\r', '\t'].contains(&c) {
        next_loc = next_loc.increment(c == '\n');
        if next_loc.index == raw.len() {
            break;
        }
        c = raw[next_loc.index];
    }

    next_loc
}

Numbers

Lexing numbers iterates through the source starting at a position until it stops seeing decimal digits (this implementation only supports integers).

fn lex_number(raw: &[char], initial_loc: Location) -> Option<(Token, Location)> {
    let mut ident = String::new();
    let mut next_loc = initial_loc;
    let mut c = raw[initial_loc.index];
    while c.is_digit(10) {
        ident.push_str(&c.to_string());
        next_loc = next_loc.increment(false);
        c = raw[next_loc.index];
    }

If there are no digits in the string then this is not a number.

    if !ident.is_empty() {
        Some((
            Token {
                value: ident,
                loc: initial_loc,
                kind: TokenKind::Number,
            },
            next_loc,
        ))
    } else {
        None
    }
}

Identifiers

Identifiers are any collection of alphabet characters, numbers, and underscores.

fn lex_identifier(raw: &Vec<char>, initial_loc: Location) -> Option<(Token, Location)> {
    let mut ident = String::new();
    let mut next_loc = initial_loc;
    let mut c = raw[initial_loc.index];
    while c.is_alphanumeric() || c == '_' {
        ident.push_str(&c.to_string());
        next_loc = next_loc.increment(false);
        c = raw[next_loc.index];
    }

But they cannot start with a number.

    // First character must not be a digit
    if ident.len() > 0 && !ident.chars().next().unwrap().is_digit(10) {
        Some((
            Token {
                value: ident,
                loc: initial_loc,
                kind: TokenKind::Identifier,
            },
            next_loc,
        ))
    } else {
        None
    }
}

Keywords

Keywords are alphabetical like identifiers are but they cannot be reused as variables by the user.

fn lex_keyword(raw: &[char], initial_loc: Location) -> Option<(Token, Location)> {
    let syntax = ["function", "end", "if", "then", "local", "return"];

    let mut next_loc = initial_loc;
    let mut value = String::new();
    'outer: for possible_syntax in syntax {
        let mut c = raw[initial_loc.index];
        next_loc = initial_loc;
        while c.is_alphanumeric() || c == '_' {
            value.push_str(&c.to_string());
            next_loc = next_loc.increment(false);
            c = raw[next_loc.index];

            let n = next_loc.index - initial_loc.index;
            if value != possible_syntax[..n] {
                value = String::new();
                continue 'outer;
            }
        }

        // Not a complete match
        if value.len() < possible_syntax.len() {
            value = String::new();
            continue;
        }

        // If it got to this point it found a match, so exit early.
        // We don't need a longest match.
        break;
    }

    if value.is_empty() {
        return None;
    }

Aside from matching a list of strings we have to make sure there is a complete match. For example function1 is not a keyword, it's a valid identifier. Whereas function 1 is a valid set of tokens (the keyword function and the number 1), even if it's not a valid Lua grammar.

    // If the next character would be part of a valid identifier, then
    // this is not a keyword.
    if next_loc.index < raw.len() - 1 {
        let next_c = raw[next_loc.index];
        if next_c.is_alphanumeric() || next_c == '_' {
            return None;
        }
    }

    Some((
        Token {
            value: value,
            loc: initial_loc,
            kind: TokenKind::Keyword,
        },
        next_loc,
    ))
}

Syntax

Syntax (in this context) is just language junk that isn't operators. Things like commas, parenthesis, etc.

fn lex_syntax(raw: &[char], initial_loc: Location) -> Option<(Token, Location)> {
    let syntax = [";", "=", "(", ")", ","];

    for possible_syntax in syntax {
        let c = raw[initial_loc.index];
        let next_loc = initial_loc.increment(false);
        // TODO: this won't work with multiple-character syntax bits like >= or ==
        if possible_syntax == c.to_string() {
            return Some((
                Token {
                    value: possible_syntax.to_string(),
                    loc: initial_loc,
                    kind: TokenKind::Syntax,
                },
                next_loc,
            ));
        }
    }

    None
}

Operators

Operators are things like plus, minus, and less than symbols. Operators are syntax but it helps us later on to break these out into a seperate type of token.

fn lex_operator(raw: &[char], initial_loc: Location) -> Option<(Token, Location)> {
    let operators = ["+", "-", "<"];

    for possible_syntax in operators {
        let c = raw[initial_loc.index];
        let next_loc = initial_loc.increment(false);
        // TODO: this won't work with multiple-character operators like >= or ==
        if possible_syntax == c.to_string() {
            return Some((
                Token {
                    value: possible_syntax.to_string(),
                    loc: initial_loc,
                    kind: TokenKind::Operator,
                },
                next_loc,
            ));
        }
    }

    None
}

And now we're all done lexing!

Grammar analysis

Parsing finds grammatical (tree) patterns in a flat list of tokens. This is called a syntax tree or abstract syntax tree (AST).

The boring part is defining the tree. Generally speaking (and specifically for this project), the syntax tree is a list of statements. Statements can be function definitions or expression statements or if statements or return statements or local declarations.

This goes in src/parse.rs.

#[derive(Debug)]
pub enum Statement {
    Expression(Expression),
    If(If),
    FunctionDeclaration(FunctionDeclaration),
    Return(Return),
    Local(Local),
}

pub type Ast = Vec<Statement>;

There's almost nothing special at all about the rest of the tree definitions.

#[derive(Debug)]
pub enum Literal {
    Identifier(Token),
    Number(Token),
}

#[derive(Debug)]
pub struct FunctionCall {
    pub name: Token,
    pub arguments: Vec<Expression>,
}

#[derive(Debug)]
pub struct BinaryOperation {
    pub operator: Token,
    pub left: Box<Expression>,
    pub right: Box<Expression>,
}

#[derive(Debug)]
pub enum Expression {
    FunctionCall(FunctionCall),
    BinaryOperation(BinaryOperation),
    Literal(Literal),
}

#[derive(Debug)]
pub struct FunctionDeclaration {
    pub name: Token,
    pub parameters: Vec<Token>,
    pub body: Vec<Statement>,
}

#[derive(Debug)]
pub struct If {
    pub test: Expression,
    pub body: Vec<Statement>,
}

#[derive(Debug)]
pub struct Local {
    pub name: Token,
    pub expression: Expression,
}

#[derive(Debug)]
pub struct Return {
    pub expression: Expression,
}

And that's it for the AST!

Some helpers

Lastly before the fun part, we'll define a few helpers for validating each kind of token.

fn expect_keyword(tokens: &[Token], index: usize, value: &str) -> bool {
    if index >= tokens.len() {
        return false;
    }

    let t = tokens[index].clone();
    t.kind == TokenKind::Keyword && t.value == value
}

fn expect_syntax(tokens: &[Token], index: usize, value: &str) -> bool {
    if index >= tokens.len() {
        return false;
    }

    let t = tokens[index].clone();
    t.kind == TokenKind::Syntax && t.value == value
}

fn expect_identifier(tokens: &[Token], index: usize) -> bool {
    if index >= tokens.len() {
        return false;
    }

    let t = tokens[index].clone();
    t.kind == TokenKind::Identifier
}

Now on to the fun part, actually detecting these trees!

Top-level parse

The top-level parse function and it's major helper, parse_statement, dispatch very similarly to the top-level lex function. For each statement in the file we look for function declarations, if statements, return statements, local declarations, and expression statements.

fn parse_statement(raw: &[char], tokens: &[Token], index: usize) -> Option<(Statement, usize)> {
    let parsers = [
        parse_if,
        parse_expression_statement,
        parse_return,
        parse_function,
        parse_local,
    ];
    for parser in parsers {
        let res = parser(raw, tokens, index);
        if res.is_some() {
            return res;
        }
    }

    None
}

pub fn parse(raw: &[char], tokens: Vec<Token>) -> Result<Ast, String> {
    let mut ast = vec![];
    let mut index = 0;
    let ntokens = tokens.len();
    while index < ntokens {
        let res = parse_statement(raw, &tokens, index);
        if let Some((stmt, next_index)) = res {
            index = next_index;
            ast.push(stmt);
            continue;
        }

        return Err(tokens[index].loc.debug(raw, "Invalid token while parsing:"));
    }

    Ok(ast)
}

Expression statements

Expression statements are just a wrapper for the Rust type system. They call parse_expression (which we'll define shortly), expect a semicolon afterward, and wrap the expression in a statement.

fn parse_expression_statement(
    raw: &[char],
    tokens: &[Token],
    index: usize,
) -> Option<(Statement, usize)> {
    let mut next_index = index;
    let res = parse_expression(raw, tokens, next_index)?;

    let (expr, next_next_index) = res;
    next_index = next_next_index;
    if !expect_syntax(tokens, next_index, ";") {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected semicolon after expression:")
        );
        return None;
    }

    next_index += 1; // Skip past semicolon

    Some((Statement::Expression(expr), next_index))
}

Expressions

Expressions in this minimal Lua are only one of function calls, literals (numbers, identifiers), or binary operations. To keep things very simple, binary operations cannot be combined. So instead of 1 + 2 + 3 we'd need to do local tmp1 = 1 + 2; local tmp2 = tmp1 + 3; and so on.

fn parse_expression(raw: &[char], tokens: &[Token], index: usize) -> Option<(Expression, usize)> {
    if index >= tokens.len() {
        return None;
    }

    let t = tokens[index].clone();
    let left = match t.kind {
        TokenKind::Number => Expression::Literal(Literal::Number(t)),
        TokenKind::Identifier => Expression::Literal(Literal::Identifier(t)),
        _ => {
            return None;
        }
    };

If what follows the first literal is an open parenthesis then we try to parse a function call.

    let mut next_index = index + 1;
    if expect_syntax(tokens, next_index, "(") {
        next_index += 1; // Skip past open paren

        // Function call
        let mut arguments: Vec<Expression> = vec![];

We need to call parse_expression recursively for every possible argument passed to the function.

        while !expect_syntax(tokens, next_index, ")") {
            if arguments.is_empty() {
                if !expect_syntax(tokens, next_index, ",") {
                    println!(
                        "{}",
                        tokens[next_index]
                            .loc
                            .debug(raw, "Expected comma between function call arguments:")
                    );
                    return None;
                }

                next_index += 1; // Skip past comma
            }

            let res = parse_expression(raw, tokens, next_index);
            if let Some((arg, next_next_index)) = res {
                next_index = next_next_index;
                arguments.push(arg);
            } else {
                println!(
                    "{}",
                    tokens[next_index]
                        .loc
                        .debug(raw, "Expected valid expression in function call arguments:")
                );
                return None;
            }
        }

        next_index += 1; // Skip past closing paren

        return Some((
            Expression::FunctionCall(FunctionCall {
                name: tokens[index].clone(),
                arguments,
            }),
            next_index,
        ));
    }

Otherwise if there isn't an opening parenthesis then we could be parsing either a literal expression or a binary operation. If the token that follows is an operator token then we know it's a binary operation.

    // Might be a literal expression
    if next_index >= tokens.len() || tokens[next_index].clone().kind != TokenKind::Operator {
        return Some((left, next_index));
    }

    // Otherwise is a binary operation
    let op = tokens[next_index].clone();
    next_index += 1; // Skip past op

    if next_index >= tokens.len() {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid right hand side binary operand:")
        );
        return None;
    }

    let rtoken = tokens[next_index].clone();

It is at this point that we could (but won't) call parse_expression recursively. I don't want to deal with operator precedence right now so we'll just require that the right hand side is another literal.

    let right = match rtoken.kind {
        TokenKind::Number => Expression::Literal(Literal::Number(rtoken)),
        TokenKind::Identifier => Expression::Literal(Literal::Identifier(rtoken)),
        _ => {
            println!(
                "{}",
                rtoken
                    .loc
                    .debug(raw, "Expected valid right hand side binary operand:")
            );
            return None;
        }
    };
    next_index += 1; // Skip past right hand operand

    Some((
        Expression::BinaryOperation(BinaryOperation {
            left: Box::new(left),
            right: Box::new(right),
            operator: op,
        }),
        next_index,
    ))
}

And now we're done parsing expressions!

Function declarations

Functions start with the function keyword, and an identifier token follows.

fn parse_function(raw: &[char], tokens: &[Token], index: usize) -> Option<(Statement, usize)> {
    if !expect_keyword(tokens, index, "function") {
        return None;
    }

    let mut next_index = index + 1;
    if !expect_identifier(tokens, next_index) {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid identifier for function name:")
        );
        return None;
    }
    let name = tokens[next_index].clone();

After the function name comes the argument list that can be empty or a comma separated list of identifiers.

    next_index += 1; // Skip past name
    if !expect_syntax(tokens, next_index, "(") {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected open parenthesis in function declaration:")
        );
        return None;
    }

    next_index += 1; // Skip past open paren
    let mut parameters: Vec<Token> = vec![];
    while !expect_syntax(tokens, next_index, ")") {
        if !parameters.is_empty() {
            if !expect_syntax(tokens, next_index, ",") {
                println!("{}", tokens[next_index].loc.debug(raw, "Expected comma or close parenthesis after parameter in function declaration:"));
                return None;
            }

            next_index += 1; // Skip past comma
        }

        parameters.push(tokens[next_index].clone());
        next_index += 1; // Skip past param
    }

    next_index += 1; // Skip past close paren

Next we parse all statements in the function body until we find the end keyword.

    let mut statements: Vec<Statement> = vec![];
    while !expect_keyword(tokens, next_index, "end") {
        let res = parse_statement(raw, tokens, next_index);
        if let Some((stmt, next_next_index)) = res {
            next_index = next_next_index;
            statements.push(stmt);
        } else {
            println!(
                "{}",
                tokens[next_index]
                    .loc
                    .debug(raw, "Expected valid statement in function declaration:")
            );
            return None;
        }
    }

    next_index += 1; // Skip past end

    Some((
        Statement::FunctionDeclaration(FunctionDeclaration {
            name,
            parameters,
            body: statements,
        }),
        next_index,
    ))
}

Phew! We're halfway through the parser.

Return statements

Return statements just check for the return keyword, an expression, and a semicolon.

fn parse_return(raw: &[char], tokens: &[Token], index: usize) -> Option<(Statement, usize)> {
    if !expect_keyword(tokens, index, "return") {
        return None;
    }

    let mut next_index = index + 1; // Skip past return
    let res = parse_expression(raw, tokens, next_index);
    if res.is_none() {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid expression in return statement:")
        );
        return None;
    }

    let (expr, next_next_index) = res.unwrap();
    next_index = next_next_index;
    if !expect_syntax(tokens, next_index, ";") {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected semicolon in return statement:")
        );
        return None;
    }

    next_index += 1; // Skip past semicolon

    Some((Statement::Return(Return { expression: expr }), next_index))
}

Local declarations

Local declarations start with the local keyword, then the local name, then an equal sign, then an expression, and then a semicolon.

fn parse_local(raw: &[char], tokens: &[Token], index: usize) -> Option<(Statement, usize)> {
    if !expect_keyword(tokens, index, "local") {
        return None;
    }

    let mut next_index = index + 1; // Skip past local

    if !expect_identifier(tokens, next_index) {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid identifier for local name:")
        );
        return None;
    }

    let name = tokens[next_index].clone();
    next_index += 1; // Skip past name

    if !expect_syntax(tokens, next_index, "=") {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected = syntax after local name:")
        );
        return None;
    }

    next_index += 1; // Skip past =

    let res = parse_expression(raw, tokens, next_index);
    if res.is_none() {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid expression in local declaration:")
        );
        return None;
    }

    let (expr, next_next_index) = res.unwrap();
    next_index = next_next_index;

    if !expect_syntax(tokens, next_index, ";") {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected semicolon in return statement:")
        );
        return None;
    }

    next_index += 1; // Skip past semicolon

    Some((
        Statement::Local(Local {
            name,
            expression: expr,
        }),
        next_index,
    ))
}

If statements

This implementation of Lua doesn't support elseif so parsing if just checks for the if keyword followed by a test expression, then the else keyword, then the if body (a list of statements), and then the end keyword.

fn parse_if(raw: &[char], tokens: &[Token], index: usize) -> Option<(Statement, usize)> {
    if !expect_keyword(tokens, index, "if") {
        return None;
    }

    let mut next_index = index + 1; // Skip past if
    let res = parse_expression(raw, tokens, next_index);
    if res.is_none() {
        println!(
            "{}",
            tokens[next_index]
                .loc
                .debug(raw, "Expected valid expression for if test:")
        );
        return None;
    }

    let (test, next_next_index) = res.unwrap();
    next_index = next_next_index;

    if !expect_keyword(tokens, next_index, "then") {
        return None;
    }

    next_index += 1; // Skip past then

    let mut statements: Vec<Statement> = vec![];
    while !expect_keyword(tokens, next_index, "end") {
        let res = parse_statement(raw, tokens, next_index);
        if let Some((stmt, next_next_index)) = res {
            next_index = next_next_index;
            statements.push(stmt);
        } else {
            println!(
                "{}",
                tokens[next_index]
                    .loc
                    .debug(raw, "Expected valid statement in if body:")
            );
            return None;
        }
    }

    next_index += 1; // Skip past end

    Some((
        Statement::If(If {
            test,
            body: statements,
        }),
        next_index,
    ))
}

And goshdarnit we're done parsing.

Compiling to a made up virtual machine

This virtual machine will be entirely stack-based other than the stack pointer and program counter.

The calling convention is that the caller will put arguments on the stack followed by the frame pointer, the program counter, and then the number of arguments (for cleanup). Then it will alter the program counter and frame pointer. Then the caller will allocate space on the stack for all arguments and locals within the function.

For simplicity in addressing modes, the function declaration once jumped to will copy the arguments from before the frame pointer to in front of it (yes I know, I know, this is silly).

The virtual machine will support add, subtract, less than operations as well as jump, jump-if-not-zero, return, and call. It will support a few more memory-specific instructions for loading literals, loading identifiers, and managing arguments.

I'll explain the non-obvious instructions as we implement them.

use crate::parse::*;
use std::collections::HashMap;

#[derive(Debug)]
enum Instruction {
    DupPlusFP(i32),
    MoveMinusFP(usize, i32),
    MovePlusFP(usize),
    Store(i32),
    Return,
    JumpIfNotZero(String),
    Jump(String),
    Call(String, usize),
    Add,
    Subtract,
    LessThan,
}

The result of compiling will be a Program instance. This instance will contain symbol information and the actual instructions to run.

#[derive(Debug)]
struct Symbol {
    location: i32,
    narguments: usize,
    nlocals: usize,
}

#[derive(Debug)]
pub struct Program {
    syms: HashMap<String, Symbol>,
    instructions: Vec<Instruction>,
}

Compiling, similar to parsing, just calls the helper compile_statement for each statement in the AST.

pub fn compile(raw: &[char], ast: Ast) -> Program {
    let mut locals: HashMap<String, i32> = HashMap::new();
    let mut pgrm = Program {
        syms: HashMap::new(),
        instructions: Vec::new(),
    };
    for stmt in ast {
        compile_statement(&mut pgrm, raw, &mut locals, stmt);
    }

    pgrm
}

And compile_statement dispatches to additional helpers based on the kind of statement.

fn compile_statement(
    pgrm: &mut Program,
    raw: &Vec<char>,
    locals: &mut HashMap<String, i32>,
    stmt: Statement,
) {
    match stmt {
        Statement::FunctionDeclaration(fd) => compile_declaration(pgrm, raw, locals, fd),
        Statement::Return(r) => compile_return(pgrm, raw, locals, r),
        Statement::If(if_) => compile_if(pgrm, raw, locals, if_),
        Statement::Local(loc) => compile_local(pgrm, raw, locals, loc),
        Statement::Expression(e) => compile_expression(pgrm, raw, locals, e),
    }
}

Function declarations

Let's do the hard one first. First off, function declarations will include an unconditional guard around them so that we can evaluate from the 0th instruction at the top-level and have only non-function-declaration statements be evaluated.

fn compile_declaration(
    pgrm: &mut Program,
    raw: &[char],
    _: &mut HashMap<String, i32>,
    fd: FunctionDeclaration,
) {
    // Jump to end of function to guard top-level
    let done_label = format!("function_done_{}", pgrm.instructions.len());
    pgrm.instructions
        .push(Instruction::Jump(done_label.clone()));

Then we'll add another limitation/simplification that local variables are only accessible within the current function scope.

For each parameter, we'll copy the parameter on the stack before the frame pointer to a place in front of the frame pointer. This gets around addressing mode limitations in our virtual machine.

    let mut new_locals = HashMap::<String, i32>::new();

    let function_index = pgrm.instructions.len() as i32;
    let narguments = fd.parameters.len();
    for (i, param) in fd.parameters.iter().enumerate() {
        pgrm.instructions.push(Instruction::MoveMinusFP(
            i,
            narguments as i32 - (i as i32 + 1),
        ));
        new_locals.insert(param.value.clone(), i as i32);
    }

Then we compile the body.

    for stmt in fd.body {
        compile_statement(pgrm, raw, &mut new_locals, stmt);
    }

Once the body is compiled we know the total number of locals so we can fill out the symbol table correctly. The location is importantly already stored because it is the location of the instruction where the function started.

    pgrm.syms.insert(
        fd.name.value,
        Symbol {
            location: function_index as i32,
            narguments,
            nlocals: new_locals.keys().len(),
        },
    );

Finally we add a symbol linking the done label for the function to the position of the end of the function. Again, this allows us to skip past the function declaration when evaluating instructions from 0 to N.

    pgrm.syms.insert(
        done_label,
        Symbol {
            location: pgrm.instructions.len() as i32,
            narguments: 0,
            nlocals: 0,
        },
    );
}

Ok that wasn't so bad. And the rest are simpler.

Local declarations

The expression for the local is compiled and then the local name is stored in a locals table mapped to the current number of locals (including arguments). This allows the compiler to turn identifier token lookups into simply an offset from the frame pointer.

fn compile_local(
    pgrm: &mut Program,
    raw: &[char],
    locals: &mut HashMap<String, i32>,
    local: Local,
) {
    let index = locals.keys().len();
    locals.insert(local.name.value, index as i32);
    compile_expression(pgrm, raw, locals, local.expression);
    pgrm.instructions.push(Instruction::MovePlusFP(index));
}

And specifically, the instruction pattern is to evaluate the expression and then copy it back into a relative position in the stack.

Literals

Number literals use the store instruction for pushing a number onto the stack. Identifier literals are copied to the top of the stack from their position relative to the frame pointer.

fn compile_literal(
    pgrm: &mut Program,
    _: &[char],
    locals: &mut HashMap<String, i32>,
    lit: Literal,
) {
    match lit {
        Literal::Number(i) => {
            let n = i.value.parse::<i32>().unwrap();
            pgrm.instructions.push(Instruction::Store(n));
        }
        Literal::Identifier(ident) => {
            pgrm.instructions
                .push(Instruction::DupPlusFP(locals[&ident.value]));
        }
    }
}

Function calls

Pretty simple: just compile all the arguments and then issue a call instruction.

fn compile_function_call(
    pgrm: &mut Program,
    raw: &Vec<char>,
    locals: &mut HashMap<String, i32>,
    fc: FunctionCall,
) {
    let len = fc.arguments.len();
    for arg in fc.arguments {
        compile_expression(pgrm, raw, locals, arg);
    }

    pgrm.instructions
        .push(Instruction::Call(fc.name.value, len));
}

Binary operations

Binary operations compile the left, then the right, and then issue an instruction based on the operator. All the operators are builtin and act on the top two elements on the stack.

fn compile_binary_operation(
    pgrm: &mut Program,
    raw: &[char],
    locals: &mut HashMap<String, i32>,
    bop: BinaryOperation,
) {
    compile_expression(pgrm, raw, locals, *bop.left);
    compile_expression(pgrm, raw, locals, *bop.right);
    match bop.operator.value.as_str() {
        "+" => {
            pgrm.instructions.push(Instruction::Add);
        }
        "-" => {
            pgrm.instructions.push(Instruction::Subtract);
        }

        "<" => {
            pgrm.instructions.push(Instruction::LessThan);
        }
        _ => panic!(
            "{}",
            bop.operator
                .loc
                .debug(raw, "Unable to compile binary operation:")
        ),
    }
}

Expressions

Compiling expressions just dispatches to a compile helper based on the type of expression. We've already written those three helpers.

fn compile_expression(
    pgrm: &mut Program,
    raw: &[char],
    locals: &mut HashMap<String, i32>,
    exp: Expression,
) {
    match exp {
        Expression::BinaryOperation(bop) => {
            compile_binary_operation(pgrm, raw, locals, bop);
        }
        Expression::FunctionCall(fc) => {
            compile_function_call(pgrm, raw, locals, fc);
        }
        Expression::Literal(lit) => {
            compile_literal(pgrm, raw, locals, lit);
        }
    }
}

If

First we compile the conditional test and then we jump to after the if the test result is not zero.

fn compile_if(pgrm: &mut Program, raw: &[char], locals: &mut HashMap<String, i32>, if_: If) {
    compile_expression(pgrm, raw, locals, if_.test);
    let done_label = format!("if_else_{}", pgrm.instructions.len());
    pgrm.instructions
        .push(Instruction::JumpIfNotZero(done_label.clone()));

Then we compile the body.

    for stmt in if_.body {
        compile_statement(pgrm, raw, locals, stmt);
    }

And finally make sure we insert the done symbol in the right place after the if.

    pgrm.syms.insert(
        done_label,
        Symbol {
            location: pgrm.instructions.len() as i32 - 1,
            nlocals: 0,
            narguments: 0,
        },
    );
}

Return

The final statement type is return. We simply compile the return expression and issue a return instruction.

fn compile_return(
    pgrm: &mut Program,
    raw: &[char],
    locals: &mut HashMap<String, i32>,
    ret: Return,
) {
    compile_expression(pgrm, raw, locals, ret.expression);
    pgrm.instructions.push(Instruction::Return);
}

That's it for the compiler! Now the trickiest part. I lost a few hours debugging and iterating on the next bit.

The virtual machine

Ok so the easy part is that there are only two registers, a program counter and a frame pointer. There's also a data stack. The frame pointer points to the location on the data stack where each function can start storing its locals.

Evaluation starts from 0 and goes until the last instruction.

pub fn eval(pgrm: Program) {
    let mut pc: i32 = 0;
    let mut fp: i32 = 0;
    let mut data: Vec<i32> = vec![];

    while pc < pgrm.instructions.len() as i32 {
        match &pgrm.instructions[pc as usize] {

Each instruction will be responsible for incrementing the program counter or having it jump around.

Addition, subtraction, less than

The easiest ones are the math operators. We just pop off the data stack, perform the operation, and store the result.

            Instruction::Add => {
                let right = data.pop().unwrap();
                let left = data.pop().unwrap();
                data.push(left + right);
                pc += 1;
            }
            Instruction::Subtract => {
                let right = data.pop().unwrap();
                let left = data.pop().unwrap();
                data.push(left - right);
                pc += 1;
            }
            Instruction::LessThan => {
                let right = data.pop().unwrap();
                let left = data.pop().unwrap();
                data.push(if left < right { 1 } else { 0 });
                pc += 1;
            }

The store instruction is another easy one. It just pushes a literal number onto the stack.

            Instruction::Store(n) => {
                data.push(*n);
                pc += 1;
            }

Jump variants

The jump variants are easy too. Just grab the location and change the program counter. If it's a conditional jump then test the condition first.

            Instruction::JumpIfNotZero(label) => {
                let top = data.pop().unwrap();
                if top == 0 {
                    pc = pgrm.syms[label].location;
                }
                pc += 1;
            }
            Instruction::Jump(label) => {
                pc = pgrm.syms[label].location;
            }

Loading from a variable

The MovePlusFP instruction copies a value from the stack (offset the frame pointer) onto the top of the stack. This is for references to arguments and locals.

            Instruction::MovePlusFP(i) => {
                let val = data.pop().unwrap();
                let index = fp as usize + *i;
                // Accounts for top-level locals
                while index >= data.len() {
                    data.push(0);
                }
                data[index] = val;
                pc += 1;
            }

Storing locals

The DupPlusFP instruction is used by compile_locals to store a local once compiled onto the stack in the relative position from the frame pointer.

            Instruction::DupPlusFP(i) => {
                data.push(data[(fp + i) as usize]);
                pc += 1;
            }

Duplicating arguments

The MoveMinusFP instruction is, again, a hack to work around limited addressing modes in this minimal virtual machine. It copies arguments from behind the frame pointer to in front of the frame pointer.

            Instruction::MoveMinusFP(local_offset, fp_offset) => {
                data[fp as usize + local_offset] = data[(fp - (fp_offset + 4)) as usize];
                pc += 1;
            }

Now we're down to the last two instructions: call and return.

Call

Call has a special dispatch for builtin functions (the only one that exists being print).

            Instruction::Call(label, narguments) => {
                // Handle builtin functions
                if label == "print" {
                    for _ in 0..*narguments {
                        print!("{}", data.pop().unwrap());
                        print!(" ");
                    }
                    println!();
                    pc += 1;
                    continue;
                }

Otherwise it pushes the current frame pointer, then the program counter, and finally the number of arguments (not locals) onto the stack for preservation. Then it sets up the new program counter and frame pointer and creates space for all locals and arguments after the new frame pointer.

                data.push(fp);
                data.push(pc + 1);
                data.push(pgrm.syms[label].narguments as i32);
                pc = pgrm.syms[label].location;
                fp = data.len() as i32;

                // Set up space for all arguments/locals
                let mut nlocals = pgrm.syms[label].nlocals;
                while nlocals > 0 {
                    data.push(0);
                    nlocals -= 1;
                }
            }

Return

The return instructions pops the return value from the stack. Then it pops off all locals and arguments. Then it restores the program counter and frame pointer, and pops off the arguments before the frame pointer. Finally it adds the return value back onto the stack.

            Instruction::Return => {
                let ret = data.pop().unwrap();

                // Clean up the local stack
                while fp < data.len() as i32 {
                    data.pop();
                }

                // Restore pc and fp
                let mut narguments = data.pop().unwrap();
                pc = data.pop().unwrap();
                fp = data.pop().unwrap();

                // Clean up arguments
                while narguments > 0 {
                    data.pop();
                    narguments -= 1;
                }

                // Add back return value
                data.push(ret);
            }

And yes, this implementation would be more efficient if instead of literally pushing and popping we just incremented/decremented a stack pointer.

And that's it! We're completely done a basic parser, compiler, and virtual machine for a subet of Lua. Is it janky? Yeah. Is it simple? Kind of? Does it work? It seems to!

Summary

Ok we've got <1200 lines of Rust enough to run some decent Lua programs. We run this fib program against this implementation and against Lua 5.4.3 (which isn't LuaJIT) and what do we see?

$ cargo build --release
$ cat test/fib.lua
function fib(n)
   if n < 2 then
      return n;
   end

   local n1 = fib(n-1);
   local n2 = fib(n-2);
   return n1 + n2;
end

print(fib(30));
$ time ./target/release/lust test/fib.lua
832040
./target/release/lust test/fib.lua  0.29s user 0.00s system 99% cpu 0.293 total
$ time lua test/fib.lua
832040
lua test/fib.lua  0.06s user 0.00s system 99% cpu 0.063 total

This implementation is a bit slower! Time to do some profiling and maybe revisit some of those aforementioned inefficiencies.

Big thanks to Christian Scott on Twitter for pointing out I should not be benchmarking with debug builds!

And thanks to reddit123123123123 on Reddit for suggesting I use cargo clippy to clean up my code.

Thanks to GiffE on Github for pointing out some key inconsistencies between this implementation and Lua. I won't modify anything because a perfect Lua subset wasn't the goal, but I'm sharing because it was good analysis and criticism of this implementation.

I wrote a new blog post on parsing, compiling, and virtual machine evaluation for a super minimal Lua implementation written from scratch in Rust!https://t.co/8qFviEecJo pic.twitter.com/d1MGArlErR
— Phil Eaton (@phil_eaton) December 28, 2021

Drones, bringing MAD to every neighborhood

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 27 Dec 2021 19:00:00 GMT

and the only protection may be to bring sanity to the same neighborhoods.

Why TurboRepo Will Be The First Big Trend of 2022

swyx's site RSS Feed — Mon, 27 Dec 2021 18:08:03 GMT

TurboRepo is a big deal for the JS community because it addresses the monorepo problem head on, bringing 85% faster build speeds and great architecture/docs/marketing.

Update to Portals

infrequent oscillations — Mon, 27 Dec 2021 11:50:49 GMT

In which we update our portal finding.

System Shock is Back Home

Nicola Iarocci — Mon, 27 Dec 2021 08:05:25 GMT

When The Digital Antiquarian released his System Shock retrospective earlier this year, I was in awe. System Shock was one of my favorite games back in the day, and yes, in the quarrel between id Software’s DOOM and Looking Glass’s System Shock, I was siding with the latter. I was so much more for immersion and storyline than shoot ’em-ups. The Antiquarian article is excellent. If you’re into gaming history or, really, into computer’s history, I urge you to read it all.

Brute-Forcing 22 Trillion Parameters

Matt Keeter — Mon, 27 Dec 2021 02:00:00 GMT

Getting started with Arch on a Macbook pro 2012 (9,2)

xnacly - blog — Mon, 27 Dec 2021 02:00:00 GMT

A guide to wifi, touchpad support and more on your macbook pro 2012 using arch

Thinking out loud

Grady Simon RSS Feed — Mon, 27 Dec 2021 02:00:00 GMT

Thinking out loud is useful, but why?

Lessons learned from 123dev newsletter

Justin Garrison's Homepage — Mon, 27 Dec 2021 02:00:00 GMT

One year of failures and tips for new and existing newsletters

Dangerous MVPs

Dan Stroot — Mon, 27 Dec 2021 02:00:00 GMT

When building a new product, we all know we should remain agile and launch something small and simple to start. Then, over time, we can make that product more functional by listening to our customers and adding more features. Unfortunately, the same process wreaks havoc when it comes to modernizing an existing system. At first, the reason why is not intuitive. In theory, the same principles should apply.

This post Dangerous MVPs first appeared on Dan Stroot's Blog

Using Datasette for an ELT Personal Twitter Data Warehouse

swyx's site RSS Feed — Sun, 26 Dec 2021 22:52:14 GMT

EDA tools like Datasette dramatically lower the cost of data analysis, with a surprisingly simple ELT contract - You handle the Extract phase, it handles the Load, and exposes a standard UI for you to do dynamic Transforms.

The best food delivery programs for the digital age...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 26 Dec 2021 19:00:00 GMT

Are the LEAST digital of all.

The Matrix Resurrections

🦉 olshansky 🦁 — Sun, 26 Dec 2021 15:42:16 GMT

Olshansky's review of The Matrix Resurrections

Book Review: About the Meaning of Life

Nicola Iarocci — Sun, 26 Dec 2021 08:05:25 GMT

I’m not a regular philosophy reader, much less of self-improvement guides. I’m wary of the latter and too ignorant for the former. Yet, theologian Vito Mancuso has intrigued me for some time. I followed his podcast on the “Four Masters of Life”¹ and found it excellent. In it, Mancuso discusses his four tutelary deities: “Socrates, the educator. Buddha, the physician. Confucius, the politician. Jesus, the prophet.” I also listened to some TV interviews where I always found him fascinating. In short, there was enough to make me decide to read some of his work. From the back cover of the book:

Loyal Citizens Being Blown into the Sky

Non-Human Party — Sun, 26 Dec 2021 05:59:05 GMT

Tornadoes are big and scary; and they’re beyond the control of ordinary civilians, so it’s the perfect scenario for helpful government intervention. It’s existential threats like these that remind us why we need governments; but perhaps too it serves as a prompt to consider if we could replace governments with something else.

Yamaha DX7 Firmware ROM Disassembly

ajxs.me — Sat, 25 Dec 2021 03:40:31 GMT

A brief update regarding the reverse-engineering of the Yamaha DX7's firmware ROM.

My 2021 reads

Alexander's Blog Blog — Sat, 25 Dec 2021 02:00:00 GMT

On A Life Well Spent: Cicero's De Senectute

Actually Creating An Idempotency Keys Gem

julik live — Sat, 25 Dec 2021 02:00:00 GMT

Things I worked on in 2021

λm.me — Sat, 25 Dec 2021 02:00:00 GMT

Writing, streaming and OSS - A recap of the things I worked on in 2021.

Christmas 2021

Lambda Land — Sat, 25 Dec 2021 02:00:00 GMT

It’s my first Christmas being a dad. The end of the school semester was unusually stressful; I didn’t have much time or energy to anticipate Christmas. But I’ve felt a marked lack of eagerness for getting presents. That feeling wanes each year—something I’m grateful for—but this year I only felt an eagerness for a time of peace and celebration with my family. Peaceful it has been.

My family isn’t perfect. (My baby daughter is about as perfect as they come, though!) The time I’ve gotten to spend so far and the time that I will spend with them is precious to me. My wife and I have focused so much on celebrating the birth of Jesus Christ that this commercialized junk we put up with has been all but absent from our home. We have a few books about Santa Claus that we read to our daughter, but we’ve listened to strictly Christ-centered music. (A Charlie Brown Christmas counts, though!) There’s been a spirit in our home that has brought me comfort and peace in this tumultuous year—more than any before it.

Merry Christmas—may the peace of God be with you.

Probabilistic spin glass - Part III

Bytepawn - Marton Trencseni — Sat, 25 Dec 2021 01:00:00 GMT

I run simulations to understand the dynamic probabilistic evolution of these toy models.

Sherlock Holmes would love rationing

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 24 Dec 2021 19:00:00 GMT

As he said, when you have eliminated all which is impossible…

Spiral Again?

Greg Herlein — Fri, 24 Dec 2021 09:00:00 GMT

Merry Christmas Eve! Playing with the Beaglebone sends me into another thought spiral.

Data cooperatives would not give EVERYBODY more power

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 23 Dec 2021 19:00:00 GMT

Just ask Kim Kardashian.

Eggnog & Cola

Dustin.Lammiman — Thu, 23 Dec 2021 02:00:00 GMT

Eggnog Ice Cream #

Ingredients #

1/2 Cup Granulated Sugar
2 Tbsp Skim Milk Powder
1/4 Tsp Xanthan Gum
1 Tsp Ground Nutmeg
1 Tbsp Imitation Rum Extract
1 1/3 Cups Milk (I used 2%)
2 Tbsp Light Corn Syrup
1 1/3 Cup Whipping Cream
3 Large Egg Yolks

Instructions #

In a bowl whisk together sugar, milk powder, nutmeg, and xanthan gum.
In a pot add milk, whipping cream, and corn syrup.
Whisk dry ingredients into the pot of liquid ingredients. Be sure to mix well.
Heat ingredients on medium/high heat, stirring well, until it comes to a boil. Remove from heat.
Temper the yolks by placing them in a bowl and slowly adding ~1/2 cup of the hot mixture while whisking well. Pour the yolks back into the main pot while continuing to whisk well.
Heat the mixture until it begins to thicken or it reaches a temperature of 180 F.
Remove from heat and allow mixture to cool for before placing in an air-tight container.
Refridgerate for 12+ hours. Then churn according to your ice cream maker's instructions.
Split the ice cream into two containers, each ~1L in size. You now have Eggnog ice cream. To make it eggnog and cola ice cream, make cola ice cream (below) and layer it on top!

Cola Ice Cream #

Note - this one didn't really turn out like I hoped. Not a lot of Cola flavour. I'd like to try finding a Cola syrup for my next attempt.

Ingredients #

5 Cups Cola
1/3 Cup Skim Milk Powder
1/4 Tsp Xanthan Gum
2 Tbsp Corn Syrup
1 1/3 Cup Whipping Cream

Instructions #

Place Cola in a medium pot and boil for ~30-45 minutes, until you have about 1 1/2 Cups of liquid remaining.
Meanwhile, in a large bowl whisk together the milk powder and xanthan gum.
Add cream and dry ingredients to the reduced cola, whisking well as you add them.
Bring the mixture to a simmer, long enough to allow the sugars to completely dissolve into the mixture.
Allow to cool, then place mixture in a sealed contained and refridgerate for 12+ hours.
Churn according to your ice cream maker's instructions.
Split the ice cream into two containers, each ~1L in size. You now have Cola ice cream. To make it eggnog and cola ice cream, layer on top of your eggnog ice cream!

Autonomous cars will kill the car industry

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 22 Dec 2021 19:00:00 GMT

Stay away from their stocks. Seriously.

2021 in Review

Jason Eckert's Website and Blog — Wed, 22 Dec 2021 02:00:00 GMT

While 2020 introduced a slew of changes to everyone’s personal and professional lives, these changes were considerably less pronounced for those of us working in tech fields. After all, most software engineering and server/cloud configuration was already performed remotely before the pandemic using long-standing processes/tools. Adapting to online tech teaching in the college was a welcome challenge, and also more relevant for the industry.

As 2020 was ending, we all knew that 2021 would bring vaccine rollouts but not a complete return to normalcy. I was very interested to see how the pandemic shakeup would transform the workplace and our use of technology in 2021, but also knew that 2021 would be more like season 2 of 2020. Consequently, I decided to make the most out of the more delightful aspects of 2020, including work-from-home, online tech conferences, and local outdoor activities.

The lazy webmaster's way of providing a RSS feed

Fabian's public notepad — Wed, 22 Dec 2021 02:00:00 GMT

I consciously setup my website as minimal as possible. It's liberating to have a the content immediately in a representation that renders good enough. Yet I'd like to offer is a RSS feed. I could of course write it manually, but that would become a chore rather soon. A small tool chain could also work, but I would lose the feeling of immediacy.

Having a regular markup structure made it possible to an external service derive the feed automatically. So, ~~here you go~~.

How I Almost Cheesed the EVM

Zefram's Blog — Wed, 22 Dec 2021 00:13:31 GMT

Genesis

Using Ethereum is expensive. Us smart contract developers are eternally doomed to the fate of locking ourselves in dank dark basements every day to optimize gas usage, so that our fancy DeFi protocols won’t make our users bankrupt.

Of all the operations done by smart contracts, storage writes & reads usually take up most of the gas usage. EVM storage is made intentionally expensive, because it’s expensive for Ethereum nodes to store data for the rest of time.

Announcing a Full-Stack Web Development cohort-based course

www.mootoday.com - RSS Feed — Tue, 21 Dec 2021 19:47:18 GMT

Today, I am excited to announce a hands-on, live, project-based course with a small, highly motivated group of professional software engineers.

Announcing the First Release of kcctl

Gunnar Morling — Tue, 21 Dec 2021 18:48:00 GMT

Table of Contents

Discussion and Outlook

🧸 It’s Casey. Casey Cuddle.

I am very happy to announce the first stable release of kcctl, a modern and intuitive command line client for Apache Kafka Connect!

Forget about having to memorize and type the right REST API paths and curl flags; with kcctl, managing your Kafka connectors is done via concise and logically structured commands, modeled after the semantics of the kubectl tool known from Kubernetes.

Would you like to have just one monthly bill... but like THIS?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 21 Dec 2021 13:50:00 GMT

Me, I think it would be dangerous.

Apple to Apple Comparison: M1 Max vs Intel 🍏

Ash's Blog — Tue, 21 Dec 2021 02:00:00 GMT

M1 Max with DDR5 challenges 64-core server workloads in hash-table benchmarks. Memory bandwidth matters more than core count for many real-world tasks.

The Value Of Not Having To Be Right

julik live — Tue, 21 Dec 2021 02:00:00 GMT

Aten: Price Convergence 2021 in CWE region

Blog Frank Boerman — Tue, 21 Dec 2021 02:00:00 GMT

Bar chart of price convergence per day in 2021 for CWE region (click to enlarge)

What is price convergence?

The electricity market in Europe is split into so-called bidding zones. The rule is that within these zones you can trade without transport restrictions, as if the entire grid there is one big conducting plate. This also means that for the day ahead marketi, there is one price per hour per bidding zone. If neighbouring bidding zones have enough interconnection capacity available to satisfy the market direction, then the price in those zones will converge: it will be the same. So, price convergence can be seen as a measure on how much transport restriction the market has run into between bidding zones.

A bit on web3, community, and the purpose of life - 123dev #51

Justin Garrison's Homepage — Tue, 21 Dec 2021 02:00:00 GMT

It's kind of fun to do the impossible. – Walt Disney

Avoid collecting without acting

Fabian's public notepad — Tue, 21 Dec 2021 02:00:00 GMT

The amount of information which is thrust upon you is orders of magnitude larger than the capacity to absorb and make use of it. This can lead to a destructive habit, called the collector's fallacy. This fallacy is a form of self-deception. It makes you feel like you are progressing where you are actually stalling. It leads to wasting your precious time on something that's not worth it.

To avoid falling prey to the collector's fallacy, you need to work actively with the collection you have. Introduce some form of rate limiting: Don't buy a new book before you finished one. Don't bookmark another article before you haven't read (or at least deleted) one of those you have bookmarked.

If it would have been important, you would have already read it. Its content likely won't be lost. There are search engines and even in case the of uncool urls there's still the wayback machine.

Why Google is rapidly losing publishers' trust

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 20 Dec 2021 19:00:00 GMT

“Don’t be evil” my foot.

It's bingo time (AoC 2021 Day 4)

geekmonkey — Mon, 20 Dec 2021 17:00:00 GMT

Our underwater journey continues. It can get dark underwater, and today is no exception. The only thing we can see apparently is a giant squid that has attached itself to the outside of our submarine.

It's only logical that we decide to play Bingo with the squid. Hey,

Self arrest on the Slippery Slope of Addiction

D13V — Mon, 20 Dec 2021 06:01:11 GMT

…distraction is just another way our brains attempt to deal with pain.

Nir Eyal, Indistractable

Welcome to part 9 of the series on “How to be a game changer”. You can find the previous posts here 1, 2, 3, 4, 5, 6, 7, 8.

There are games which can be win-win when played a limited amount, but become overall negative if you play them excessively. For example, unhealthy addictions can feel like a slippery slope - starting with a little bit, and then becoming a regular user, who is more and more attracted to the game. With such slippery slope games, one way to opt out is to define and maintain a stopping point - a.k.a. Shelling - a point beyond which we don’t venture, as the slope is too slippery for us, and we’re going to slide away. I highly recommend reading this article from Less Wrong, as my article largely applies its ideas, and illustrates them with stories from my life. I will share two stories of slippery slope games - substances and entertainment.

Defining the point a-priori allows us to self-arrest the way a mountaineer would be able to stop themselves from falling on an actual icy slippery slope.

Addictive Substances

I used to drink a lot, but today I don’t. Here’s what happened. I think Alcohol can be a very effective way to quickly relax after a stressful day, and in the right social environment, can help people who drink it to open up to social connections, be more empathetic, and foster relationships. I used to drink regularly, and I found it hard to control how much I drank. When I was buzzed, I wanted to drink more and more.

Fifteen years ago, when I was in college I used to drink a lot. I enjoyed and craved drinking. But one day, someone close called me an alcoholic. I didn’t identify as alcoholic and this criticism made me feel painful, and in doubt of myself. After another binge, and a really bad hangover the next day, I decided to take a break from alcohol, and didn’t drink for a month. I wrote more about the story here. I kept the practice of no-alcohol month every year for 12 years, and I gradually started noticing that after an alcohol fast, I would start back drinking slowly, one drink a week or so, but gradually the volume would increase. It’s a slippery slope! Few years ago I started adding a second month of no alcohol, which lead to adding a third month the year after, which easily transformed to my current relationship to alcohol. Today, I’d only have a drink on a special occasion, and I’m in a near permanent alcohol fast.

For me, the Shelling point of alcohol has moved very close to the no-alcohol zone, and I think of my relationship to alcohol as healthy at that level. Some people can maintain a personal restriction of never going more than 1-2 drinks per night, and that can also be a sustainable relationship to it. People who are past alcoholics, or others who just went cold turkey are staying at completely zero alcohol in their life. That’s where they put their boundary and it’s OK. They stay in self-arrest all of the time, as they wouldn’t be able to stop if they slide even a little bit. If I didn’t take 10 years to gradually wean off it my boundary would be different. I think I can self-arrest if I notice myself sliding, so I allow myself a drink on special occasions.

Other substances may be more addictive than alcohol. I’ve never had heroin, which is known to be extremely addictive, and I cannot speak about it. But I’ve had another extremely addictive drug, which 90% of US population is currently addicted to - caffeine. It started with black tea, and in college I upgraded to coffee. Caffeine is a reasonably healthy drug, unlike alcohol which is toxic. Caffeine can also boost mental focus for brief periods, and a lot of our economy runs on coffee and tea, perhaps as much as it runs on gasoline.

For me, caffeine’s addiction slope was less slippery than alcohol - I could easily maintain a certain amount of coffee per day for a long time, without needing to increase - it felt more like a staircase, than slope. The reason I decided to opt-out of caffeine is that I thought was abusing it to force myself to do work I didn’t actually want to do. I also had the hypothesis that the human body should be fine without it, as we’ve had most of our biological evolution without it. A wolf doesn’t drink an espresso before chasing a deer, after all. I didn’t find good long term research on caffeine addiction - it is hard to find subjects for a control study when nearly everyone uses, and who the hell wants to give up coffee and suffer. Whatever research I found seemed to only focus on short term, and I was curious about the long term effects of caffeine.

So, on March 14th 2018 (pi day!), after weaning out caffeine for a few weeks, I quit it. The self-arrest shook me up, and caffeine withdrawal felt much worse than alcohol withdrawal. For a few weeks, I had headaches, exhaustion and other withdrawal symptoms. I was between jobs, and I had the privilege to afford the space for the withdrawal, and to realize that I can be productive without caffeine, perhaps even more productive over the long term. Since then, I’ve learned to recognize how music, exercise, cold showers and pomodoro timers can induce the same level of focus. And I’ve learned to appreciate how I can be more motivated by having skin in the game.

Today, I will only have caffeine in “emergency” or if I am climbing a high elevation. I also occasionally have small amounts of dark chocolate, which has caffeine too. My Shelling point is very close to zero, and honestly, caffeine doesn’t feel that good to me any more, after the first few minutes of focus. I feel grateful for having reduced my dependency on it.

Information slipping into entertainment

YouTube contains a lot of useful videos. It has helped me do a few minor repairs, and taught about a bunch of technologies through instructional videos. But the YouTube’s video recommendations can cause me to stick around for much longer. I come for the information, but stay for the entertainment. The information is valuable, it makes my life objectively better if I learn how to repair a light fixture. And some entertainment is also valuable as helps me relax. But as I get entertained I want more, more, more. I find excessive entertainment is not only not helpful - it is not that enjoyable either! Excessive use of such media is mostly a lose-lose game. We, as users waste our time, and society loses our potentially valuable contributions. And even YouTube may lose in the long term, as its users churn cold turkey when they realize it is a problem. The only people who win in the short term are advertisers, which buy our time for pennies per hour.

Other media outlets like TV(CNN, Fox News, etc.), Facebook, Instagram, Twitter, and so on, can be just as slippery. They may seem valuable, but beyond providing some basic information, these media outlets optimize for sheepish behavior, and turn their consumers into sheep. They add more and more addictive entertainment to their content. Even if that entertainment takes the form of making us angry or scared. Such entertainment is empty and it doesn’t make our lives better.

I also strongly suspect it affects our body chemistry - we get a dopamine boost from every funny video we watch. Opting out of this game is hard, as our bodies will literally go into withdrawal, and we’ll try to replace our dose of dopamine with something else. We may think that what happens in our minds stays in our mind, but it doesn’t. It has physical effects on us, and we’re literally poisoning ourselves with our screens. As of writing this, I still have an excessive scrolling habit. I have not yet been able to redefine my relationship to it.

And I’m still experimenting with ways to self-arrest and opt out of it.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Upgrade tolerant way of setting the path to Oracle SQLcl with Homebrew

Philipp Hartenfeller Blog RSS Feed — Mon, 20 Dec 2021 02:00:00 GMT

With two lines in your zsh config file, Oracle SQLcl will still work after an upgrade with Homebrew.

End of Year Reading 2021

THINK@RJ — Sun, 19 Dec 2021 17:40:32 GMT

Top book recommendations for end of year reading in 2021

Getting started with Hakyll on Fedora

blog.marbu.eu — Sun, 19 Dec 2021 13:53:10 GMT

Posted on 28 March 2021

Tags: Fedora, Haskell

Table of Contents:

Installing Hakyll on Fedora
Initializing Hakyll project
Building the example site
Next steps

Upstream installation tutorial of Hakyll static site generator shows how to use cabal or stack to get hakyll installed and build example site with it. And while the tutorial also mentions an option to use distribution packages on Debian or Fedora, the rest of the tutorial doesn’t explore this option further. In this post I provide a quick overview of installation and usage when Hakyll is installed from rpm packages on Fedora 33.

Hakyll is a static-site generator written in haskell, inspired by other such tools like Jekyll, nanoc or yst. It uses haskell DSL for configuration in a similar way how xmonad does, and for html building, it relies on versatile pandoc document converter.

Reasons I use Hakyll to build my personal blog include:

I simply prefer static-site generator approach for this use case, it allows me to write posts in a simple plain text format and then store them in a git repository.
I already use pandoc for various tasks (converting between wiki formats, generating epub files, exporting my markdown or org mode notes …) because it can convert between lot of markup and document formats. Possibility to use pandoc features in hakyll makes all these familiar possibilities available, which is a plus.
I’m little familiar with haskell and as xmonad user, I find xmonad’s approach to configuration flexible and powerful.

Installing Hakyll on Fedora

Fortunately Hakyll is already packaged in Fedora, so we don’t need to build it themselves via cabal or stack just to have it installed if we don’t have other reason to do so (eg. using latest version of hakyll for testing or development of hakyll itself).

The package we need to install is ghc-hakyll-devel:

# dnf install ghc-hakyll-devel

And everything else will be installed as a dependency. Installing the devel package is necessary because Hakyll’s reconfiguration process includes haskell compilation.

Initializing Hakyll project

First of all, we are going to use hakyll-init tool to create new directory example-site with both content and configuraion files of the example project:

$ hakyll-init example-site
Creating example-site/templates/post-list.html
Creating example-site/templates/post.html
Creating example-site/templates/default.html
Creating example-site/templates/archive.html
Creating example-site/index.html
Creating example-site/images/haskell-logo.png
Creating example-site/site.hs
Creating example-site/contact.markdown
Creating example-site/css/default.css
Creating example-site/about.rst
Creating example-site/posts/2015-10-07-rosa-rosa-rosam.markdown
Creating example-site/posts/2015-08-12-spqr.markdown
Creating example-site/posts/2015-11-28-carpe-diem.markdown
Creating example-site/posts/2015-12-07-tu-quoque.markdown
Creating example-site/example-site.cabal
$ cd example-site

Now we need to compile site.hs, which is haskell source code file with hakyll configuration for the site, into site binary file:

$ ghc --make site.hs
[1 of 1] Compiling Main             ( site.hs, site.o )
Linking site ...

We could also use cabal instead as explained in upstream hakyll installation tutorial, but since we have Hakyll installed from the rpm package, we already have all the devel dependencies installed as well, and there is no need to use cabal just for building of the site executable.

The resulting site tool then provides all hakyll’s functionality:

$ ./site -h
Usage: site [-v|--verbose] COMMAND
  site - Static site compiler created with Hakyll

Available options:
  -h,--help                Show this help text
  -v,--verbose             Run in verbose mode

Available commands:
  build                    Generate the site
  check                    Validate the site output
  clean                    Clean up and remove cache
  deploy                   Upload/deploy your site
  preview                  [DEPRECATED] Please use the watch command
  rebuild                  Clean and build again
  server                   Start a preview server
  watch                    Autocompile on changes and start a preview server.
                           You can watch and recompile without running a server
                           with --no-server.

When we check the size of the executable, we see that it’s quite large. That is because like golang, haskell compiler uses static linking by default. From the system perspective, the executable is dynamically linked though. If we stripe the binary, we can save about 25% of the executable size.

$ ls -lh site
-rwxrwxr-x. 1 martin martin 148M Mar 28 19:13 site
$ strip site
$ ls -lh site
-rwxrwxr-x. 1 martin martin 110M Mar 28 19:47 site

But when the size is really bothering us, we can tell ghc to use dynamic linking instead. This will link the executable with system haskell libraries making the binary file orders of magnitude smaller.

$ ghc -dynamic --make site.hs
[1 of 1] Compiling Main             ( site.hs, site.o )
Linking site ...
$ ls -lh site
-rwxrwxr-x. 1 martin martin 78K Mar 28 19:52 site

Building the example site

So now we can generate html files of the site via:

$ ./site build
Initialising...
  Creating store...
  Creating provider...
  Running rules...
Checking for out-of-date items
Compiling
  updated templates/default.html
  updated about.rst
  updated templates/post.html
  updated posts/2015-08-12-spqr.markdown
  updated posts/2015-10-07-rosa-rosa-rosam.markdown
  updated posts/2015-11-28-carpe-diem.markdown
  updated posts/2015-12-07-tu-quoque.markdown
  updated templates/archive.html
  updated templates/post-list.html
  updated archive.html
  updated contact.markdown
  updated css/default.css
  updated images/haskell-logo.png
  updated index.html
Success

And check the result in _site/ directory. We can also start hakyll preview http server which will present the site on http://127.0.0.1:8000 and auto recompile html files when it’s source files changes:

$ ./site watch
Listening on http://127.0.0.1:8000
Initialising...
  Creating store...
  Creating provider...
  Running rules...
Checking for out-of-date items
Compiling
Success

Next steps

As noted above, the configuration of a hakyll project is done via site.hs file, which contains haskell source code. And as you have most likely already guessed, when you update configuration in site.hs file, you need to recompile site executable.

Domain specific language approach combined with good upstream tutorials and real world examples makes changing hakyll configuration possible even without fully understanding haskell language, but having some haskell knowledge will definitely help a lot. That said I have to admit that if you don’t like idea of learning a bit of haskell (or finding an excuse for that), you won’t be very happy with hakyll.

Do you have a comment or question? You can contact me.

Consuming Articles Offline

Beetle Space — Sun, 19 Dec 2021 10:00:00 GMT

My desktop is my portal to the Internet. [1] Lately, I’ve needed to spend significant amounts of time away from my machine. So I took to printing articles and taking them with me.

The experience has been transformative.

As an example, consider this article:

And as I scroll further …

My Favorite Board Games and How I Play Them

Jim Westergren — Sat, 18 Dec 2021 23:04:00 GMT

Board games is one of my passions. Both as a player and now also as an aspiring game designer. And I can sometimes get obsessed in trying to improve rules of games to better suit the tastes of me and those I play with (usually my wife, son and brother-in-law). I normally play as a [...]

Digital transformation in Eastern Europe

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 18 Dec 2021 20:00:00 GMT

Tell me more about it!

Tick, Tick... Boom!

🦉 olshansky 🦁 — Sat, 18 Dec 2021 13:50:33 GMT

Olshansky's review of Tick, Tick... Boom!

TypeScript <3

Liza Shulyayeva — Sat, 18 Dec 2021 11:45:00 GMT

I’ve started playing around with TypeScript for the first time over the last couple of weeks and it is excellent. I’ve made a few hobby JS games in the past as you can see from the games page. JavaScript was always fun for me, but after years of statically typed languages I have grown to prefer them.

Class Management Reviews: Fall 2021

Lambda Land — Sat, 18 Dec 2021 02:00:00 GMT

A collection of what worked well and what didn’t in classes that I took this semester. This is partially for me to record what things reduced friction for me as a student so that one day, should I become a professor, I’ll be able to run the lowest-friction class ever!

Probabilistic spin glass - Part II

Bytepawn - Marton Trencseni — Sat, 18 Dec 2021 01:00:00 GMT

This is a continuation of the previous article on probabilistic spin glasses, with improvements to the simulation code and improved entropy computation.

Update to Game of Ur Background

ℤ→ℤ — Fri, 17 Dec 2021 20:16:48 GMT

I’ve made an update to the Game of Ur - Background post. A reader took up my challenge to find a closed form solution for the number of valid game placements and found a solution.

Overthinking TypeScript Types

TOGoS's Project Log — Fri, 17 Dec 2021 20:13:37 GMT

All those millions of online meetings are WASTED...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 17 Dec 2021 19:00:00 GMT

…if they don’t bring about LASTING changes in HUMANS.

Iterating on Club Leaderboards

Mike Kasberg's Blog — Fri, 17 Dec 2021 18:00:00 GMT

Iteration is an important part of my development workflow, and it’s an important part of the way we work at Strava. Over the course of my own career, I’ve learned to really value the process of incremental development. By shipping relatively small changes quickly, we can gather feedback, observe important metrics, and continue the cycle with targeted improvements. Ultimately, this helps us continually deliver athlete value on a rapid timeline.

Creating Fastmail Masked Emails without 1Password anytime and everywhere

David Mohl — Fri, 17 Dec 2021 17:39:23 GMT

I love Masked Emails, Fastmails take on one-time-use email addresses. I’ve been doing something similar myself for years now, but it is a bit of a shame that it’s so tightly coupled to 1Password. Sure you can create them through the webinterface now, but it’s still kind of a pain to use and not very convenient (unless you use 1Password).

You can’t even manually trigger creation of one within 1Password, so when the extension doesn’t detect an email signup field, you’re sorta out of luck.

What I want is something like this:

Let’s get it working.

The first thing I noticed is that the request to create a new masked email (through 1Password) isn’t actually coming from the 1Password servers like I first thought, it goes directly from the client to api.fastmail.com, like seen on a reverse-proxy like Proxyman:

By the way, Fastmail also stated they put this functionality right into their open-source JMAP project in a blogpost.

After a bit of reverse-engineering the request format to the API server and replicating it in Golang:

I present you: https://github.com/dvcrn/maskedemail-cli

maskedemail-cli -accountid xxxx -token abcdef12345 create facebook.com

… and you get a new masked email for facebook.com, right in your shell!

Glueing the CLI to shortcuts with KeyboardMaestro

I wanted this to be available everywhere whenever I need to. A full extension for a product that isn’t mine would have been too much work, and I wanted it to work outside browsers as well. With something like KeyboardMaestro this is trivial to do

Here’s the macro from above:

Insert “…”
Retrieve the URL from the current browser window. I had to use some glue applescript here because Firefox has no scripting interface
Pipe it into the maskedemail-cli
Remove the “…” and paste the newly created email

Other thoughts

I am sure this will become more available in the near future. Fastmail likely has some kind of exclusivity thing with 1Password for now, but it’s great to see that we can hack ontop of it so easily. It’s also very cool that this is built right into JMAP. I hope we’ll see more email providers implementing it!

Advent of Code 2021 - Day 3

geekmonkey — Fri, 17 Dec 2021 17:00:00 GMT

Deep under the water, we find ourselves hearing creaking noises. Our submarine handily gives us a diagnostics report. Sadly, as is oft too common in AoC, the report format is complete garbage, to put it mildly.

Our puzzle input comes in the form of binary numbers. The task is to

Monte Carlo Options Pricing

Sasha Kuznetsov's Blog — Fri, 17 Dec 2021 10:00:00 GMT

The options casino

Predicting the movement of stock prices is an alluring challenge with the promise of riches. Unfortunately, predicting future stock prices consistently and reliably is generally considered impossible. However, we can use models to make useful predictions, manage risk, and profit probalistically.

Geometric Brownian Motion

One such commonly used model is geometric Brownian motion (1, 2) - in fact, the famous Black-Scholes options pricing formula assumes this model as well. This has the form

$$ \begin{align} S(t) = S(0)\exp{((\mu-\frac{\sigma^2}{2})t + \sigma B(t))} \tag{0} \label{eq:gbm} \end{align} $$

where $B(t)$ is standard Brownian motion. If we let $S(0)$ to be the initial stock price, $\mu$ to be the mean compounding return, and $\sigma$ to be the volatility of a stock, then we can generate very realistic price paths.

$\mu$ 0

$\sigma$ 0

So, what does this mean, and why is this a good model for security prices?

Logarithmic Returns

In quantitative finance it is common to deal with continuously compounded returns rather than simple returns. What that means is that instead of examining a simple return between trading periods $1 + R_t = \frac{S(t)}{S(t-1)}$ we will examine the compounding return $r_t = \ln{(1 + R_t)} = \ln{\frac{S(t)}{S(t-1))}}$. This has the very nice property that one can recover the price of a stock by simply summing up the logarithmic returns as:

$$ \begin{align} S(t) = S(0)\exp{(r_1 + r_2 + ... + r_{t-1})} \tag{1} \label{eq:rr} \end{align} $$

This is also convenient because a common assumption in finance is that the logarithmic returns of a stock are independent and normally distributed with mean $\mu$ and variance $\sigma^2$. Note that for small returns, simple returns $R_t$ approximate logarthmic returns $r_t$ since $r_t = \ln{(1 + R_t)} \approx R_t$, so the difference between the two returns is not substantial, and thus one can assume both simple and logarathmic returns are normally distributed. Next, if the logarithmic returns are normally distributed, then this in turn implies the stock price is lognormally distributed. This has been observed empirically to some extent, although in real life stock prices tend to have “fatter tails” - meaning rare events happen more often than a normal distribution would predict. Here are the monthly returns for IBM stock showing this behaviour.

Monthly simple and logarithmic returns of IBM stock with fitted normal distributions. We observe more returns at the extremities of the distributions than the normal distribution would predict. We call these 'fat tails' or 'excess kurtosis'. This image has been taken from 'Analysis of Financial Time Series, 3rd Ed.' by Ruey S. Tsay.

This model also works well because it only permits positive security prices. Securities tend to only have positive prices (although in some cases they can go negative - see the oil crash of 2020).

So, we want a process that generates a series of normally distributed returns $r_t$ - which invites Brownian motion as a natural choice.

Brownian motion

Brownian motion describes the motion of a particle in a fluid or gas. Such a particle bounces around “randomly” within the fluid surrounding it. The path it creates appears quite noisy and random. It can essentially be considered a limiting form of the random walk where both the time between steps and the step length approach zero (with some caveats: namely, if the time step is $t$, the step length must be $\sqrt{t}$ - this produces a process whose variance scales exactly with time). Consider flipping a coin to determine whether to take a step forward or backward. Now let the time between steps approach zero. You would trace a path that looks like this:

1 dimensional [arithmetic] Brownian motion. Geometric Brownian motion is the exponential of this process. CC BY-SA 3.0, Source

Brownian motion has the very nice property that the trajectories are normally distributed, which is exactly what we are looking for. We can simulate a Brownian process with discrete steps by sampling from a normal distribution (although if you only cared about the terminal value of a time series, you could sample from any distribution with unit variance; due to the central limit theorem the sum of random variables would approach a Gaussian distribution in the long term anyway).

Hence, Brownian motion provides us with the series of normally distributed logarathmic returns $r_t$ as required in $\eqref{eq:rr}$:

$$ \begin{align} r(t) = (\mu-\frac{\sigma^2}{2})t + \sigma B(t) \tag{2} \label{eq:bm} \end{align} $$

Taking the exponential of this process, we’ve recovered an equation for modelling stock price movement.

Of particular interest here is the drift rate $\mu - \frac{\sigma^2}{2}$ which causes the process to drift linearly, as the name suggests. $\mu$ is obvious - it’s the mean continuosly compounding return for the stock, and in the absence of noise (or volatility), the stock price would only grow (or decrease) by this compounding amount. But what about $\frac{\sigma^2}{2}$? Well, it turns out that the expected value of a lognormally distributed variable is $\exp(\mu + \sigma^2/2)$. Since the stock price is lognormally distributed (while returns are normally distributed), and since we require that the expected value of the stock price after continuous compounding to be $S(0)\exp(\mu t)$, we must apply the correction term of $- \frac{\sigma^2}{2}$ to the process.

There’s also a philsophical argument to be made for choosing Brownian motion as the basis for this model. A common assumption made in finance is that markets are efficient, so there should be no arbitrage opportunities. Hence, the market should be unpredictable - if an actor could predict it, that would be an arbitrage opportunity that would be exploited away. Hence, random noise (which is what Brownian motion is, essentially) is a good choice for modelling markets.

Option Pricing

Monte-Carlo simulation is a statistical technique inspired by the casinos of Monaco. Much like gamblers resigning their fates to probability, we hand over the results of statistical analysis to chance. By running enough trials, we can make conclusions with statistical significance.

Consider evaluating a call option with a strike prices of $105 for this stock. What would be the expected value of the option at expiry, given a geometric Brownian model for the stock’s movement? We can estimate this by generating a series of price paths, each of which can be considered a trial. The terminal value of each trial is related to the intrinsic value of the option via the relation $V = \max(0, S - K)$ where $S$ is the stock price and $K$ is the strike price. The mean terminal value provides us with an estimate of the option’s value at expiry, while the standard deviation of terminal values provides us with a measure of how certain we are that the option will expire close to this value.

We can increase the number of trials to increase the statistical certainty of the estimate. Re-running the simulation (by clicking “Regenerate”) on a large number of trials (below) yields a much smaller discrepency between experiments than on a small number of trials (above).

More Advanced Scenarios

Monte-Carlo simulation of terminal values is a relatively simple simulation, and one that is not too useful as this analysis can probably be completed analytically. The true power of Monte-Carlo simulation is unlocked when analysing scenarios that are more difficult to solve analytically, if not impossible. For example, if we wanted to analyse an American-style option, which can be exercised anytime, we might want to count the probability of a stock price exceeding the strike price at any time during the lifetime of that option. We could do this by counting how many trials cross the strike price boundry, which is easy to do with a Monte-Carlo simulation.

My Family Minecraft Survival Multiplayer Server built on AWS

Stories by Zach Wolfe on Medium — Fri, 17 Dec 2021 08:47:19 GMT

The armada of family and friends off to defeat the Ender Dragon

Last month (November 18, 2021) marked the one-year birthday of my family Minecraft Survival Multiplayer server.

After over 1,500 hours of combined play time and 11 players at peak, I thought I’d share details on the setup and what I’ve learned so far hosting an active Minecraft server on AWS.

Some technologies I use include AWS EC2, ECS, S3, EFS, CDK, CloudWatch, Docker, Typescript, and PaperMC. One day I hope to release these AWS CDK Constructs to Github, so look out for that!

Background

In the winter of 2020, like many other people in our lives, my wife and I were in a rut of depressing news, pandemic anxiety, work stress, and local unrest in Seattle. Cut off from typical face-to-face social interactions at work and being unable to hang out with friends on the weekends, life was lonely and weird.

Enter: Familycraft 2020!

Looking for an escape and an excuse to hang out with loved ones for a dose of normalcy, I asked close friends and family members if they wanted to play Minecraft together over Thanksgiving holiday break.

As a software engineer familiar with hosting Java applications professionally, I also took it as an opportunity for a fun side project to host the Java-based Minecraft Server software myself, with a “backup plan” of renting a server from Apex Minecraft Hosting or similar if it didn’t work out or I lost enthusiasm.

Self-Hosted Minecraft: Requirements

Having hosted two Minecraft servers prior, a private world with my wife and a Happy Hour server with co-workers, both on a single cheap AWS EC2 instance, I had learned a set of requirements if I was going to host a world that others would cherish.

For “Version 2” of my attempts of self-hosting Minecraft, I wanted the following:

Automatic, frequent, durable backups of the world in case of disaster
Lots of RAM, a fast CPU, with 4 or more cores for minimal lag when up to 8 players connected
Fast networking (players were scattered around the US)
A Minecraft server version optimized for many players that receives frequent updates
Hosting infrastructure that was “one-click” simple to tweak
A locally-reproducible method of running the server so I could test new Minecraft versions, plugins and configuration changes on my PC before pushing them to “production”
An easy way to instantly “pull the plug” to save on hardware costs if people stopped playing that wouldn’t interfere with my other side projects
A voice channel and chatroom to rouse anyone interested in ‘crafting (Discord)

AWS: Go With What You Know

hangin’ out at the in-game tavern with cousins across the country

Being familiar with AWS professionally, I opted to go with what I know best. I knew EC2 would offer instances that could achieve the low network latency I needed for cheap enough, and S3 was an obvious world backup solution.

Docker: Packaged, Reproducible Runtime

To run, the Minecraft server software needs a Java installation, a .jar file, and a writable directory for the world files, which seems easy enough.

To achieve my goal of being able to quickly run a carbon copy of the Minecraft world on my PC to test out new plugins, Minecraft versions, new EC2 instance type, or do any world file surgery, I chose Docker so the server was runnable as one neatly wrapped package.

After getting half-way through building my own Docker image, I found an outstanding open source version that did everything I wanted and more. The itzg/docker-minecraft-server image acts as a configurable wrapper around the Minecraft server software, using environment variables to configure plugins, mods, admins, seeds, and version. Kudos to itzg for the awesome work.

I also found a configurable cronjob-PUT-to-S3 Docker image that would suit my needs for regular backups.

AWS CDK: One-line Infrastructure Updates

the “amber Waynes of grain”

A challenge I had with a previously hosted Minecraft server was it was hard to change the instance, instance type, and JVM parameters powering the world.

This meant as the world got bigger and wanted an instance type with more RAM to run without lag, I needed to manually zip and scp files around, manually reconfigure firewall settings, reconfigure backup cronjobs, etc.

To avoid that headache this time around, I decided to spend the time up-front and to use AWS Cloud Development Kit (CDK) to define all of my Minecraft server resources as code and check them in to Git. This makes it simple to tweak things like instance type, make a “carbon copy” of the server, test out new instance types or versions, or roll back if something broke.

After getting a messy version to work, I broke up the stack into subcomponents, called CDK Constructs, to allow me to run multiple Minecraft worlds on a unique cluster or shared cluster, unique filesystem or shared, depending on what I wanted to do, and also to allow me to reason about components individually.

CDK Constructs: Logical Pieces of Minecraft Infrastructure

Logically, I was thinking about the infrastructure in the following pieces:

WorldCluster: a collection of EC2 compute resources to run the Minecraft server instance
WorldPersistence: a collection of runtime storage resources
WorldService: the Minecraft game runtime
WorldBackup: a component that periodically backs up the WorldService’s game files
WorldConfig: a collection of per-instance configurations that are unique to the game — eg: the list of Admins, the Minecraft version (1.17, 1.18.1, etc), a list of plugins/mods, EC2 instance type, RAM
World: Wraps all the above into one package, allowing me to easily add more server instances with a few lines of code.
Stack: A collection of Worlds (maybe a Universe would be a better name?)

A World ends up looking something like this:

https://medium.com/media/d53d14871cd583aa8aed090c0709c5bd/href

..and if I ever want to add another server for testing, another group of people, playing with a new Minecraft update, etc, it’s a one-ish-liner to the stack:

https://medium.com/media/ea5a7e379add60b747495608807a48e6/href

..and a one-liner to update the stack

https://medium.com/media/78adc805c0cc2260d2c723891df713aa/href

Looking under the hood, each logical component has several AWS resources that interact with each other.

https://medium.com/media/57899c9645d93892cabb570859e609f4/href

Making Changes: CDK Shines

“the colonies”

A few months into Familycraft we decided to defeat the Ender Dragon. By that time, our server had gained a few new members, and we wanted everyone to be online together for the big boss fight.

To prepare for that event, I bumped up the EC2 instance type from a t3.medium with 1GB RAM dedicated to the game to a more powerful c5a.large with more CPU cores, a dedicated processor, 2GB RAM and faster networking, which was as simple as making this small diff then rerunning cdk deploy:

https://medium.com/media/ab57541ff369fd6ce2d978f199168856/href

It turns out that this was a good idea! During the event, in the hour-ish that everyone was in the same few chunks during The End Dragon fight, the CPU spiked up to > 47%, which might have overwhelmed the t3.medium.

defeating the Ender Dragon

Minecraft Server version updates were almost as simple, bumping the “VERSION” parameter from 1.16.4 to 1.17.1, and utilizing the hard work itzg put into getting the right Java versions set up.

https://medium.com/media/faf2c76944ca7c7dd32868a108faa52f/href

In Case Of Emergency: World Backups

she can fly!

To prevent against any in-game disasters, glitches, etc, or my mistakes when hosting the server, I needed a ‘fire-and-forget’ backup system.

To achieve this, the Docker image peterrus/s3-cron-backup runs as a sidecar to the Minecraft Server Docker image, peeking at the Minecraft World’s runtime EFS volume with read-only permissions, throwing them in a zip, and pushing to S3.

The image operates on a cronjob of every other day at 7pm which balances storage costs with recovering in-game projects. I’ll also occasionally manually trigger a backup after someone completes something epic. Again, CDK really shines here, allowing me to configure the backup image and schedule using auto-generated resource names.

https://medium.com/media/c23c8fcdac61659d76f1558109bd6f1c/href

Optimized For Multiplayer: PaperMC

In previous worlds I had used the vanilla Minecraft server jar directly from Mojang. These ran “okay”, but I saw frequent micro-stutters, rubber banding, and ghost blocks (where the client thinks there’s only air in a voxel but the server has a block), and after some Googling I found a few different versions of optimized server instances that claimed to have better performance when multiple people were online.

There’s a hilarious number of choices to pick from, including Spigot, PaperMC, Forge, Fabric, Airplane, Purpur, etc, etc, and I didn’t want to try them all. I read an article by Maddy Miller suggesting PaperMC was the best, and after making some tweaks (re-enabling TNT duping), it definitely runs more smoothly than vanilla.

This doesn’t come without a cost though. For example, Minecraft 1.18: Caves and Cliffs was officially released Nov 30, 2021, however as of writing (Dec 16), PaperMC has blocking bugs, which was a disappointment to have to wait.

Docker Run: Testing Dangerous Changes Locally

When Minecraft releases a new version, or there’s a new plugin/tool we want to try, it’s extremely useful to run the server locally on my PC before pushing those changes out to everyone that plays on the server.

To do this, I download a recent automatic update and run a few commands to start the server

https://medium.com/media/da95df052408d9617e3e021c5992a443/href

and then configure the client to connect to localhost

..then I run around the world, which is both served by and running on my laptop, and play around with the important farms and areas of the world.

This was crucial recently, where I discovered some breaking bugs in the 1.18 PaperMC update that would have griefed everyone.

Craftin’ Tonight? Player Join Notifications

creeper farm + perimeter — beacon for scale

One kind-of awkward part of playing with a group was getting people to join around the same time to play together.

We use Discord for voice and video calls while we’re playing, so I spent a few hours writing an open source Spigot/Bukkit/Paper plugin, minecraft-player-notification, that sends a message to a Discord channel whenever a player joins a Minecraft server.

This nudges folks that someone’s around, and to pop in and say “hey” in-game.

I built this with all intentions of it only ever being used by our server, but to my delight it’s been used by something like 30 others as well!

The plugin works by intercepting PlayerJoinEvents, and using HTTP POST Webhooks to notify a Discord channel.

https://medium.com/media/c2825a20c85eb5327a45ad4ff486f872/href

It has a few configuration knobs, changed with a config file like this, and it would be easily extended to other notification types, like rich Discord messages, SMSes, etc.

https://medium.com/media/1f0760cdc0da17710ee39e20a380c559/href

Fiddly Bits: ECS + EFS = :(

rare blue axolotyl!

The most annoying component to get set up was the runtime world persistence. Getting EC2, Docker, ECS, and EFS to play nice took the most time of anything, and required a lot of trial and error.

The difficulty was in getting the Docker Container to allow connections to the EFS service backend, which required BRIDGE network mode and a few in/out ports punched in the EC2 Security Group.

I wasn’t able to find much documentation or prior art of how to achieve this in CDK, so this took a long time to figure out.

https://medium.com/media/7c2dca3e782c4911c0c659de3ab3e8b4/href

What’s next?

One thing I want to add to the setup is a CloudWatch alarm-based auto-sleep system to save a few bucks per month on hosting. Whenever there’s a period of X hours with nobody connected, the ECS service should be scaled down and the EC2 instance paused. The only reason I haven’t done that yet is to make sure the server is available for players when they want it, so I also want to build a simple web portal that players can use to turn it on without my help.

Another thing I’d like to add is DNS. Occasionally when making updates to infrastructure, the EC2 Autoscaling Group decides it needs a fresh EC2 instance, which rotates the IPv4 address. To support that, I’ll eventually get around to a Route53 DNS address and subdomain for the Minecraft server.

Appendix

Familycraft name

We call our family Minecraft server Familycraft, and were unaware of “Familycraft Dad” at the time. We’re not associated with him in any way, but he seems like a cool guy.

List of Mormon History Association "Best Book" award books

David Schlachter — Fri, 17 Dec 2021 02:36:00 GMT

The MHA's list of their "Best Book" award winners only lists authors, not the book titles. Here's an easier list to browse.

Finding Sierpiński in the Oddest Places

ℤ→ℤ — Fri, 17 Dec 2021 02:23:08 GMT

This is a tale of finding fractals in an unexpected place and why the appearance makes sense in hindsight.

I was looking for patterns in valid board game configurations, specifically the middle contesting row in the Game of Ur. In the game, there are eight slots either of the two players may occupy, but a slot may only be occupied by at most one player. I represented the available slots as two bit strings, with each bit representing a single slot. A valid placement is one where the two corresponding bits are both false or only one is true, but never both are true. Formally, this is equivalent to the bitwise formula $\neg (p \land q)$ or $p \uparrow q$.

Plotting the p and q bitstrings as axes within a matrix, I saw:

I recognized this set of repeated triangles as a Sierpiński Triangle (or Gasket). Described by Wacław Franciszek Sierpiński in 1915 (mathematically; there may have been artistic representations in the 13th century by the Cosmati¹), the Gasket is an iterative structure with infinite scaling. The Triangle is considered a fractal (and one of the first discussed in any primer), but the discovery predates the field of fractal geometry. A typical way to describe the construction² is to inscribe a triangle with coordinates (0,0), (0,1), and (1,0). Then, inscribe three triangles via the transformations:

$(x,y) \rightarrow (\frac{1}{2}x, \frac{1}{2}y) $
$(x,y) \rightarrow (\frac{1}{2}x + \frac{1}{2}, \frac{1}{2}y) $
$(x,y) \rightarrow (\frac{1}{2}x, \frac{1}{2}y + \frac{1}{2}) $

Animating this construction (note that this figure has been rotated versus the matrix orientation):

Of course, the Triangle’s representation can also be rotated and projected without impacting the structure. Equilateral constructions are very popular.

There are many ways to construct the Triangle, including many that aren’t graphical. The Triangle is registered as OEIS A001317 Sierpiński’s triangle (Pascal’s triangle mod 2) converted to decimal. The catalog entry contains 16 formulas and 17 links. Due to the link to Pascal’s Triangle, which has a tremendous number of constructions, this is a small subset of the known set.

Looking through the catalog, I tried to find one that corresponded to how I was generating the Triangle but didn’t find one. The exclusive-or method — $a_{n+1} = a_n \otimes 2*a_n$ — seemed like a good candidate, but I did not find an an algebraic equivalence.

Since my formulation wasn’t generating integers but rather a binary matrix, I found a different formulation that was defined for binary matrices:

$$a_{i=1,j=*}=1$$

$$a_{i=*,j=1}=1$$

$$a_{i,j} = a_{i-1,j} \otimes a_{i,j-1}$$

This formulation can be easier to see in smaller matrices:

I was still having difficulty finding an equivalence between this formula and mine, although the above seemed closer. In particular, I started thinking of the representations of binary numbers and how my ordering incorporated binary operations. Thinking that I might have discovered a novel representation, I expanded my research and found that, yes, someone had discussed this construction before (and they didn’t even treat it as novel). Alas.

Fractal Images of Formal Systems³ explores the intersection of formal systems (e.g. propositional calculus) and fractals via visualizations. (Academic works tend to be what they say on the tin.) In one of the examples, the authors explore computing NAND (not and; often depicted via the Sheffer stroke $\uparrow$) of two sentences of propositions. They found that ordering these sentences into ascending order and plotting the results yielded a Sierpinski Triangle. This approach is analogous to my application of NAND’ing two binary strings.

So, why is the Triangle generated by the NAND of two bitstrings, projected onto a two-dimensional matrix? Drawing upon the discussion in Fractal Images:

First, we can consider one of the classic constructions (similar to before): For any given triangle, the interior points can be divided into two classes. The first class of points are ones where, if we measure the distance from them to the closest vertex of the surrounding triangle, a doubling of that distance would be outside the triangle. The second class of points are the ones where the doubling of the distance would be inside the triangle. The first class will form an empty inverted triangle in the center. Of the points in the second class, we can repeat the process by separating points that would be outside the triangle by twice doubling the distance versus ones that would be inside. A Sierpiński Triangle is formed via the infinite repetition of this process.

How can we map the above construction of the Triangle into our NAND matrix world? Consider the NAND logic inscribed on a unit square, where the true values are connected to form a triangle:

If we implement the doubling logic within this unit square, the logic acts as:

Closest Vertex	Translation
A (Origin)	$ (x,y) \rightarrow (2x, 2y) $
B (1,0)	$ (x,y) \rightarrow (1 - 2(1-x), 2y) \rightarrow (2x - 1, 2y) $
C (0,1)	$ (x,y) \rightarrow (2x, 1 - 2(1-y)) \rightarrow (2x, 2y - 1) $

(It may be helpful to think of an inward pointing vector from the vertex.)

What does doubling mean in terms of numbers encoded as binary strings? Multiplying a binary number by 2 is equivalent to a left shift (and halving is equivalent to a right shift). Subtracting a one will invert the lowest bit (among other potential changes).

Shifts either insert a zero at the lowest bit or remove the lowest bit from the string. A pair of zeros will not change the result of a NAND computation, so if two strings are valid, we can left shift them infinitely with no change. Similarly, if two strings are valid under NAND, stripping the lowest bits will not change the result of the computation.

Subtracting the one will eliminate any one-bits on the lowest bit. If the lowest bit is not set, this leads to an arithmetic subtraction that may invert a number of bits depending on the original string (borrowing). How does borrowing avoid violating the NAND rules? For the two original strings, there are three possibilities for the lowest bits: x=0 y=0, x=0 y=1, and x=1 y=0. (The case of x=1 y=1 would be invalid per NAND.) For the second and third case, the subtraction will borrow from a placement of 1 leading to a valid string. For the first case, rule A does not require borrowing. Thus, the recursive nature of the construction remains valid under the rules of NAND and the Triangle is approximated by NANDing binary numbers.

Brunori P., Magrone P., Lalli L.T. (2019) Imperial Porphiry and Golden Leaf: Sierpinski Triangle in a Medieval Roman Cloister. In: Cocchiarella L. (eds) ICGG 2018 - Proceedings of the 18th International Conference on Geometry and Graphics. ICGG 2018. Advances in Intelligent Systems and Computing, vol 809. Springer, Cham. Author PDF ↩︎
Falconer, Kenneth. Fractals: A very short introduction. OUP Oxford, 2013. ↩︎
Denis, Paul St, and Patrick Grim. “Fractal images of formal systems.” Journal of philosophical logic 26.2 (1997): 181-222. PDF HTML ↩︎

Millionaire's Shortbread

Dustin.Lammiman — Fri, 17 Dec 2021 02:00:00 GMT

Millionaire's Shortbread Ice Cream #

Ingredients #

1/2 Cup Granulated Sugar
2 Tbsp Skim Milk Powder
1/4 Tsp Xanthan Gum
3 Tbsp Cocoa Powder
1 1/3 Cups Milk (I used 2%)
3 Tbsp Water
2 Tbsp Light Corn Syrup
1 1/3 Cup Whipping Cream
Shortbread Cookies

Instructions #

In a bowl whisk together sugar, milk powder, cocoa powder, and xanthan gum.
In a pot add milk and corn syrup.
Whisk dry ingredients into the pot of liquid ingredients. Be sure to mix well.
Heat ingredients for ~5 minutes on medium heat, stirring well, until sugar is disolved.
Remove from heat and add remaining cream and allow mixture to cool for a few minutes.
Refridgerate for 12+ hours. Then churn according to your ice cream maker's instructions.
When ice cream is almost finished churning, cut shortbread cookies into small pieces and add to the ice cream dasher.
Once it is well mixed, move your ice cream into an airtight container. Layer half the ice cream, then caramel sauce, then the rest of the ice cream.
Freeze the ice cream for 4+ hours before serving.

Caramel Sauce #

Ingredients #

3/8 Cup Salted Butter (start with 1/2 cup of butter, and use 3/4 of that)
3/4 Cup Brown Sugar (packed)
1 Tbsp Water
Pinch of Salt
1/4 Cup Evaporated Milk (I know this is a small amount from a large can - every Christmas I make a batch of fantasy fudge and so I'm always looking for recipes that will use the excess evaporate milk. If you don't already have an open can, there are plenty of great caramel recipes that use heavy cream that should work fine instead.)
1 Tsp Vanilla Extract

Instructions #

Melt butter, sugar, water, and salt together in small pan.
Bring to a boil, then reduce to a high simmer for about 5 minutes.
Remove from heat and add evaporated milk and vanilla.
Mix well, then move to an airtight container and allow to cool completely in the refridgerator before adding to your ice cream.

On digital gardening

Fabian's public notepad — Fri, 17 Dec 2021 02:00:00 GMT

On content-focused websites the prevalent pattern for the organizing content is the blog format: a list of articles/posts presented in reverse-chronologically order. Amy Hoy wrote a criticism of this organizational pattern, which found some resonance. So nowadays - as everything old is new again - a bunch of personal websites started to conscientiously style and organize themselves as digital gardens.

Now, there is no copyright to that term, but one overarching concept I see and very much like is, that articles are not seen as something that is written once and then finished after the publication, but as a seed which might grow into something that eventually becomes evergreen, more akin to a well-organized public (read-only) wiki.

I haven't yet figured out how I will this page in the long run, currently it is much more a blog than a garden, yet I find the idea to go back to existing article and enhance them over time very intriguing.

Device neutrality coming to Europe?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 16 Dec 2021 20:00:00 GMT

Yay. Partially, that is.

Strangers from the internet paid for my MacBook Air: on my 4th year of indiehacking

Max Rozen — Thu, 16 Dec 2021 07:52:00 GMT

In which I finally make money from strangers off the internet.

Static Pods in Kubernetes

Ye Lin's Random stuff — Thu, 16 Dec 2021 06:23:13 GMT

After taking the CKAD exam, I am planning to take the (harder) CKA exam. As I was studying for it, I stumbled upon Static Pods in Kubernetes. It is slightly different from the regular Pods.

This is the normal Kubernetes cluster components

From Cloud9 Kubernetes Concepts and Architecture

Learn more about the Control plane components here.

Now, let’s imagine there is a Kubernetes cluster with no Schedular or Controller or even no Master Node. There are worker nodes. On each worker node, there is kubelet running. Now, how can we tell kubelet to run a Pod itself?

The Intermediate Engineer Syndrome

PseudoFreedom — Thu, 16 Dec 2021 03:39:25 GMT

Graduating with a Computer Science degree from Duke University, Wen followed some of his friends into a well respected tech company. Wen worked hard, consistently hit his goals at work, and gained a great deal of technical competency in the first 18 months. A timely promotion to Software Engineer II came just in time for a confidence booster.

Kubernetes Quickstart

Jason Eckert's Website and Blog — Thu, 16 Dec 2021 02:00:00 GMT

Over the past two decades, we’ve shifted from running servers bare metal to running them almost exclusively within virtual machines (VMs). More recently we’ve seen another shift to running servers within containers. Like VMs, containers are servers that contain a unique filesystem, IP stack and software. But unlike VMs, containers do not contain a full operating system or operating system kernel. Instead they rely on an underlying operating system that runs the container using a container runtime, such as Docker. Consequently, containers are much smaller than VMs and ideal for running software within cloud environments where resource utilization must be kept to a minimum to control costs.

Running SQL Server in a container on Github Actions

Notes on software development — Thu, 16 Dec 2021 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Why you want labels for software, just like for food

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 15 Dec 2021 20:00:00 GMT

Even more, possibly.

TIL how to create Github repos with Github

zufallsheld — Wed, 15 Dec 2021 15:55:00 GMT

I needed to create multiple Github repositories and grant access to them for different users. After creating the first repository by hand and granting access to it, I had enough of it. There are just too many damn clicks!

So I used the CLI. Github has a nice cli-program called …

Exploring the Nature of a PC

one mikro2nd — Wed, 15 Dec 2021 14:37:10 GMT

In which I speculate about some of the attributes I’d want to see in a Personal Computer that takes seriously its job of uplifting my ability to think, understand, explore and organise information. An Upsight: When it comes to taking notes during meetings and reviews I reach for pencil and paper. Not the keyboard (though others seem comfortable doing so). Is it just that other folk are more fluent at the keyboard than I?

New York, 2021

Avy Faingezicht — Wed, 15 Dec 2021 02:00:00 GMT

After a weekend in Philly, our Thanksgiving trip continued with my family in New York City. My mom flew in from Costa Rica, while Max and Einat drove from Boston with Rollo in the back. Hopefully next time Danny, Caro and the kids will get to join us. I also brought my film camera, so expect a few stragglers to come from that in a second post later on.

How not to learn Rust

dystroy|Canop / blog — Wed, 15 Dec 2021 02:00:00 GMT

I've seen too many good programmers struggle learning Rust, or even give up.

Here are the mistakes I've seen which may make you fail at learning Rust. I hope this list will help you avoid them.

The Foolproof Scones

Brent's Blog — Wed, 15 Dec 2021 02:00:00 GMT

These scones take max 30 mins to prep and cook and are very simple and enjoyable.

Certain "fact checks" are nothing more than opinion

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 15 Dec 2021 01:50:00 GMT

At least on Facebook, that is.

How to find a namespace number

River Writes - A MediaWiki Blog — Tue, 14 Dec 2021 17:28:29 GMT

Here’s how you can find the number of a particular namespace

Advent of Code 2021 - Day 2

geekmonkey — Tue, 14 Dec 2021 17:00:00 GMT

The advent of code is an annual coding challenge in the month of December. This article covers the second day's puzzle with a solution written in Go.

Converting my init.vim to lua

Imran's Blog — Tue, 14 Dec 2021 02:00:00 GMT

I use neovim as my main driver for all my coding and text editing needs. They released version 0.5 (up to 0.6 now) which supports having your vim configuration in lua (5.1 via LuaJIT) instead of in vimscript.

I've never used lua before but it's a nice small language. LuaJIT is also crazy fast. You can find most of what you need to know with either :h lua and some syntax reference. One of the nicest changes is the ability to use maps for storing configuration for plugins. For example the following ale configuration:

let g:ale_fixers = {
\   '*': ['remove_trailing_lines', 'trim_whitespace'],
\   'python': ['autopep8', 'black', 'isort'],
\   'javascript': ['eslint'],
\   'vue': ['eslint'],
\   'dhall': ['dhall-format'],
\   'go': ['gofmt'],
\   'rust': ['rustfmt']
\}

Becomes this:

vim.g.ale_fixers = {
    ["*"] = { "remove_trailing_lines", "trim_whitespace" },
    python = { "autopep8", "black", "isort" },
    javascript = { "eslint" },
    vue = { "eslint" },
    dhall = { "dhall-format" },
    go = { "gofmt" },
    rust = { "rustfmt" }
}

Some rough spots still exist (e.g. auto commands) but there is a nice vim.cmd escape hatch that can still take regular vimscript. Super excited to see what else is in store for this project!

Hide and Seek by Fiona Benson

ho.dges.online — Tue, 14 Dec 2021 02:00:00 GMT

*This poem really affected me when I read it.

After her swim I wrap my child warm
and take her to the changing room
and lay her down to dry. She holds the corners
of the towel up over her face
like a soft, turquoise tent and yells
‘Hide and seek! Hide and seek!’
I life an edge and shout ‘Boo!’
and she shrieks with laughter -
I can feel the heat rising from her body
and smell the chlorine - she hides again,
and again. I peek under and she’s beside herself
with happiness - she’s at an age where she thinks
that if she just stands still in the middle of the lawn
I will not see her, that somehow she is gone -
but always, in the pockets behind this game,
there is this residue, this constriction,
families squeezed behind false walls
or hidden under the floor. I think of the soldier
sensing the hollow under his sole
and prying up the board on all those cramped
and flinching humans; but mostly I think
of the mothers, their hearts jumping out of their mouths
trying to shush their children - my first-born now,
who’s never been able to do as she’s told,
how she’d have writhed and screamed and bitten like a cat
if I’d tried to hold her quiet, how I’d have hurt her,
clamping her mouth, trying to keep her still.
The trapdoor is always opening, the women and children
are herded into the yard - and I ask myself if,
when my daughters were pulled from me,
I would fight and scream to keep them,
or let them go gently, know
there was nothing to be done?
If we were pushed into the showers
would I pretend it was only time to get them clean?
We are not meant to write of the Sho’ah,
we who were not there, but on bad days it’s all I can think of,
the mothers trying to shield their children with their bodies
under the showers, screaming for mercy, begging for rain.
And it’s never over - here are the children
riding to the border in fridges as the air becomes hot and thin,
their tiny bodies glowing like bright sardines
on the customs officer’s hand-held scan;
and here is the tribesman carrying your husband’s genitals
and a bloody machete, and you are a mother
running for your life with a baby tied to you back
and two children by the hand
but one small son is falling behind;
Jesus fucking Christ, I don’t know who
I’m teaching you to hide from, but look
how eagerly you learn.

Intentional weakness, getting a job in tech, and OSS funding - 123dev #50

Justin Garrison's Homepage — Tue, 14 Dec 2021 02:00:00 GMT

I'm back to custom issue titles. I'll add a small intro here too. Welcome all th

On Satisficing

Fabian's public notepad — Tue, 14 Dec 2021 02:00:00 GMT

Every day we need to decide many things and many of those decisions are of minor importance. Lots of trivial decisions to make can create neural fatigue and lead to unproductivity and loss of focus.

Herbert Simon coined the word satisficing for a strategy that concentrates on getting an option that is good enough, but possibly not the very best. Satisficing is one of the foundations of productive human behaviour because it avoids wasting time on finding improvements that are not going to make a significant difference.

Yet, satisficing should not be applied to very high-priority endeavours where much is at stake. There pursuing excellence might actually be the better strategy. But those are few and far between. To have the capacity to make them happen, satisficing is an imperative.

WhatsApp and the domestication of users

All content on Seirdy’s Home — Tue, 14 Dec 2021 01:26:03 GMT

WhatsApp's rise and recent PR efforts highlight a class of business models that I call "user domestication".

Setting up Katex - Maths in Hugo

Yasha Solutions — Tue, 14 Dec 2021 01:00:00 GMT

So, i had this planned for a while to add some more math content to this blog. Mostly for the content I do in french, where I do a lot of school related content for the kids but also for here, in english, where I will likely need it later when adding content about recommender system and other ML topics. Anyway, figured out that Latex is the way to go usually when you want to do some maths in a paper, and well, turns out there is a library for this name Katex and it’s super simple to integrate in Hugo.

Will my new blood pressure monitor be the thing to get me to finally meditate?

Liza Shulyayeva — Mon, 13 Dec 2021 21:02:00 GMT

I know meditation is good for me, but I’ve never been able to stick with the darn stuff. I just…don’t do it. I do something else - anything else - instead. I’m always too busy, or too anxious or excited to move on to other things.

Link rot, explained with just one post

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 13 Dec 2021 18:00:00 GMT

One post, and 45 images. Missing images.

Guide to 3D OSM Building Models - Milan Edition

Geography & Coding — Mon, 13 Dec 2021 16:54:06 GMT

OSM data offers a vast amount of information about our cities. In this guide I focus on working and styling OSM building data for 3D-modeling in Aerialod for close-to-reality models.

Tl;dr: Use OSM building levels, emulate roofs with shapeburst effect and use Sketchfab for interactive 3D models

There and Back Again: GraphQL at strongDM

Evan Todd — Mon, 13 Dec 2021 15:00:00 GMT

This post originally appeared on the strongDM engineering blog.

Once upon a time, strongDM had no dedicated frontend engineers. We backend engineers dipped our toes in the React frontend as infrequently as possible. It relied on dusty, bespoke, private REST endpoints that returned schema-less JSON blobs. We let these languish while we built a shiny new public API with code-generated SDKs in five languages. Occasionally I was forced to write a new private endpoint for the frontend. I did so with shame, dreaming of a future when the whole frontend would only talk to the public API. Then one day, we hired a frontend engineer, and everything changed.

ZFS RAIDZ2 - Achieving 157 GB/s

Neil Panchal — Mon, 13 Dec 2021 05:31:34 GMT

Update: See Note 5 below . 157 GB/s is a misleading bandwidth due to the way fio lib handles the --filename option. Actual bandwidth is approximately 22 GB/s, which is still mighty impressive.

I built a new server that's also going to serve as a NAS. It

Aten: introduction and the first plot

Blog Frank Boerman — Mon, 13 Dec 2021 02:00:00 GMT

As those that I have spoken with personally recently are well aware by now, I am working at TenneT at the System Operation Transport division, mainly on flowbased capacity calculation and bidding zone review topics. I have chosen TenneT because I wanted to play a part in something that both fascinates my and in which I hope to make a small difference for good: the energytransition and the energy market.
The more I work on it the more I become fascinated by all the intricate details and endless possibilities to analyse it, encouraged by my helpful co-workers. From this fascination a new hobby was born: project Aten. I like to name my projects after greek-roman, old norse or ancient egyptian mythology and this one is named after the egyptian sungod Aten.

The benefits of a classless approach to CSS

Fabian's public notepad — Mon, 13 Dec 2021 02:00:00 GMT

I first experienced the web in the mid-nineties. At that time CSS wasn't really around and even if the first browsers supported it, many pages were just text-heavy unstyled documents. Today you would consider such an approach as luddite, yet since the H in HTML stands for hypertext you could argue that keeping styling concerns out of markup is the purest form of authoring a document for the web.

Today's default choices are quite distinguishably different from that. A minified version of Bootstrap 4 ships 117kB. As a comparison: I would have to write about 60 to 80 additional articles on this site to match that weight with an equal portion of content. And while at it, I'd have to clutter my markup with classes over classes. There are of course slimmer css frameworks than Bootstrap, but nonetheless, my main motivation is in the writing down useful things, therefore I decided to approach the styling of this site quite radically with classless CSS. All styling decisions are taken by CSS which is targeted by element selectors only. If I'd bother to minify it, that would clock in little over half a kilobyte, and even unminified it is less than 1% of bootstrap.

But page weight isn't the most important thing. A classless CSS approach enables me to author content without intermediate representations (e.g. the Markdown flavour du jour in combination with a static site generator) directly in HTML while still allowing me to focus on the content.

Opt Out of Cynicism

D13V — Sun, 12 Dec 2021 23:44:28 GMT

A cynic is a man who knows the price of everything, and the value of nothing.

Oscar Wilde

Welcome to part 8 of the series on “How to be a game changer”. You can find the previous posts here 1, 2, 3, 4, 5, 6, 7.

Growing up in post-socialist-turned-cowboy-capitalist Bulgaria, I grew up around a lot of cynical behavior and absorbed it deep into me. It was the water I was swimming in, and I knew no better. There was always this feeling that attempts at improvements are futile. If anyone tried to improve the system in any way, they will face a great opposition, and any value they bring forward will be immediately vultured away. This made it obvious for me to see how any changes will be abused and rendred futile. I also became good at rationalizing the existing status quo. There’s this example which stuck with me, that if someone created a coin operated parking meter, another one will quickly figure out how to steal the coins out of it. Thus, the attempt to bring order will fail, and the rationalization is that we are a motivated but backstabbing people which get in our own interest.

And today, I still don’t know much better, and sometimes view the world through cynical eyes. A cynicism frame and mindset that turns a lot of my life into a zero-sum game. When I recently realized that I’ve been carrying this cynical mindset for a long time, my legs felt soft, and my body started shaking. I was reckoning all the mistakes I’ve made with this mindset.

But what do I mean by being cynical? When you’re conversing with someone, it is often easy to detect the frame through which they view the world and the interaction with you. Cynics think that the world is never changing on a fundamental level, but it only changes cosmetically, and superficially. Cynics may look at some new trend and call it “lipstick on a pig”. According to cynics, the world operates in a certain way, the elites are always the same, and will always be the same. The cynics think we are always playing the same game that has always been played. For example, a cynic may think that media’s only purpose is to sell your attention to advertisers - like I did here. And often they might be right.

We never try to be cynical. At least not consciously. Instead, we try to make models of the world. And we often simplify those models by ignoring the fact that the world today is different than the world ten years ago, and alien to the world a hundred years ago. We take a mental shortcut, but we end up in a dead end.

Optimists, and even sometimes pessimists, see the world changing, for better or worse, and with it, the games we play change as well. They’re also both right to some degree. Cynics have fixed mindset, whereas optimists have growth mindset. The opposite of an optimist isn’t a pessimist, as they both realize that the world is changing and time moves forward. A pessimist also has a growth mindset, except they have a negative sign in front of it. The opposite of an optimist is the cynic.

We are more likely to detect cynicism in others than in ourselves. For example if you approach a friend suggesting you both bake a pie together, they start talking about dividing the pie - they have the cynical fixed mindset. Or if a friend complains a lot, that’s also a cynical mindset - they don’t see a way to change the world, or their environment, and all they have left is complaining.

When we detect in others, we’ll be more likely to detect in ourselves as well, and not only filter cynics out of our lives, but sometimes re-align them to a growth mindset, and attract other partners with growth mindset. Growth mindset says “yes and…” instead of “yes, but…”, or “no, but…”. And we may even re-align ourselves, to start recognizing all the ways the world may change - for better or worse.

One of the strongest medicines against cynical mindset which I’ve tried is to do a “no-complain challenge”. I find its effects to be enormous and durable for years. In this challenge, you move a bracelet from one wrist to the other when you catch yourself complaining. The goal is to make it 21 days without having move the wrist. I did it years ago, and by the end of it, it felt as if I am in a different universe. With the years, some of its effect has rubbed off, and I am due a refresher.

By realizing that the world is always changing, and by stopping to complain, we opt out of seeing the world through cynical glasses, and become part of the change. When we realize that our life can change, we can change it.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Advent of Code 2021 - Day 1

geekmonkey — Sun, 12 Dec 2021 16:00:00 GMT

The advent of code is an annual coding challenge in the month of December. This article covers the first day's puzzle with a solution written in Go.

Plain Org v1.1 released 🎄☃️

xenodium.com @alvaro — Sun, 12 Dec 2021 02:00:00 GMT

Plain Org v1.1 is now available on the App Store. The update receives new features and bugfixes.

If you're finding Plain Org useful, please help support this effort by getting the word out. Tell your friends, tweet, or blog about it.

What is Plain Org?

Ok, now on to what's included in the v1.1 release…

Compact mode

By default, Plain Org layout uses generous padding. The new option Menu -> View -> Compact mode packs more content into your screen.

Regroup active and inactive tasks

Regrouping tasks now bubbles active ones up. Similarly, inactive tasks drop to the bottom of their node. Changes are persisted to the org file.

Native table rendering

Tables are now rendered natively but also support displaying links and other formatting within cells.

Open local ID links

If your file provider supports granting access to folders, local ID links (ie. id:eb155a82-92b2-4f25-a3c6-0304591af2f9) can now be resolved and opened from Plain Org. Note that for ID links to resolve, other org files must live in either the same directory or a subdirectory.

Fill paragraphs

If your org paragraphs contain newlines optimizing for bigger screens, you can toggle Menu -> View -> Fill paragraph to optimize rendering for your iPhone. This rendering option makes no file modifications.

By the way, the previous screenshot text comes from Org Mode - Organize Your Life In Plain Text, a magnificent org resource.

Show/hide basic scheduling

Use the new Menu -> View -> Scheduling to toggle showing SCHEDULED or DEADLINE dates.

Show/hide tags

Similarly, the new Menu -> View -> Tags option toggles displaying tags.

Native list rendering

Lists are now rendered natively. With the exception of numbered cases, list items now share a common bullet icon. Description lists are also recognized and receive additional formatting when rendered.

- First list item
* Second list item
+ Third list item
1. Numbered list item
+ Term :: Description for term

Numbered checkboxes are now recognized and receive the same formatting and interaction as their non-numbered counterparts.

1. [ ] First checkbox
2. [X] Second checkbox
3. [X] Third checkbox

Reload current file

Plain Org may not be able to automatically reload files for some syncing providers. In those instances, use Menu -> Reload to explicitly request a reload.

Open .txt files

Although .org files are plain text files, they aren't always recognized by other text-editing apps. This release enables opening .txt files, so you can choose to render them in Plain Org, while giving you the option to edit elsewhere.

Bugfixes

Improve vertical whitespace handling.
Fixes rendering edge cases.
Fail gracefully when creating new files on unsupported cloud providers.
Prevent creating new files with redundant extensions.
File access improvements.
Replicates property spacing behaviour using Emacs's org-property-format default value.
Fixes keyword picker border rendering.
Improves rendering performance for large nodes.

"Open source" is not broken

Blog on Kailash Nadh / Personal homepage — Sun, 12 Dec 2021 02:00:00 GMT

I read this article (“Open Source” is Broken by Xe) written in the aftermath of the unfortunate log4j2 fiasco. The author discusses a pertinent problem that has plagued the FOSS (Free and Open Source) world ever since large for-profit corporations started their widespread consumption of FOSS, ever since countless “unicorns” raised infinite amounts of funding on valuations built pretty much entirely on FOSS, ever since FOSS got co-opted into corporatisation and capitalisation. And yet, countless maintainers of critical and widely used FOSS struggle to make a living.

Berkeley Mono December Update

Neil Panchal — Sun, 12 Dec 2021 01:23:29 GMT

It's been a while since I had announced the release of Berkeley Mono. The more I examine it under scrutinous eyes, the more I find that there is more work to be done. At one point, I had the following masters going through the final tuning:

Berkeley Mono

Zen of python

Yasha Solutions — Sun, 12 Dec 2021 01:00:00 GMT

A known thing - I am just posting this here as a reference Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren’t special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess.

2021–12–12: Pinephone Keyboard – p-boot landscape mode

megi's PinePhone Development Log — Sun, 12 Dec 2021 01:00:00 GMT

Analog or digital, prepping makes no sense

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 11 Dec 2021 20:00:00 GMT

Not that it ever did, of course

Guide to 3D OSM City Models - Trier Edition

Geography & Coding — Sat, 11 Dec 2021 17:14:06 GMT

There are few cities providing open source LIDAR data for 3D-modeling. However, leveraging OSM building, road and landuse data gives you neat city models in no time!

After a year in development, my friend Jonatan and I are releasing a new game for all the word game…

ka-yich — Sat, 11 Dec 2021 11:36:56 GMT

After a year in development, my friend Jonatan and I are releasing a new game for all the word game lovers out there:

https://ronigame.com

Migrating a Windows 10 VM to Windows 11 in Parallels Desktop: a story of TPM chips and BIOS upgrades

Nicola Iarocci — Sat, 11 Dec 2021 08:05:25 GMT

This weekend assignment was to upgrade a couple of old Windows 10 VMs to Windows 11 in Parallels Desktop 17. I couldn’t do that right away because Windows Update was complaining about the lack of the TPM chip. A little research revealed that TPM chips only work on UEFI BIOS. To check which BIOS version was being used in my VMs, I used the msinfo32 (System Information) application. It showed the BIOS to be of “Legacy” type. So my task was now to switch it to UEFI.

Philadelphia

Avy Faingezicht — Sat, 11 Dec 2021 02:00:00 GMT

After briefly seeing our friends Annie and Michael in Louisville for their wedding party, we planned to meet them for a bit longer in Philadelphia. We had planned an East Coast trip to see my family in NYC for Thanksgiving, so adding one more city to the itinerary was pretty easy.

Detecting if a PWA/TWA is Installed

Thomas Hunter II — Sat, 11 Dec 2021 02:00:00 GMT

My Kindle 3rd Gen (2010-2021)

Elliott Slaughter — Sat, 11 Dec 2021 02:00:00 GMT

My first exposure to ebooks was through the Kindle platform. I had money saved from a summer internship, and (among other things) used it to purchase a then-new third-generation Kindle. While I have continued to enjoy some aspects of paper, my Kindle had distinct advantages—most notably that it was actually lighter than many of the books I was reading. With it, I was able to maintain longer reading sessions without having to deal with my arms getting sore.

Probabilistic spin glass - Part I

Bytepawn - Marton Trencseni — Sat, 11 Dec 2021 01:00:00 GMT

I run Monte Carlo simulations on probabilistic spin glasses, a simple mathematical model of magnetized matter with short range interactions. I use entropy to characterize the model's order-disorder transition.

Nudification is for women NEAR you

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 10 Dec 2021 19:00:00 GMT

And that is its worst part.

Thoughts on Metaplex, candymachine and NFTs on Solana

David Mohl — Fri, 10 Dec 2021 13:00:00 GMT

I have a few comments on metaplex, the batteries-included go-to solution for everything NFT related out-of-the-box, that I wanted to share with the world.

These are mostly thoughts I had when dealing with metaplex during the lifetime of a few projects, from the perspective of someone who just joined the Solana ecosystem.

About myself: I worked on a few production projects on Solana, am sometimes active on the Solana discord #developer-support channel to help newcomers out, and currently work on other (NFT related) projects. The following points are my thoughts and experience from building projects in this space.

Instructions (often) unclear

When you arrive in Solana land and want to do anything NFT related, chances are that you end up on the metaplex GitHub repo, or documentation at https://docs.metaplex.com/.

Coming from Ethereum, we have the ERC721 standard that dictates a handful of functions a smart contract has to implement to be considered ERC721 compliant.

In solana there are no smart contracts like in Ethereum, instead we’re dealing with pRoGRaMs. Tokens are also not handled by an individual ERC20/ERC721 contract per token, but by the central spl-token program. In fact, each token on Solana is owned by the spl-token program (spl stands for Solana Program Library).

After clicking around you might land on this page describing a metadata standard on Solana. Okay so far so good, but how do you actually use it? What is the address of this metadata program that you’re supposed to use? How does the function call look like to register metadata to something? Don’t know.

There is a metadata overview docs page available at https://docs.metaplex.com/overviews/metadata_overview but at the time of writing is still empty.

After a bit more of digging and wrapping my head around it, I found the metadata program address defined in the lib.rs file of the program itself:

 	solana_program::declare_id!("metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s");

Why is such a crucial bit of information not easier to find?

Then there is a single spl-token-metadata crate available on crates.io that you might want to use to submit metadata from your onchain rust programs, but after banging my head against it for literal hours and trying to get it working, it turned out this isn’t an official package and has a bug that makes it impossible to submit metadata - whoops. (Thanks for the kind folks over at #developer-support that helped me debug this)

The issue I have is that for something that wants to be the standard of something like NFTs on the entire chain, the experience can’t be frustrating but has to be as welcoming and easy to use as possible. Right now, from a newcomers perspective, figuring out how to simply add metadata for a token is incredibly frustrating. (Nowadays there is a JS SDK available that has functions to create metadata, but try to figure that out without initial guidance)

When dealing with other metaplex components such as candymachine (a program to sell mints of a NFT collection) we are running into a similar situation - the official documentation available here is missing half it’s pages* and instead of giving a technical overview of how it works, relies on commands to copy. If something isn’t working, which it often doesn’t, people flock into the discord to ask questions because they aren’t able to debug it themselves.

Instead of official documentation guidance, people relied on blog posts by other people that explain how the candymachine is working and how to set one up properly.

*/EDIT: in the time it took me to finish this post, the documentation has been updated and has more pages now. Thanks!

That brings me to my next point

Copy & Pasting encouraged over thinking and understanding

While dealing with metaplex, be it the metadata program, candymachine or other components, it became very apparent to me that the creators of metaplex had envisioned one very specific way of dealing with their projects.

Candymachine, fair-launch and other tools have a very specific way to set them up that they want users to copy and paste, rather than trying to understand and reason with. Of course being friendly to non-technical people is good, but this doesn’t teach users (technical and non-technical alike) how things are actually working, but instead encourages to stop thinking and follow guides. When technical people (like myself) try to understand and reason with the programs, we run into more road blocks due to lack of documentation, code quality and resources available.

In tech, copy-pasting and using big blobs of programs without attempts to understand them is usually a bad idea, especially when money flow and your business depends on it. If anything breaks you need to be able to reason with the program and figure out what is wrong.

When using metaplex like the creators intended to, so following candymachine instructions step by step with the provided CLI, and submit NFT metadata through candymachine itself, then everything works fine (unless you run any of the small issue resulting in one of the many crypto errors that have no reply on github). But when you try to use them individually, such as only using the metadata program without any of the master edition stuff, or candymachine components without the CLI, things can get fuzzy. (I still don’t know if it’s okay to remove the minting authority from a token that needs metadata - https://github.com/metaplex-foundation/metaplex/issues/279)

One example is IPFS. Candymachine supports IPFS, but only through Infura instead of allowing to actually use IPFS p2p. Without paying for Infura it is not even possible to create a candymachine with 10 mints without running into rate limitations and constant retries (I had to find the code and add a blocking sleep for things to go smooth). Of course when using arweave, like the creators wanted it to be used with, everything works fine. It’s also what 80% of the candymachine projects use because the instructions use that as default, often without understanding what arweave actually is.

Of course biased tools are completely fine, but my problem is more the messaging that everyone in Solana is supposed to be using this otherwise you’ll have a “bad time”. There are occasionally even people that don’t want to deal with you if you created your own minting onchain program and don’t use candymachine.

And that’s a shame because…

Candymachine is centralized and heavily botted

I mentioned above that in Solana, the spl-token program is kind of a central program (like the system program) that handles one thing - tokens.

For a system program I think that’s fair, but candymachine (and metadata but that’s not that big of an issue) are following the same pattern and trying to be the centralized thing that your application or dApp is talking to. Imagine it a little bit like having a “NFT sale server” and to buy an NFT you make a request like this (imaginary example):

POST candymachine.com/buy
{
	candymachineID: "abc123",
}

The candymachine program manages all candymachines. This program can also be upgraded at will (upgrade authority has not been revoked - https://explorer.solana.com/address/cndyAnrLdpjq1Ssp1z8xxDsB8dxe7u4HL5Nxi2K5WXZ by it’s creators and functionality can be changed without prior notice).

Being a central program, it is very trivial to see the transactions from other projects to setup a new NFT mint. It usually goes like this (pseudocode):

POST candymachine/setup-machine
POST candymachine/add-mint project1 1 "My Mint #1"
POST candymachine/add-mint project2 2 "Another Mint #2"
POST candymachine/add-mint project2 5 "Another Mint #5"
...

Once you monitor the onchain program you can very easily extract all available NFTs including their mint order from any project that uses candymachine. This sparked a huge amount of botters that take advantage of this, to either snipe rare NFTs, buy-out huge amounts of NFTs or in some cases even accidentally buying all available NFTs.

In other words: If you are a NFT buyer on Solana, you have almost no chance to mint a rare one. Bots know ahead of time if a mint is rare, and will be faster than you buying it.

To combat this, projects are now creating multiple candymachines with some fake ones, just to confuse botters.

If, however, candymachine was setup to be a program that you fork, adjust and deploy yourself, it would be a different story because botters would be unable to figure the program address ahead of time and thus can’t bot it (let alone knowing what input is needed to submit a mint transaction).

I see no reason that candymachine should be a central program. A strength of blockchains is decentralization so let’s take more advantage of it instead of relying on central entities that aren’t immutable.

Wrapping up

I sound critical and grumpy, but it’s good that there is so much NFT adoption happening on Solana. Candymachine being this easy to setup is certainly a reason why we have a huuuge amount of new projects popping up every day (although some with very low quality).

I hope the Solana foundation can pickup the Metadata standard or make an official one, and I hope that candymachinev2 will solve some of the issues I mentioned above. Or even better: Make something slim that people can fork, modify and deploy themselves, that would be nice!

Beware of transit gateways bearing large packets

skouf.com — Fri, 10 Dec 2021 04:30:18 GMT

Investigating an outage caused by AWS Transit Gateways and some quirks of handling TCP.

Search Result Relevance

Weblog on marginalia.nu — Fri, 10 Dec 2021 02:00:00 GMT

This entry is about a few problems the search engine has been struggling with lately, and how I’ve been attempting to remedy them.

Before the article starts, I wanted to share an amusing new thing in the world of Internet spam.

For a while, people have been adding things like “reddit” to the end of their Google queries to get less blog spam. Well, guess what? The blog spammers are adding “reddit” to the end of their titles now.

Your cautionary tale before purchasing any "smart home" stuff

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 09 Dec 2021 15:50:00 GMT

DO listen to these words of wisdom:

A big nail in the coffin of MySQL

Nicola Iarocci — Thu, 09 Dec 2021 08:05:25 GMT

After five years in Oracle’s MySQL team, Steinar H. Gunderson resigned a few days ago. On the same day, he dropped the bomb on his blog:

let me point out something that I’ve been saying both internally and externally for the last five years (although never on a stage—which explains why I’ve been staying away from stages talking about MySQL): MySQL is a pretty poor database, and you should strongly consider using Postgres instead¹.

Self-hosting Wikipedia using Kiwix

./techtipsy — Thu, 09 Dec 2021 06:00:00 GMT

Hosting your own Wikipedia instance has never been this easy!

On the, er... limits of tiny houses

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 09 Dec 2021 01:50:00 GMT

When fools meet sustainability fads, bad things ensue.

Leveraging Gatsby Incremental Builds With ButterCMS

Software Engineering - Rafael Quintanilha — Wed, 08 Dec 2021 20:08:00 GMT

Learn how to speed up your static builds with code examples.

On the incredible opportunities offered by Starship

Nicola Iarocci — Wed, 08 Dec 2021 08:05:25 GMT

Space-junkie me is back, this time reading about the innumerable opportunities that SpaceX’s Starship will offer once it becomes operational, hopefully no later than this year or the next. In his Science Upside for Starship, the exceptionally knowledgeable Casey Handmer makes a case for Starship relevance in the future of space exploration by listing an astounding number of reasonable use-cases for the vessel.

I think it is relatively straightforward to think of cool things to do with SpaceX Starships, so recent posts have focused on trying to understand the more mixed consequences for incumbent industrial organizations that are not ideally positioned to exploit the coming advances. It is, however, a fun exercise to enumerate all the ways in which Starship and related technologies can help execute bold, ambitious missions of scientific discovery.

Thoughts on Managing Managers

James Carr — Wed, 08 Dec 2021 02:00:00 GMT

In any organization, the role of a manager is crucial to ensure that the team performs to its full potential. As teams grow larger and more complex, managing them effectively becomes increasingly challenging. At a certain point, you need to subdivide the team into several teams and make the decision to either promote or hire additional managers. This is where a director takes on the role of a “manager of managers.

Vim tip 1: increment/decrement numbers

learnbyexample — Wed, 08 Dec 2021 02:00:00 GMT

Did you know that you can easily increment or decrement a number in Vim?

Ctrl+a will increment the number under the cursor or the first occurrence of a number to the right of the cursor
Ctrl+x will decrement the number under the cursor or the first occurrence of a number to the right of the cursor

You can also provide a count prefix:

3 followed by Ctrl+a will add 3
1000 followed by Ctrl+x will subtract 1000

Numbers prefixed with 0, 0x and 0b will be treated as octal, hexadecimal and binary respectively. You can also use uppercase for x and b. What if you want numbers prefixed with 0 to be treated as decimal? You can use the nrformats setting as shown below:

set nrformats-=octal

Decimal numbers prefixed with - will be treated as negative numbers. For example, using Ctrl+a on -100 will give you -99. While this is handy, this trips me up often when dealing with date formats like 2021-12-08.

See :h ctrl-a, :h ctrl-x and :h nrformats for documentation.

Video demo:

Running IntelliJ IDEA with JDK 17 for Better Render Performance with Metal

Mustafa Akın — Wed, 08 Dec 2021 02:00:00 GMT

Over the years, although a tremendously valuable tool, IntelliJ IDEA started to feel slowed down, even on these latest M1 Pro machines. It’s become frustrating. I disabled as many plugins as not needed. However, it’s still not sufficient. I have discovered that you can improve the performance by switching IDEA to the latest JDK. JetBrains has its runtime with its patches for HiDPI supports, performance improvements, and bugfixes for running the IntelliJ family more smoothly.

Debugging Electron.js native crashes on macOS

Juan Cruz Viotti — Wed, 08 Dec 2021 02:00:00 GMT

This article describes in detail how to symbolicalize Electron.js native crashes on macOS

From porch cameras to police states

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 08 Dec 2021 01:50:00 GMT

It’s just a short step.

Make lots of sandbox pages

River Writes - A MediaWiki Blog — Wed, 08 Dec 2021 01:41:34 GMT

Instead of putting all of your code experiments on a single page, make lots of little sandbox pages!

Procedural Christmas cards

Qubyte Codes — Tue, 07 Dec 2021 18:01:05 GMT

Below is a procedural snowflake. I'm using a little code to create Christmas cards this year, and I wanted each to be unique! If you received a card from me, you may see something like ?seed=1234567890 in the URL bar. That will be the random seed which generated your snowflake (and it's yours to keep). To see a random snowflake, remove everything after the question mark and hit enter. Refresh the page to see a fresh random snowflake!

The red colour is a convenience to make it show up in the software I use to cut card.

What stats do people want to learn?

Minireference blog — Tue, 07 Dec 2021 17:53:58 GMT

If I’ve learned anything about the startup world, it is that you have to listen to your customers, which in my case are the readers of the No Bullshit Guide textbooks. With this principle in mind, I sent out a survey to readers interested in the upcoming statistics requesting feedback on the general direction for […]

Deploy My Startup

Nothing Easy About This — Tue, 07 Dec 2021 15:00:00 GMT

There is a real opportunity here to make DevSecOps best practices an easy opt-in for SaaS founders.

Thirty Years Ago Today

Greg Herlein — Tue, 07 Dec 2021 09:00:00 GMT

Thirty years ago today I left the US Navy. A reflection.

Camping in the Headlands

Avy Faingezicht — Tue, 07 Dec 2021 02:00:00 GMT

After far too long since our last camping trip, Hannah and I decided to spontaneously book one for the upcoming weekend. There happened to be an opening at Bicentennial Campground in Marin, a tiny 3-site campsite near Point Bonita.

Hyperscaler Shopping List: 2022 Data Center Tech Frenzy ☁️

Ash's Blog — Tue, 07 Dec 2021 02:00:00 GMT

DDR5, PCIe Gen5, CXL, and new CPUs from Intel, AMD, and NVIDIA converge in 2022. The biggest datacenter hardware refresh in a decade.

Skills, stories, and software every dev should know - 123dev #49

Justin Garrison's Homepage — Tue, 07 Dec 2021 02:00:00 GMT

Comments

Problems

Isn’t it great when a problem fixes itself? I’ve seen this happen more often with people systems than with technology. Ignore that email a couple of days and it’s likely not to be a problem anymore.

With technology we usually have to take some action. Reboot a server, add a sleep, or add more resources. It’s not a “fix” but the problem isn’t visible for a little while so we can go back to doing other things.

Foundations of High-Modernist Ideology in Metropolis

Lambda Land — Tue, 07 Dec 2021 02:00:00 GMT

The following is from a essay from a class on German literature and film.

Fritz Lang’s movie Metropolis is primarily about the struggle between the oppressed working class and the ruling elite. What drives this tension, however, is a particular view of technology and technological progress that exacerbates the problems the film focuses on. This mentality is called high modernist ideology by Scott in his book Seeing Like a State:

What is high modernism, then? It is best conceived as a strong (one might even say muscle-bound) version of the beliefs in scientific and technical progress that were associated with industrialization in Western Europe and in North America from roughly 1830 until World War I. At its center was a supreme self-confidence about continued linear progress, the development of scientific and technical knowledge, the expansion of production, the rational design of social order, the growing satisfaction of human needs, and, not least, an increasing control over nature (including human nature) commensurate with scientific understanding of natural laws.

(Scott, 89)

Note that this is different from the literary definition of high modernism. For the purposes of this paper, high modernist ideology is understood as a belief that rational, scientific systems will solve all of mankind’s ailments. Metropolis is situated right at the tail end of the reign of high modernist ideology, and illustrates some of the failings of that mentality.

In this paper we will examine how Metropolis illustrates the dangers of high modernist thinking in how it portrays man’s subservient relationship to machinery and the fragility of the city of Metropolis resulting from top-down, rational planning. Ironically, even though these themes of centralized failure have been common place in film for more than a century now,¹ the dream of a perfectly organized Utopia persists in our culture today. Metropolis therefore remains an important case study of the dangers and failings of high modernism.

Man Serves the Machine

The opening of Metropolis shows a stark example of men working as slaves to the machine. Freder goes down to the lower levels to see how the workers live. He passes workers mechanically going from pose to pose to keep the machines running. Their jerky, repetitive motions cast the human workers merely as organic components of each machine that they tend.

When Freder arrives at the M-Machine, he witnesses an explosion that kills or seriously maims several workers, all because one worker was unable to reach a particular valve in time. The machine was obviously not designed to be ergonomic and was lacking critical safety features that would have prevented such a devastating accident. The machine’s complexity and human-hostile design make accidents like this essentially inevitable. The massive requirements that the machines impose on the humans shows the high modernist idea that humans should submit themselves to rational ideas, and that human nature can be overcome by rational, technical means.

Yet this submission to rational, mechanical systems is in itself an irrational one. Freder’s vision of Moloch gobbling up workers who dutifully march into the gaping jaws of the machine speaks to the almost fanatic devotion certain planners of large systems had to rationality. The vision closes and Freder sees some workers wordlessly² collecting the bodies of the injured and dead, whilst others take up their posts to keep the machine working. There’s some redundancy in how the human workers are allocated to account for mechanical failures or accidents, but there’s no redundancy in the machines to account for human failures.

We get other hints throughout the film of the submission of men to the machine. Metropolis looks to be absolutely unlivable. In the upper levels, we do see crowds gathered for entertainment and sport, but the city streets are either entirely devoted to automobiles, or are so foreboding and unwelcoming that no pedestrian-centric economy could ever flourish there. The streets are sterile and perfectly regular; the buildings are all blocks and perfectly featureless. This prefigures some of the architecture and city planning that was present during the DDR: massive apartment complexes built out of prefabricated concrete slabs were often placed far away from the city center, so as to make the Plattenbau districts feel stifling. The straight lines might look good on paper, but maps and city plans are of necessity a simplification over what real life looks like. Life flourishes in the hand-crafted and in the unique. Top-down imposed grids crush what makes life interesting and livable in the city.

The Fragility of Monoculture and Centralized Planning

After the accident at the M-Machine, Freder rushes to tell his father, the architect and ruler of Metropolis, about what he saw. In his office at the top of the New Tower of Babel, Jon Frederson controls the city from a central vantage point: aides digest and bring him information, while others record his decisions and carry out his orders. He is the very model of a high modernist: his rationality has brought order and prosperity to the citizens of the upper levels, and everyone above is free to spend their time as they please. It seems that the city is working perfectly well. However, Metropolis turns out to be very fragile.

Frederson’s control begins unraveling when his aids fail to bring him the information that he wants, and continues when Rotwang plots against him. Eventually the city collapses when the striking workers’ protest goes further than he had foreseen. The destruction of the Heart Machine causes the lower levels of the city to flood and the upper levels to loose power. Cars halt in the streets, and elevators plummet down their shafts. The apparently well-ordered city with bumper-to-bumper cars moving smoothly down the streets suffers cardiac arrest: broken elevators and stopped cars hinder the protagonists as they try to reach safety.

This fragility has a good analogue from the mistakes of high modernist ideology. Scientific forestry came about in the 1700s in Germany as a way to improve lumber yield. A plot of forest would be cleared of all “junk” trees and underbrush, and new trees of a single species were planted in straight lines. The ground was cleared to make way for the saplings, and each year there was a neat rotation between different parcels of land from which mature wood could easily and quickly be harvested. Initially, this was a great success: the wood was strong, straight, and plentiful. There was no junk wood that interfered, and lumber outputs could be reliably predicted.

It was not, however, without its drawbacks. In a book published in 1986, Richard Plochmann describes how yields dropped after the first few batches (Lang and Pye). The reasons for the drop are complex, but the cause was simple: the “clean” forest that produced such predictable quantities of lumber was a monoculture: there was no variety to protect against a pest that targeted a single species of tree. The soil fauna that are critical to a flourishing forest died off, and the trees withered when they didn’t get the nutrients that they needed.

One of the hallmarks of high modernist ideology is that problems can be solved by large, complex systems, be they mechanical, political, or bureaucratic. Such aspirations usually fail to take account of the difficulties in getting complex systems to run for long periods of time without encountering significant failures. Metropolis vividly illustrates the fall of Jon Frederson’s pride as the fragility in the massive system he created causes everything to fail around him.

Mediation

Amidst the warnings of the dangers of high modernist ideology, the message of Metropolis is tentatively hopeful: “the mediator between the hands and the head must be the heart!” By the end of the movie, we see the robot masquerading as a human unmasked and destroyed, and the beginnings of greater understanding between the architect of the city, and those that are actually required to live in the city and maintain it. This ultimately is also the cure for high modernist ideology: the designers of complex systems must take into account the needs and the experiences of those who would suffer the brunt of its effects. Metropolis casts this lesson in the discourse of class struggle, but it can also apply to those who design cities, automobiles, software, and other systems.

Works Cited

Scott, James C. Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed. Nachdr., Yale Univ. Press, 2008.

Lang, Chris, and Oliver Pye. “Blinded by Science: The Invention of Scientific Forestry and Its Influence in the Mekong Region.” Chrislang.Org, 1 Nov. 2000, https://chrislang.org/2000/11/01/blinded-by-science-the-invention-of-scientific-forestry-and-its-influence-in-the-mekong-region/.

Footnotes

¹ Other films with central-point-of-failure plots include Star Wars, (the thermal exhaust port on the Death Star) The Lord of the Rings, (cast it into the fire!) Batman Begins, (the train with the microwave generator) etc.

² The failure of Texas’s power grid is an extremely interesting case study about the dangers of fragile systems.

Constitution DAO, thanks for a good laugh

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 07 Dec 2021 01:50:00 GMT

Let’s just hope it brings more wisdom, too.

The Sorting Hat of Sales Mindset

D13V — Mon, 06 Dec 2021 02:21:34 GMT

Always be closing. Always be closing.

Alec Baldwin’s character in Glengarry Glen Ross

Welcome to part 7 of the series on “How to be a game changer”. You can find the previous posts here 1, 2, 3, 4, 5, 6.

As an engineer, I’ve had an aversion to sales for a long time. It seemed like something which doesn’t add value in the grand scheme of things, but instead it just shuffles it around. I had this view of sales being adversarial, and zero-sum. I win, you lose, or vice versa. It seemed to me that when a salesperson makes a deal, they are somehow tricking the other person into getting something bad for them. It felt like it does a lot of persuasion to twist someone’s arm into a bad deal for them. Over time, my mindset about sales changed, and I started seeing salesmanship as a service to the customer.

I started to look at sales not through the prism of how much do you close, but through the prism of how much you and your counterpart LEARN about whether what you’re selling fits for them. If as a salesperson you uncover whether and how your product will best help the customer, you are actually providing a service to that customer in the form of education. If you focus on maximizing the amount of education service you provide, and uncovering situations where the product is a natural fit, then you are no longer playing a zero-sum adversarial game, and instead you are now playing a win-win game.

You opt out of thinking of your customer as an adversary.

This becomes harder to do in a competitive market. You may be selling a product, or an idea, or an action or a philosophy, or your own personal brand. The general “selling” is the act of getting a customer to choose to “buy your product”. For this, they not only have to pick your product among many others, but they also need to have enough conviction to buy any product at all! There are two steps - the first is “buy”, and the second is “yours”. By focusing on providing education and uncovering the natural “buy” and solving for the benefit of the customer, it becomes much easier to transition to the “yours” step afterwards. The customer can be annoyed to hear reasons why product A is better than product B if they don’t want either one!

It is also a valid thing to recommend to “not buy”. Some people may think this is a reverse psychology trick. But another way to look at why this reverse psychology works at all, is that this is exactly what a you as a salesperson focused on the “buy” problem will do if you think it is not the right fit - recommend a no-buy and let everyone move on. By doing that you have already provided a win to the customer, by saving them from a bad decision, and you’ve added a win for yourself, by improving that relationship with the customer. That same customer may not be buying this time, but their circumstances may change, or they may have friends who are better fit, and spread the word. Furthermore, there is an extra win there. That extra win is turning the market away from being a “lemon market” where everyone is selling shit, and one step closer to a useful market where both customers and sellers profit. This extra win can help grow the market itself.

By helping the customer, you put on your sorting-hat hat. You help the customer find their best outcome, and sometimes they end up buying from you. Over time, you also learn from them, about which types of customers are the best fit, and how you can change your product to address the ones for whom it is not.

Today, as an entrepreneur I may need to pitch my company to customers, investors, or potential employees. There can be an explicit “ask” in customer interviews (buy my product), elevator pitches (give me money) or job interviews (come work for me). By trying to opt out of fixating on the ask, and turning towards communicating and conversing, I can help potential employees find their happiness better, and potential investors see more clearly if this would be a great fit. The “buyer” might not volunteer any information explicitly, but there are always cues, and those turn a pre-determined pitch to a dynamic conversation. If both of us are open and listening, we often reach a much more amicable solution which we didn’t even anticipate. I believe that being systematically focusing on improving their outcomes will also improve my outcome over time.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Candy Cane Ice Cream

Dustin.Lammiman — Mon, 06 Dec 2021 02:00:00 GMT

Candy Cane Ice Cream #

Ingredients #

1/2 Cup Granulated Sugar
2 Tbsp Skim Milk Powder
1/4 Tsp Xanthan Gum
1 1/3 Cups Milk (I used 2%)
2 Tbsp Light Corn Syrup
1 1/3 Cup Whipping Cream
1/2 Tsp Vanilla Extract
1/4 Tsp Mint Extract
2 Ounces Dark (70%) Chocolate
1 Tsp Vegetable Oil
3 Regular Size Candy Canes

Instructions #

Use a food processor to break the candy canes into small pieces.
In a bowl whisk together sugar, milk powder, and xanthan gum.
In a pot add milk and corn syrup.
Whisk dry ingredients into the pot of liquid ingredients. Be sure to mix well.
Heat ingredients for ~5 minutes on medium heat, stirring well, until sugar is disolved.
Remove from heat and add remaining cream and allow mixture to cool for a few minutes.
Refridgerate for 12+ hours. Then churn according to your ice cream maker's instructions.
While ice cream is churning, place a heatproof bowl above a pot that contains about 1 inch of water. The bowl should rest on the pot, but not touch the water.
Combine chocolate and vegetable oil in the bowl. Turn the heat to low and stir occasionally until the chocolate is melted. Remove from heat, but leave the bowl on top of the pot until your ice cream is almost ready.
When your ice cream is almost ready, slowly drizzle the chocolate into the ice cream. Pour in a thin, steady stream. This will allow the chocolate to harden as it touches the ice cream, creating tasty chocolate speckles.
Add candy cane pieces to the ice cream. Once it is well mixed, move your ice cream into an airtight container. Layer half the ice cream, then chocolate sauce, then the rest of the ice cream.
Freeze the ice cream for 4+ hours before serving.

Chocolate Sauce #

Chocolate sauce is optional. It's delicious without it too!

Ingredients #

1/4 Cup Cocoa Powder
1/4 Cup Sugar
1/4 Cup Water
Pinch of Salt
1/2 Tsp Vanilla

Instructions #

Whisk together cocoa and sugar in a saucepan.
Add water and salt and bring to a boil over medium-high heat, continuing to whisk the mixture.
Reduce to a simmer and continue to whisk for about 30 seconds.
Remove from heat and add vanilla.
Store in the refridgerator in a sealed container.

A Public Slipbox

RSS — Mon, 06 Dec 2021 01:17:00 GMT

Introducing my slip-box: https://explog.in/slipbox/.

I take org notes in logseq; which are then processed with a tiny python script & published using Emacs's org-export with an automatic Github Action every few hours.

Big Mouth: Season 3

🦉 olshansky 🦁 — Sun, 05 Dec 2021 09:36:01 GMT

Olshansky's review of Big Mouth: Season 3

Gunpowder Milkshake

🦉 olshansky 🦁 — Sun, 05 Dec 2021 09:27:22 GMT

Olshansky's review of Gunpowder Milkshake

Big Mouth: Season 2

🦉 olshansky 🦁 — Sun, 05 Dec 2021 09:10:01 GMT

Olshansky's review of Big Mouth: Season 2

Big Mouth: Season 1

🦉 olshansky 🦁 — Sun, 05 Dec 2021 09:09:19 GMT

Olshansky's review of Big Mouth: Season 1

My First Amazon Blog... ML/AI Chatbot over the Phone!

Greg Herlein — Sun, 05 Dec 2021 09:00:00 GMT

A month ago I told you I changed roles at AWS. Now I can show you what I’ve done since then!

Louisville, Kentucky & Washington DC

Avy Faingezicht — Sun, 05 Dec 2021 02:00:00 GMT

As part of the post-COVID wedding marathon, we flew to Louisville and DC back to back.

Christmas in Europe Is No Joke!

Godsip Club — Sun, 05 Dec 2021 02:00:00 GMT

First: Avoid Being Chased Down #

I live in Italy and, when I was a child, I didn’t really believe in Santa – I was already asking too many questions. I didn’t mind too much being a good child to receive nice gifts, but a few hundreds of kilometers from me, things were (are!) quite different. In the Eastern Alps, in Austria, Slovenia and other Slavic countries, naughty children are not gifted with black coal by the Befana, as in Italy. In those countries, bad kids are actually hunted down by a horned demon – the Krampus.

Every year, the 5th of December, during the Krampusnacht (“Krampus Night”), men and women march down the street dressed up like demons, with horrifying masks that take the whole year to make. The Krampus Run, or Krampuslauf, is a joyful parade for adults, mostly drunk, but a bit less cherish for children, who are continuously scared and chased by these horrible figures. Large bonfires are lit in the snowy streets as the quiet Alpine villages come to life.

The Krampus is clearly St. Nichola’s counterpart, or Santa’s villain from a modern perspective – a very bad Grinch. He leaves wooden sticks for bad children and takes the meanest with him. Christianity tried to banish these figures and traditions, but in this case, at the end of 19th century, this folklore was rediscovered because of… consumerism. Since when Austrian government quit the monopolisation of postcards, Krampuskarten boomed: Krampus-themed holiday cards filled with dark humor.

“Greetings from Krampus”… isn’t it lovely? I feel like I designed it myself.

Second: Chase the Origins #

As always, the fascinating aspect of this tradition is its origin – even if these are hypotheses and we will never have the absolute certainty. What is sure, though, is that the Percht (female: Perchta) was widely known in the same regions before the Krampus itself. It’s the figure of a woman considered the goddess of the Winter. She has two forms: one is a beautiful young lady white as snow, while the other one is an ugly elderly woman.

The name Perchta may mean “the bright one”, suggesting the lady’s pale skin, but there are also common aspects with the Krampus and the old hag: in historical descriptions, both have two different feet – one human and a larger one, called “goose foot”. Overall, though, Percht-themed masks were already worn during festivals in past times – and this tradition continues nowadays. They reflected the duality of the figure: some costumes were beautiful and bright, while others were horrible horned demons wearing animal furs. Those festivals are still called Perchtenlaufen, probably an earlier form of the Krampus runs.

This costume represents the duality of the Perchta – both beauty and beast… Maybe a bit tending to the latter.

Probably, the Percht evolved in the Krampus after St. Nichola’s introduction. Nowadays, the two shapes may a bit intertwined, but what is interesting is their past. In fact, similarities with the Perchta and other European figures and names can be observed: Bertha, Holda (who also appears in a Grimm’s fairy tale), the Russian Baba Yaga, and even the Italian Befana that I cited before. All these folklore characters, with so many common traits… must have some common roots! This was one of the thoughts fascinating me when I decided to explore our lore.

Third: Connect the Gods Dots #

A simple answer may be also the best one: those were goddesses, or a sort of. All of them were often referred as “guardians of the animals” and, in the form of beautiful women, probably they were Freya. She, Odin’s wife, was often linked to the nature and the animals – she was often identified with boars and cats. Also, the latter ones are a universal symbol for witches. Freya, indeed, was the most powerful sorceress of the whole Norse pantheon; so powerful that she could have been Odin’s teacher regarding magical arts.

Freya by Johannes Gehrts. Note the cats at her feet… they used to tow her chariot!

It is possible that, in Slavic countries, Freya had been a “Winter queen” and then evolved into figures as Holda: the Middle Ages may have shaped her from Norse goddess into a folklore character. But if Freya had been undoubtedly beautiful, why was the Perchta depicted also as an ugly witch?

Short answer: as often happened in the past centuries, probably some roles were mixed. Freya herself is already a notable example of how deities could be merged together – or even split! Frigg, in fact, is another name with which she is referred to in the poems; it is never clear if they are two distinct goddesses and what are the differences between them. One hypothesis is that, at some point, there had been the tendency to split her in two different beings¹.

But if Freya had been undoubtedly beautiful, why was the Perchta depicted also as an ugly witch?

Another resemblance with Norse mythology is again about the Krampus, maybe inherited from the Percht: it is told to be the son of Hel, the ruler of the homonymous world, the Land of the Dead. Hel was one of Loki’s daughters² and she was portrayed with a gloomy appearance. Her name means “the hidden one” – do you know which name could have the same meaning, according to the Old German language? Percht, that’s correct!

It’s not over yet: it is told that the Perchta howled in the woods in winter nights, as the Wild Hunt did, and lived in the elderberry bush, which is often linked to Freya. The list of common features could be so much longer! But of course (it is mandatory to remind that) all of these are thesis with no “real” proofs. I feel so heavy-hearted when I think about all that past knowledge that we won’t be able to recover.

What is certain, though, is that today is the 5th of December, and tonight the Krampus will run through the snowy streets of small rural Alpine villages, chasing down children – and it will happen again next year, and the next, and the next…

That glance… Enough said. Children, beware! 👀

Unfortunately, we have only relatively “modern” poems of Norse mythology since, until less one thousand years ago, those stories were retold verbally and with rock carvings. ↩︎
Loki had offspring as father… but as mother too! Did you know that he gave birth to an eight-legged horse? We’ll talk about that. ↩︎

Flashing the Supermicro variant of SAS9201-16e (SAS2116)

du.nkel.dev — Sun, 05 Dec 2021 02:00:00 GMT

Flashing SAS Cards feels like a trip to the 90s, with all the peculiarities included. This particular LSI SAS9201-16e card left me puzzled, until I realized it is the Supermicro variant that needs special handling. Let's see if I can pull together the important pieces, so that those who come after me don't need to give up on their cards.

Conspirituality is pervasive, but why?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 05 Dec 2021 01:50:00 GMT

Because the Web is too fast, of course.

Contacts

Home on mfashby.net — Sat, 04 Dec 2021 17:05:47 GMT

Fixing AMD OpenCL on Windows

bastibe.de — Sat, 04 Dec 2021 11:19:00 GMT

When I recently installed Windows 11 on my desktop, my photo editor Darktable suddenly got much slower than it used to be. When I looked into its preferences, I noticed OpenCL was no longer available.

As it turns out, some versions of the AMD graphics driver apparently no longer ship with OpenCL support on Windows. However, they do ship with the necessary libraries, it's just that these libraries are not registered any longer.

To register them, open the Registry Editor (aka regedit.exe), navigate to the key HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors (if the key does not exist, create it), and create a new DWORD of Value 0. Now rename the DWORD to be the path to your amdocl64.dll. Mine is C:\Windows\System32\DriverStore\FileRepository\u0344717.inf_amd64_d38cec78c83eee99\B343886\amdocl64.dll. Search C:\Windows\System32\ for amdocl64.dll to find the correct path on your computer.

With that, OpenCL was once again recognized by Darktable and the rest of my photo editing programs.

(source)

Book Review: A useless man

Nicola Iarocci — Sat, 04 Dec 2021 08:05:25 GMT

Sait Faik Abasıyanık is an acclaimed Turkish storyteller. A useless man is a collection of short stories that spans nearly two decades of the author’s output, offering a glimpse into his imaginative and troubled mind. His overflowing love for others (even sensual, with a preference for street kids) combined with a “mal de vivre” that pushes him towards self-destruction are apparent.

His passion for the most popular areas of Istanbul and, in contrast, the atavistic nostalgia for the simple life of the nearby fishermen islets exudes from these stories, which often run similar one after another. The composition is sometimes complex, making the text hard to parse and comprehend. Several novels are noteworthy, though, especially those narrated in third-person, where the narrator is not involved. The author is holding back his poetic prowess and the resulting text is more linear. Notable examples are The samovar, on the elaboration of mourning, and A story for two, a touching story of friendship between a bird and a fisherman. Some first-person ones are exceptional, too. For example, Milk is Sait Faik’s worthy tribute to Proust’s madeleine. One short story on the protagonist’s struggle with a street kid was profoundly touching.

Editing and making myself finish the first series

Liza Shulyayeva — Sat, 04 Dec 2021 08:02:00 GMT

I have been editing the next science fiction book. The plan is to hire an editor to go through it as well, since the previous books made enough income to pay for that. Editing will be the first service I pay for in my self-publishing journey. If things continue to go well, I might start buying proper on-market covers as the next step (but I also really like designing my own covers, so we’ll see…it’s just that I know nothing I come up with will be as good as a professional cover designer in my genre).

Wasted Resources

Weblog on marginalia.nu — Sat, 04 Dec 2021 02:00:00 GMT

At a previous job, we had a new and fancy office. The light switches were state of the art. There was an on button, and a separate off button. When you pressed the on button, the lights would fade on. When you pressed the off button, they would fade off. In the cloud somewhere was two functions that presumably looked a bit like this:

fun turnOnLamp() {
 while (!bright()) increaseBrightness();
}
fun turnOffLamp() {
 while (!dark()) decreaseBrightness();
}

I have deduced this from the fact that if you pressed both buttons at the same time, the lights would flicker on and off until someone was contacted to restart something. It is a marvellous time to be alive when you need to reboot your light switches because of a race condition. Modern computers are so fast that we often don’t even recognize when we are doing things inefficiently. We can end messages half way around the world to turn on the lights and it seems like it’s just a wire between the switch and the lamp.

The Whole World is Your Antilibrary

Avy Faingezicht — Sat, 04 Dec 2021 02:00:00 GMT

Este ensayo también está disponible en español. In _How to Travel with a Salmon_, Umberto Eco writes about the shock visitors to his home experienced upon encountering the thousands of tomes in his personal library. The essence of the essay became famous when Nassim Taleb paraphrased it in his bestseller _Black Swan_. Much like the japanese concept of "Tsundoku," Taleb's opening quote now adorns many book lovers' blog posts who share it without giving it much thought: > The writer Umberto Eco belongs to that small class of scholars who are encyclopedic, insightful, and nondull. He is the owner of a large personal library (containing thirty thousand books), and separates visitors into two categories: those who react with “Wow! Signore professore dottore Eco, what a library you have. How many of these books have you read?” and the others —a very small minority— who get the point that a private library is not an ego-boosting appendage but a research tool. The library should contain as much of what you do not know as your financial means ... allow you to put there. You will accumulate more knowledge and more books as you grow older, and the growing number of unread books on the shelves will look at you menacingly. Indeed, the more you know, the larger the rows of unread books. Let us call this collection of unread books an _antilibrary_. I'm a book hoarder, so the idea of an _antilibrary_ clearly resonates. I've known Taleb's version of the story focusing on the known unknowns for years, but had never looked into its source until this week. Funnily enough, after an online search for the title of the Eco book where the essay appears, I realized I had picked up the book years ago and put it into my own _antilibrary_ to sit and wait for an opportune moment. My books spreadsheet is nearing a thousand titles, and it metaphorically sits next to the ever-expanding shelf in my living room that, as of today, includes another four volumes by Eco (out of which I've only read two). According to Taleb, the value of owning unread books is in having unknown information ready at arms length at the right time. Today, however, there's an inherent paradox in the activity of literary accumulation. Eco and his fish first traveled together back in the summer of '94. Coincidentally, Jeff Bezos founded Amazon around the same time, setting in motion a machine that'd ultimately change that dynamic and redefine the value of physical books. The potential value stored in our shelves was sizable then, but has since diminished. To write this essay I got Libby to instantly deliver one of the San Francisco Public Library's copies of _Black Swan_ to my Kindle, and if I wanted to purchase the paper version I could have had it delivered to my door within 48 hours. Thanks, Jeff. Today, unknown unknowns aren't resolved merely by holding more books. Personal libraries are useful research tools as long as the books in them wouldn't be otherwise available. While a good library may shine in terms of curation, that quality could also be achieved with a simple list and no actual books. There's a reason libraries abstract their collections into classification systems and indices: physicality, almost by definition, represents a loss in discoverability compared to a pared down index or, more controversially, a search box. When your local library can send the digital version of any book to your smartphone for free, and when purchasing a copy online is a couple of taps away, the availability constraint is no longer binding, even for Taleb's skeptical _antischolars_. Except for rare and out of print books, annotated works, or other oddities, I'd argue that the value of a physical _antilibrary_ no longer holds. Somewhat irrationally, this hasn't stopped me from building one. Reading is exponential by nature. Intertextuality means that for every book we read, we're likely to find references to another N that we'll want to add to our stack. I accumulate unread books, and I enjoy it. Browsing bookstores and getting lost between their shelves is always in my checklist when visiting new cities. I prefer reading on paper than on a screen, and I like the feeling I get when in conversation someone brings up a book that I know is sitting on my shelf. And yet, if I am honest, I have no idea why I return home so often with books to add to my collection. There's no rational justification behind a personal antilibrary, but there doesn't need to be one. Ultimately, Taleb's point was not that accumulating unread books is good, but that we can only thrive when we admit the limitation of our knowledge. An _antilibrary_ is valuable because it reminds us of how much we don't know. These days, nearly the whole output of the human enterprise is at our fingertips. That fact is more daunting than the rows of books of any physical _antilibrary_.

Photo: Lungarno degli Archibusieri, Florence, by me. Previously posted on Italia II - Firenze.

How to Solve Incidents - a research proposal

ℤ→ℤ — Sat, 04 Dec 2021 00:09:56 GMT

How to Solve It, but for operational incidents.

Engineers with site reliability responsibilities are often faced with operational issues (incidents) that have an unknown cause and uncertain solution. How to Solve It is a classic work describing heuristics for solving mathematical problems. There have been adaptations of this book for different domains, but none yet for software-intensive operations. Within this study, we aim to build a taxonomy or categorization of techniques/heuristics in practice for understanding and overcoming operational issues. The catalog will be based on a survey of experienced engineers and real incident logs and postmortems from a variety of organizations. Readers will be able to leverage the catalog to solve tricky and novel production issues.

Research Plan

This project will use both survey and field study methods to catalog “in practice” incident mitigation techniques and heuristics. We expect the project to take between 6 and 9 months (to the point of a submission-ready article), although the timeline may be extended due to data access issues since operational data is often sensitive and negotiations may be prolonged.

Data Collection

We will reach out to organizations (primarily SaaS or computing intensive companies) for:

Interviews with senior engineers — how do they approach an unfamiliar production problem? do they use any explicit mental models, heuristics, or existing processes?
Access to incident logs, chat/ticket threads, and/or root cause analyses. Basically, documents describing how the team is working to resolve incidents, preferably “raw”. The research team will then:
1. Document problem solving threads by incident — i.e., faced with x situation, attempted y or sought to determine z status. Was this successful?
2. Document what mental models (if any) were being used? Was this mental model correct?
3. Note: Raw incident logs are preferred to RCAs. For example, Slack’s DNSSEC RCA is very detailed on the cause of the problem, the prior attempts to prevent an outage, and actions to correct the outage once it happened, but is light on details about specific troubleshooting steps and unsuccessful threads. RCAs can provide useful data, but usually gloss over the troubleshooting activity.
Access to “Try if everything else fails”-style runbooks (if they exist).

A suitable organization should have at least 30 relevant incidents. We are targeting at least 10 organizations within the study.

A sample of incidents will be excluded from the coding effort to form an “out of band” pool. We will also augment the out of band pool with publicly documented incident RCAs.

Data Protection

We anticipate that all organizations that participate in the study will be acknowledged within the paper and that individuals interviewed may be quoted with their consent. The paper will not link heuristics to specific companies or incidents; heuristics and techniques are generalized and anonymized. A heuristic must be practiced by at least two organizations to be referenced.

We expect to sign NDAs or similar legal agreements to protect sensitive data and may accept reasonable constraints on access and storage of any shared data.

As members of the Association of Computing Machinery, we are bound by the Code of Ethics, in particular section 1.7 Honor Confidentiality:

Computing professionals are often entrusted with confidential information such as trade secrets, client data, nonpublic business strategies, financial information, research data, pre-publication scholarly articles, and patent applications. Computing professionals should protect confidentiality except in cases where it is evidence of the violation of law, of organizational regulations, or of the Code. In these cases, the nature or contents of that information should not be disclosed except to appropriate authorities. A computing professional should consider thoughtfully whether such disclosures are consistent with the Code.

Create taxonomy/categorization

Using the collected data, we expect to code it via a faceted coding approach. An example of a facet coding is Ranganathan’s five core categories, typically labeled PMEST. P is the personality or most specific subject, M is the material or component, E is the energy or activity, operation, or process, S is the space or location, and T is the time. A specific heuristic may not have all five facets.

As an example, let’s say that within an incident the engaging team has detected a drop in production rate from a distributed group of queue-based workers. For coding, the personality is the queue-based job workers. The material could be the workers, the queue, the underlying compute infrastructure, a worker dependency, or even ancillary items like the logging and monitoring subsystem. The energy will be the specific action, such as restarting workers, checking connections to the queue, or verifying health of dependencies.

The coded values can then be grouped into a taxonomy by grouping similar items by facets and frequency.

Why go through the effort to create a taxonomy? How to Solve It organizes the heuristics alphabetically, How to Solve It: Modern Heuristics organizes items hierarchically by academic discipline, and Discussion of the Method : Conducting the Engineer’s Approach to Problem Solving lacks any reference-able organization. There are two reasons to go through the effort: construction and reference. Construction-wise, a taxonomy of heuristics can identify gaps in the catalog (similar to gaps in the Periodic Table identified elements). Reference-wise, a good organization will make it easier for readers to apply to incidents.

Typology versus Taxonomy. Both are terms for categorization, but in some domains a typology is conceptual while a taxonomy is empirical. Typologies are organized by dimensions that a human specifies, while a taxonomy is created via cluster analysis based on natural attributes or dimensions selected by humans. Since heuristics lack natural attributes, the more correct term for this activity is typology, albeit more obscure.

Testing the Taxonomy

Ideally, the true test of efficacy would track reduction in repair time due to improved problem solving with and without the catalog. However, this would require controlled experiments or a large statistical study, both of which are studies in their own right. Instead, the validity of the catalog will be evaluated for (practical) completeness and likelihood of utility.

Completeness will be evaluated via out-of-band examples. For the incidents and public RCAs marked as out-of-band during the data collection phase, we will map the incident’s narrative to the coded catalog to check for any absences in the catalog or difficulties in following the organization. Out-of-band examples may be added to the catalog (assuming they meet our inclusion threshold), but their initial absence is a useful statistical indicator of completeness.

Likelihood of utility will be evaluated via an outside group of both inexperienced and experienced engineers who will be asked to use the catalog on prior incidents (of their choosing) and current incidents and record their experiences. (See Use testing for more context.) Their experiences will be recorded via survey and aggregated.

Hypothetical Categorization

To provide an idea of what the categorization might look like:

Have you tried turning it off and on again? (Blind luck strategies)
1. Restarting the complaining service
2. Restarting the complaining service’s dependencies
3. Redeploying
4. Moving to another host / cluster
5. Rollback
6. Search internet for similar error messages
Configuration management
1. Restore previous configuration
2. Attempt to isolate problem to certain instances / shard
3. Verify configuration same between instances
4. Restore previous configuration (dependency)
Capacity management
1. Increase allocation
2. Decrease allocation
3. Rebalance among shards
Traffic management
1. Throttle traffic source
2. Eliminate traffic source
3. Route traffic to another shard
4. Verify traffic along route
5. Eliminate traffic magnification
Broaden scope
1. Orchestration issue
2. Operating System/Host Issue
3. DNS / Service Lookup Issue
4. Network Issue

Potentially, the categorization could become a decision tree, depending on the statistical strength of the data.

Each entry would contain a description, including any known failure modes and additional context.

Prior Art (and why this is novel)

How to Solve It by G.Polya

This is a classic work on mathematical methods to solve problems. Although some of the heuristics described (e.g. Do you know a related problem, Draw a figure, Generalization) are applicable beyond mathematical problems, most of the heuristics fall within the mathematics domain. In contrast, the proposed project targets operating software intensive systems and also seeks to quantitatively find which heuristics seem to work most often.

How to Solve It: Modern Heuristics by Z. Michalewicz and B. Fogel

This book follows a similar approach to How to Solve It, but changes the domain to computer science algorithms by discussing approaches like constraint solving, search, and evolution. None of the heuristics are particularly relevant to operations.

Effective Troubleshooting by Chris Jones (Google’s SRE Book); Atlassian Incident Management Handbook

Both focus on the management of incidents (e.g. communication, maintaining focus, leadership) and the general process of solving problems rather than specific actions or techniques the engineering group can perform, particularly if the cause is novel. Although a scientific or structured approach certainly aids in mitigating issues, our work will be focused on specific techniques to be used within that greater framework.

How to mitigate the incident? An effective troubleshooting guide recommendation technique for online service systems¹ This paper studies “troubleshooting guides” (TSGs) at Microsoft and develops a recommendation system for suggesting specific actions operators can take based on an incident description. Examining incident records, they find a lower-bound of 27.2% of incidents that had an existing TSG and a large proportion (~60%) of incidents contained novel failures (no existing or relevant guide). The proposed work is not focused on making it easier to find existing run books to solve recurring or known failure modes, but shorten the time to restore for novel or ambiguous failure modes, which the paper suggests are the majority of cases within their sample.²

Publication Venue

Peer reviewed with free access to final paper (e.g. Open Access). Potential venues include SREcon, DevOpsDays, and magazines with broader scope like Communications of the ACM (Contributed Articles section).

If there is a large amount of material, this could become a book or long-form website, but we would prefer to start with an article.

Jiajun Jiang, Weihai Lu, Junjie Chen, Qingwei Lin, Pu Zhao, Yu Kang, Hongyu Zhang, Yingfei Xiong, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2020. How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1410–1420. DOI:https://doi.org/10.1145/3368089.3417054 ↩︎
As noted in the paper, Microsoft does not require TSGs to be developed nor maintained. We expect that the availability of runbooks to vary widely between organizations based on their culture. ↩︎

Why aren't the most useful Mac apps on the App Store?

Copper • A blog about conductive layers — Fri, 03 Dec 2021 19:28:39 GMT

Let’s set the stage first. So, it’s Tuesday night and I’m Command Tab-ing my way through 10 different apps, some with 3-4 windows, while trying to patch bugs in Lunar faster than the users can submit the reports. I’m definitely failing.

I feel my brain pulsing and my ring finger going numb on the Tab key. I stop switching apps and just stare at the Xcode window, containing what I knew was Swift code but looked like gibberish right now.

“Feels like burnout” I think. Wasn’t that what I ran away from when I quit my job to make apps for a living?

I heard a joke recently:

Show joke

It’s probably only funny for a small group of workaholics, but the reality of those words struck me in the middle of the hysterical laughter I was trying to stop.

Why am I still developing this app?

Why am I adding all the features the users are asking for, then deal with the flood of frustrated emails saying “what an overcomplicated stupid app, I just want to make my screen brighter!!”, then try to hide advanced features to make it simpler, then get assaulted with the confused “I can’t change volume anymore fix this ASAP!!!” because UI changes can very easily introduce bugs by simply forgetting to bind a slider to a value, then get back to scotch taping broken parts slower than the users can report them?

Those features should have probably been their own independent app.

I start to feel my fingers again, press Command Tab once more, and while looking at the list of app icons I realise something.

Maybe pressing Tab 4-5 times while visually assessing if the selected app icon is the one I want to focus, isn’t the best solution for this kind of workflow.

So what does my brain do when I feel burnt out? Gives me ideas for even more apps…

rcmd

That’s how the idea of rcmd began. We have two Command keys on a Mac keyboard, and the right hand side one is almost never used. What if I use it exclusively for switching apps?

rcmd app screenshot

When I used Windows for reverse engineering malware, I liked switching apps using Win + Number where the number meant the position of the app icon in the taskbar. I didn’t like counting apps however.

Using the app name felt the most natural. I remembered using Contexts for a while, which provides a Spotlight like search bar for fuzzy searching your running apps. But that needed a bit more key presses than I wanted (that is 1) and more attention than I wanted to give (which is none).

My idea sounded a bit simpler: Right Command + the first letter of the app name

So simple that people were offended by it…

I pitched this idea to Ovidiu Rusu, a very good friend of mine, who surprisingly seemed to have the same need as me. We created the first prototype in about a week (icons and graphics take so much time…) and started using it in our day to day work to see if it made sense.

In less than a day, rcmd became so ingrained in our app switching that we got incredibly annoyed when we had to quit the app for recompiling and debugging. We just kept pressing Right Command X and staring at the screen like complete idiots, not understanding why Xcode wasn’t being focused.

What most people overlook when they have a simple idea is that 80% of the effort goes into handling edge cases that are not visible in the original idea.

Just for this simple app we had to solve the following problems:

What do I do when there are multiple running apps with the same first letter?
- How do I decide which one gets priority so that I meet user expectations?
- How do I get to the other apps with less priority in as few key presses as possible?
How do I persist key assignments if necessary?
What if the user assigned the key to an app that’s no longer running?
- Should I launch the app when the key is pressed?
- Where is the app located?
There are dozens of running apps while only about 10% of them are actual apps launched by the user. How do I filter those out? The others could be:
- Menu bar apps (e.g. Alfred, Lunar etc)
- OS services (e.g. Spotlight, Siri etc.)
- Daemons (e.g. CoreAudio etc)
How do I only listen to the Right Command key?

This last question is what led me to write this article. It turned out we needed to do quite a few hacks if we wanted to publish this app in the App Store.

The Sandbox

Every app that is submitted to the App Store must be compiled to run within a sandbox. This means that the app will run in a container which will have the same structure as your home directory, but with mostly empty folders.
The sandbox also limits what APIs you can use, and which system components you can communicate with.

The defacto way of reacting to Right Command + some other key is to monitor all key events (yes, just like a keylogger), and discard events that don’t contain the Right Command modifier flag.

1
2
3
4
5
6
7
8
9


public extension NSEvent.ModifierFlags {
    static let rightCommand = NSEvent.ModifierFlags(rawValue: UInt(NX_DEVICERCMDKEYMASK))
}

NSEvent.addGlobalMonitorForEvents(matching: .keyDown) { event in
    guard event.modifierFlags.contains(.rightCommand) else { return }

    // do your thing
}

Easy peasy, right? Well no, because that’s not allowed on the App Store.

To use that API you need to first request Accessibility Permissions from the user. Those permissions are prohibited inside the Sandbox, because with those permissions, an app would be able to do all kinds of nasty stuff:

Log all your key presses (including passwords if you aren’t using a Secure Input field)
Extract text from all the running apps
Click on buttons inside other apps
Write text in fields or send key combinations to the system
Render elements like buttons and tooltips over the interface of other apps

Those are perfectly reasonable things in the context of assistive software, because you need the computer to do stuff for you when you aren’t able to use a keyboard or a mouse/trackpad.

And you need the computer to read out text from other apps, or show choice buttons which you can trigger with your voice.

Technical content ahead. Click to skip this section if you’re not interested in macOS internals.

But for rcmd’s use case, we’re restricted to APIs that don’t require these permissions. APIs so old that 64-bit wasn’t even a thing when they launched and they require passing C function pointers instead of our beloved powerful Swift closures.

That’s the Carbon API and it goes a little something like this:

Registering a Command+R hotkey

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


// Install the key event handler
var pressedEventType = EventTypeSpec()
pressedEventType.eventClass = OSType(kEventClassKeyboard)
pressedEventType.eventKind = OSType(kEventHotKeyPressed)

InstallEventHandler(GetEventDispatcherTarget(), { _, inEvent, _ -> OSStatus in
    return handlePressedKeyboardEvent(inEvent!)
}, 1, &pressedEventType, nil, nil)


// Register the hotkey
let hotKeyId = EventHotKeyID(signature: UTGetOSTypeFromString("some-unique-identifier" as CFString), id: 0)
var carbonHotKey: EventHotKeyRef?

RegisterEventHotKey(UInt32(kVK_ANSI_R),
                    UInt32(cmdKey),
                    hotKeyId,
                    GetEventDispatcherTarget(),
                    0,
                    &carbonHotKey)

// Handle the event
func handlePressedKeyboardEvent(_ event: EventRef) -> OSStatus {
    assert(Int(GetEventClass(event)) == kEventClassKeyboard, "Unknown event class")

    var hotKeyId = EventHotKeyID()
    let error = GetEventParameter(event,
                                  EventParamName(kEventParamDirectObject),
                                  EventParamName(typeEventHotKeyID),
                                  nil,
                                  MemoryLayout<EventHotKeyID>.size,
                                  nil,
                                  &hotKeyId)

    guard error == noErr else { return error }
    assert(hotKeyId.signature == UTGetOSTypeFromString("some-unique-identifier" as CFString), "Invalid hot key id")

    switch GetEventKind(event) {
    case EventParamName(kEventHotKeyPressed):
        // do your thing.. eventually
    default:
        assert(false, "Unknown event kind")
    }
    return noErr
}

Not so pretty as the NSEvent method, but does the job. Kind of.

You see, that beautiful code macaroni above only lets us listen to Any Command + R, not specifically the Right Command. There’s no way to pass something like rightCmdKey into RegisterEventHotKey.

A workaround I found for this was:

Listen for flagsChanged
Set a global boolean to true when the there’s a rightCommand modifier
Discard the event if the boolean is false

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


var rcmd = false

NSEvent.addGlobalMonitorForEvents(matching: .flagsChanged) { event in
    rcmd = event.modifierFlags.contains(.rightCommand)
}

func handleHotkey(key: String) {
    guard rcmd else { return }

    focusApp(with: key)
}

Doing this reminded me of the days I worked with Rust, and how wonderfully impossible a task like this would be. I don’t think I’m touching it again, I like my global atomic booleans.

Now the weirdest limitation hits me. There’s no way to discard a hotkey event and forward it back to the system so it can use it for the next handler.

Say I register Command C and I only want to do something when Right Command is held. If I do nothing when Left Command is held, then you can’t copy text anymore using Command C.

I tried returning the inappropriately named OSStatus(eventNotHandledErr) but the event still doesn’t return to the handler chain.

At this point we seriously considered dropping the App Store idea and just going the self publishing route.

But just thinking what we would have to do for that triggered something akin to PTSD.

Here’s a list with what I can remember off the top of my head from Lunar:

Implement the Paddle SDK for licensing
Add Sentry for error reporting
Lose the useful App Store analytics
Lose ratings and reviews
Add Sparkle for auto-updating
- Generate signing keys
- Create DMG with the app
- Sign every single update
- Try not to lose the signing key
- Generate the appcast XML
- Host that appcast somewhere
Host the app bundle for download somewhere

Finding yet another workaround seemed much easier.

Thankfully it really was easy. It turns out that RegisterEventHotKey is plenty fast. So fast that we were able to register the hotkeys only when Right Command was being held, and unregister them when the key was released.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import Atomics

var _rcmd = ManagedAtomic<Bool>(false)
var rcmd: Bool {
    get { _rcmd.load(ordering: .relaxed) }
    set { _rcmd.store(newValue, ordering: .sequentiallyConsistent) }
}

NSEvent.addGlobalMonitorForEvents(matching: .flagsChanged) { event in
    rcmd = event.modifierFlags.contains(.rightCommand)

    if rcmd {
        registerHotkeys()
    } else {
        unregisterHotkeys()
    }
}

The App Store review

Now rcmd was ready for publishing on the App Store.

There was one little thing that bothered me though. I usually keep 4-5 separate projects open in Sublime Text, each with its own window. Because of the sandbox, there’s no way to get a list of windows for an app and, say, focus a specific one, or cycle between them.

But I found a little gem while I was customising my fork of yabai, a way to trigger Exposé for a single app:

1

CoreDockSendNotification("com.apple.expose.front.awake" as CFString)

app window expose screenshot

We decided to show Exposé if for example you press rcmd s while Sublime Text is already focused. It was good enough for us.

Not for the App Store reviewer though.

app store review rejecting the expose feature

I knew private and undocumented APIs are not seen well on the App Store. But I had no idea they will guarantee a rejection.

Free Trials

I like breaking the norm with my creations. Some of them will be flukes, some will be criticised into oblivion, but a small number of them might turn out to be something a lot of people wanted but didn’t know they needed.

rcmd is one of those things: a bit quirky, unique in its approach, and incredibly useful for a specific group of people.

That is also its weak point though. It’s hard to communicate this usefulness without being able to try the app first. But as it turns out, the App Store doesn’t provide any support for creating a free XX-day trial before buying an app.

Free trials for non-subscription apps have been allowed since mid-2018 on the App Store, and are supposed to be implemented using in-app purchases. Unfortunately, this approach has a lot of inconveniences which are very well detailed in this article: Ersatz Free Trials | Bitsplitting.org

These are the biggest shortcomings for my case:

The trial doesn’t start immediately
- The users have to be nagged with a custom Activate Trial UI to start the trial manually
- Starting the trial means buying a $0.00 in-app purchase
- If the users skip the trial UI out of frustration, they’ll be left with a dumbed down version of the app, possibly missing the functionality they downloaded it for
You have to create your own UI for starting a trial, buying the full app after the trial ends, showing the trial status etc.
You have to provide some kind of limited functionality if the trial ends
- In the context of simple and small apps like rcmd, this doesn’t really make sense

I tried a few dozen apps on the App Store and I couldn’t find a single one offering a free trial for a non-subscription purchase using the above method.

Having to pay upfront is steering away a lot of possible users, but with all that bad UX, we decided to not implement any free trial and just sell the app for a one-time fair price.

3 years ago, I would have probably chosen to make the app open source and give it away for free, just like I did with Lunar.

I would have thought:

I’m making a ton of money at this company, what I would get by selling a small app would be peanuts anyway.

Only recently I realised that this approach kept me dependent on having a job where I click-clack useless programs 8 hours a day, only to get 1-2 hours after work for my projects, and sacrifice my health and sanity in the process.

In my whole 7-year career as a professional API Glue Technician and experienced Wheel Reinventer, I never did anything remotely as useful as even the simplest app I can code and publish in 2 months right now. At those companies, most of my work was scraped anyway when the redesign period of the year came.

So I’d rather have those peanuts please.

Everyone’s choosing to be left out

Now, with so many limitations, I think we can take a fair guess at why most indie developers choose to distribute their app outside the App Store.

Here are some of the apps I find most useful, and what I think is the main reason for them not being in the App Store:

Alfred

The app’s main functionality (searching the filesystem) needs Full Disk Access permissions which are not allowed inside the sandbox.

It also uses Accessibility Permissions for auto-expanding snippets and other custom workflows.

BetterTouchTool

Capturing and responding to all kinds of keyboard and trackpad events needs Accessibility Permissions.

The app also encapsulates the older BetterSnapTool utility for snapping windows to a grid. Resizing windows requires the same permissions.

Karabiner-Elements

Reacting to and changing keyboard input in realtime needs a special keyboard driver which is only allowed by Apple on a case by case basis. You have to request DriverKit entitlements from Apple, and they have to deem you worthy of those entitlements.

Needless to say, they won’t give hardware driver entitlements for a software app mimicking a keyboard.

Sublime Text

Full Disk Access is probably the biggest requirement here.

Of course, there are other code editors on the App Store like BBEdit but they have this initial phase where you have to manually give them access to your / (root) directory.

bbedit sandbox access dialog

bbedit allow access

Compared to Sublime Text’s launch and edit instantly first time experience, I feel this is a bit annoying. I’m pretty sure this confuses a lot of first time users, and they will probably blame the developer, not knowing that this is the only way to access files from the Sandbox.

Swish

Resizing windows, listening for global trackpad gestures, detecting titlebars, moving windows to other spaces/screens. All of these need Accessibility Permissions.

There’s even an FAQ for that on their page:

Why is Swish not on the App Store?

Apple only allows sandboxed apps on the App Store. Swish needs to perform low-level system operations which prevent it from being sandboxed. Read more here.

Sip

As outlined in their 2017 article, Moving from Mac App Store, the sandbox limitation is the primary reason

CleanShot X

Their Screen Recording feature has three very useful functions:

Audio recording (this needs installing an audio loopback device)
Pressed keys overlay (needs Input Monitoring permissions)
Select single window (needs Accessibility Permissions for getting the window position and size)

Sketch

Honestly, I’m not sure about this one. The App Store is full of image editors and graphic content creation tools.

I thing the unique pricing model is something they would have a hard time implementing on the App Store.

The unique pricing model of Sketch

sketch pricing model

Parallels Desktop

They actually have an App Store edition, but it’s severely limited.

Sharing things between the host and the VM is probably the largest functionality affected by the sandbox.

They provide a table with everything that’s missing in their App Store version of the app: KB Parallels: What is the difference between Parallels Desktop App Store Edition and Standard Edition?

Lunar

Low-level communication with monitors is only possible by using a lot of private and reverse engineered APIs (IOKit, DisplayServices, IOAVService etc.)

Accessibility Permissions are also needed for listening and reacting to brightness and volume key events.

Because of the sandbox, the lite App Store version of Lunar only supports software dimming and can only react to F1/F2 keys.

Soulver

I think the free trial limitation is the only thing keeping such a self-contained app outside the App Store.

Soulver is incredibly complex and useful in its functionality, but I don’t think too many people would splurge $35 on a notepad-calculator app without trying it first. It deserves every single dollar of that price, that I can say for sure.

Certain risks of drones were wildly exhagerated

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 03 Dec 2021 18:00:00 GMT

But only because we were looking the wrong way.

How to customize your Transistor.fm Website with JS and CSS

swyx's site RSS Feed — Fri, 03 Dec 2021 09:58:15 GMT

The default Transistor.fm website is kinda ugly. Here's how to customize your Transistor.fm website if you use Transistor. But also it's a simple guide to do clientside customizations of almost any website whose code you don't control.

Will VR in Space be Indispensable?

Thoughts by Thomas Derflinger — Fri, 03 Dec 2021 02:00:00 GMT

With the proposed flight to Mars and its exploration, there is a long stay required by the astronauts under claustrophobic conditions. Could Virtual Reality (VR) systems alleviate some stress of long stays in space?

The job of regulation is to make decentralization work

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 02 Dec 2021 20:00:00 GMT

Especially in a digitally networked age.

Introducing the OSS Quickstart Archetype

Gunnar Morling — Thu, 02 Dec 2021 19:30:00 GMT

Table of Contents

The Challenge
The OSS Quickstart Archetype
Outlook

I am very happy to announce the availability of the OSS Quickstart Archetype!

Part of the ModiTect family of open-source projects, this is a Maven archetype which makes it very easy to bootstrap new Maven-based open-source projects, satisfying common requirements such as configuring plug-in versions, and adhering to best practices like auto-formatting the source code. Think Maven Quickstart Archetype and friends, but more modern, complete, and opinionated.

XKB With Sway

Blog on blog.gnoack.org — Thu, 02 Dec 2021 19:24:17 GMT

This explains how to configure multiple keyboards in Sway, and how to use advanced configuration when regular xkb_options are not enough.

Sway configuration

Luckily, Sway has great support for multiple keyboards with different layouts, so it nicely adjusts to whatever keyboard you have plugged in or can use separate configurations for a physical keyboard and the laptop keyboard.

First, identify the identifiers for the keyboards you have plugged in, using the following command:

$ swaymsg -t get_inputs

Look for the lines starting with Identifier: for the device you’re interested in.

Next, add one section for each of your keyboards to ~/.config/sway/config:

input "1278:32:PFU_Limited_HHKB-Classic" {
	xkb_layout "us"
	xkb_model "hhk"
	xkb_options "compose:ralt"
	xkb_capslock "disabled"
}

input "1133:49948:Logitech_USB_Keyboard" {
	xkb_file ".xkb/keymap/logitech"
}

The example above lists a HHKB (“Happy Hacking”) keyboard where right Alt acts as the Compose key. More options are documented in the man pages sway-input(5) as well as xkeyboard-config(7).

In the case where the existing XKB options are not sufficient, you need to refer to a separate XKB configuration file, as described below.

XKB configuration

To configure my Logitech keyboard, the plan was to have the key mapping reasonably close to the HHKB layout.

In my case, I have two files under ~/.xkb (the directories need to first be created): This is the ~/.xkb/keymap/logitech file:

// Derived from `setxkbmap -print`,
// added the "+unixy" part to xkb_symbols.
xkb_keymap {
	xkb_keycodes  { include "evdev+aliases(qwerty)"	};
	xkb_types     { include "complete"	};
	xkb_compat    { include "complete"	};
	xkb_symbols   { include "pc+us+inet(evdev)+unixy"	};
	xkb_geometry  { include "pc(pc105)"	};
};

This file is the output from running setxkbmap -print, but adds the +unixy part to the xkb_symbols section. This refers to the ~/.xkb/symbols/unixy file with the following content:

partial alphanumeric_keys
xkb_symbols "unixy" {
	// exchange backspace and backslash
	key <BKSL> {	[ BackSpace, Backspace ] };
	key <BKSP> {	[ backslash, bar ] };

	// caps is ctrl
	key <CAPS> {	[ Control_L ] };
	modifier_map Control { <CAPS> };

	// right alt is compose
	key <RALT> {	[ Multi_key ] };
};

Looking for a Serious-Use Travel Mouse

trv.rs — Thu, 02 Dec 2021 15:33:00 GMT

I've been looking for a high-quality wireless mouse that supports USB-C: I found it in the Steelseries Prime Mini.

For years, I have been using gaming mice for code and design. I used MX501 for a long time, then a steady series of Razers… currently a Razer Deathadder. I'm just used to the feel, the speed, the quality. I can't use the bars of soap that the fruit company calls magic¹.

As Mac has slowly moved to USB-C… so have I. I have tried to keep an eye on mice that support USB-C, and thought I'd take another look.

Earlier in the year, I had picked up the latest Logitech MX Master 3. I did not like it. It didn't fit how I hold mice, I don't use 60% of the buttons and features, and it felt very laggy.

We are considering doing some traveling, so more coffee shop work could be a possibility. I want something USB-C, travel friendly, and fast. I don't like the lag I've seen in previous wireless mice.

The last good travel mouse that I ever had was the first version Razer Orochi. It supported both wired and bluetooth, used AA batteries, and was my weapon of choice for a long time.

Orochi V1, G Pro Superlight, Prime Mini

Look how small the Orochi is!

What I'm looking for:

OSX-friendly²
USB-C
low lag
wireless, but supporting wired for charging and use
not loud… no garish gaming logos and light effects
travel friendly: small, onboard battery preferred

For the purposes of this search, I'm not going to try and evaluate:

ergonomics
tracking on weird surfaces. I bring a small mouse pad.
how delightfully clicky are the buttons (so clicky!)
build quality
customer support

This is also not one of my long-use reviews… yet. I'm making a purchase decision and returning the others. I put in an order for three options, and have spent the last day evaluating the following:

Razer Orochi V2

(official link, $69.99 MSRP)

Because of my long fondness for the first version and the fact that my Deathadder has been so successful on OSX, this was the first one that I reached for. Of the three, this is the one that is billed as a "travel mouse."

The Orochi v2 is also the only one of the three that supports wireless without a dongle through bluetooth, just like the first version. The Orochi v2 does not support any wired mode.

The Orochi v2 is significantly bigger than the v1, and a touch smaller than the other two mice. It has that familiar Razer grip.

You have to download Razer Synapse to get the mouse speed up to my customary speeds. The default three settings weren't fast enough. I'd tell you how it went, but Synapse doesn't work after OSX Mojave. 👎 Fail.

I don't like that it requires batteries… nothing else in my bag does. It's nice that you have the choice of using either AA or AAA, but it's a dealbreaker for me. I want my wire dopp to just have some USB-C cables, a power brick, and go.

Logitech G Pro Superlight

(official link, $149.99 MSRP)

Immediately after taking this one out of the box I gasped. It's really light.

It fails my USB-C requirement… twice. It has a trim little USB-A dongle, and it's charging port has a custom fitted USB-micro cable. It's not a bad mouse cable, but it's stiff and not as nice as the one with the Prime.

The USB-A dongle does have the neat feature of being stowable inside the mouse's shell for travel… but I'll need to carry an adapter anyway which kind of ruins it for me.

I put it on the list because I saw a lot of people using it online, and was curious how a wireless mouse held up in terms of responsiveness and latency. Of the three I tested, it has the most options and the highest DPI/CPI, but there's a catch:

There's no onboard DPI/polling controls! Out of the box, it's set to the lowest setting, a paiiiinful crawl. You have to install Logitech's G Hub software (brew install logitech-g-hub) which is powerful and works on Mac, but I just need a mouse, not separate sniping macros for each game. G Hub wanted me to do a firmware update on both the mouse and the dongle as soon as it booted, which required me finding two USB-A adapters because they have to both be connected to update.

While it took overly long to setup (made painful by the default sloooooow settings) once connected the mouse is fast and responsive. I had no clue that a wireless mouse could feel so fast. I am really impressed, but in order to use this mouse on the road, I have to keep track of a couple custom cables just for the G Pro. Not ideal.

Steelseries Prime Mini

(official link, $59.99 MSRP)

I am a little surprised how much I liked this mouse. I had two Steelseries mice in the past, the Xai and the Sensei. Both had very strange tracking issues I was never able to nail down. It could have to do with my lack of knowledge on polling and such, but it was frustrating.

The Steelseries Prime Mini is the first USB-C only gaming mouse that I have yet found. It's very right-handed, so lefties beware. It's a touch heavier than the ridiculously light G Pro, but not overwhelmingly. Like the G Pro, I was impressed with it's wireless latency… it feels as fast as a wired mouse.

The included dongle for the connection is USB-C! It's fat though… but will barely permit both the default Mac charging USB-C cable and the old Earpods TRRS. It doesn't store in the body like the G Pro, so don't lose it³.

More importantly, the port on the front of the mouse is also USB-C, and seems to work fine with the number of USB-C cables I have lying around. They have provided a USB-C -> USB-A cable with it, it's paracord wrapped, very light and indistinguishable from a normal wired mouse. It's as nice or nicer than the fixed cable I've got with my Deathadder. It actually feels as light as the one on my friend's Glorious Model O.

I appreciate that the Prime Mini requires no software to update or maintain, there's a hardware DPI/polling button on the bottom. There is Steelseries GG, which does work on the mac and supports even higher DPI/CPI settings. Didn't need it anyway, the external button was enough.

Conclusion

For me, the Steelseries Prime Mini wins hands down. Because of my travel objectives, the fact that it uses the same cables as my keyboards, iPads, and computer charging makes it an easy choice to grab on my way to a coffeeshop. I like that I could use it wired on my standing desk, grab it and throw it in my backpack to take to a remote location to use with the dongle. Best, at half the price of the G Pro Superlight I feel less guilty buying another peripheral. If I was choosing purely for gaming/desk feel, I would probably be keeping the G Pro Superlight. It just feels that nice… but not nice enough to maintain yet another special cable that only works for it.

It will serve as my primary mouse for a season, both at my desk and on the road. The Prime Mini. A serious-use mouse for serious people, just plug and play. 👍

If you or someone you love insists on using the Magic Mouse, consider the wonderful Elevation Labs Magic Grips. ↩
You'd be surprised how many mice don't work quite right on OSX. I also recommend turning off mouse acceleration. ↩
It does look like Steelseries lets you buy replacement dongles for $9.99, which isn't bad. ↩

How to set up Sphinx for Read the Docs

River Writes - A MediaWiki Blog — Thu, 02 Dec 2021 11:54:37 GMT

A tutorial for configuring Sphinx for Read the Docs for your Python projects

Fave New Podcasts in 2021

swyx's site RSS Feed — Thu, 02 Dec 2021 09:25:13 GMT

My picks of best podcasts I found in 2021, across tech, business, and creator categories. I also pick up on best of the best from 2020.

Becoming physically immune to brute-force attacks

All content on Seirdy’s Home — Thu, 02 Dec 2021 08:01:21 GMT

Using thermal physics, cosmology, and computer science to calculate password vulnerability to the biggest possible brute-force attack.

Engineering vs. Technology

Misleading Metaphors — Thu, 02 Dec 2021 03:16:38 GMT

In this article, I attempt to chronicle how I arrived at what I think is a layman explanation of the difference between the terms engineering and technology .

An Incomplete Devil’s Dictionary

My goal with this series is to offer layman definitions of words used in the technology industry.

Italia V - Positano Revisited, Napoli

Avy Faingezicht — Thu, 02 Dec 2021 02:00:00 GMT

In retrospect, I should have taken a lot more film photos during our trip to Italy. I still don't fully trust myself with film, but there's something that makes these feel very special. Below are a few more pics from our time in Positano, and then the end of the trip in Napoli. _This is the fourth in a series shared throughout Fall 2021. When you're done here, make sure to also check out parts one, two, three, and four._

LEAD Capstone

Sam Solomon — Thu, 02 Dec 2021 02:00:00 GMT

Leadership Exploration, Application and Development—or LEAD as it’s known—is a 6-month leadership course offered by Salesloft.

An incoming risk from Google, Apple, Twitter etc.. is coming

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 01 Dec 2021 18:00:00 GMT

Trouble is, WHEN?

Raster-Based 3D Models with QGIS, OSM and Aerialod

Geography & Coding — Wed, 01 Dec 2021 17:58:12 GMT

Creating 3D models from raster or vector data has never been that easy! This tutorial takes you through the whole process from vector/raster data to 3D in just 3 steps.

STEM: Science, Technology, Engineering, and Mathematics

Misleading Metaphors — Wed, 01 Dec 2021 17:56:38 GMT

A clear and concise definition for each discipline that makes up the STEM acronym.

How to avoid a climate disaster

Bajkowski.com — Wed, 01 Dec 2021 06:44:36 GMT

A brief review of Bill Gates' book How to Avoid a Climate Disaster: The Solutions We Have and the Breakthroughs We Need

CKAD Experience

Ye Lin's Random stuff — Wed, 01 Dec 2021 06:08:12 GMT

I recently took the Certified Kubernetes Application Developer (CKAD) exam and passed with 97 out of 100. Some background about myself. I have been a backend + infra engineer working with Kubernetes via Google Cloud’s managed service: GKE for the past few years. I am also a (Ubuntu) Linux user for both the desktop and servers. I am familiar with basic linux commands, bash, tmux and vim.

The primary motivations for me were

Advent of Code 2021, day 1

viralinstruction — Wed, 01 Dec 2021 02:00:00 GMT

Curious about Julia? Why not use Advent of Code 2021 to learn it!

DICOM File Format Basics

vladsiv — Wed, 01 Dec 2021 02:00:00 GMT

Explore the fundamentals of the DICOM file format! This quick introduction covers the basics of DICOM's structure, its essential uses, and tips for easily navigating its complex and abstract components.

Math may be just math, but algorithms are written by people

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 01 Dec 2021 01:50:00 GMT

And people have biases. Just ask Safiya Noble.

Develop Ali Express Scraper in Python with Scraper API

Adnan's Random bytes — Tue, 30 Nov 2021 13:30:33 GMT

This is another post in ScrapeTheFamous, in which I will be parsing some famous websites and will discuss my development process. The posts will be using Scraper API for parsing purposes which makes me free from all worries about blocking and rendering dynamic sites since Scraper API takes care of everything. In this post, we are going to scrape AliExpress. AliExpress is a Chinese B2C portal to buy stuff. The script I am going to make consists of two parts, or I say, two functions: fetch and parse. The fetch will accept a category and return all links of individual items and parse will parse an individual entry and returns a few data points in JSON format. So, let’s begin! def fetch(url): links = [] headers = { 'authority': 'www.aliexpress.com', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': '"macOS"', 'dnt': '1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'none', 'sec-fetch-mode': 'navigate', 'sec-fetch-user': '?1', 'sec-fetch-dest': 'document', 'accept-language': 'en-US,en;q=0.9,ur;q=0.8,zh-CN;q=0.7,zh;q=0.6', } with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() URL_TO_SCRAPE = url BASE_URL = 'https://www.aliexpress.com' payload = {'api_key': API_KEY, 'url': URL_TO_SCRAPE, 'render': 'true'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60, headers=headers) if r.status_code == 200: html = r.text.strip() soup = BeautifulSoup(html, 'lxml') links_section = soup.select('._3KNwG') for link in links_section: links.append(BASE_URL + link['href']) return links The API_KEY.txt file contains the Scraper API key. After that, I fetch all elements that have the class ._3KNwG in them and then store them in a list(). You may store them in a file or database, up to you. The second function is parse() and it looks like below: def parse(url): title = '' price = '' image = '' store_name = '' record = {} headers = { 'authority': 'www.aliexpress.com', 'pragma': 'no-cache', 'cache-control': 'no-cache', 'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': '"macOS"', 'dnt': '1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'none', 'sec-fetch-mode': 'navigate', 'sec-fetch-user': '?1', 'sec-fetch-dest': 'document', 'accept-language': 'en-US,en;q=0.9,ur;q=0.8,zh-CN;q=0.7,zh;q=0.6', } with open('API_KEY.txt', encoding='utf8') as f: API_KEY = f.read() payload = {'api_key': API_KEY, 'url': url, 'render': 'true', 'country_code': 'pk', 'keep_headers': 'true'} r = requests.get('http://api.scraperapi.com', params=payload, timeout=60, headers=headers) if r.status_code == 200: html = r.text.strip() soup = BeautifulSoup(html, 'lxml') title_section = soup.select('.product-title-text') if title_section: title = title_section[0].text.strip() price_section = soup.select('.uniform-banner-box-price') if price_section: price = price_section[0].text.strip() image_section = soup.select('.image-viewer img') if image_section: image = image_section[0]['src'] store_section = soup.select('.shop-name a') if store_section: store_name = store_section[0].text.strip() record = {'title': title, 'price': price, 'image': image, 'store': store_name} return record You need to pay attention to the payload variable where I introduced two new parameters: country_code and keep_headers. The reason I set country_code here because AliExpress is available in multiple countries. Since ScraperAPI uses proxies by default, it was picking the proxy IP of a random country and was not displaying price in local Pakistani currency. By setting the country I am forcing ScraperAPI to use Pakistan-based proxies. It is quite a useful feature because many websites allow local traffic only and don’t allow global visitors to visit them. The rest of the code is self-explanatory. The reason I am using render as True because AliExpress is generating content by parsing Javascript. Conclusion So in this post, you learned how to scrape e-commerce sites like AliExpress to get the desired data by using Scraper API. You do not have to worry about Proxy IPs either nor do you have to pay hundreds of dollars, especially when you are an individual or working in a startup. Companies spend 100s of dollars on a monthly basis just for the proxy IPs. Oh if you sign up here with my referral link or enter promo code adnan10, you’ll get a 10% discount on it. In case you do not get the discount then just let me know via email on my site and I’d sure help you out. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Develop Ali Express Scraper in Python with Scraper API first appeared on Adnan's Random bytes.

Skills, stories, and software every dev should know - 123dev #48

Justin Garrison's Homepage — Tue, 30 Nov 2021 02:00:00 GMT

Comments

Logic

How people think has been on the mind of philosophers for ages. Aristotle had the most famous works on logic for around 2000 years. In the 1600’s Gottfried Leibniz attempted to improve on Aristotle’s logic by simplifying it and representing it in mathematical notation.

Two hundred years later George Boole further simplified Leibniz’s logic and notation into what we know today as Boolean algebra. Every introduction to computer science will teach you AND, OR, and NOT which are the foundational components of human logic.

What I learnt building a SaaS

Gaganpreet Arora — Mon, 29 Nov 2021 23:26:02 GMT

Background #

It was around the start of the pandemic in 2020 that I started hacking away at some of the ideas that I had collected but didn’t have a chance to work on. I had come across the FingerprintJS project, and I had an idea to use similar fingerprints for a cookie-free analytics platform. At that time, other competitors in this market were still in their infancy, and I wanted to build an analytics product that will have more information for marketeers to base business decisions on. I started building No Cookie Analytics.

O Kafka, Where Art Thou?

Gunnar Morling — Mon, 29 Nov 2021 19:55:00 GMT

The other day, I came across an interesting thread in the Java sub-reddit, with someone asking: "Has anyone attempted to write logs directly to Kafka?". This triggered a number of thoughts and questions for myself, in particular how one should deal in an application when an attempt to send messages to Kafka fails, for instance due to some network connectivity issue? What do you do when you cannot reach the Kafka broker?

一些实用工具列表

ipfans's Blog — Mon, 29 Nov 2021 19:48:00 GMT

一些在工作中经常使用的一些工具。如果有什么推荐的，也欢迎在评论中提供。这个列表后续会持续更新

HTTP工具

curlie - httpie-like 工具，底层是curl
lego - Let’s Encrypt证书工具
mkcert - 方便导入本地证书
paw.cloud - 原生的macOS HTTP调试工具，现在每年都会免费送，有兴趣关注一下

编译工具

go-task - 我用来替代Makefile，并无什么特殊必要，主要是不想写Makefile

代码质量

golangci-lint - 感觉无需介绍了，集成了很多实用工具，重复的就不列举了
pre-commit - 提交前检查代码质量，比如代码风格，缩进，空格等等
dcd - 查找代码中的重复代码

代码统计

scc - 高性能统计代码行数

图表工具

go-diagrams - 使用Go语言描述系统架构图
ndiag - 如果不想用Go描述，也可以选择用YAML描述系统架构
draft - 另外一个用YAML描述的工具，风格不一样
k8sviz - 你也可以从现成的K8s环境中生成系统架构图
archview - 通过代码中注释生成应用内部分层结构
go-plantuml - 根据Go代码生成结构体的PlantUML图
goplantuml - 另外一种生成PlantUML的工具
go-erd - 不想用PlantUML也可以换这种风格
asciiflow - 可以画ASCII图，ASCII图好处是可以放在代码里，如果你愿意的话
sequence - 嫌弃asciiflow比较原始，做时序图的时候可以用这个
mermaid-js - 方便集成在网页中，也可以导出成图片
kroki - 上面没提到的图类型的生成？看看这个

IaC

pulumi - Terraform业界比较常用，不过要学习HCL比较蛋疼，我个人比较喜欢pulumi，可以选择自己的习惯的语言，tf-cdk目前还比较初级。

The meditation that keeps capitalism happy...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 29 Nov 2021 19:00:00 GMT

…would have never happened without social media.

A life-changing trip to Mexico on the Day of the Dead

🦉 olshansky 🦁 — Mon, 29 Nov 2021 18:04:52 GMT

My personal journey into joining the Crypto space full-time at Pocket

Chet Baker, born to be cool

Nicola Iarocci — Mon, 29 Nov 2021 08:05:25 GMT

A great piece of writing on jazz has recently been posted on The Smart Set. In Born to be cool, Matthew Duffus writes about jazz trumpeter Chet Baker, reporting about his troubled life, musical prowess, influence, and legacy. Some facts are well known, like the reception and then the competition with trumpet legends such as Miles Davis and Dizzy Gillespie; other tidbits are less known (to me at least). On Baker legacy:

Autonomous Teams

Craig Pardey — Mon, 29 Nov 2021 02:00:00 GMT

One thing that development teams constantly struggle with is striking a balance between building features, fixing bugs, paying down tech debt, and embarking on more ambitious activities like replacing core components in their architecture. It’s also something that Agile tries to solve but I’ve never seen it solved to everyone’s satisfaction.

And I think it’s because many Agile Product Owners are focused on the specifics of what features they’d like to see built instead of setting the direction for the dev team and letting them figure out how to get there. They allocate some random percentage of the sprint to tech debt just to keep the team happy but it’s hard to make significant changes with 10% of the sprint capacity.

Encanto

Code & Visuals — Mon, 29 Nov 2021 02:00:00 GMT

For the first time since 2016, Walt Disney Animation Studios is releasing not just one animated feature in a year, but two! The second Disney Animation release of 2021 is Encanto, which marks a major milestone as Disney Animation’s 60th animated feature film. Encanto is a musical set in Colombia about a girl named Mirabel and her family: the amazing, fantastical, magical Madrigals. I’m proud of every Disney Animation project that I’ve had the privilege to work on, but I have to admit that this year was something different and something very special to me, because this year we completed both Raya and the Last Dragon and Encanto, which are together two of my favorite Disney Animation projects so far. Earlier this year, I wrote about the amazing work that went into Raya and the Last Dragon and why I loved working on that project; with Encanto now in theaters, I now get to share why I’ve loved working on Encanto so much as well!

Disney Animation feature films take many years and hundreds of people to make, and often the film’s story can remain in a state of flux for much of the film’s production. All of the above isn’t unusual; large-scale creative endeavors like filmmaking often entail an extremely complex and challenging process. More often than not, a film requires time and many iterations to really find its voice and gain that spark that makes it a great film. Encanto, however, is a film that a lot of my coworkers and I realized was going to be really special very early on in production. Now obviously, that hunch didn’t mean that making Encanto was easy by any means; every film requires tons of hard work from the most amazing, inspiring, talented artists and engineers that I know. But, I think in the end, that initial hunch about Encanto was proven correct: the finished Encanto has a story that is bursting with warmth and meaning, has one of Disney Animation’s best main characters to date with a huge cast of charming supporting characters, has the most beautiful, magical animation and visuals we’ve ever done, and sets all of the above to a wonderful soundtrack with a bunch of catchy, really cleverly written new songs. Both the production process and final film for Encanto were a strong reminder for me of why I love working on Disney Animation films in the first place.

From a technical perspective, Encanto also represents something very special in the history of Disney Animation’s continual advancements in animation technology. To understand why this is, a very brief history review about Disney Animation’s modern production pipeline and toolset is helpful. In retrospect, Disney Animation’s 50th animated feature film, Tangled, was probably one of the most important films the studio has ever made from a technical perspective, because the production of Tangled required a near-total ground-up rebuild of the studio’s production pipeline and tools that wound up laying the technical foundations for Disney Animation’s modern era. While every film we’ve made since Tangled has seen us make enormous technical strides in a variety of eras, the starting point of the production pipeline we’ve used and evolved for every CG film up until Encanto were put into place during Tangled. The fact that Encanto is Disney Animation’s 60th animated feature film is therefore fitting; Encanto is the first film made using the USD-based successor [Miller et al. 2022] to the production pipeline that was first built for Tangled, and just like how Tangled laid the technical foundations for the subsequent ten films that followed, Encanto lays the technical foundations for many more future films to come! As presented in the USD Birds of a Feather session at SIGGRAPH 2021, this new production pipeline is built on the open-source Universal Scene Description project and brings massive upgrades to almost every piece of software and every custom tool that our artists use. Already Encanto’s upgraded production pipeline has enabled cool new tools that would have been much harder to create previously, such as a new command center tool that gives a bird’s eye unified overview of stats and data across the entire movie [Tennant et al. 2022]. An absolutely monumental amount of work was put into building a new USD-based world at Disney Animation, but I think the effort was extremely worthwhile: thanks to the work done on Encanto, Disney Animation is now well set up for another decade of technical innovation and another decade of pushing animation as a medium forward.

Moving to a new production pipeline meant also moving Disney’s Hyperion Renderer to work in the new production pipeline. To me, one of the biggest advantages of an in-house production renderer is the ability for the renderer development team to work extremely closely with other teams in the studio in an integrated fashion, and moving Hyperion to work well in the new USD-based world exemplifies just how important this collaboration is. We couldn’t have pulled off this effort without the huge amount of amazing work that engineers and TDs and artists from many other departments pitched in. However, having to move an existing renderer to a new pipeline isn’t the only impact on rendering that the new USD-based world has had. One of the most exciting things about the new pipeline is all of the new possibilities and capabilities that USD and Hydra unlocks; one of the biggest projects our rendering team worked on during Encanto’s production was a new, very exciting next-generation rendering project. I can’t talk too much about this project yet; all I can say is that we see it as a major step towards the future of rendering at Disney Animation, and that even in its initial deployment on Encanto, we’ve already seen huge fundamental improvements to how our lighters work every day. Hopefully we’ll be able to reveal more soon!

Of course, just because Encanto saw huge foundational changes to how we make movies doesn’t mean that there weren’t the usual fun and interesting show-specific challenges as well. Encanto presented many new, weird, fun problems for the rendering team to think about. Geometry fracturing was a major effect used extensively throughout Encanto, and in order to author and render fractured geometry as efficiently as possible, the rendering team had to devise some really clever new geometry-processing features in Hyperion [Burley and Rodriguez 2021]. Encanto’s cinematography direction called for a beautiful, really colorful look [Robinson 2022] that required pushing artistic controllability in our lighting capabilities even further, and to that end our team developed a bunch of cool new artistic control enhancements in Hyperion’s volume rendering and light shaping systems. One of my favorite show-specific challenges that I got to work on for Encanto was for the holographic effect in Bruno’s emerald crystal prophecies [Butler et al. 2021]. For a variety of reasons, the artists wanted this effect done completely in-render; coming up with an in-render solution required many iterations and prototypes and experiments carried out over several months through a close collaboration between a number of artists and TDs and the rendering team.

Encanto also saw continued advancements to Hyperion’s state-of-the-art deep-learning denoiser and stereo rendering solutions and saw continued advancements in Hyperion’s shading models and traversal system. A particularly notable advancement in our shading model is the addition of a new physically accurate practical multiple-scattering sheen lobe [Zeltner et al. 2022] to the Disney BSDF [Burley 2015]; I think this new sheen model is going to catch on widely in industry due to its combination of accuracy, ease of implementation, and performance, all of which improves greatly over previously existing sheen models [Conty and Kulla 2017]. These advancements helped us tackle many of the interesting complexity and scaling challenges that Encanto presented; effects like Isabella’s flowers and the glowing magical particles associated with the Madrigal family’s miracle pushed instancing counts to incredible new record levels [Finley et al. 2022], and for the first time ever on a Disney Animation film, we actually rendered some of the gorgeous costumes in the movie not as displaced triangle meshes with fuzz on top, but as actual woven curves at the thread-level [Velasquez et al. 2022]. The latter proved crucial to creating the chiffon and tulle in Isabella’s outfit and was a huge part in creating the look of Mirabel’s characteristic custom-embroidered skirt. My mind was thoroughly blown when I saw those renders for the first time; on every film, I’m constantly amazed and impressed by what our artists can do with the tools we provide them with.

Encanto also saw rendering features that we first developed for previous films pushed even further and used in interesting new ways. We first deployed a path guiding implementation [Müller et al. 2017] in Hyperion back on Frozen 2 [Müller 2019], but path guiding wound up not seeing too much use on Raya and the Last Dragon since Raya’s setting was mostly outdoors, and path guiding doesn’t help as much in direct-lighting dominant scenarios such as outdoor scenes. However, since a huge part of Encanto takes place inside of the magical Madrigal casita, indoor indirect illumination was a huge component of Encanto’s lighting. We found that path guiding provided enormous benefits to render times in many indoor scenes, and especially in settings like the Madrigal family’s kitchen at night, where lighting was almost entirely provided by outdoor light sources coming in through windows and from candles and stuff. I think this case was a great example of how we benefit from how closely our lighting artists and our rendering engineers work together on many shows over time; because we had all worked together on similar problems before, we all had shared experiences with past solutions that we were able to draw on together to quickly arrive at a common understanding of the new challenges on Encanto. Another good example of how this collaboration continues to pay dividends over time is in the choices of lens and bokeh effects that were used on Encanto. For Raya and the Last Dragon, we learned a lot about creating non-uniform bokeh and interesting lensing effects, and what we learned on Raya in turn helped further inform early cinematography and lensing experiments on Encanto. One more great example can be found in how eyes are shaded in Encanto- over our last few shows, we’ve been steadily moving our eye shading approach over to a next-generation shading model with advanced, physically accurate iris caustics [Chiang and Burley 2018] sampled using manifold next event estimation, and Encanto is the first show to use this new eye shading model on 100% of characters. The way we push technology further and further on each film isn’t limited to just rendering either; I mostly write about only lighting/shading/rendering topics here because that’s my home domain, but there are countless other examples in things like rigging, animation, simulation, procedural authoring, interactive visualization, and more about how we each film tech advances on top of the previous film. A great example published at SIGGRAPH 2022 is the new hair simulation technique that was developed for Mirabel’s bouncy curly hair [Liu 2022]; ever since Tangled, Disney Animation has been great at hair, but with each movie we still keep advancing what we can do!

In addition to all of the cool renderer development work that I usually do, I also got to take part in something a little bit different on Encanto. Every year, the lighting department brings on a handful of trainees, who are put through several months of in-studio “lighting school” to learn our tools and pipeline and approach to lighting before lighting real shots on the film itself. This year, I got to join in with the lighting trainees while they were going through lighting training; this experience wound up being one of my favorites from the past year. I think that having to sit down and actually learn and use software the same way that the users have to is an extraordinarily valuable experience for any software engineer that is building tools for users. Even though I’ve been working at Disney Animation for six years now, and even though I know the internals of how our renderer works extensively, I still learned a ton from having to actually use Hyperion to light shots and address notes from lighting supervisors and stuff! Encanto’s lighting style required really leaning on the tools that we have for art-directing and pushing and modifying fully physical lighting, which really changed my perspective on some of these tools. For most rendering engineers and researchers, features that allow for breaking purely physical light transport are often seen as annoying and difficult to implement but necessary concessions to the artists. Having now used these features in order to hit artistic notes on short time frames though, I now have a better understanding of just how critical a component these features can be in an artist’s toolbox. I owe a huge amount of thanks to Disney Animation’s technology department leadership and to the lighting department for having made this experience possible and for having strongly supported this entire “exchange program”; I’d strongly recommend that every rendering engineer should go try lighting some shots sometime!

Finally, here are some stills from the movie pulled from the Blu-ray, 100% created using Disney’s Hyperion Renderer by our amazing artists. I’ve ordered the frames randomly, to try to prevent spoiling anything important. These frames showcase just how gorgeous Encanto looks, but they only represent a small fraction of how breathtakingly beautiful and colorful the total film is. I highly recommend seeing Encanto on the biggest screen you can; if you are a computer graphics enthusiast, go see it twice: the first time for the wonderful, magical story and the second time for the incredible artistry that went into every single shot and every single frame! I love working on Disney Animation films because Disney Animation is a place where some of the most amazing artists and engineers in the world work together to simultaneously advance animation as a storytelling medium, as a visual medium, and as a technology goal. Art being inspired by technology and technology being challenged by art is a legacy that is deeply baked into the very DNA of Disney Animation, and that approach is exemplified by every single frame in Encanto:

Here is the credits frame for Disney Animation’s rendering and visualization teams! These two teams collectively are responsible for generating all of the pixels at Disney Animation, be it final frames from Hyperion, or interactive viewports using our internal realtime rasterizer:

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

Also, be sure to catch our new short, Far from the Tree, which is accompanying Encanto in theaters. Far from the Tree deserves its own discussion later; all I’ll write here is that I’m sure it’s going to be fascinating for rendering and computer graphics enthusiasts to see! Far from the Tree tells the story of a parent and child raccoon as they explore a beach; the short has a beautiful hand-drawn watercolor look that is actually CG rendered out of Disney’s Hyperion Renderer and extensively augmented with hand-crafted elements. Be sure to see Far from the Tree in theaters with Encanto!

References

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley and Francisco Rodriguez. 2022. Fracture-Aware Tessellation of Subdivision Surfaces. In ACM SIGGRAPH 2022 Talks. Article 10.

Corey Butler, Brent Burley, Wei-Feng Wayne Huang, Yining Karl Li, and Benjamin Huang. 2022. “Encanto” - Let’s Talk About Bruno’s Visions. In ACM SIGGRAPH 2022 Talks. Article 8.

Matt Jen-Yuan Chiang and Brent Burley. 2018. Plausible Iris Caustics and Limbal Arc Rendering. In ACM SIGGRAPH 2018 Talks. Article 15.

Alejandro Conty and Christopher Kulla. 2017. Production Friendly Microfacet Sheen BRDF. In ACM SIGGRAPH 2017 Course Notes: Physically Based Shading in Theory and Practice.

Henrik Dahlberg, David Adler, and Jeremy Newlin. 2019. Machine-Learning Denoising in Feature Film Production. In ACM SIGGRAPH 2019 Talks. Article 21.

Andrew Finley, Jesse Erickson, Peter De Mund, and Ying Liu. 2022. Modeling Animated Jumbo Floral Display on Disney’s “Encanto”. In ACM SIGGRAPH 2022 Talks. Article 43.

Haixiang Liu. 2022. Gravity Preloading for Maintaining Hair Shape Using the Simulator as a Closed-box Function. In ACM SIGGRAPH 2022 Talks. Article 40.

Tad Miller, Harmony M. Li, Neelima Karanam, Nadim Sinno, and Todd Scopio. 2022. Making Encanto with USD: Rebuilding a Production Pipeline Working from Home. In ACM SIGGRAPH 2022 Talks. Article 12.

Thomas Müller. 2019. Practical Path Guiding in Production. In ACM SIGGRAPH 2019 Course Notes: Path Guiding in Production. 37-50.

Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 36, 4 (Jun. 2017), 91-100.

Michelle Robinson, Michael Woodside, Daniel Rice, Tad Miller, Scott Kersavage, and Tyler Kupferer. 2022. We Don’t Talk About Bruno - An Encanto Musical Sequence Unveiled. In ACM SIGGRAPH 2022 Production Sessions. Article 2.

Justin Tennant, Mitch Counsell, Far Jangtrakool, Salina Ortega, Rajesh Sharma, Tad Miller, and Scott Kersavage. 2022. Visualizing the Production Process of “Encanto” with the Command Center. In ACM SIGGRAPH 2022 Talks. Article 11.

Jose Velasquez, Alexander Alvarado, Ying Liu, and Maryann Simmons. 2022. Embroidery and Cloth Fiber Workflows on Disney’s “Encanto”. In ACM SIGGRAPH 2022 Talks. Article 22.

Tizian Zeltner, Brent Burley, and Matt Jen-Yuan Chiang. 2022. Practical Multiple-Scattering Sheen Using Linearly Transformed Cosines. In ACM SIGGRAPH 2022 Talks. Article 7.

“It’s not peaches and cream either for men”

Avik Das — Mon, 29 Nov 2021 02:00:00 GMT

I spend a lot of time talking about men’s mental health because it’s what I, as a man, know about. And like with everything, the truth is complicated. We live in a patriarchal society that privileges men in certain ways, but also hurts them in other ways. The harmful effects on our heavily gendered society especially show themselves when racial or class oppression enter the picture. But ultimately, I have to listen to the experiences of others to piece together this complicated web of good and bad. So, when I came across Crossing the Divide, four accounts of transgender men who have experienced life being treated as women and now living as men, I was fascinated.

Today I start a brand new adventure!

swaits.com — Mon, 29 Nov 2021 02:00:00 GMT

I normally ignore recruiter calls. But when I saw an email with the subject "Distinguished Engineer at Zwift", I instantly knew I had to look into it.

I first knew of Zwift in early 2016. I tried it, but didn't stick with it. I was already deep into TrainerRoad and happy with it, and Zwift was still early days and not nearly as polished as it is today.

I was really into cycling those days. I worked my way up on the road from Category 5 to 3, and raced just about every chance I could anywhere in San Diego or even across SoCal and sometimes Arizona. Eventually, I gave up racing and dialed my cycling way back. But, I was still aware of Zwift. When the pandemic hit, all of my cycling buddies instantly moved over to Zwift. And as things opened up and in-person group riding became okay again, many of them stuck with Zwift. And though I'm not as active a cyclist as I once was, I'm still passionate about it. Zwift gives me an opportunity to combine my work with my passion for cycling.

Before Amazon, I spent 19 years making video games professionally. I chose to leave the industry in 2015 and never regretted it. But, I did miss some of the interesting technical challenges. Zwift is basically an MMO video game where the controller is a connected smart trainer. Working at Zwift gets me closer to the most interesting technical challenges I know.

I spent the past five years at Amazon learning, growing, and practicing my skills at broad leadership as a Tech IC. I've worked across hundreds of SDEs on everything from the tactical minutiae to the strategic tech decisions in very large businesses. As a Distinguished Engineer at Zwift, I hope to apply what I've been refining and practicing as a PE at Amazon the past five years.

It's the combination of these three things, a passion for cycling and indoor fitness, getting closer again to really interesting technical challenges like I worked on in games, and landing in a role where I can influence a large tech organization for the better, that makes me extremely excited about this new job!

Climb your own status Everest

D13V — Mon, 29 Nov 2021 01:51:19 GMT

The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; … and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.

Teddy Roosevelt

Welcome to part 6 of the series on “How to be a game changer”. You can find the previous posts here 1, 2, 3, 4, 5.

Driving a new Land Rover is a flex in most parts of the world. By having, and showing a luxury car to others, its owner signals that they are affluent enough to be able to afford it. It can be a Veblen good which separates its owner from the rest, and increases their status. Until it doesn’t. I was fortunate enough to go on a vacation recently in Los Angeles, and booked a modest studio in a nice neighborhood. There I saw the highest concentration of Land Rovers and Range Rovers in my life. One out of every 5 houses seemed to have this status symbol parked in front of it. Which made me think - is it still a status symbol for these neighbors? Or does it merely become a luxury trinket? How many people can have Land Rovers and still claim the status boost? In the luxury car dimension, there are only a limited number of people who can flex. This makes fighting for the status of the owner of the nicest car in the neighborhood a game with limited resources. And you already know my opinion, that we should want to opt out of such games.

However, fighting for status seem similar to fighting over a limited resource, as status may seem limited, but it doesn’t have to be. Status is more abundant than we think. Let’s look one of the most clear cases of status - winning an olympics gold medal, and let’s consider running. It seems like a zero sum - there is a winner, and losers, and only one winner at a time. At least on the surface, the winner gets the majority of the status, while everyone else gets the much lesser status of being a finalist. The other runners are not really worse-off than they would’ve been if they didn’t participate in the running game. They are still getting paid, and they gain fame. Their loss is only relative, and running is a win-win game. The amount of fame and happiness from a single race might seem limited but it is not - it varies with the quality of the performance. For example at the 400m hurdles race at the 2021 Olympics, Karsten Warholm destroyed his previous world record of 46.70s by running it in 45.94s. This is 760 milliseconds faster, three quarters of a second. Beating a world record by so much is wild! But what made the race even more amazing, and Warholm’s win even more impressive is that his main challenger Rai Benjamin also destroyed the previous world record by running 46.17. In order for Warholm to win, he had no choice but to break the world record by a lot. The top two runners each got more glory, fame and status by pushing each other. They both become heroes worthy of praise and both win status, as runners who broke the world record, and creating an unforgettable experience for everyone watching.

Let’s now look at a different type of race - the race to be the first person to make a scientific contribution, and the first company to get a certain innovation to the market. Someone might say, with a dose of cynicism, that the scientists in the forefront are engaging in a zero-sum race with each other and that this scientific discovery was going to get made anyway. Yet it will be a boon for society if that discovery happens 10% faster and brought to market 10% faster. New discoveries build up on top of old and over ten years, this gradual speedup will amount to a whole year of advancement. Over 60 years, that’s whole 6 years of advancement. For healthcare, that may be the difference between having the treatment to a cancer or Alzheimer disease before or after someone you love catches it. Even if we look at a much shorter time frame - the race to create and deploy a COVID vaccine saved millions of lives. The outcome of any seemingly trivial science and innovation race can be a matter of life and death eventually. In the meantime, the race resets every time there’s a new discovery, all the scientists can build up on top of it, and win status. The faster the discoveries happen, the faster new status is “mined”. Some scientists become giants, and that allows other to stand on their shoulders.

In a given dimension, there may be only so much status - one best runner/scientist, one employee promoted to the next level, one president elected. But if we look for more dimensions, we realize that there are more untapped opportunities for gaining status. For example, if two employees are fighting for the same promotion, they might instead build up their areas of ownership to the level that they are providing so much value, and they both get promoted to expand their work. Or, if people want the clout associated with being a president, they can instead focus on achieving what a president would achieve and improving other social and environmental issues such as poverty, pollution, healthcare, etc. If they have a major success there, they might be remembered by the future generations even more than whomever was the president at the time. Martin Luther King Jr was not a president, but has a more important place in history than LBJ, who was president at a time, and even has an official holiday dedicated to him.

In my opinion, the highest status is achievable by each of us, simultaneously, and without detracting from anyone else’s status. That’s the status and confidence of being in internal harmony, and in harmony with Nature. In such state, my internal angels and demons will be dancing together instead of fighting, and my being will be dancing with others instead of fighting them. This is my goal in life, and I wouldn’t feel any less accomplished if others also achieve it - in fact it will make me happier. We are all different, and each of us has a different Everest to climb.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

The physical Sackur-Tetrode entropy of an ideal gas

Bytepawn - Marton Trencseni — Mon, 29 Nov 2021 01:00:00 GMT

I derive the Sackur-Tetrode equation for entropy of a monatomic ideal gas.

Book Review: Endurance, Shackleton's Incredible Voyage

Nicola Iarocci — Sun, 28 Nov 2021 08:05:25 GMT

Of all the stories of maritime adventures I’ve read, that of the Endurance, masterfully told by Alfred Lansing in this book, is the most incredible and shocking. Unbelievable to say, given the premise (a crew of 28 men stranded on the Antarctic pack, camped on floating slabs of ice hundreds of miles from any human settlement, at the gates of the Antarctic winter), but the story does not end in tragedy. The original expedition failed, but what followed the capture of Endurance (never was the name of a ship more prophetic) by the Antarctic ice borders on the impossible, if not the miraculous.

Brief thoughts on crypto

utf9k — Sun, 28 Nov 2021 04:04:00 GMT

It really is what it says on the tin: I have some thoughts about crypto

Emacs bends again

xenodium.com @alvaro — Sun, 28 Nov 2021 02:00:00 GMT

While adding more rendering capabilities to Plain Org, it soon became apparent some sort of screenshot/snapshot testing was necessary to prevent regressing existing features. That is, we first generate a rendered snapshot from a given org snippet, followed by some visual inspection, right before we go and save the blessed snapshot (often referred to as golden) to our project. Future changes are validated against the golden snapshot to ensure rendering is still behaving as expected.

Let's say we'd like to validate table rendering with links, we can write a test as follows:

func testTableWithLinks() throws {
  assertSnapshot(
    matching: OrgMarkupText.make(
      """
      | URL                    | Org link    |
      |------------------------+-------------|
      | https://flathabits.com | [[https://flathabits.com][Flat Habits]] |
      | Regular text           | Here too    |
      |------------------------+-------------|
      """),
    as: .image(layout: .sizeThatFits))
}

The corresponding snapshot golden can be seen below.

This is all done rather effortlessly thanks to Point Free's wonderful swift-snapshot-testing utilities.

So what does any of this have to do with Emacs? You see, as I added more snapshot tests and made modifications to the rendering logic, I needed a quick way to visually inspect and override all goldens. All the main pieces were already there, I just needed some elisp glue to bend Emacs my way™.

First, I needed to run my Xcode builds from the command line. This is already supported via xcodebuild. Next, I needed a way to parse test execution data to extract failing tests. David House's xcodebuild-to-json handles this perfectly. What's left? Glue it all up with some elisp.

Beware, the following code snippet is packed with assumptions about my project, it's messy, surely has bugs, can be optimized, etc. But the important point here is that Emacs is such an amazing malleable power tool. Throw some elisp at it and you can to bend it to your liking. After all, it's [your]{.underline} editor.

And so here we are, I can now run snapshot tests from Emacs using my hacked up plainorg-snapshot-test-all function and quickly override (or ignore) all newly generated snapshots by merely pressing y/n keys. Oh, and our beloved web browser was also invited to the party. Press "d" to open two browser tabs if you'd like to take a closer look (not demoed below).

Success. Emacs bends again.

;;; -*- lexical-binding: t; -*-

(defun plainorg-snapshot-test-all ()
  "Invoke xcodebuild, compare failed tests screenshots side-to-side,
and offer to override them."
  (interactive)
  (let* ((project (cdr (project-current)))
         (json-tmp-file (make-temp-file "PlainOrg_Tests_" nil ".json"))
         (default-directory project))
    (unless (file-exists-p (concat project "PlainOrg.xcodeproj"))
      (user-error "Not in PlainOrg project"))
    (set-process-sentinel
     (start-process
      "xcodebuild"
      (with-current-buffer
          (get-buffer-create "*xcodebuild*")
        (let ((inhibit-read-only t))
          (erase-buffer))
        (current-buffer))
      "/usr/bin/xcodebuild"
      "-scheme" "PlainOrg" "-target" "PlainOrgTests" "-destination" "name=iPhone 13" "-quiet" "test")
     (lambda (p e)
       (with-current-buffer (get-buffer "*xcodebuild*")
         (let ((inhibit-read-only t))
           (insert (format "xcodebuild exit code: %d\n\n" (process-exit-status p)))))
       (when (not (eq 0 (process-exit-status p)))
         (set-process-sentinel
          (start-process
           "xcodebuild-to-json"
           "*xcodebuild*"
           "/opt/homebrew/bin/xcodebuild-to-json"
           "--derived-data-folder" (format "/Users/%s/Library/Developer/Xcode/DerivedData/"
                                           (user-login-name)) "--output" json-tmp-file)
          (lambda (p e)
            (with-current-buffer (get-buffer "*xcodebuild*")
              (let ((inhibit-read-only t))
                (insert (format "xcodebuild-to-json exit code: %d\n\n" (process-exit-status p)))))
            (when (= 0 (process-exit-status p))
              (with-current-buffer (get-buffer "*xcodebuild*")
                (let ((inhibit-read-only t))
                  (insert "Screenshot comparison started\n\n")))
              (plainorg--snapshot-process-json (get-buffer "*xcodebuild*") json-tmp-file)
              (with-current-buffer (get-buffer "*xcodebuild*")
                (let ((inhibit-read-only t))
                  (insert "\nScreenshot comparison finished\n"))
                (read-only-mode +1))))))))
    (switch-to-buffer-other-window "*xcodebuild*")))

(defun plainorg--snapshot-process-json (result-buffer json)
  "Find all failed snapshot tests in JSON and offer to override
 screenshots, comparing them side to side."
  (let ((hashtable (with-current-buffer (get-buffer-create "*build json*")
                     (erase-buffer)
                     (insert-file-contents json)
                     (json-parse-buffer))))
    (mapc
     (lambda (item)
       (when (equal (gethash "id" item)
                    "SnapshotTests")
         (mapc
          (lambda (testCase)
            (when (and (gethash "failureMessage" testCase)
                       (string-match-p "Snapshot does not match reference"
                                       (gethash "failureMessage" testCase)))
              (let* ((paths (plainorg--snapshot-screenshot-paths
                             (gethash "failureMessage" testCase)))
                     (override-result (plainorg--snapshot-override-image
                                       "Expected screenshot"
                                       (nth 0 paths) ;; old
                                       "Actual screenshot"
                                       (nth 1 paths) ;; new
                                       (nth 0 paths))))
                (when override-result
                  (with-current-buffer result-buffer
                    (let ((inhibit-read-only t))
                      (insert override-result)
                      (insert "\n")))))))
          (gethash "testCases" item))))
     (gethash "classes" (gethash "details" hashtable)))))

(defun plainorg--snapshot-screenshot-paths (failure-message)
  "Extract a paths list from FAILURE-MESSAGE of the form:

failed - Snapshot does not match reference.

@−
\"/path/to/expected/screenshot.1.png\"
@+
\"/path/to/actual/screenshot.1.png\"

Newly-taken snapshot does not match reference.
"
  (mapcar
   (lambda (line)
     (string-remove-suffix "\""
                           (string-remove-prefix "\"" line)))
   (seq-filter
    (lambda (line)
      (string-prefix-p "\"" line))
    (split-string failure-message "\n"))))

(defun plainorg--snapshot-override-image (old-buffer old new-buffer new destination)
  (let ((window-configuration (current-window-configuration))
        (action)
        (result))
    (unwind-protect
        (progn
          (delete-other-windows)
          (split-window-horizontally)
          (switch-to-buffer (with-current-buffer (get-buffer-create old-buffer)
                              (let ((inhibit-read-only t))
                                (erase-buffer))
                              (insert-file-contents old)
                              (image-mode)
                              (current-buffer)))
          (switch-to-buffer-other-window (with-current-buffer (get-buffer-create new-buffer)
                                           (let ((inhibit-read-only t))
                                             (erase-buffer))
                                           (insert-file-contents new)
                                           (image-mode)
                                           (current-buffer)))
          (while (null result)
            (setq action (read-char-choice (format "Override %s? (y)es (n)o (d)iff in browser? "
                                                   (file-name-base old))
                                           '(?y ?n ?d ?q)))
            (cond ((eq action ?n)
                   (setq result
                         (format "Keeping old %s" (file-name-base old))))
                  ((eq action ?y)
                   (copy-file new old t)
                   (setq result
                         (format "Overriding old %s" (file-name-base old))))
                  ((eq action ?d)
                   (shell-command (format "open -a Firefox %s --args --new-tab" old))
                   (shell-command (format "open -a Firefox %s --args --new-tab" new)))
                  ((eq action ?q)
                   (set-window-configuration window-configuration)
                   (setq result (format "Quit %s" (file-name-base old)))))))
      (set-window-configuration window-configuration)
      (kill-buffer old-buffer)
      (kill-buffer new-buffer))
    result))

How to Receive Audio Messages from Your Users

Thoughts by Thomas Derflinger — Sun, 28 Nov 2021 02:00:00 GMT

Everybody knows the contact forms on website where users can send text message to the website owner. But how could you design a contact form that allows the recording of an audio message? In this article I want to show you how to do that.

First Post

Speculative Branches — Sat, 27 Nov 2021 20:28:52 GMT

Hello everyone, and welcome to my blog.

I am an ex-Google senior engineer focused on systems programming and performance optimization. My past experience includes hardware engineering, numerical analysis, and high-frequency trading.

Here, we will be talking about software engineering, performance, computer systems and foundations, interesting math concepts, electrical engineering, hardware acceleration, FPGAs, and more. We may also branch out to non-technical topics including companies, organizational psychology, and the stock market.

Your Smart Car may spy on your smartphone

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 27 Nov 2021 19:00:00 GMT

But how can it be smart, if it spies?

We Should All Be Feminists

Apurva Shukla's RSS Feed — Sat, 27 Nov 2021 14:29:48 GMT

⭐ ⭐ ⭐ ⭐ ⭐ Great message! My key take aways are that the lived experiences of people are real and valid - one cannot dismiss them and try…

Local Data Collection and Command Distribution

Greg Herlein — Sat, 27 Nov 2021 09:00:00 GMT

I’m revisiting why I don’t like MQTT, and what I am doing about it.

I'm a Moka guy

Nicola Iarocci — Sat, 27 Nov 2021 08:05:25 GMT

I’m a Moka guy, always have been. Admittedly, I also like so-called American coffee and, of course, espresso. But every day at my place, I’ll have a Moka-brewed coffee. Twice. As I wake up, and then in the afternoon before getting back to work. I’ve been observing the pods frenzy spreading all around me with curiosity and bewilderment in recent years, with dedicated retailers opening (and often closing soon after) everywhere in my town. Bialetti’s Moka coffee pots have always been around my life. Not disposable, for they are pretty expensive even here in Italy, but common, everyday kitchen gadgets? Absolutely.

M5 Paper e-ink Calculator

Brent's Blog — Sat, 27 Nov 2021 02:00:00 GMT

I added an M5 Paper to my collection of dev boards and decided to make a basic calculator to teach my son to count. Since he is only a few months old he mostly uses it as an expensive teething device.

Limits of files you can store in a directory

theblackcat102 — Sat, 27 Nov 2021 02:00:00 GMT

04/03/2022 Update: Instead of using ext4, you should simply consider other file format such as xfs, zfs, btrfs for high file counts storage

If Trees Could Talk

Godsip Club — Sat, 27 Nov 2021 02:00:00 GMT

It Came from The North #

More than one thousand years ago, Vikings ruled the North Sea. They weren’t mainly sailors, but they had a great knowledge about building ships and taking long journeys. Legend says that, while travelling to colonise the new Icelandic land, wooden pillars were brought aboard the long ships. When the coast was on sight, the great poles were thrown overboard; wherever they had touched the shoreline, new settlements would be built. In their villages, those posts were put in the center of the farmhouses, with other two lower ones on the sides, because they were believed to have a sacred power.

Norse people had a special connection with woods and trees – and they weren’t the only ones. First of all, they hadn’t actual temples, so they primarily held celebrations outdoor. They had special places, or altars, named hörgar, where the presence of the otherworld was stronger. That is a general definition for “all the worlds other than Midgard”, where all the supernatural beings live. In these revered sites, during particular moments –as the start of their year, at the end of the modern October–, the boundary between the worlds was narrower, and strange events could happen.

Among the pantheon of deities, Thor was undoubtedly the most famous. He was the mighty slayer of the Jötnar, the giants which wanted to bring chaos in this world. It was told that, when a lightning was seen, it was Thor hitting the target with his hammer Mjölnir; the thunder, then, was the giant crumbling to the ground. He was the god praised by common people, while the most powerful Odin was worshipped mostly by chiefs or, generally, by “major figures”. So, why was Thor so popular?

Marvel’s Thor is totally different than the Norse god described in the myths (Marvel Studios)

One of the reasons could have been his temper: almost every Norse god had his unique personality, and Thor was particularly stubborn, touchy, and wrathful. That’s pretty much the stereotype of a Viking. More than that, he was also related to important notions as fertility¹ and… trees; oaks, more accurately. It may seem odd that a slaughterer of giants was associated to a shrub, but maybe it wasn’t so uncommon.

Let Me Stand Next To Your Fire #

Fun fact: what is the tree most hit by the lightning? You can guess the answer; or just ~~Google~~ DuckDuckGo it. In the last few centuries, oaks have been observed as especially tending to be hit during a thunderstorm. It probably isn’t a coincidence that this is also the tree related to the god of thunder, but the fascinating fact is that the Norse one is one of many.

See? Told ya! 🌳⚡️ (I didn’t use DuckDuckGo just because Google is better known)

Perkūnas was the thunder god in the Baltic area, and was also related to the same tree. In the Slavic mythology, Perun was its counterpart. The Roman Jupiter and the Greek Zeus were thunder gods, too. Of course, there were many more, but they all had curiously similar traits. If these already seem cool coincidences… sit back and grab the popcorns.

Amid cultures from Europe, that due to the same latitude had the same calendar, which depended on harvest or shepherding, it was common to light up large bonfires in specific dates, among which:

First days of Spring
Summer Solstice
All Hallows’ Eve (Halloween)
Winter Solstice

Of course, each country had its own customs, names and traditions, and obviously the days may have differed. Norse people, for instance, called the midsummer fires “Balder’s balefires” (Balder’s Balar), which recalled the god’s funeral pyre. In the Highlands of Scotland, in the first day of May, “Beltane fires” were set on – probably with human sacrifices until some centuries ago. Also, they could have different meanings: Celtic people, for instance, were mostly shepherds and probably their observances were different from the ones of the farmers who lived on the Mediterranean Sea.

Bonfires always have a cozy charm… but the ones described were MUCH larger than this!

There were some notable common points between all these fires, though. All of them symbolised the power of the sun, that dissolved all the negative influences and blessed the harvest and the farm animals. Interestingly, fire also was the only weapon known against witches, that long ago were considered the main (or even the only) cause of illness, curses, etc. In many countries, the last sheaf of corn was called “the Old Woman” or “the Witch” and it was burned; but this is so intriguing that it is worth a further discussion.

Fertility could have been another great mutual peculiarity: the sun had the power to make the vegetation grow, so the fire could have resembled the star’s fertilising strength. This could explain why even the farm animals were driven through the flames, sometimes hurting themselves badly.

You’ve Been Thunderstruck #

A lot more could be said about these pyres, and surely there will be occasion, but what links them to Thor and his fellow thunder gods… are the gods themselves, in the form of oak trees. In fact, it was an ordinary tradition to burn a big log of oak in these special events. The Yule log is probably the best-known heritage from those times, but it was not the only one.

The log was burned slowly: it was taken off the fire soon and kept safe for the whole year. In case of thunderstorms, it was put again on the fire. It was said to have the power to keep the house safe from thunders and to protect it from wildfires. Also, chunks of the burnt wood were put in the animal’s drinker to increase their fertility.

To conclude this brief trip through European fires, a question: what grows on trees, very noticeable during the winter, while the plant itself seems dead? Mistletoe. Since it is so flourishing through the winter, while the tree itself flounders, from a primitive point of view it could have been normal to associate it to the soul. It contains its vitality, the life-giving power of the Sun, which will be reborn in the next Spring. And yes, it often can be found on oak trees.

What if the mistletoe kiss tradition was born to commemorate someone? (Paul Zoetemeijer/Unsplash)

Though it is a parasitic plant, that may be why mistletoe is widely picked up during winter, and why it was so important in Europe. It was believed to cure epilepsy because, since it was so tangled to the branches on the tree, a person who had kept it in the pocket surely wouldn’t fall with seizures. Its modern Celtic name in Brittany, Wales or Ireland still means “all-healer” because of its healing power – which is not proven by scientific studies. Druids had specific golden tools to pick the plant, and special rituals: mistletoe had not to touch the ground, otherwise it would have lost its powers, so a white cloth was used to let it fall on it.

Let’s close the loop. Fire, thunder, trees. Three concepts so distant, yet identical in certain ways. A few paragraphs before, the Norse god Balder was quoted. In one version of the myth of his death, he was killed by his brother Hod, who had been tricked by Loki. Classic Loki. With what object has he been able to kill him, even if the god was invulnerable? With a branch of mistletoe.

So, a god was killed by a plant, which is related to the power of the Sun, which is enclosed in an oak tree, related to a god. From another perspective, we could say that Balder was killed by its own power – or by himself. An impossible paradox? Not at all, we just have to explore a little deeper our past, and maybe ourselves.

Sif, Thor’s wife, had a lush blonde hair that was affiliated to wheat. His husband was related to rain, so he had the responsibility to fertilise the crop. ↩︎

Programming deals

learnbyexample — Sat, 27 Nov 2021 02:00:00 GMT

Hello!

Here's some exciting programming deals for my own ebooks as well sales details from other creators.

Offers for my ebooks (valid till Nov 30)

Practice Python Projects — FREE (normal price $10)
Learn by example Python bundle — $2 (normal price $12)
All 11 Books Bundle — $5 (normal price $22)
Giveaway contest on twitter — a chance to get a single ebook for free

Indie creators

Python Problem-Solving Bootcamp — 40% OFF today, 30% OFF tomorrow and so on (boost your Python problem-solving skills)
Python books by Michael Driscoll — $10 OFF on any book using coupon code "black21" (Python 101/201, Image/PDF/Excel processing, etc)
- see also author's blog post for links to other Python sales
Python Morsels — 50% OFF until Nov 30 (skill-building service with short videos and hands-on bite-sized Python exercises)

Miscellaneous

The Leanpub Monthly Sale for November 2021
Humble Book Bundle: Code Like a Pro by Manning Publications — various sale options starting from $1
Various programming deals discussion on Hacker News
DevUtils.app — 30% OFF this week (Powerful developer tools for your everyday tasks, Native macOS app)
How to Journal to Live your Best Life? — $1.99 for 30 customers, $3.99 for the next 30 customers and so on (not strictly related to programming, applicable for life events, career, etc)

Happy learning :)

Quantum mechanics for everyone

diff × weyl — Fri, 26 Nov 2021 23:46:13 GMT

This post explains quantum mechanics (QM) without any advanced math. Unlike most introductions, I will focus on the interpretation of QM: what the objects in the theory mean and how they fit into a broader philosophy of doing physics. Specifically, I explain why the Von Neumann-Wigner interpretation, a variant of the standard Copenhagen interpretation, is … Continue reading

Renewables are not THE solution

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 26 Nov 2021 19:00:00 GMT

THE ONE solution, that is. Obvious, but never said often enough.

Red Notice

🦉 olshansky 🦁 — Fri, 26 Nov 2021 18:12:09 GMT

Olshansky's review of Red Notice

On "Powered by AI / ML" marketing

Blog on Kailash Nadh / Personal homepage — Fri, 26 Nov 2021 02:00:00 GMT

An email I had sent in response to a survey on the use of “AI / ML” and the “AI-first mindset” in our organisation and in the industry was shared on social media^{[1] [2]} sparking surprising amounts of interest. I did candidly state the simple fact that we haven’t come across any big problems that warrant any specific “AI / ML” solutions in our organisation yet (Zerodha - stock broker that offers online investment and trading platforms), and that the bulk of the “powered by AI” claims we have seen across industries and in the numerous startup pitches that we receive, have been cases of hollow marketing. There is also a general expectation (delusion) that any sufficiently large technology company should be using “AI / ML” somehow, somewhere, for some reason.

Employers should not own open-source contributions

Juan Cruz Viotti — Fri, 26 Nov 2021 02:00:00 GMT

This article makes a case on why employers should not attempt to own the IP of the contributions of their employees

The best Arduino projects are those that look like OLD Legos

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 25 Nov 2021 19:00:00 GMT

And I mean OLD Legos.

Happy Thanksgiving 🦃

Ingo Richter on Today is Tomorrow's Past — Thu, 25 Nov 2021 10:00:00 GMT

Happy Thanksgiving

I wish everybody a great day with their families and friends. I hope you can all meet in person and make up for last year. I’m grateful for having a family and friends, a roof over my head, and a fulfilling job.

Happy Turkey Day, or whatever suits your mood. Stay safe wherever you are and get some time for yourself.

Beaglebone Black - Getting Started

Greg Herlein — Thu, 25 Nov 2021 09:00:00 GMT

Getting Started with the Beaglebone Black (Again)

Book Review: Mathematics is politics

Nicola Iarocci — Thu, 25 Nov 2021 08:05:25 GMT

Mathematics as the study of relationships: in this aspect lies the similarity and affinity with politics. And then the need in both cases to proceed with stubbornness and trust, without fearing error which, as in all difficult things, is not only lying in wait but inherent, and often, when it is discovered, it is the stimulus and engine of new successes and goals. Hence the need to respect rules and (not or, mind you) the compelling need for revolutions. The parallel narration of the author’s life and the change, over the years, of her own beliefs and points of view, makes this short essay more credible and streamlines the story.

🙏

Blog posts of Andreas Klinger — Thu, 25 Nov 2021 02:00:00 GMT

Over the years i met several people that left quite an impact on me and in many cases helped me along the way. I tried to summariz...

Numeric Palindrome

learnbyexample — Thu, 25 Nov 2021 02:00:00 GMT

I posted a coding challenge in the second issue of learnbyexample weekly. I discuss the problem and Python/Perl solutions in this blog post.

Problem statement🔗

Find numbers from 1 to 10000 (inclusive) which reads the same in reversed form in both binary and decimal formats. For example, 33 in decimal is 100001 in binary and both of these are palindromic.

Python solution🔗

Here's one possible solution for this problem:

for n in range(1, 10001):
    dec_s = f'{n}'
    bin_s = f'{n:b}'
    if dec_s == dec_s[::-1] and bin_s == bin_s[::-1]:
        print(n)

Extending the above solution to include more comparisons is easy with built-in features:

for n in range(1, 10001):
    dec_s = f'{n}'
    bin_s = f'{n:b}'
    oct_s = f'{n:o}'
    if all(s == s[::-1] for s in (dec_s, bin_s, oct_s)):
        print(n)

As an exercise, extend this program further to include hexadecimal number comparison as well. Can you find out what's the first number that is greater than ten to satisfy all the four numeric formats?

Perl one-liner🔗

Here's a solution for CLI enthusiasts:

$ perl -le 'for (1..10000) { $bn = sprintf("%b", $_);
                print if ($_ eq reverse) && ($bn eq reverse $bn) }'
1
3
5
7
9
33
99
313
585
717
7447
9009

See my Perl One-Liners Guide ebook if you are interested in learning to use Perl from the command-line.

A Very Overengineered Audio Splitter Panel

TOGoS's Project Log — Wed, 24 Nov 2021 20:13:37 GMT

I used up all my #6 hex nuts to make, over 3 days, a thing that should have been made in a couple hours using more conventional wiring techniques

The big paradox of green, recyclable batteries

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 24 Nov 2021 20:00:00 GMT

More exactly, of green AND recyclable batteries.

mwcleric, my Python MediaWiki client library

River Writes - A MediaWiki Blog — Wed, 24 Nov 2021 08:21:26 GMT

An overview of mwcleric

Development Practices

Ben E. C. Boyter — Wed, 24 Nov 2021 02:00:00 GMT

Development Practices

Follow a development practice of developer empowerment. Encourage change and experimentation with zero or low penalty for defects allowing them to be found early and resolved with additional tests or automation.
I embrace the fact that bugs and problems will be found, but by making the penalty for them exceedingly low we can ensure that they are identified and resolved quickly with little to no production impact. All processes from CI/CD and development are built around this idea and ensure that rapid changes can be made, allowing for extensible solutions, with as low a cost of development as possible while keeping quality high. This also has the added benefit of being a developer friendly approach that ensures teams remain engaged with as little buck passing as possible.
The following development practices are also supported and where appropriate are implemented to ensure the quality of any solution delivered is high and that development remains as easy as possible with minimal blockers.
Use of source control. Appropriate branches, with progress branches checked in allowing for rapid feedback and quick checks to ensure the wrong direction is not taken.
Frequent incremental commits using source control.
Use of static analysis and linting tools prior to committing to ensure that code is consistent and remains of high quality.
Ability to spin up the environment locally. While difficult with serverless infrastructure with up front effort it is possible to run integrations locally to allow for faster development. This can be achieved by using shims in the code allowing for the use of the local filesystem for S3 components and things such as DynamoDB Local for local development. Docker can be used to make this easier.
Give developers a sandbox AWS account for validation of code. This is to allow them to set up integrations such as SQS and develop against it locally to ensure code works, saving them having to deploy to validate, which saves on development and debugging costs.
Build unit and integration tests as appropriate, with the emphasis being on delivering quality code.
Use security scanners to ensure that secrets and keys are never committed to source control, and in the case they are, remove them. If using git this can be achieved using git-filter-branch or tools such as BFG.
Where possible apply S.O.L.I.D development principles.
Either commit vendor dependencies or have a managed place to store them in order to allow control of the version and ensure that development can continue in the case that 3rd party providers are down. This also allows for security management scanning and auditing.
Everyone is responsible for the quality of the solution. This means that developers own the quality of code they write equally with everyone else on the project, ensuring the highest quality of development.
Defence in depth solution design. This ensures that system failures should not degrade the performance of the application as a whole. An example would be that queues and other techniques are implemented such that when the target system recovers the system processes as normal.
Event driven architecture. This applies to defence in depth systems but also ensures minimal cost, as the system scales and grows to meet the needs as required.
Serverless as a preference. This lowers costs, ensures that the system is generally recoverable and self healing.
Managed services as a preference. This lowers maintenance costs and ensures backups are performed on an automated basis as well as scaling.
Encryption by default. Everything is encrypted at rest and in transport. No clear text communication across the stack ever. Where required encrypted memory though the use of secure string implementations can be used.
Stick to AWS best practices with extensions as required.
Continually improve processes to resolve issues. If something causes a problem now, fixing it with a test or other tool solves it for everyone.
Traceable logging. All logs are written to be traceable by transaction, caller and to a specific location within the codebase allowing for prompt investigation where a problem is found.

Game Wizards

ℤ→ℤ — Tue, 23 Nov 2021 20:50:58 GMT

Game Wizards: The Epic Battle for Dungeons & Dragons by Jon Peterson

Publisher’s Website

Game Wizards is a business history spanning the creation of Dungeons & Dragons and TSR to the firing of Gary Gygax in 1985. The book has a special focus on the legal battles between Gary Gygax and Dave Arneson around creator and royalty rights for D&D, but also covers the rivalry between GenCon and Origin and the overall growth of TSR. Jon Peterson assembles the story from a broad-range of sources, including media portrayals, fanzines, interviews, legal documents, unexpected artifacts, and recreations of financial statements.

Peterson lays out the pivotal and mundane operations by year. This approach allows a very deep recalling of each year’s happenings and an “uncompressing” of history. As D&D takes off in the early 1980s, TSR becomes profligate with spending and bad habits become entrenched. By maintaining a constant velocity through history, the narrative is less of a “bubble burst” business story and more a nuanced look at the mechanics of selling and producing the games. The book’s detail will make it a primary resource for later historians and should be of interest to game entrepreneurs for details on marketing, distribution, and rights management.

The result is far-less hagiographic than Empire of Imagination: Gary Gygax and the Birth of Dungeons & Dragons (2015) and showcases more of the personality issues, lousy business practices, and messiness inherit in starting a business within a novel market. The book spends little time on game design or why roleplaying games were successful or fun; for example, non-gamers may be mystified why modules were so lucrative as the reader is assumed to understand the mechanics of hosting a gaming session. However, audience restrictions aside, the book is a very valuable addition to the relatively young field of game studies and should be very useful for comparative studies for how other entertainment industries managed their growth and talent.

Equality needs rituals, NOT flattening

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 23 Nov 2021 19:00:00 GMT

If you care about equality, beware of what flattens society.Especially…

14 years of game dev

Liza Shulyayeva — Tue, 23 Nov 2021 08:02:00 GMT

It feels like just yesterday I got my first job in game dev back in Australia, but it’s been about 14 years! I feel like I’ve done a little bit of everything, and was very fortunate to have gotten to grow as an SE on the job while working with different languages, frameworks, and teams. I guess the main phases of the last 14 years have been:

From the archives [2014]: a speculative interview about public transport

Michael Honey: Writing - michaelhoney.com — Tue, 23 Nov 2021 07:25:51 GMT

Interviewer: Kerry Ingram, thanks for talking to us. You’ve been the CEO of Transport for Canberra for five years now, and it’s been a very exciting five years for the organisation and for the city. The first thing that people from outside notice, of course, is that the TfC network is free.

Ingram: That’s right. In almost every other public transport network, there’s either full or partial cost-recovery via tickets. Its very hard to make public transport profitable, though: the more you charge, the less accessible it is, and the fewer people use the service. Even with pricing set to maximise income, we only used to derive about 20% of our revenue from ticketing. That’s a significant chunk of money, to be sure, but we also spent a significant amount of money installing and maintaining machines, running the ticketing network, enforcing compliance and so on.

The first and most important thing I wanted to do when I entered this job was to eliminate fares. Money’s strange: a dollar here is exactly the same as a dollar there. But people don’t necessarily see it that way. Some services, like roads, are provided “free” by the government, while others – like the old bus services – were user-pays – except in this case, the user was only paying less than a quarter of the full cost. So it’s always been subsidised to a large percentage – we just made that percentage 100.

Dropping fares cost about $25 million, but what we bought with that is far more valuable. We bought access throughout the city as a right for citizens, something which they could expect, in the same way as they expect the streets to be lit, garbage to be collected, fires to be extinguished. Put that way, not making transport free would be perverse.

Collecting fees – the ticketing, the the compliance – itself costs money and time and hassle. Going free means we save that money (none of which helped people travel on our network), but it also means that we make our services faster to board, easier to use, and much more widely available to all members of the community, no matter what their financial status.

But that’s not how everyone saw it.

Sure. We had a backlash from those those stood to lose money, particularly taxi drivers and even the operators of pay parking in the city. The taxi operators had a point: governments in the past had derived significant revenue from selling taxi licenses as exorbitant rates. We ended up buying back those licenses and folding taxi services into the TfC network. There were also cranky letter writers whose basic objection, when you you boiled it down, was to “subsidising” poor people, as if they themselves had paid directly for all the government services they use.

Once we went free, there was a huge increase in network use: it was pretty hard to meet the demand at first, but people understood there’d be growing pains. Now, we’d never go back to the old way: free transport is a defining characteristic of our city. You’re overseas and mention you’re from Canberra and people ask about it. Going free means it’s normalised: public transport isn’t just for people who can’t drive, it’s for everyone. When we became an independent for-benefit corporation, I didn’t want its direction set by the usual bunch of middle-aged businesspeople, people like myself. Canberra’s citizens are shareholders and stakeholders, and I’m responsible to a board which is demographically proportional to our population: we have schoolchildren, young parents, older Canberrans, students, as well as workers of all kinds. We encourage people to contribute their ideas, their energy, their perspectives to the way we work.

How much is it worth to live in a connected community, one where everyone from every walk of life has the same access to the city? How much is it worth to have residents the outer suburbs able to get to work, to play, to visit their families without having to pay extra? And having a popular transport network also means we can have a pervasive network. More people use it to go more places every year. We’re continually extending the network, making it better, faster, more accessible.

It’s a very different network now, compared to five years ago.

Absolutely. I came on board as the head of ACTION: the ACT Internal Omnibus Network. That name specified a mode of transport: naturally, that limited the scope of its ambitions. Changing the name to Transport for Canberra wasn’t just an exercise in rebranding: it reconfigured our thinking around what we’re here for. We’re not in the bus business: we’re in the transport business. We should be mode-agnostic. Today, most users of our network use at least two different vehicles each trip – typically an electric microbus to the nearest local hub, then an express vehicle – tram, train, bus – to a regional centre.

Switching to microbuses for that first five kilometres was a no-brainer. Big buses are great for moving fifty, a hundred people, and above that number, trains excel. But big buses are noisy, they don’t take corners well, and they accelerate and decelerate badly, which is exactly what you don’t want when you’re picking up passengers every few hundred metres. The electric microbuses are cheaper to run and easier to maintain – and they use green energy, so it’s a win all around.

The new network – and the organisation which runs it - is– much more flexible, more responsible, more able to cope with shifting demands. Requesting a microbus via smartphone and knowing the exact moment it’ll be there beats standing in the rain waiting for a big bus to come and pick you up. We track every trip via face-recognition, so we know who’s going where. Better data leads to better decisions: our systems can predict demand and meet it efficiently. We’re seeing people use the network outside of normal times, to do things that they never would have done before. Groups of friends call up a microbus and go out to dinner: picking people up and dropping them home is part of the night.

What’s next, then?

We’re going to see further diversification in our vehicle fleet. Usage patterns are notoriously lumpy: we get much more traffic at the start and end of the working day, of course, but weather and school holidays and major events all feed into it. Ideally we’ll always be able to meet demand. We’re looking at bringing in casual, on-demand microbuses or even smaller vehicles, owned privately, which will be available when required to meet peak needs. A semiretired person might do a couple of hours at each end of the day, making contacts with their local community and earning some money while they’re at it. A uni student could go out late on a Friday night and be the designated driver for their whole suburb.

We’re seeing some unexpected benefits too, with our shift to electric vehicles. Deciding to build the vehicles ourselves rather than going out to tender has literally electrified our engineering and maintenance teams: we’re getting orders from other cities for vehicles of their own, and there’s a growing cottage industry of people renting the workshops and our peoples’ skills after-hours, to do their own electric vehicle conversions. We’re not only the free-transport city: we’re now the electric vehicle city.

We don’t have the infrastructure for it yet, but I’m also interested in how an electric fleet can help power our homes and our workplaces. Most of our vehicles are still idle for much of the time: they could be feeding power back into the grid, heating our buildings, making our energy network more flexible, more resilient. Further out, I imagine autonomous, self-driving vehicles will make a big impact: salary costs are three-quarters of our budget. If we could save that money, we could move a lot more people…

Interesting times. Thanks for talking to us.

Thank you.

Skills, stories, and software every dev should know - 123dev #47

Justin Garrison's Homepage — Tue, 23 Nov 2021 02:00:00 GMT

Comments

Moment of impact

If you watch the gif again look at the moment the ball hits the glass. The glass appears to shatter before the ball makes contact. How is that possible?

It turns out glass often breaks from tension rather than compression. The ball pushing on the glass makes it break from the opposite side from where it hit.

Implementing zip archiving in Golang: unzipping

Notes on software development — Tue, 23 Nov 2021 02:00:00 GMT

All code for this post is available on Github.

Let's take a look at how zip files work. Take a small file for example:

$ cat hello.text
Hello!

Let's zip it up.

$ zip test.zip hello.text
adding: hello.text (stored 0%)
$ ls -lah test.zip
-rw-r--r-- 1 phil phil 177 Nov 23 23:04 test.zip

So a 6 byte text file becomes a 177 byte zip file. That is pretty small! Parsing 177 bytes sounds like it can't possibly be too complicated!

Let's hexdump the zip file.

$ hexdump -C test.zip
00000000  50 4b 03 04 0a 00 00 00  00 00 8a b8 77 53 9e d8  |PK..........wS..|
00000010  42 b0 07 00 00 00 07 00  00 00 0a 00 1c 00 68 65  |B.............he|
00000020  6c 6c 6f 2e 74 65 78 74  55 54 09 00 03 74 73 9d  |llo.textUT...ts.|
00000030  61 74 73 9d 61 75 78 0b  00 01 04 eb 03 00 00 04  |ats.aux.........|
00000040  eb 03 00 00 48 65 6c 6c  6f 21 0a 50 4b 01 02 1e  |....Hello!.PK...|
00000050  03 0a 00 00 00 00 00 8a  b8 77 53 9e d8 42 b0 07  |.........wS..B..|
00000060  00 00 00 07 00 00 00 0a  00 18 00 00 00 00 00 01  |................|
00000070  00 00 00 a4 81 00 00 00  00 68 65 6c 6c 6f 2e 74  |.........hello.t|
00000080  65 78 74 55 54 05 00 03  74 73 9d 61 75 78 0b 00  |extUT...ts.aux..|
00000090  01 04 eb 03 00 00 04 eb  03 00 00 50 4b 05 06 00  |...........PK...|
000000a0  00 00 00 01 00 01 00 50  00 00 00 4b 00 00 00 00  |.......P...K....|
000000b0  00                                                |.|
000000b1

We can see both the file name and the file contents in there.

Structure

Let's take a look at the zip structure defined here. Based on section 4.3.6 it looks like file metadata followed by the file contents are stored one after another with a final chunk of "central directory" metadata.

Image Credit

The local header metadata looks like this:

Field	Size
local file header signature	4 bytes
version needed to extract	2 bytes
general purpose bit flag	2 bytes
compression method	2 bytes
last mod file time	2 bytes
last mod file date	2 bytes
crc-32	4 bytes
compressed size	4 bytes
uncompressed size	4 bytes
file name length	2 bytes
extra field length	2 bytes
file name	variable
extra field	variable

The header signature is a single integer (0x04034b50) in a valid zip file. We'll ignore version, the general purpose flag, and the checksum. Compression is either 0 for no compression or 8 for DEFLATE compression/decompression.

Last modified time and date is MSDOS-style date/time format which is pretty funky.

Let's translate this roughly to Go with some high level flourishes.

package main

import (
    "os"
    "bytes"
    "compress/flate"
    "io/ioutil"
    "encoding/binary"
    "time"
    "fmt"
)

type compression uint8
const (
    noCompression compression = iota
    deflateCompression
)

type localFileHeader struct {
    signature uint32
    version uint16
    bitFlag uint16
    compression compression
    lastModified time.Time
    crc32 uint32
    compressedSize uint32
    uncompressedSize uint32
    fileName string
    extraField []byte
    fileContents string
}

main

Our entrypoint will read a zip file and keep walking through the file until we stop being able to parse zip file entries.

func main() {
    f, err := ioutil.ReadFile(os.Args[1])
    if err != nil {
        panic(err)
    }

    end := 0
    for end < len(f) {
        var err error
        var lfh *localFileHeader
        var next int
        lfh, next, err = parseLocalFileHeader(f, end)
        if err == errNotZip && end > 0 {
            break
        }
        if err != nil {
            panic(err)
        }

        end = next

        fmt.Println(lfh.lastModified, lfh.fileName, lfh.fileContents)
    }
}

Files

For each file we'll fail early if the first four bytes are not the magic zip signature.

var errNotZip = fmt.Errorf("Not a zip file")

func parseLocalFileHeader(bs []byte, start int) (*localFileHeader, int, error) {
    signature, i, err := readUint32(bs, start)
    if signature != 0x04034b50 {
        return nil, 0, errNotZip
    }
    if err != nil {
        return nil, 0, err
    }

The basic pattern is that one of these read helpers will take an offset and return a Go value and a new offset. The read helper will do bounds checking. We'll define the read helpers further down.

Let's follow the same pattern to the end of the struct:

    version, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }

    bitFlag, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }

    compression := noCompression
    compressionRaw, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }
    if compressionRaw == 8 {
        compression = deflateCompression
    }

    lmTime, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }

    lmDate, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }
    lastModified := msdosTimeToGoTime(lmDate, lmTime)

    crc32, i, err := readUint32(bs, i)
    if err != nil {
        return nil, 0, err
    }

    compressedSize, i, err := readUint32(bs, i)
    if err != nil {
        return nil, 0, err
    }

    uncompressedSize, i, err := readUint32(bs, i)
    if err != nil {
        return nil, 0, err
    }

    fileNameLength, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }

    extraFieldLength, i, err := readUint16(bs, i)
    if err != nil {
        return nil, 0, err
    }

    fileName, i, err := readString(bs, i, int(fileNameLength))
    if err != nil {
        return nil, 0, err
    }

    extraField, i, err := readBytes(bs, i, int(extraFieldLength))
    if err != nil {
        return nil, 0, err
    }

Now if the file contents are uncompressed we can just copy bytes after the file header. If the file contents are compressed though we'll use Go's builtin DEFLATE support to decompress the bytes after the file header.

    var fileContents string
    if compression == noCompression {
        fileContents, i, err = readString(bs, i, int(uncompressedSize))
        if err != nil {
            return nil, 0, err
        }
    } else {
        end := i + int(compressedSize)
        if end > len(bs) {
            return nil, 0, errOverranBuffer
        }
        flateReader := flate.NewReader(bytes.NewReader(bs[i:end]))

        defer flateReader.Close()
        read, err := ioutil.ReadAll(flateReader)
        if err != nil {
            return nil, 0, err
        }

        fileContents = string(read)

        i = end
    }

And return the filled out representation:

    return &localFileHeader{
        signature: signature,
        version: version,
        bitFlag: bitFlag,
        compression: compression,
        lastModified: lastModified,
        crc32: crc32,
        compressedSize: compressedSize,
        uncompressedSize: uncompressedSize,
        fileName: fileName,
        extraField: extraField,
        fileContents: fileContents,
    }, i, nil
}

Read helpers

Now we just define those read helpers with bounds checking, using Go's builtin libraries for dealing with binary encodings.

var errOverranBuffer = fmt.Errorf("Overran buffer")

func readUint32(bs []byte, offset int) (uint32, int, error) {
    end := offset + 4
    if end > len(bs) {
        return 0, 0, errOverranBuffer
    }

    return binary.LittleEndian.Uint32(bs[offset:end]), end, nil
}

func readUint16(bs []byte, offset int) (uint16, int, error) {
    end := offset+2
    if end > len(bs) {
        return 0, 0, errOverranBuffer
    }

    return binary.LittleEndian.Uint16(bs[offset:end]), end, nil
}

And basically only bounds checking for grabbing bytes and strings.

func readBytes(bs []byte, offset int, n int) ([]byte, int, error) {
    end := offset + n
    if end > len(bs) {
        return nil, 0, errOverranBuffer
    }

    return bs[offset:offset+n], end, nil
}

func readString(bs []byte, offset int, n int) (string, int, error) {
    read, end, err := readBytes(bs, offset, n)
    return string(read), end, err
}

MSDOS time

At the time zip was created, MSDOS time format was popular, I guess. But it's not popular today so it took a bit of work to finally find an explanation of the format with some code (in C).

func msdosTimeToGoTime(d uint16, t uint16) time.Time {
    seconds := int((t & 0x1F) * 2)
    minutes := int((t >> 5) & 0x3F)
    hours := int(t >> 11)

    day := int(d & 0x1F)
    month := time.Month((d >> 5) & 0x0F)
    year := int((d >> 9) & 0x7F) + 1980
    return time.Date(year, month, day, hours, minutes, seconds, 0, time.Local)
}

Tout ensemble

Running it we get:

$ go build
$ ./gozip test.zip
2021-11-23 23:04:20 +0000 UTC hello.text Hello!

That looks good! Now let's try zipping more than one file.

$ cat bye.text
Au revoir!
$ rm test.zip
$ zip test.zip *.text
  adding: bye.text (stored 0%)
  adding: hello.text (stored 0%)
$ ./gozip test.zip
2021-11-24 03:40:00 +0000 UTC bye.text Au revoir!

2021-11-23 23:04:20 +0000 UTC hello.text Hello!

Fab.

Notes

There are many parts of the standard to deal with (e.g. directories) and many common extensions. I'm ignoring them.

There's some space left at the end of the file which is probably the "central directory" metadata but I haven't dug into that. Understanding those last remaining bits are probably necessary if I want to be able to create zip archives.

I wrote a new post on building a zip archive reader in Go!https://t.co/U0Yg2powlP pic.twitter.com/ns5dF3mjIx
— Phil Eaton (@phil_eaton) November 24, 2021

Refining The Return Type Of Java Methods Without Breaking Backwards-Compatibility

Gunnar Morling — Mon, 22 Nov 2021 19:30:00 GMT

Table of Contents

The Problem
Bridge Methods to the Rescue
Creating Bridge Methods Ourselves

If you work on any kind of software library, ensuring backwards-compatibility is a key concern: if there’s one thing which users really dislike, it is breaking changes in a new version of a library. The rules of what can (and cannot) be changed in a Java API without breaking existing consumers are well defined in the Java language specification (JLS), but things can get pretty interesting in certain corner cases.

The Eclipse team provides a comprehensive overview about API evolution guidelines in their wiki. When I shared the link to this great resource on Twitter the other day, I received an interesting reply from Lukas Eder:

I wish Java had a few tools to prevent some cases of binary compatibility breakages. E.g. when refining a method return type, I’d like to keep the old method around in byte code (but not in source code).

I think kotlin has such tools?

In the remainder of this post, I’d like to provide some more insight into that problem mentioned by Lukas, and how it can be addressed using an open-source tool called Bridger.

NEVER mix your employer and your smartphone

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 22 Nov 2021 19:00:00 GMT

Seriously. Never, ever mix them.

Butterfly scales tile the plane, expanding spiral

Michael Honey: Writing - michaelhoney.com — Mon, 22 Nov 2021 07:26:04 GMT

Butterfly scales tile the plane, expanding spiral. Using p5js

Franklin Falls

Sam Hooke — Mon, 22 Nov 2021 05:00:00 GMT

No trip report yet.

Staff Engineer: Leadership beyond the management track

Home on Pinecoder blog — Mon, 22 Nov 2021 02:00:00 GMT

This post is a review of (and thoughts about) Will Larson’s Staff Engineering book. Introduction I’ve worked at a few successful1 startups, yet rarely one’s that had proper engineering roles as such. If your tech career spawned out of IT like me, you might lack a conceptual framework for thinking about engineering leadership. There are hero programmer’s and prestige seeker’s. If there is an Engineering Manager it might be someone from finance with an MBA.

Art Science Taste And Clean Code

julik live — Mon, 22 Nov 2021 02:00:00 GMT

Creeper Host API wrapper in TypeScript

Max Leiter — Mon, 22 Nov 2021 02:00:00 GMT

I've had a server on Creeper Host for around eight years now, and have used it for small side projects and hackathons. Lately, I've been trying to automate more of my life and discovered that the API their in-house control panel uses is available for general use. I'm making this post in case someone else ever wants to interact with their API via JavaScript. Maybe someday I'll create a proper repository, but for now my wrapper is incomplete and it's easier to just include here. It was also a good chance to write some TypeScript from scratch, something I haven't had the opportunty to do very much of.

To get started, you'll need to generate a Developer Token on the control panel. Navigate to the `Sub-accounts` page and generate a new API key at the bottom of the page:

Also, install axios (or change the below code to your favorite request library, I don't care):

    yarn add axios

Then, incorporate these two files:

import axios, { AxiosRequestConfig } from 'axios'
import { InitializationOptions, MetricResponse, RestartResponse } from './types'

export default class Client {
  private instance: string
  constructor({
    apiKey,
    apiSecret,
    apiUrl = 'https://api.creeper.host',
    instanceId,
  }: InitializationOptions) {
    axios.defaults.baseURL = apiUrl

    axios.defaults.headers.common['key'] = apiKey
    axios.defaults.headers.common['secret'] = apiSecret
    axios.defaults.headers.common['Content-Type'] = 'application/json'

    this.instance = instanceId
  }

  private async request<T = any>({
    method = 'GET',
    apiRoute,
    body,
  }: {
    method: 'GET' | 'POST'
    apiRoute: string
    body?: any
  }) {
    const config: AxiosRequestConfig = {
      method,
      url: apiRoute,
      data: body,
    }

    console.log(
      `[${method}] ${apiRoute}`,
      body ? `\t ${JSON.stringify(body)}` : '',
    )

    const request = await axios.request<T>(config)
    return request.data
  }

  public os = {
    getram: async () => {
      return this.request<MetricResponse>({
        method: 'GET',
        apiRoute: 'os/getram',
      })
    },
    getssd: async () => {
      return this.request<MetricResponse>({
        method: 'GET',
        apiRoute: 'os/getssd',
      })
    },
  }

  public minecraft = {
    restartserver: async () => {
      return this.request<RestartResponse>({
        method: 'POST',
        apiRoute: 'minecraft/restartserver',
        body: { instance: this.instance },
      })
    },
  }
}

export type MetricResponse =
  | {
      status: 'success'
      free: number
      used: number
    }
  | {
      status: 'error'
      message: string
    }

export type RestartResponse =
  | {
      status: 'success'
      message: string
    }
  | {
      status: 'error'
      message: string
    }

Finally, create an instance via `new Client({ apiKey, apiSecret, instanceId });`
You can use it by finding what you want to call on the Creeper Host docs and accessing that on the client; I purposefully used the same naming scheme. For example,

Client.os.getram(): Promise<MetricResponse>

corresponds to the https://api.creeper.host/os/getram endpoint.

I removed some methods from the above to avoid cluttering this page, but it's straight forward to add your own. If you really want to use this, contact me and I can set up a repo for us to collaborate. Also, if you have suggestions on how I can improve this, let me know via email or on Twitter.

Exiting Artificial Scarcity

D13V — Sun, 21 Nov 2021 23:57:55 GMT

He who controls the spice, controls the universe

Baron Harkonnen, character in “Dune”

This is part 5 of the series on “How to be a game changer”. You can find the last four here: 1, 2, 3, 4. Today we continue with examples of negative games you don’t want to play - and we’ll discuss artificial scarcity.

Let’s start with my dog, Euler. Originally he didn’t always want to eat his dog food kibble. So me and my partner started mixing in pieces of meat, or sauce - something to add more real flavor. Euler would then devour it. He wasn’t sick - he was just a picky eater. We were glad we could help him get the nutrition he needs, and enjoy the flavor, but then we realized we were having a hard time attracting his attention, he became bratty, and treats lost their persuasion power. What happened was that Euler expected something better than the treats - meat in his dog food - for granted. We then stopped flavoring his kibble, and at first he didn’t eat much again. But a stomach is a powerful motivator and he of course got back to eating. Not only that, but now he was a lot more interested in treats in other situations, which helped us teach him other behaviors and distract him when others are around.

We definitely could afford to put a little bit of meat in his food every meal, heated on the microwave so the juices flavor the dogfood. But we introduced an artificial scarcity for him, and as a result his behavior improved. We didn’t even use the words artificial scarcity when discussing this policy - we just reasoned about the behavior he would exhibit with or without the extra meat in his food.

So what really is scarcity, and when is it artificial? And how do you escape the artificial scarcity? Scarcity has a simple meaning - some resource is only available in small quantities - fewer than people want. Gold is somewhat scarce on Earth, as we tend to want a lot more gold than we have on Earth, but on a lot of asteroids it may be abundant. When we can mine these asteroids, gold will no longer be scarce for us.

When something is scarce and people want it, it becomes valuable. And this is the dynamic which causes artificial scarcity - when someone wants an item to be valuable, if they can make this item appear scarce, it will be valuable. They can then sell that item for a large profit, and incentivize behavior in others who want that item. For example, seeing a picture of Mona Lisa is not scarce, as you can just google it, but seeing the original in the Louvre in Paris is scarce.

Ultimately, artificial scarcity is about control. There is no need for a conspiracy here, as those with the means to control scarcity will simply do so to maximize their wellbeing, profitability, and power.

The way to exit scarcity is being truthful to yourself what you truly want inside, and not what other people have made you to believe you want. One way to exit scarcity is to not be swung by hype. There was no scarcity of smart phones in 1920, as nobody wanted them, but there was scarcity in 2007 because more people wanted one than there were smartphones. For the first few years Apple made the new model of iPhone scarce by adding some feature which wasn’t available on the old phones. But today they haven’t innovated as much and people are not as hyped up on getting the newest iPhone.

Another way is to not buy what is artificially scarce, and adopt a mindset of abundance. Realize that FOMO is just visibility bias, the things that we crave, we crave not because we really need them and they are scarce, but because we see them more often. Monkey see, monkey want. But we can notice that monkey in our mind and kindly tell it about all the other things it doesn’t see. See, the monkey’s got the steering wheel, but we can influence it ourselves, similar to how our environment influence it. We can tell the monkey about the actual abundance available around us. Materially, humanity is the most advanced it has ever been, but we still often look at the world through artificial scarcity glasses.

Today California, and especially the Bay Area has artificial scarcity of housing, which creates high real estate and rent prices, and exacerbates homelessness. Buying overpriced housing would mean giving in to the artificial scarcity narrative. We have the technological and economic power to make this area a lot more welcoming and affordable, but that means people who recently bought houses would experience a big loss in paper value. One way to exit is to not buy, because if you buy, one day the artificial scarcity will meet reality, one way or another, and someone will be the “biggest fool”, who bought an ultra expensive house, but lost all paper value. And that reality check might be sooner than you expect - Silicon Valley is no longer the only hub for high tech development, and a lot of the demand for its scarce housing may be redirected to other locations. If you have to be there - rent. That way you’re still playing an artificial scarcity game, but at least the stakes are much lower.

In a sense, Bay Area homeownership is a proxy for status today. And status is another type of artificial scarcity, but it is so important that it deserves its own article. We’ll discuss status in the next week’s essay.

rsync on macOS: Matters Arising

Misleading Metaphors — Sun, 21 Nov 2021 13:27:26 GMT

Setting up your macOS to use rsync v3.2.3 from Homebrew, the right way.

AWKGo, an AWK-to-Go compiler

Ben Hoyt's technical writing — Sun, 21 Nov 2021 13:10:00 GMT

A discussion of AWKGo, a simple compiler that translates a useful subset of the AWK text processing language into Go source code.

WFH Day 616

Ingo Richter on Today is Tomorrow's Past — Sun, 21 Nov 2021 10:00:00 GMT

WFH Day 616

I spent four weeks in Germany with my wife and some of my children. Part of the time, I had some PTO and was able to meet with friends and family. The remaining time I was either working or renovating the new place for my wife. Usually working at my desk using my brain, it was a welcome change of pace to work with my hands and do more manual labor tasks. After seven consecutive days of manual work stripping carpet, removing wallpaper, painting, removing old furniture and crazy wood ceiling decorations, I was looking forward to having a small break. My whole body was aching, and every step hurts :-) I’m on my way back to California before returning to Germany in 4 weeks for Christmas break and to finalize the new place. There is always something to do in a new place… I always wanted to say that I’m grateful to live in California and periodically work remotely from Germany. Glad my employer gives me that freedom. Especially during these challenging times when traveling is not always straightforward. I’m looking forward to the day that my family will travel to California without any restrictions. I need to remind myself to stay positive and hope for the best as time goes by. That’s the only way to keep my sanity and don’t lose sight of better times for all of us.

How to automatically pull and deploy updated Docker images

Nicola Iarocci — Sun, 21 Nov 2021 08:05:25 GMT

We want our test and production stacks to be automatically updated every time something new is pushed to the test or release branch. CI builds the docker image on successful test runs, then stores it in our private registry. But how do you automatically pull and deploy those updated images?

I looked into the Watchtower project, which is interesting. You add Watchtower to the stack, and it will diligently check for new versions of the images used by the containers in the stack, pulling, building and deploying as needed while the stack is up and running. In my experiments, however, I had little luck in making it talk with our private registry. Also, I’m not too fond of polluting my stack with foreign containers. I want my docker stack to be simple, tidy, clean, and single-tasked.

Only 1% of Software Benefits from SIMD Instructions

Ash's Blog — Sun, 21 Nov 2021 02:00:00 GMT

Analysis of 2,000+ Linux and macOS binaries reveals less than 1% of instructions use SIMD, despite CPUs dedicating massive die area to vector processing.

Munchie and Mangia

Sam Solomon — Sun, 21 Nov 2021 02:00:00 GMT

We moved into our new house in late September. It felt enormous compared to the studio where Holly, Remi, and I spent most of the pandemic.

Please support every "Polar Bear" like this

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 20 Nov 2021 19:00:00 GMT

Because we need Open Hardware manufacturing. A lot.

Dune

🦉 olshansky 🦁 — Sat, 20 Nov 2021 10:47:58 GMT

Olshansky's review of Dune

Catastrophic Wealth

Greater Dan Or Equal To — Sat, 20 Nov 2021 04:07:25 GMT

Sometimes I worry about becoming rich.

You Better Shut Your Mouth (While You Sleep)

Godsip Club — Sat, 20 Nov 2021 02:00:00 GMT

A Tiny White Mouse #

Vampires are not the only popular myths from Transylvania: there are other peculiar beliefs, often broaden to other countries – or maybe borrowed from them, who knows. One is about the soul and it is told by mothers to their children: if you sleep with your mouth open, your soul will escape your body in the form of a tiny white mouse.

That’s a powerful image and it’s hard to forget – I’d bet that you will recall it next time you’ll see your partner sleeping. ’Silly Europeans, they have such a fantasy!’ one would think, blaming Medieval exaggerations… Well, sorry to tell you, but centuries before, in Guinea, if a person had waken up with bone pain, he would think that his soul was kicked by another one in the sleep.

It was quite common opinion that, while we are dreaming, all that we see and feel reflects what the soul is doing: if a primitive hunter dreamt about chasing an animal, his soul was actually doing that. In the sleep, something leaves our body and continues living. Because of that, to cover the mouth of a sleeping person would be considered as despicable at least: what would happen if the spirit wouldn’t be able to return to the owner?

Please note how this tiny kitten keeps his mouth wisely closed while he sleeps.

This could also be the reason why it is a common rule to avoid to wake up a sleeper abruptly: his soul wouldn’t have time to get back to the body. Someone says that he would get sicker and sicker until it won’t come back. Worst of all, in Bombay it was considered murder to change a sleeper’s aspect, for instance painting the face. The soul then wouldn’t recognise the person and hence couldn’t return to the body.

When In Doubt, Close It #

In general, the soul is conceived –literally– as a bird ready to take flight. It’s easy to “scare it” or to lose it, especially when a person is:

Asleep
Sick (or about to die)
Eating
Terrified

In all these cases, the spirit could just exit an orifice and fly out. An orifice, that’s correct, not necessarily the mouth. Sometimes the nose and the eyes can be a way out, too!

Note that I don’t want to make fun of ancient beliefs, but sometimes, as a foolish game, I try to imagine what could have meant to live in a society with such great taboos. Just consider that in Celebes, in Indonesia, fishing hooks were fastened to a sick man’s nose, navel and feet to “catch” the soul in case it would escape. Fishing hooks. Marquesans, from French Polynesia, held the mouth and nose of dying people in order to prevent his spirit to fade away – imagine those poor souls¹, suffocated while trying to grasp to their life. We really can’t realise how such far and ancient civilisations lived one or two thousand years ago… Even without going too far in time, there are some intriguing examples from recent times.

Look At This Photograph #

Shame on you, Nickelback, you shouldn’t expose your spirit so much! You see, people from Andaman archipelago, in the Indian Ocean, thought that the soul is in one person’s reflection, even in water pools. People are more vulnerable when they walk next to ponds, so they have to keep their eyes low to avoid hurting themselves and the others. Also, it was believed that those were sorcerers’ hunting grounds, who had the power to steal a man’s soul and therefore make him fall sick. In primitive cultures, a great knowledge meant a great power.

This man clearly has no idea about the risk he’s taking. (Via Marc Olivier-Jodoin/Unsplash)

Since the reflection could be harmful, when someone passed away, in the home, all the mirrors had to be covered with a cloth. Otherwise, the alive one’s spirits could be taken away by the one of the dead, which continued to stay there for some days.

As the title says, spirits could also lie in photographs. Tepehuanes people from Mexico believed that photographers had the power to catch their souls and then devour them as they pleased. Also, a chief of a village by the Yukon River, when he examined an explorer’s camera, after a minute of silence, announced to his people: “He has all of your shades in this box.”

Even in shadows the souls could lie. In Nepal, the Grand Lama was able to defeat Sankara while the latter was floating mid-air, by sticking a sword in his shade. Also, in China, at funerals people next to the coffin take a step back when it is about to be closed or when it is about to be lowered into the ground – their shadows could get caught or fall into the grave. Now it’s a little clearer why grave-diggers tie a strip of cloth firmly around their waist: to avoid such accidents.

One of the most curious facts, though, is the awe and the dread that several cultures, especially in Australia, have towards… mothers-in-law. In some tribes of New South Wales, a man could not have any communication with his wife’s mother; it was prohibited to even look in her direction. It’s a bad omen if the mother-in-law’s shade fell on the legs or any part of the man’s body. In the New Britain Island, one of the solemn oaths was: “Sir, if I am not telling the truth, I hope I may shake hands with my mother-in-law.” Understandable, I guess. 😬

Profit Maximisation #

Until now I quoted some examples that point out how fragile and somehow volatile the soul was believed to be. The question comes naturally: ‘Are there any positive aspects in all of this?’ Of course! People always find a way to yield profit of something. In particular, that’s the case of somebody with a greater knowledge who understands how spirits work: sorcerers and shamans.

Even in the Middle Ages, there were plenty of stories and legends with powerful mages who were invincible because their soul was kept safe somewhere outside their bodies. Russian Koschei is one of the many characters who hid their spirit in nested objects and cannot be defeated until the hero of the story is able to find the magical item that holds their life (which usually is tiny and fragile).

Often, in the lore, the trial to find this treasure is quite absurd. A classic example goes like this: “On a remote island, there is a forest with a circle of trees. In the middle, six buckets of water lie. Under the last one, there is wild boar. In the wild boar’s stomach, there is a goose. Inside the goose, there is an egg. In that egg, lies the soul.” What can we say? 👀

Some videogame bosses require atypical strategies to be defeated, too.

Despite folk tales, souls depicted as objects or animals are quite common: totems are a great example of a deep connection between men and certain animals who are treated as sacred. Dyaks of Borneo believed that humans have seven souls; it’s perfectly possible, from a primitive point of view, to trust one of them to a totem animal. The indivisibility of the spirit is a “modern” religious dogma that never touched many civilisation, even if it was considered by Plato.

The Little People (not those ones!) #

To close the circle, we could say that soul’s perception is quite… diversified around the globe, but indeed evident. Another form it can take is the same of the host – just smaller. Nootkas Indians believed that we have a small man, one-inch tall, in our head. If it holds a standing position, we are healthy; if it happens to lose its upright stance, the owner loses the senses.

By combining some of the concepts above, we get interesting cocktails. Among the Indian tribes of the Lower Fraser River, a person has four souls: the first one has the shape of a mannikin, while the others are the shadows of it. Moreover, until now we didn’t mention trees! It is a firm conviction that a bond exists between men and woods, and there are so many fascinating stories.

Little People, a concept widely spread across the globe — “1Q84” by Degdeg on DeviantArt.com

Despite the credence that spirits inhabit the forestland, which is so vast that it will be addressed in a separate article, it is quite common to plant a fruit tree when a baby is born, even in modern times. Since then, the fate of both beings will be connected. Maori people had the same belief and used to place the baby’s umbilical cord and a tiny plant in a sacred field. In England, to cure rupture or rickets, children were passed through cracked ash trees; therefore, a sympathetic connection was established between them.

Last, but not least, genealogical trees are quite spread. Though, it is told that some families have special affinities with particular physical trees, as the Scottish Edgwells. Nearby Castle of Dalhousie, an old oak tree stands from decades; it was told that every time a branch fell, a member of the family was about to pass away. Even Sir Walter Scott noted it in his journal.

All of these curiosities have been extracted from The Golden Bough by James G. Frazer. #

Poor… souls! Ah! 🥶 ↩︎

2021–11–20: Pinephone Pro – audio and modem power up support

megi's PinePhone Development Log — Sat, 20 Nov 2021 01:00:00 GMT

Where I hope to go (just NOT with SubStack)

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 19 Nov 2021 19:00:00 GMT

Uncomfortable center, yes. Substack and friends, no.

The Jaunt

Arun Mani — Fri, 19 Nov 2021 02:00:00 GMT

A future that is scary because of science.

Working at Amazon. A life changing experience.

swaits.com — Fri, 19 Nov 2021 02:00:00 GMT

Today is my last effective day with Amazon - my official last day is next Friday, November 26. But I'm taking most of next week off in observance of Thanksgiving in the US. Thus, today is really it!

Inevitably people will ask "why?" It's no secret that I planned to finish my career at Amazon; after all, my hair isn't getting thicker or less gray as each day goes by. And this is a damn good gig!

But, life is full of unexpected twists and turns. Something in the "oh wow, I gotta go try this" category landed in my inbox. More on that later. For now, I want to take a moment to celebrate Amazon.

This has been the best five years in my career, by far. Amazon was truly life-changing for me. From getting to work with an amazing Principal Engineer community, alongside industry luminaries, collaborating with engineering teams to build distributed systems at unmatched scale, working with hundreds of amazing, brilliant people in all kinds of roles who taught me new stuff everyday, realizing the power of Amazon's Leadership Principles and Working Backwards… I just can't say enough great things about the whole experience.

Lots of people here greatly impacted me, professionally and personally. So many, it's impossible to thank everyone by name, but I feel compelled to share a few.

I had the three best managers any person in the world could possibly have. Infinite thanks to Paul Horvath, Tony Bacos, and Margot Johnson. They showed me what truly compassionate, empathetic, people-first leadership looks like. It turns out you can deeply care about the people you work with, be extremely respectful and nice to each other, and still get lots of stuff done!

Along the way, I worked with many fantastic PEs, Sr. PEs, and even DEs. But there were a handful that were always by my side supporting me. Thank you Jenn Lin, Porter Schermerhorn, Sean Blakey, and Simon Johnston. They each taught me so much more than they probably know.

And a hat tip to Nate Wiger who started it all when he called me up a little over five years ago and said, "hey dude I wrote a six pager and now Amazon's hiring in San Diego, want to check it out?" Without him, I wouldn't be here today.

How could any person be so fortunate to experience what I've experienced at Amazon? I still don't know the answer to that. Everyday at work I want to pinch myself and verbalize: why me??? Nevertheless, I'm so grateful for it all, and not just because it's almost Thanksgiving. I'm genuinely, deeply grateful, humbled, and forever appreciative.

To everyone I worked with at Amazon, from the bottom of my heart, thank you! ❤️

My approach to problem solving

Fri, 19 Nov 2021 01:36:52 GMT

The last few days, I’ve had the opportunity to help a few different people solve problems. Regardless of the problem I am up against, I use a few basic principles to solve it. I thought I’d share them here, in case they are useful to you.

Reduce your reproduction

Reproduce your problem, then reduce your reproduction. What do I mean by reduce? I mean, try to remove as much of the extra stuff around your problem as possible – this may be as simple as taking the time to run only the broken test case instead of all of them (or writing a new test case to exercise your bug).

2021–11–19: Pinephone Pro LCD panel

megi's PinePhone Development Log — Fri, 19 Nov 2021 01:00:00 GMT

Entropy of an ideal gas with coarse-graining

Bytepawn - Marton Trencseni — Fri, 19 Nov 2021 01:00:00 GMT

I show the first steps of how to arrive at a definition of entropy for a monatomic ideal gas modeled as hard billiard balls.

Our URGENT need for simplicity, summarized

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 18 Nov 2021 20:00:00 GMT

It’s high time to listen to Einstein, really.

Achieving 10 GB/s Scanning Rate in a Single Cloud VM

Tanel Poder Blog — Thu, 18 Nov 2021 19:52:00 GMT

You may remember my previous article about high-performance I/O for cloud databases, where I tested out the Silk Platform and got a single Azure VM to do over 5 GB/s (gigaBytes/s) of I/O for my large database query workload. The magic was in using iSCSI over the switched cloud compute network to multiple “data VMs” at the virtualized storage layer for high I/O throughput and scalability.

Well, the Silk folks are at it again and Chris Buckel (@flashdba) just sent me screenshots of similar test runs using the latest Azure “v5” instance as the single large database node and they achieved over 10 GB/s scanning rate for reads and 6 GB/s write rate with large I/Os!

Working in Castle Legacy

trv.rs — Thu, 18 Nov 2021 12:53:00 GMT

Certain terrain lends itself to human flourishing. Food, water, shelter… often next to a source of wealth or trade.

Because the terrain drives demand humans settle, destroy, and resettle in the same desirable locations… often over centuries.

If we look at the historical record, the prosperity of a good location is often marked by violence. There are still layers of burned ash and melted nails under Rome, and spearheads and arrowheads under most cities.

Conflict doesn't always bring progress. Sometimes a rival civilization makes an example of a city, or their own internal conflict causes them to fail. The ashes and wreckage too find their way into the earth.

But if the ground is good, someone will always start again.

This cycle can continue in desirable locations for centuries. As successive generations or civilizations build on top of the infrastructure left by the previous occupants, the future settlement is shaped both by the natural contours of the land and by the remaining previous infrastructure. The pile of dirt that contains all these layers of histories is called a “tell.”¹

Many great European cities contain such tells. Many owe their shape and character to medieval or even Roman architecture and decisions: Paris’s 14th century sewers², or Roman aqueducts³ and roads and bridges⁴.

The original terrain or the changes people made to that terrain drives any new construction. The new builders will take to jumpstart their progress. It's no surprise that the surviving bits of those old European cities that stay maintained and used are invisible infrastructure. Sometimes the original plan conflicts with our mental image of the construction in surprising ways.⁵ Other times you'll see the same triumphs and mistakes repeated time and time again. This makes sense… there are only a few places to build a port, defend a tower, or build a road.

Just like the physical construction a legacy software code case contains contains historical layers… where each successive attempt and architecture on the same business domain borrows abstractions and concepts from the previous iterations.

Digging through the digital "tell"s that I've worked on before, I've found these layers in:

Routers / URLs⁶
Namespaces
Business Rules (Permissions!)
Glue Code
Legacy APIs
Code Specific to Old Clients or Business Models

Thinking about software as layers of a city⁷ has helped me a lot. When working in a legacy software “tell,” I’ll find myself shifting my mindset from an engineer to that of an archaeologist. A digital archaeologist understands that the previous occupants of this terrain might have had similar goals, but a different language, culture, belief system that drove their decisions. They’ll take a human perspective to the remains they find, and use some forensic tools⁸ to try and build a picture of the story.

When working in a digital "tell", it is more important to learn to dig than to build. It's partly why I love vim: it's whole paradigm centers around the critical skill of editing what is already there rather than focusing on writing from scratch.

I've also carried this thinking into writing "new" code. What kind of decisions are going to be so ingrained into this "terrain" that I should really document for future generations.

Tell (archaeology) - wikipedia.org ↩
Paris sewer - wikipedia.org ↩
Aqua Virgo - Wikipedia.org ↩
Ponte di Tiberio (Rimini) ↩
My favorite example: NYC doesn't really have alleys in Manhattan because of the designed grid. Cortland Alley - 99percentinvisible.com ↩
I once read about a decision for Twitter URIs that slowed down response time… as the platform grew, they couldn't undo the change without breaking the value of the platform, so they were stuck with the decision. (Trying to find the source.) ↩
I think I have to credit Vernor Vinge's book A Deepness in the Sky for planting this image in my head. Thinking about this process in software development as building a continous stack over millenia… fascinating. Great book. ↩
Spelunking in Git - evantravers.com ↩

Hosting

Liza Shulyayeva — Thu, 18 Nov 2021 08:02:00 GMT

I’ve tried quite a few blogging platforms over the years: Wordpress, Squarespace, even Ghost at some point, I think.

I opted for self-hosting years ago, always on S3, first via Octopress and now (and for the last few years) Hugo.

useRainbow()

Rafał Pastuszak — Thu, 18 Nov 2021 02:00:00 GMT

Hi there! It's been a long time, I know. I took a break from work and started working on some small, personal projects (toys).

One of those small projects is potato.horse where I keep all of my doodles, visual short stories and jokes. Check it out!

However, this post is not about my break from work, other experiments, or the site itself. People seem to like one particular technique I used in the design, notably, the backround effect applied when the user browses the content:

Some asked me how this effect was implemented (including going as far as reading the minified code, which is very flattering).

So, here's a quick gist, followed up with some context:

export const useRainbowBg = () =>
    useEffect(() => {
        const cb = () => {
            const viewportHeight = window.innerHeight
            const contentHeight = document.body.getBoundingClientRect().height
            const viewportsPerRotation = Math.min(
                3,
                contentHeight / viewportHeight
            )
            const from = 51
            const progress =
                window.scrollY / (viewportHeight * viewportsPerRotation)
            const h = (from + 360 * progress) % 360

            document.body.style.backgroundColor = `hsl(${h}deg, 100%, 50%)`
        }
        window.addEventListener('scroll', cb, { passive: true })
        return () => window.removeEventListener('scroll', cb)
    })

In short, I map the scroll position into the hue in the HSL colour notation. Let's break this down.

Colour models

There are many ways of describing colours in CSS, with the two most common ones being RGB (left) and HSL (right):

RGB is an additive colour palette. This means that mixing 100% of red, green and blue will produce white, mixing 100% red and 100% green but 0% blue, will produce yellow and so on. This is different from, say, using oil paint or the CMYK colour model, where the resulting tone would be black(-ish)1.

We're used to this approach because it's easy to describe in code but specifying colours in terms of hue, saturation and luminosity seems more natural, especially if you come from a design background, or... you know, are a human being using a human language.

We've gotten used to RGB as developers, but in spoken language using it would feel unnatural and confusing. Façade would be very hard to use in RGB.

On the other hand, HSL can often be much more intuitive to work with. For instance if I want to make a colour slightly colder, I can just move the hue slider a bit towards blue and I should get closer to what I have in mind. With RGB if we make the colour appear colder by including more blue, the resulting tone will be a bit brighter as the blue component contributes to the overall lightness. This means that you'd have to lower the red and green values to compensate.

To see how this works in practice, try maxing out the blue colour in the example below.

The first thing that stands out is that all tones are shifted towards blue and the overall brightness of the picture is increased. In the case of the effect we're discussing, that would be undesirable.

Now, let's try to do the same with the HSL colour circle. Drag the slider to the left, by ca. 90 degrees:

In this scenario using HSL not only turns Susan into a vampire, but also maintains a similar2 level of brightness. And that's exactly what I'm looking for.

So, what did I mean by saying:

How does this work? In short, I map the scroll position into the hue in the HSL colour notation. Rafal, 2 days earlier

I meant that every time we detect a scroll event, I try to map it to an angle on the hue circle:

I didn't want to start with red as it would make me hungry and the base yellow fits the design a bit better, so I applied a small initial shift, hence const from = 51 set as the initial offset.

And, as far as the basic implementation goes, that's it!

Now, there are three other areas of improvement:

Performance

We're triggering a repaint on every scroll, so I was a bit worried that older mobile devices, or even some hi-end laptops plugged in to 4k screens might not be able to maintain solid 60fps. But, I'm happy with the results so far. Using passive event listeners provided a bit of a boost, especially on mobile.

When/if I realise that performance is a problem, especially with more content down the line, I'll probably focus on these areas first:

removing the unnecessary call to getBoundingClientRect on every scroll handler call
deferring or throttling background colour changes using requestAnimationFrame

I expect the first improvement to have some impact, but the benefits of the second one should be negligible.

Measure before optimising. Obsessing about the performance only makes sense when issues become noticeable, be it through a drop in framerate or battery impact. Your iPhone Pro has more computing power than many low-end laptops, so it's a good idea to test on those devices too. It's good to have a crappy old Android phone exacly for that purpose if you can spare a few quid.

Perceptually uniform colour spaces

You might've noticed that in the previous illustrations some fully saturated colours seemed darker than the others. That's because the colour spaces we use normally when coding don't reflect the way the human eye works. I'll leave the in-depth explanation to someone much more experienced than me, but suffice to say (gross oversimplification alert!) that generally the same amount of red/green/yellow will appear brighter than blue. This means that in some cases the text on the page will be harder to read.

HSL
⬇

⬆
HSLUV (perceptually uniform)

For now, this isn't an issue as I've just put this thing online and titles serve a secondary purpose. But there's a solution to the problem and it's not overly complicated: use a perceptually uniform colour space. There's a bunch of libraries that do it out of the box, both in JS/TS and CSS/SASS/<pick your CSS flavour here>. hsluv seems like a good starting point.

Accessibility

Note that I'll be focusing on the visual effect itself and not discussing the rest of the site (e.g. alt tags, document structure, etc...). I'd like to focus on contrast, colour blindness and people who rely on prefers-reduced-motion. The site is a living document, there's always so much to improve. For instance, contrast can be an issue in a few, non-critical places. I'm happy to accept feedback and implement it: hit me up!.

Colour blindness

I wanted to make sure that the effect doesn't break the site completely for people with colour blindness. So I focused on the most common types: Deuteranomaly and Protanomaly (red-green colour blindness), but also ran wider tests. I used Photoshop and Colorblindly (Chrome extension) for some rudimentary checks.

`prefers-reduced-motion`

The prefers-reduced-motion CSS media feature is used to detect if the user has requested that the system minimize the amount of non-essential motion it uses. MDN

This site doesn't contain many animations (besides the Little Sausage Angels you'll see if you hit 'Share'), but I was wondering if people who rely on prefers-reduced-motion would like the background colour to stay constant.

The short answer is: I don't know. My intuition is that rotating colours doesn't really qualify as motion, but my experience and understanding of the problem is, to say the least, limited. In situations like this I'd rather depend on user research instead of guesses.

Luckily, the site had its five minutes of fame on Reddit which proved to be a decent opportunity to collect feedback. None of the users brought up an issue with the background effect so far. I'm also lucky enough to know a bunch of accessibility specialists, such as Sandrina Pereira. Her suggestion was that a) background animations definitely qualify as motion, b) perhaps the effect feels natural because it's a direct result of a user interaction.

Summary

The late 90s/Geocities web felt playful and weird. It was fun in an uninhibited, somewhat less performative, way. I wanted to incorporate some of this look and feel in the site. But still, I didn't want to make it feel esoteric to the point where you'd need to up your hipsterdom-level to 9000 and browse it exclusively throught Netscape 7. All of that, while listening to the new Nirvana: Unplugged.

I still wanted decent UX on mobile and desktop, and some space for easter eggs (something you can't do when living in the strange and abusive relationship with social media we've grown so accustomed to).

As a kid I had build 6 websites before I even got access to the Internet for the first time. Now, after being burned out for 3 years, even considering changing my job, it was the first time I genuinely enjoyed coding. Fuck, I forgot how much fun it was.

Now, go out, pet your cat and make some shit!

PS. Check https://www.cameronsworld.net/

PPS. The code for interactive diagrams can be found here

Footnotes

hence the K component in CMYK meaning black. Using B would be confusing as it means blue in other colour models.
it's not perfect, since the perceptual colour space differs from what's described using RGB/HSL.

run-pr.sh

Max Leiter — Thu, 18 Nov 2021 02:00:00 GMT

Today I found myself once again trying to run a pull request that was created in a different GitHub repository. If it was in the same repo, it would be a simple `git checkout`, however because GitHub insisted on the creation of "forks" I now need to go through the extra steps of fetching that specific ref. Here's what I do:

git fetch https://github.com/${team}/${respository}.git refs/pull/${prId}/head
git checkout FETCH_HEAD
git rebase master # or main

Just replace `team`, `repository` and `prId` and you're all set.

I picked this script up from The Lounge: https://github.com/thelounge/thelounge/blob/master/scripts/run-pr.sh. It's a more complete example if you want to start using a script like this. Notice the \` and bash substitution (\`) at the end of it — these allow you to pass in command-line arguments to `yarn start` (or whatever your run script is).

Individuals Matter

Dan Stroot — Thu, 18 Nov 2021 02:00:00 GMT

One of the most common mistakes I see is thinking of people as interchangeable, as if “who” doesn’t matter. Companies will plan out the next few quarters of work and then assign some number of people to it, one person for one quarter to a project, two people for three quarters to another, etc. The roadmap creation and review process perpetuates the polite fiction that people are interchangeable, that people can be generically abstracted over.

This post Individuals Matter first appeared on Dan Stroot's Blog

The epic notability battle between pornstars and scientists

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 17 Nov 2021 18:00:00 GMT

I wonder if it still going on…

Improving the code from the official Go RESTful API tutorial

Ben Hoyt's technical writing — Tue, 16 Nov 2021 20:00:00 GMT

My re-implementation of the code from the official Go tutorial 'Developing a RESTful API with Go and Gin', using only the standard library, adding tests, and fixing issues.

Watching the World Bank must become easier

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 16 Nov 2021 19:00:00 GMT

Especially because now it IS much easier.

Funding Not Secured

Nothing Easy About This — Tue, 16 Nov 2021 16:00:00 GMT

I can't fund an acquisition. This post outlines why this hasn't worked, and shares an investment model that I think someone could take up.

VPN Advice (November 2021 Edition)

IvyMike.dev — Tue, 16 Nov 2021 10:00:00 GMT

I had a short discussion with a friend about his VPN needs; here's a summary of that discussion.

Executive summary:

Most users don't need a VPN for privacy anymore; the majority of your web traffic is encrypted by default.
If you need a VPN so you can access servers at …

Learn in public

Nicola Iarocci — Tue, 16 Nov 2021 08:05:25 GMT

Today I searched the internet for something, and the first result I got from @duckduckgo was a note I wrote months ago to my future self; how meta is that.

Learn in public, it gives superpowers¹. Also, in recent years, adopting POSSE was the best thing I did for my personal development.

I should do better. Post more TILs, for example. [rss]: https://nicolaiarocci.com/index.xml [tw]: http://twitter.com/nicolaiarocci [nl]: https://buttondown.email/nicolaiarocci ↩︎

Skills, stories, and software every dev should know - 123dev #46

Justin Garrison's Homepage — Tue, 16 Nov 2021 02:00:00 GMT

Comments

Skill

Some developers are very skilled. They can do things seemingly impossible for others. It might be developing a feature faster than others, debugging a hard problem, or reaching new scale with systems.

Many times those skills are accompanied by a decent amount of luck. While a skilled developer can replicate something with high accuracy, it’s not always skill alone that helped the first time.

2021–11–16: A bunch of mostly Pinephone Pro news

megi's PinePhone Development Log — Tue, 16 Nov 2021 01:00:00 GMT

Engagement Insider Threat

ℤ→ℤ — Mon, 15 Nov 2021 20:38:16 GMT

Warning: Contains spoilers for the season 6, episode 5 “Tethics” of the show Silicon Valley which aired in 2019.

In “Tethics”, two managers of Pied Piper, Gilfoyle and Monica, are threatened by HR due to their low employee engagement scores — there’s a consensus among the employees that the two managers are jerks. They both promise to raise their scores, but stupidly promise to raise the scores from “hate” to “love” within a week. Faced with this impossible task, they hatch a plan to use social engineering to steal employee’s passwords and then, using the compromised employee’s accounts, modify the engagement scores to show that employees now love them. How realistic is this plan? How can we mitigate these kinds of attacks? Let’s use a threat modeling approach to answer both.

More formally, the attacker’s objective is to modify the engagement data to benefit themselves. As managers within the company, they will know:

what system is being used to collect and analyze the data (we will assume this is commercial, non-self hosted SaaS software),
who (in their company) would be administering the project,
who would be providing engagement scores (the respondents), and
the high-level project plan and process.

The data needs to be modified surreptitiously; if the engagement data is deleted or is exposed as doctored, the attackers may have gained a short reprieve but haven’t accomplished their goal of appearing as better managers. In addition, we will assume the attackers do not have advanced technical skills to attack the SaaS software directly¹ and the time budget is closer to days than months.

The proposed attack is two-stage:

Obtain employee passwords by using social engineering/knowledge of the employee’s backgrounds to guess passwords
Leveraging those accounts, change the employee’s ratings for the managers in the engagement monitoring system

In the STRIDE threat model², this is a Spoofing threat (obtaining account access) that can then allow a Tampering threat (modifying engagement score data). The asset to be protected in the first stage are passwords and, in the second stage, the employee ratings. Spoofing threats are mitigated via authentication and tampering threats are mitigated via ensuring data integrity.

(If you find this kind of analysis interesting, I strongly recommend Adam Shostack Threat Modeling³ book. Also, Ross Anderson’s Security Engineering⁴ covers a broader systems-approach to security; the two books are complementary.)

Spoofing mitigations

While the employee engagement responses may be anonymized or otherwise have some sort of privacy protections, the actual act of providing feedback is not anonymous. Instead, the collection system requires some ability to know who the respondent is in order to tie the responses to the respondent’s place in the organization. (Engagement studies often include questions specific to certain roles or organizations and results need to be aggregated under the appropriate manager.) Thus, the collection process requires respondent authentication. Within the episode, the authentication is handled via passwords, so that is where we’ll focus our model.

Password policy/recovery policy

Using personal information to aid in guessing a user’s password is not a novel attack; it’s a standard trope in fiction but one that still works in the real world. As a straight-forward attack, it relies on users choosing poor passwords and on the systems being vulnerable to guessing attacks. Better passwords can be encouraged via the use of password managers (which reduces the pain of a unique password per site), single sign-on (which increases the value of a strong password), or policies that encourage higher entropy passwords (although onerous policies can encourage users to use their creativity to perversely choose worse passwords).

Additionally, if the system locks out (even temporarily) an account after a number of bad logins, this will slow the rate an attacker can guess passwords. Within this scenario, the managers do not have a large amount of time to guess passwords and there is no indication they can guess passwords “off-line”, so a good authentication system can throttle their attack.

An alternative interpretation of their attack is they are not attacking passwords directly, but rather using personal details to attack the password recovery mechanism. Password recovery mechanisms typically require the user to answer a handful of personal questions and, if they answer correctly, the system will initiate a password change. Recovery questions often suffer from a low number of possible answers and answers can often be sourced with some public background checks. Good password recovery systems will notify the user (ideally through multiple channels) about the password change, so this can cause the attack to become known if the users see the notifications and decide to flag them as improper. However, not all recovery systems include these features, so the attackers could maintain their cover.

Two-factor authentication

The three-factors are something you know, something you have, and something you are. Two-factor authentication requires two of these factors, normally a password and a key (often via a multi-factor authentication app on a phone or a physical device). If providing an engagement score requires two-factor authentication, then the cost of spoofing dramatically increases because a password, by itself, is not useful. However, this also increases the cost to the respondent (and the complexity of the collection) since the respondent will need to go through a longer and more error-prone authentication process. If this creates too much hassle, there will be a decrease in the response rate which may invalidate the utility of the engagement project.

A single sign-on (SSO) system, which may require an onerous authentication process but will re-use the authentication across multiple systems, can spread the pain sufficiently to mitigate the friction concerns above. However, SSO is more complex to setup and not all vendors support it.

Alternative authentication approaches

Could we mitigate the attack by not using passwords at all? In The quest to replace passwords⁵, the authors review three dozen approaches against two dozen criteria and find that none of the proposed alternatives to passwords is in-arguably superior across usability, deployability, and security dimensions. There are trade-offs in all approaches and all examined approaches have at least one feature that is “worse” than passwords.

For example, take the “One Time Password over Email” approach⁶. In this approach, users submit their email address to a server which persists one-half of a split token as a cookie and sends the other half to the user’s email. The user supplies the email token and the server combines and verifies the two tokens. Versus passwords, this scheme has the advantages that the user does not need to remember any long-term tokens, is resilient to someone watching the user type, is resilient to both throttled and unthrottled guessing, resilient against release of information from other parties, and resilient to phishing. However, this scheme has the disadvantages of being inefficient for use ⁷, is incompatible with existing password-based systems, is immature versus other schemes, requires trusting a third-party (email system and intermediaries), and allows linking of accounts between disparate systems.

For the employee engagement application, the fact that emails are required is likely a non-issue due to the system likely knowing quite a bit more about the employee; it’s already a PII-sensitive application. Adding friction to the login process is non-ideal as the project needs to maintain response rates, although employee engagement projects tend to have motivated respondents. The immaturity of the scheme may be acceptable for the respondent use case. Respondents don’t need typical user accounts as they are only supplying data to the system. However, using a novel system will require more development effort to implement, may incur a larger user training cost, and will have a higher cost in security audits.

Tampering mitigations

Tampering attacks involved modifying the asset under protection, in this case, the response data. (Tampering could also involve modifying software or hardware within the data flow, but those kinds of attacks would be beyond this kind of attacker.) Maintaining the integrity of the data relies on either preventing improper modification or having a path to restore the proper data after an attack.

Enforce deadline on updates

In a standard employee engagement study, there will be a collection period followed by an analysis period where the project teams assembles and disseminates the reports. If updates to responses are restricted to the collection period, this reduces the window of opportunity for attacks.

Measure trends rather than moments

In the episode, they only need to have good scores on the next review date. This means that actions will be taken on anomalous readings; the attackers only need to adjust scores for one review. If the project instead looked at trends over time, the attackers would need to adjust scores over multiple phases, which would greatly increase the chance that their attack would be noticed.

Report back to the respondents on updates

If changes in data are notified to multiple systems, it can make tampering more apparent. However, notifications and logs are only useful if people look at them as there is a cost to monitoring, particularly if updates are expected in the normal course of business (i.e. false positive rate is significant). One way to counter the economic cost is to notify the respondent, as their volume of notifications will be small. Any additional notifications seen by them will likely be treated as suspicious. Additionally, some respondents will like a copy for their own records, making a security feature a user feature as well. Sending the respondent an email copy of whatever is submitted is an easy mitigation to improve detection of tampering. However, this approach also allows a new vector of information disclosure, since attackers can now access the response data from other data sources. An employee’s company email account is generally more accessible to management and IT staff than an engagement response database.

Use robust aggregate statistics

The attacker’s goal is to increase their engagement score, which is an aggregate of individual scores. Which aggregate is used can greatly influence how the attack must be performed. If individual responses include a score, typically aggregates include mean and median. Alternatively, the program may use a system like Net Promoter Score® (where responses are classified into the three categories of promoter, detractor, and passive and the derived score is the delta of the percentage of promoters versus percentage of detractors) or some linear regression model. The choice of the aggregate determines how robust the calculation is towards outliers and the more robust the statistic, the more expensive the attack. (The use of “outlier” here is informal; the SaaS software presumably restricts value to a valid range, so someone can’t break the calculation by setting an individual’s score to a million.)

For example, let’s say the score is based on the average (arithmetic mean) of individual employee’s scores. The sample includes 15 employees who have given the scores {1,1,1,2,2,3,3,3,3,3,4,4,5,5,6}. The mean of these scores is 3.1 and the median is 3. The attackers have a good idea of who will be giving the worst scores. By changing the scores to {10,10,10,2,2,10,3,3,3,3,10,4,5,5,6}, they’ve managed to improve the mean to 5.7 and the median to 5. As the mean is less robust, the attackers can improve it (here by 2.6) more easily by modifying one-third of the scores. The median is more robust and has only increased by two.

If clamping is not enforced, then the attackers just need to modify one score to allow them to set the mean score to any arbitrary value. To modify the median to an arbitrary value, they would need to modify one half of all scores.

Retain historical data (and report on it)

There are two vectors for a data change: an updated response from the respondent and an administrative edit. Particularly for this kind of study, respondents may naturally update their responses if they remember some new detail or tempers cool from a rash response. Thus, collection systems often allow updates to a response, even after the respondent has hit ‘submit’. On the administrative side, project administrators may edit the data to correct errors introduced in the project setup, may perform some data clean-up such as re-scaling values, or may edit the data for broader release⁷ or upon request by the respondent. Reports and analytics resulting from the project should reflect the “latest” data and incorporate all edits.

For both vectors the appropriate security feature that the SaaS developers and project administrators should maintain is “non-repudiation”. This feature means that someone can say “I didn’t do that” and the system can prove them wrong (or at least evidence that they did). In bank applications, this means someone can’t withdraw some money and then protest they didn’t do the withdrawal and should get the money back; the bank’s records will identify who performed the withdrawal, where it was done, and likely include photographic evidence of the act.

Non-repudiation requires both data to be logged or persisted appropriately but the details of the changes to be available and evident to the project administrators. On the persistence side, developers will often use an immutable data design or event log to provide that audit trail, with some sort of “latest” snapshot to make analytical processing efficient. On the presentation side, this can be a challenge since there is often a “natural” amount of updates within a project that can obscure nefarious activity. Highlighting altered data within a data grid or showing the number of versions of each row are two straight-forward ways of exposing this data, as long as the volume of changes is small. In the case of a high volume of change, there is an opportunity for a system to provide anomaly or fraud detection to help administrators identify odd patterns.

“Next Best Alternative” Threats

Attackers are rational and will attempt to use the least risky and least expensive attacks available to them. Defenders, faced with their own resource limitations, should invest to raise the costs of attacks beyond the attack’s acceptable threshold, but not over-invest such that the investment exceeds the risk. If the above mitigations are successful against the manager’s attack, what would the managers do instead?

First, instead of attacking respondent’s accounts, they could instead attack a project administrator’s account. Using the admin account, they could then edit the data or delete responses. Administrator accounts are more likely to include additional authentication protections than respondent’s accounts, so this approach is more risky, although once the account is accessed they have greater power to accomplish their goals. Protecting the data against administrative modifications have similar strategies as in “retain historical data”.

Secondly, they could try to discredit the study or process itself. This is not a typical “security” attack, but if sufficient doubt is placed on the accuracy of the study, then the engagement scores will be ignored or treated with little weight. In the context of the episode, though, they are well-known for being anti-social, so they would need to argue that although the results match expectations, they are somehow invalid. Within the episode, the attackers use an alternative form of this attack by making the HR lead’s engagement scores terrible. The HR lead then cancels the study.

Thirdly, an alternative form of discrediting could be accomplished through a denial of service attack. If the software becomes unavailable, the schedule for the study can be disrupted or random failures could shed doubt on the SaaS vendor’s ability to collect all the relevant data or present the complete picture. While a denial of service attack launched directly against the SaaS vendor would be very public and risky, a company’s network can be configured (perhaps through http proxies) to randomly drop packets or requests to the vendor’s sites, thus making it seem like the symptoms of overloaded or unreliable software. Based on our profile of the attackers, this would be outside their ability unless they are part of an IT group or could gain support from someone managing the network.

Conclusion

By incorporating standard security controls and incorporating data integrity controls into their software, the cost and difficulty of the attack can be increased beyond the attacker’s abilities. Since the Pied Piper company is run so shabbily, it is believable that their in-house engagement software could suffer from such an attack, but since these styles of attacks are well-studied and have off-the-shelf mitigations, they should not work against a professional SaaS vendor’s solution.

This is in contrast to the show, in which one of the managers, Gilfoyle, is an extremely skilled engineer. The episode does not explain why Gilfoyle does not attempt to change the data within the database as the episode implies their collection systems are in-house rather than hosted by a vendor. ↩︎
Shostack, Adam. Threat modeling: Designing for security. John Wiley & Sons, 2014. ↩︎
ibid. ↩︎
Anderson, Ross. Security engineering: a guide to building dependable distributed systems. John Wiley & Sons, 2020. ↩︎
Bonneau, Joseph, et al. “The quest to replace passwords: A framework for comparative evaluation of web authentication schemes.” 2012 IEEE Symposium on Security and Privacy. IEEE, 2012. pdf ↩︎
Van Der Horst, Timothy W., and Kent E. Seamons. “Simple authentication for the web.” 2007 Third International Conference on Security and Privacy in Communications Networks and the Workshops-SecureComm 2007. IEEE, 2007. pdf ↩︎
Every time a user attempts to login, they will need to wait for an email (or some similar transmission). ↩︎

Of course, even NATO is doing Artificial Intelligence

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 15 Nov 2021 19:00:00 GMT

Trouble is, with whom?

Interactive Gradient Descent Demo

Sasha Kuznetsov's Blog — Mon, 15 Nov 2021 10:00:00 GMT

What is this special algorithm?

Gradient descent is an optimization algorithm for finding the (local) minimum of a function.

Why is that useful?

Say you have some function $f(x,y)$. This function may represent a cost to you, and $x$ and $y$ are some inputs that vary the cost. Naturally, you are going to try to find the values of $x$ and $y$ that minimize the cost. This is a problem of finding the minimum of $f(x,y)$. For example, we may want to find the minimum of this function, which is plotted below:

$$ \begin{align} f(x,y)=3(1 - x)^2 e^{- x^2 - (y + 1)^2} - 10(\frac{x}{5} - x^3 - y^5) e^{-x^2 - y^2} - \frac{1}{3} e^{-(x+1)^2 - y^2} \tag{0} \label{eq:cost_func} \end{align} $$

This is where gradient descent can help.

Why gradient descent?

From elementary calculus, you may remember a procedure for finding the minimum of a function. You could find the first derivative, set it to zero, and solve that equation:

$$ \begin{align} f'(x,y)=0 \tag{1} \label{eq:first_deriv} \end{align} $$

This will reveal the locations of the extrema (maxima and minima) of the function. The sign of the second derivative $f''(x,y)$ can reveal whether the extrema are local maxima or minima. However, solving the first derivative equation $\eqref{eq:first_deriv}$ can be non-trivial, and often require numeric methods. And as your cost function involves higher and higher dimensions, e.g. $f(x,y,z,w,q,u,v,...)$ the problem can become unmanageable. Gradient descent provides an efficient procedure for locating minima by utilizing the first derivative in a clever way.

The gradient descent algorithm

The motivation behind gradient descent is intuitive, and I bet you would arrive at the same procedure in an analogous physical situation. Imagine you find yourself in a thick fog, and you can only see a few feet in front of you and feel the slope of the ground beneath you. You are trying to get to the bottom of a valley as quickly as possible. Surely, you won’t waste your time walking along a flat slope - or worse, head uphill. To descend quickly, you will inspect the slope at your location, and proceed in the direction where the slope is steepest downwards. Gradient is a synonym for slope, hence, you would be performing gradient descent. Conversely, if you were trying to find the top of the hill, you would follow the steepest part of the surface up, wherever you are. This is gradient ascent.

Gradient

Gradient is essentially another word for slope. Most people will be familiar with finding a slope in a 1-dimensional situation. In 1-D, since one can only proceed either forward or backward, the slope is clearly defined. However, in higher dimensions - imagine you find yourself on a hike - the slope may have a different value depending on which direction you go. How does one define the slope at this point?

In vector calculus, one uses the gradient operator $\nabla$, which is a vector of the slopes (or first derivatives) along each dimension.

$$ \begin{align} \nabla f(x,y)= \begin{bmatrix} \frac{\partial f}{dx} (x,y) \\ \frac{\partial f}{dy} (x,y) \end{bmatrix} \tag{2} \label{eq:gradient} \end{align} $$

On a 2D surface defined by $f(x,y)$, this would be the slope along the x-axis and the slope along the y-axis. The slope in any particular direction (which is important, since we are not constrained to move only along either the x-axis or the y-axis) can be found by summing the corresponding x and y components of the slope in that direction. In other words, the dot product of a direction vector with the gradient produces the slope in that direction.

Another interesting property of the gradient is that it gives the direction of the steepest slope. One can see this mathematically - since the slope in any particular direction can be found by dotting that direction vector with the gradient, the dot product is maximized when the gradient is dotted with itself!

Below, I plot the negative gradient of $f(x,y)$ with Mathematica. This gives us the direction of steepest descent at every point on $f$.

Gradient of f plotted as a vector field.

Follow the steepest part down

Earlier we agreed that if you were blindly navigating down a valley, you would follow the steepest direction of the ground down if you wanted to get to the bottom as quickly as possible. Mathematically, you are traversing down the valley along the gradient at each point, since the gradient vector at each point is the direction of the steepest descent. Looking at the image above, we can see that means following the arrows will lead us to a minimum (but be wary - each minimum has its own basin of attraction! More on that later).

Thus we can define a procedure. It may go something like this:

Inspect the slope at your position and determine the direction of steepest descent
Walk 5 steps in that direction
If you are still descending, go to 1. Otherwise, you’re at the bottom!

You may be wondering how I arrived at 5 steps. Well, it was totally arbitrary. But we can let it be a parameter of our procedure. Let’s call it a descent rate, and assign it the symbol $\alpha$. The larger the descent rate, the less often we’ll have to inspect the slope around us. Computationally, this is more efficient - but as you’ll see, this can lead to pitfalls.

Furthermore, we can make another optimization. When the local slope is still steep, we are probably far from the bottom of the valley. But as the slope gets flatter, we can figure that we are probably approaching the bottom. Hence we may want to descend at a rate proportional to the gradient. Let’s define the gradient with the symbol $\bar{g}$, and redefine our gradient descent procedure:

Inspect the slope at your position and determine the direction of steepest descent
Walk $\alpha \times \|\bar{g}\|$ steps in that direction
If you are still descending, go to 1. Otherwise, you’re at the bottom!

Next, how do we determine if we are still descending? One approach is to check how much we’ve descended on every iteration. If we’re barely changing altitude, then we’re probably essentially at the bottom already.

Mathematically, our procedure looks like:

$$ \begin{align} \bar{p}_{n+1} = \bar{p}_n - \alpha \nabla f(x,y) \tag{3} \label{eq:grad_descent} \end{align} $$

where $p_n$ is our position at the $n^{th}$ iteration of our descent. Try it out on the demo below. Varying the rate $\alpha$ varies the rate of the descent, while varying $X$ and $Y$ vary the initial starting point on the surface. What do you notice?

X 0

Y 0

α 0

Observations

You may notice that gradient descent is sensitive to the initial position. It will pull you towards a local minimum, which may not be the global minimum. The area of the domain that causes gradient descent to converge to a particular minimum is known as that minima’s basin of attraction.

You may also notice that increasing the descent rate $\alpha$ reduces the number of iterations required to arrive at the minimum. But making it too large causes the algorithm to overshoot, and possibly diverge.

Machine learning

You’ve probably heard of gradient descent in the context of machine learning. In that context, the cost function represents the error of our machine learning model. For example, it may be the error of classifying images, and the inputs to the cost function are the weights of the model we are varying. Hence, we can use gradient descent to find the optimal configuration of weights that minimize the classification error of the neural network! In this field, the descent rate $\alpha$ has a special name: it is known as the learning rate. Maybe it’s what our brains have been doing all along.

Develop Google scraper in Python with Scraper API

Adnan's Random bytes — Mon, 15 Nov 2021 09:42:23 GMT

The post Develop Google scraper in Python with Scraper API first appeared on Adnan's Random bytes.

Ankibutton

voussoir.net/writing — Mon, 15 Nov 2021 02:55:54 GMT

Back to writing

[3dprinting] [electronics]

Ankibutton

Glamour shots
Introduction
The problem
False starts
True starts
The fourth button
Conclusion
Download
Appendix: Installing Seeeduino Xiao into the Arduino IDE

Glamour shots (§)

Introduction (§)

One of the most effective ways to procrastinate on a task is to spend time working on something else that will supposedly help you do the original task more efficiently, but in actuality will never recoup the invested time. You can make a new tool, or design a new process, or rearrange your work environment, just as long as you spend more time on it than it could ever possibly pay you back. Whatever you do, never consult this chart. This technique will ensure that you come out behind on your goals while giving you the defense of having worked hard the entire time.

In this article, I'll show you a new tool I made.

The problem (§)

Over the past three years, I've done almost 100% of my Anki studying on my smartphone, simply because I don't enjoy the ergonomics of using Anki on the computer. I don't like having to sit with my hands on the keyboard for a long time, pressing the spacebar. I'd rather be able to lean back or sit farther away.

At the same time, the ergonomics of the smartphone are not ideal either. The hands are most comfortable at the lap but this is bad for the neck. Researching words and copy-pasting unicode is slow and cumbersome on mobile.

I thought it might be nice to have a handheld controller that I can use with my computer from a distance. I already have an Xbox controller and joytokey, but if I used those things then I'd be done procrastinating already.

So there I was, with no choice but to make my own.

False starts (§)

When I assembled my keyboard I used a microcontroller that cost $18 plus $5 shipping, which is more than I wanted to spend on a silly thing like the ankibutton. I happened to have an old rubber-dome style keyboard that was broken and useless, so I thought I'd take the controller out of that and wire up some of my leftover keyboard switches onto it. By following the sheets of electrical contacts you can learn which pins you'd need to solder switches to to make a keypress. I never did solder anything, but I used a piece of wire to connect some pins and figure out where the keys I want are.

I designed a case for this controller and three cherry switches.

I decided not to use that controller for two reasons. Mainly because it's not possible to reprogram the firmware on it, so if I ever got tired of using it as an ankibutton I'd be unable to change any of the keys without unsoldering and resoldering it in a new configuration. Secondly because I want to include a Ctrl+Z button on the ankibutton, and I don't think soldering a single switch onto the pads for both Ctrl and Z would work properly.

True starts (§)

I set this idea aside for a few months until one day I got an idea for a different project and started researching for other microcontrollers. I discovered the Seeeduino Xiao, and was able to get a two-pack for $14. Seven dollars for an arduino-compatible board with USB-C is a pretty good price.

I also decided not to use the gateron keyboard switches, because for a handheld device these switches were just too big and would require lots of thumb movement during normal use. Instead, I harvested some small pushbuttons from a plate that had come out of a fax machine.

I redesigned the enclosure around the Seeeduino and 6mm pushbuttons.

I used superglue to hold the buttons in place, and double-sided tape to fix the controller in its recess. I used doubled-sided tape to attach a piece of felt on the underside of the lid, so that when the lid is firmly shut it provides more support to the controller.

The wiring was straightforward, though I'm still not very good at soldering. The idea is the same as a full-sized keyboard matrix. One wire comes out of the controller and visits all four switches, providing what I am calling the "outbound" signal. Then, the other leg of each switch is wired to a separate pin on the controller so that there are four distinct "inbound" signals. The controller simulates a keypress whenever an inbound signal is received.

I was surprised to find that the Seeeduino does not support QMK firmware like I used on my keyboard. Or at least, I was not able to make QMK Toolbox discover the device even after switching the MCU type to atmega328p. I had to write the firmware myself as Arduino code which took me a while but was overall not too difficult.

Programming for hardware is slower than programming computer software. Normally when I write a program that crashes I can just read the traceback, but in hardware there are no tracebacks. It just stops working. And I have to keep bisecting the code until I find the part that does the least amount of wrongness, each time using a piece of wire to short RESET to GROUND to go back to bootloader mode because regular mode crashes before Arduino can reprogram it.

Even though the first several prints I made were able to snap shut with a bit of friction, for some reason the final one didn't hold itself together very well. But I was tired of printing it so I wrapped it with some blue tape instead of doing it again.

The fourth button (§)

My original plan was for three buttons: Spacebar (right answer), 1 (wrong answer), and Ctrl+Z (go back). After switching to the little buttons it made more sense to have four. It would be nice to map it to Anki's "easy" answer, but that's not straightforward since easy is sometimes 3 and sometimes 4. I am not quite sure what I'll do with it yet.

Conclusion (§)

The finished product functions very well. With a long USB cable I can lean back and review my flashcards comfortably. I added code to debounce the button presses and they seem reliable.

Having said that, I will probably use the ankibutton a few times and then forget about it. The burden of plugging it in, or worse, leaving it plugged in and cluttering my desk space, is simply higher than the value it provides. Mission accomplished!

Download (§)

ankibutton.scad

ankibutton.ino

Appendix: Installing Seeeduino Xiao into the Arduino IDE (§)

Open File > Preferences, and into the "Additional Boards Manager URLs", paste this: https://files.seeedstudio.com/arduino/package_seeeduino_boards_index.json.
Open Tools > Board > Boards Manager, and download "Seeed SAMD Boards", which should appear in search results for "xiao".
Select Tools > Board > Seeed SAMD > Seeeduino Xiao.

When the Seeeduino is in normal operating mode, it will occupy one COM port. If you use a piece of wire to short RESET to GROUND twice in a row, it will enter bootloader mode and occupy a different COM port. During bootloader mode it will also appear in Windows as an external storage device.

When uploading the code from Arduino IDE to the board, it will automatically go from normal mode to bootloader mode and back again. But if you need to reset it manually due to a crash, you need to go back to Tools > Ports and pick the bootloader port before uploading again.

I found that uploading the code would often make my computer reset all of its USB connections, including my external hard drives, which was problematic for me. I did the remainder of the programming on a separate laptop to avoid this.

At first, my laptop was not able to recognize the device at all. I had to download seeed_usb_serial.inf, and then tell Device Manager to install a driver from that file. When Device Manager asks for a driver location it only allows you to select folders, not files, so just select the folder into which you downloaded the inf file.

After these steps I did not encounter any more issues with programming the controller.

View this document's history

2021-11-14 Add ankibutton.md.

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Shortn

june.kim — Mon, 15 Nov 2021 02:00:00 GMT

Salesloft Pause Cadences

Sam Solomon — Mon, 15 Nov 2021 02:00:00 GMT

Cadences are at the core of what makes Salesloft such a fantastic tool. It is a workflow that groups like tasks together to reduce context switching.

Installing a fast Linux server on an M1-based Mac

Jason Eckert's Website and Blog — Mon, 15 Nov 2021 02:00:00 GMT

One of the biggest worries that IT users have regarding Apple’s new ARM-based M1 platform is the ability to run virtual machines (VMs) of other operating systems using hypervisor software. And while it’s possible to run virtual machines of both Windows and Linux operating systems made for the Intel platform on the M1, the performance cost of translating Intel instructions to ARM is high (even with the optimizations Apple has within their M1 CPU). Luckily, the most common VM that an IT systems administrator or developer will install is Linux, and ARM versions of Linux have been mainstream for well over a decade now.

A guy at work asked, where's your blog?

swaits.com — Mon, 15 Nov 2021 02:00:00 GMT

✅ here's my blog

A guy at work said, "where's your blog?" So, here's a quickly hacked together site using zola with the lightspeed theme. It's hosted in a private repo on sourcehut and, at least for now, built and deployed to sourcehut pages.

I blogged many years ago. Like, decades ago. Don't know what I might put here, but I'll give it a shot!

Games Not Worth Playing: Limited Resources

D13V — Mon, 15 Nov 2021 01:44:14 GMT

He who has learned to disagree without being disagreeable has discovered the most valuable secret of negotiation.

Chris Voss, Never Split the Difference

This is the fourth post in a series. You can find the last three here: 1, 2, 3. There we set the basics of opting out of negative games, and now we can start diving into examples.

A classic example of a negative game is the adversarial distribution of limited resources. In such situations, you might be fighting with others over the same set of stuff, whether it is the proverbial pie, or that promotion which only one person will get, the custody of a child, or even who gets to speak in a conversation.

One way to exit such game, is to change the vibe from competition to cooperation, to increase these limited resources.

For example, if we consider a divorcing couple which fights over the custody of their child, we can see that what they’re really fighting for is the satisfaction of their own egos, and not the best outcome for the child. By treating that child as a limited resource to fight over, instead of a person, which both want to support, and instead of discussing how to make the transition less traumatizing for the kid, they end up creating more trauma for the child, as the child will lose one parent. However, if the parents care more about their children’s future than about their own egos, they might find an arrangement where the kids experience more love from both parents, and grow up to be more fulfilled. The parents can opt out of bickering by realizing they still have a shared vested interest in their kid’s wellbeing, and change the “vibe” of the conversation.

More generally, in any conversation you may try to convince the others of something, and think that the way to do it is to spend more time talking. You might see the time during the conversation as a limited resource and try to capture most of that time by speaking more, and speaking louder than others. And doing so, you might make it harder for others to express themselves, creating a competitive argument. A shouting match of loudly saying “Me, me, me!” As a result, you’re not going to be effective at convincing, and will not even contribute to information transfer. When you’re trying to talk, you’re not trying to listen. If you instead try to play the game of increasing information transfer, regardless of the direction, you stand a better chance of persuading others, and you are also allowing yourself to have your worldview improved.

In the last example today, we look at the Red Queen’s race. A company entering a crowded market may be fighting for market share with other companies, selling to the same customers. That company may even be successful if the incumbent are not providing a good service. And if these companies are fighting over the same market share, they end up both cutting costs and price, and undermining each other’s profit. A way out for a company is to figure out how to grow the market and capture that new growth, by serving an underserved population or use case which wasn’t in the market previously. Focusing on beating the competition is zero-sum thinking, but focusing on the customer is a win-win thinking.

Next week, we will discuss artificial scarcity.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Backups

Home on mfashby.net — Sun, 14 Nov 2021 13:12:44 GMT

My ASP.NET 5 migration to .NET 6

Nicola Iarocci — Sun, 14 Nov 2021 08:05:25 GMT

I spent the last few days migrating our ASP.NET REST services, MVC web applications and Blazor server apps to .NET 6. Overall the process was pretty straightforward. The few issues I went through were easy to solve and well documented. Things got more involved with the EF Core 6 transition, especially with the Npgsql Entity Framework Core Provider.

The official ASP.NET Core 5.0 to 6.0 migration guide was my first stop. It offers the perfect entry point, rich with in-depth links. At this stage, I am not interested in switching to the new .NET 6 minimal hosting model (aka Minimal APIs). I think it’s a significant improvement, and we will likely adopt it for new projects, but our production projects aren’t going to be refactored right away. Should minimal APIs also prove to be remarkably performant, we’ll reconsider them¹.

Keep secrets in python

Yasha Solutions — Sun, 14 Nov 2021 01:00:00 GMT

Keeping Secrets Secret in Python Based on an article by originally published by Jason Goldberger but unfortunately the original is not online anymore so I have rewritten a short version, since this is something I have been looking for a while. Fernet has one classmethod called generate_key() and two instance methods encrypt(plaintext_binary) and decrypt(cipher_binary). Step 1 : Generate a key Generate a key and save it to the OS’s environment variables:

Three Editor Use Cases

duncanlock.net — Sat, 13 Nov 2021 21:14:33 GMT

I have three different text editors, that I use for three distinct use cases.

I’m a software developer, so I basically edit text files for a living, and the editing software that I use for this is fairly important to me.

Projects

For any substantial editing, either for work, or for personal projects, I currently use Visual Studio Code - generally known as “vscode”. It’s good enough: it’s extremely actively developed, so always up to date, and it has all the plug-ins you could ever want. Performance is good enough, once it’s started up.

I tend to just start my project editor once and leave it open permanently.

I’m still aggrieved that Microsoft killed the Atom editor when they bought GitHub, despite promising that they wouldn’t. It was a better editor - better UX, Tree-sitter, etc…

Occasional or One-Off Editing

If I’m just doing a quick edit of a config file or something - I generally don’t use the project editor. It’s too heavy, too slow to start and will remember all the previous open stuff - which I want in project mode, but not for one-off editing.

Because I have less baggage and attachment to this editor, this is where I experiment with new editors.

I have used Mousepad, CudaText, Lite XL - but currently mostly using Lapce. Really fast startup, cross-platform, tree-sitter, good enough.

CLI Editing

I mostly use a desktop GUI, but always have terminals open all over the place and often do some editing there. I’ve never learned to sing the song of vi - and I don’t feel the need, at this point. I use the excellent micro editor for all my terminal editing needs. I have this set in my .bashrc:

export EDITOR=micro
export MICRO_TRUECOLOR=1

Micro is excellent, open source and written by Zachary Yedidia.

For me, editing in the CLI is generally one-off editing too. If it turns into a project, I switch to using my project editor.

We HAD been warned about coming digital stupidity

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 13 Nov 2021 18:00:00 GMT

Another of those obvious things that are never repeated too much.

git gc - You're not going to need it

Donat Studios — Sat, 13 Nov 2021 12:43:37 GMT

tl;dr

I recommend every developer turn off automatic garbage collection in git. It's one of the first things I do setting up a new machine - and you should do so too.

For a happier life, go to your terminal right now and run:

$ git config --global gc.auto 0

I promise you won't regret it.

The full story

Turn off garbage collection? Surely, I must be crazy, right? Automatic garbage collection is important, right? They wouldn't turn it on by default if it wasn't!? Eh, not really. Not for you. I'm not being sarcastic. You just probably don't work at Meta or Microsoft scale.

Garbage collection for git in most every use case just isn't worth it. That is unless you're very regularly committing very large files, and then orphaning those commits. I don't see that happening in most uses cases; garbage just doesn't build up that quickly in normal usage.

Don't believe me?

I have been working with the same git repo every day for ten solid years with gc turned off. My .git folder is currently 554 MB whereas a freshly cloned copy is 112 MB. I could be saving a whole 442 MB - whoop de doo.

I don't know about you, but I'll happily pay 442 MB for 10 years of improved data safety.

But why turn it off?

Having it on means detached commits could disappear at any time. It's a roll of the dice.

Turning off automatic GC means that anything you once committed will remain restorable, forever. If a rebase goes badly, you can always safely get back to the previous state of a now-orphaned commit. There is zero chance of it ever disappearing.

Basically, turning automatic GC off means any amount of history rewriting is always undoable.

Example

A simple call to git reflog will show you all recent commits, including ones that are no longer on any branch.

$ git reflog
98a9b6b HEAD@{1}: commit (amend): Increase footer font-size and line-height
edc192c HEAD@{2}: commit: Increase footer line height
0360c78 HEAD@{3}: commit: Update footer color to quiet lighthouse
9231f91 HEAD@{4}: commit: Burn down a bunch of old css
…

In the example above, commit edc192c became 98a9b6b when it was amended.

To undo that amend, I can reset the HEAD of my current branch back to edc192c. I have no fear of edc192c ever randomly disappearing before I get around to it.

Turning off GC is a safety net that has saved me many times.

Turning off automatic GC means you could safely work on a detached HEAD if you're wild like that. You probably shouldn't do that unless you know exactly what you're doing, but you certainly could.

And of course…

Keep in mind, you can always just run GC manually if you feel like a lot of garbage has piled up.

$ git gc

Things You Are Allowed To Do, Academic Edition

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sat, 13 Nov 2021 12:00:00 GMT

Since it’s application season again, I thought that I would share some things that you are ‘allowed’ to do in academia but may not be aware of. This post is inspired by Milan Cvitkovic’s post on ‘Things you’re allowed to do’ and Veronika Cheplygina’s posts on advice for Ph.D. students, in particular her post ‘7 things I wish I had done during my PhD’.

Interpolating between the two posts, I thus present you a list of seven items that you might want to try out if you are somehow connected to academia.

Give someone feedback on a preprint or a paper you liked. Academics are mostly used to negative feedback loops: reviewers criticising a paper, evaluators criticising a proposal, etc. Why not flip it around for a change and provide positive feedback? It does not have to be long, but a brief, genuine message along the lines of ‘Your paper on X was really inspiring.’ may go a long way, in particular if you are writing it to an early-career researcher!
Ask someone for access to a paper or its materials. If you want to read a paper but cannot find an accessible version, most authors will be more than happy to accommodate your request. We are not paid anything by the publishers when you buy our paper from them, by the way.

(This piece of advice also pertains to other materials in a paper, such as code or even the original sources for figures. In the worst case, your request will be ignored.¹ I have shared a lot of my own materials in the past and I am always happy to see stuff I created being actually useful to people.)
Ask someone for a potential collaboration. This one might require more guts, but can be even more rewarding. If you know that someone has a specific set of skills that would come in handy for your project, why not try to include them and ask them for a collaboration?

When you do this, however, make sure that your request is as specific as possible. The chances that you will receive a response will go up immediately. At the same time, you should also be prepared to receive no response; especially the ‘big shots’ in every field are probably inundated by both external and internal collaboration requests.
Ask someone to mentor you. Similar to the previous point, this one might not get you a direct response. But if you are looking for some advice and think you know exactly the person that should give it to you, reach out to them and tell them. Again, be aware that mentoring is a large time commitment, so you might not always get a positive response—but it’s worth a try!
Invite someone for a workshop or another event. If you are planning an event of any sort, why not consider inviting someone whose work you always admired? You would be surprised about the response rate. There are already programmes such as Skype a Scientist that aim to connect researchers with the general public, but if you want to have a specific person at your event, why not invite them directly? You could invite them to be speakers for your reading group, your workshop, your lab meeting, …

Previously, this used to be much more problematic as it required a large amount of logistics, but now that many meetings are of a virtual or hybrid nature, there is a higher probability that people can easily join your event. Use that to your advantage!
Contribute to someone’s research project. Did you find an error in a paper? Were you able—or unable—to reproduce certain results? Did you like the code? Even if something has been published, most authors will be happy to learn that someone is using their project or thinking more deeply about it.

Contributing to someone’s (research) project, even if it’s only a seemingly small contribution, can be a nice way to brighten their day. Personally, I am delighted when someone opens a pull request on one of my projects, because that implies that the project was useful to at least one other person!
Acknowledge someone. If someone had a positive impact on you in any way, why not take the time to acknowledge that person? Academia currently only ‘permits’ acknowledgements in dissertations or papers, but that should not be stopping you!² You could have a separate page or write a nice personalised message, for instance. Even if you do not receive a response, being aware of the positive things in your life makes you more grateful and more balanced in general!

Hope you try out a few of these things and find them useful! Let me know if I missed your personal favourite ‘permission’ in the list. Until next time!

Probably due to time reasons or because we lost control of our inbox. ↩︎
Moreover, since academia consists of people like you and me, we should also get a vote about what is and is not allowed. ↩︎

Empty Handshakes

onKeyPress — Sat, 13 Nov 2021 11:07:23 GMT

Eurovision Song Contest: The Story of Fire Saga

🦉 olshansky 🦁 — Sat, 13 Nov 2021 09:57:13 GMT

Olshansky's review of Eurovision Song Contest: The Story of Fire Saga

for vs list comprehension

Ye Lin's Random stuff — Sat, 13 Nov 2021 06:07:13 GMT

This is me writing down my discussion with someone from work regarding for loop vs list comprehension in Python.

During a code review, I was commenting on a chunk of code to change from for loop to a list comprehension. We also talked about why I wanted it that way. One reason is list comprehension could be (arguably) more readable than a for loop. For a simple loop, that might be the case. Another reason is list comprehension is faster than the for loop. How fast? I have never tried comparing them before. So I made a few examples to showcase that. These are from the excellent For Loop vs List Comprehension article.

A brief hypothesis about "normies"

Weblog on marginalia.nu — Sat, 13 Nov 2021 02:00:00 GMT

The phenomenon of “normies” is an interesting one. The term itself is a bit problematic and not one I’d typically use, but as a phenomenon they are still worth investigating.

Their perhaps biggest distinguishing feature is that they don’t get “it”, whatever it is. It’s tempting to think that these are an especially mindless type of person with no personality and little in terms of thought going on.

I have a theory that normies may not actually exist. That is, you can’t actually show me a person that is a normie. They are a mirage.

Systems, Scale, Value

Aditya Athalye's Eval/Apply Blog — Sat, 13 Nov 2021 02:00:00 GMT

Creating things is a delicate endeavour, fraught with peril. People struggle forward through crazy marketplace and environmental complexities just to get from one day to the other. Yet I can't shake off the feeling that we make it harder for ourselves than it should be. I've been trying to work out why. There's a lot to unpack. This post is a start at thinking about it in public.

The Golden Bough

Godsip Club — Sat, 13 Nov 2021 02:00:00 GMT

An Unexpected Journey #

First of all, thank you if you are reading this. It will be different from the other blog articles I have in mind: as an introduction, I want to let you know how the idea about this website was born and what it is meant to be – in my mind, at least.

Now, I’d like to point out that I usually didn’t like talking about religion — or generally, that I didnt’ like religions. Despite that, in the summer of 2021, a friend of mine, a neighbor, while chatting about religion(!) and myths, recommended a text: “You should read The Golden Bough by Frazer. Therein, there is everything”. I never heard about Frazer before, so I immediately searched on Wikipedia. The book abstract looked interesting, but not great: starting from the legend behind a painting, the author explores the reasons of a brutal yet methodical ritual that was committed in central Italy, in the woods near Aricia (Ariccia in modern Italian).

“The Golden Bough” by J. M. W. Turner, which ingnited James G. Frazer’s curiosity.

‘Cool, I know that area,’ I thought while my mind immediately went to the famous porchetta. Since it was very cheap, I immediately bought the volume; only the next day, when I could hold it in my hands, I realised how massive it was. In that pocket edition, it is still a 840-page bold compendium about lore. Nevertheless, I wasn’t discouraged and slowly started flipping through its pages.

A New Perspective #

After a while, it was clear to me that it was necessary to take notes about that enormous amount of information. Until then, I used a paper moleskine-like notebook filled with bad writings, but in this case it would have been a mess. Luckily, in those days I became aware of Obsidian, a powerful tool which allows you to link easily one note with another, so I immediately started writing down annotations.

The first hundreds of pages introduce the story of Aricia’s King of the Wood (a high-priest) and the mysterious ritual regarding his succession: he was killed violently using a blade, every year. To simplify, the question is: why using such violence in the Classical era (Greek and Roman), when barbarian traditions should have been abandoned? It was a very interesting question, but the book rhythm is very – VERY – slow. To do a comparison, it was like asking: ‘After how much time should a pancake be flipped on the pan?’ and answering: ‘Well, I’ll first explain why Earth is round’. To be honest, my interest grew very slowly through the first half of the book.

It was like asking: ‘After how much time should a pancake be flipped on the pan?’ and answering: ‘Well, I’ll first explain why Earth is round’.

My enthusiasm was dropping quicker than I expected, but then one of the great strong points of the tome came out: logical connections. Some terrific observations started to make me think and the interest was kept alive with examples and hypotheses: one above all, the evolution of magic into religion. I began to really appreciate the author’s style, but anyway it was a very time-consuming read.

Therein, There Is Everything #

Work duties needed attention, Summer passed and Fall came quickly. Leaves were already on the ground when I opened again the tome; but something had changed in the meanwhile. During these last few weeks –or better, days– I finished the book with a sprint. In the last two-hundred pages, as the end of the journey was approaching, I realised what the big change was: the author’s talent as teacher erupted like a volcano.

I was aware that Frazer’s intention was to explain the common roots of traditions of civilisations from all the globe since the first pages, but only towards the end I became conscious of his great ability to gradually prepare the reader to accept his hypotheses. Of course, he humbly repeats that he could be wrong through his passages; but while the two of them (author and reader) finally close the loop in the very final chapters, everything makes perfect sense. For me, the last pages were almost emotional after such a long trip through time and space!

A little spoiler about the book – meme edition.

Now, despite the huge pleasure that it gave me, I don’t know if I would ever suggest everyone to read The Golden Bough. I’d only make them angry because it can be so heavy. But that’s when the idea for this website came it. Wouldn’t a collection of cool findings about folklore be awesome? An explanation (not to be taken too seriously) about, for instance, why the oak tree is so important for many civilisations? Why Vampirism was accepted in all the continents? Or, simply, a fun fact about why the ocean’s low tide is considered a bad omen?

Out there, in the internet wilderness, it’s full of blogs like this. But this is mine, and it will be replenished of interesting notes as long as I’ll keep doing –in my opinion– remarkable discoveries in my spare time. I LOVE reading and writing about these topics, so I hope to shake up your interest.

Last thing: if I got your attention, you could consider reading or downloading The Golden Bough for free on Project Gutenberg – I hope you will be hooked as I was.

Benchmarking esbuild, swc, tsc, and babel for React/JSX projects

Notes on software development — Sat, 13 Nov 2021 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Living longer with THESE social media? NOT good

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 12 Nov 2021 20:00:00 GMT

I mean, just look at the Spacers.

JDK Flight Recorder Events in GraalVM Native Binaries

Gunnar Morling — Fri, 12 Nov 2021 19:30:00 GMT

If you have followed this blog for a while, you’ll know that I am a big fan of JDK Flight Recorder (JFR), the low-overhead diagnostics and profiling framework built into the HotSpot Java virtual machine. And indeed, until recently, this meant only HotSpot: Folks compiling their Java applications into GraalVM native binaries could not benefit from all the JFR goodness so far.

Old and New

Weblog on marginalia.nu — Fri, 12 Nov 2021 02:00:00 GMT

I’ve been thinking recently about the emphasis put on “new”, specifically for search engines, but the discussion has some merit even in a wider context. I will start wide and narrow down.

It is common to conflate new with good, and most being young sometime between 1950-2000 will indeed have seen marvellous improvements in quality of life and technology with each passing year. In the light of that, it’s at least easy to explain how one might confuse the two.

Italia IV - Pompeii & Positano

Avy Faingezicht — Fri, 12 Nov 2021 02:00:00 GMT

After a few days in Rome, we drove South. We stopped at Pompeii on our way to the Amalfi Coast, where we spent a few days in Positano. _This is the last in a series shared throughout Fall 2021. When you're done here, make sure to also check out parts one, two, three, and five._

Making technology choices

@sebinsua — Fri, 12 Nov 2021 02:00:00 GMT

Back in 2010, the adage “the right tool for the job” was prevalent within the developer community, but by 2015 it was seen by some as motivated reasoning and in general harmful advice. Nowadays it’s common to hear that we should “Choose Boring Technology” and “optimize our engineering decisions for iteration speed”.

My view is that the adage “the right tool for the job” suffered from being transmitted by engineers in an almost folkloric manner. It’s a truism (“obviously you want to pick the right tool!”) but in practice was often wrong since shoe-horning an old bit of common sense advice into the domain of software engineering gave licence to the self-interests of software engineers who could avoid thinking about its costs (operational complexity, training/ramp-up costs of employees, etc).

On the other hand, “Choose Boring Technology” treats technology choices as if the only system they interact with is a technical one and human incentives can be disregarded — it pays no attention to why “the right tool” was so-often endorsed, and how this related to the polyglot movement (both were, in a sense, marketing that allowed startups and employees that wanted to chart uncharted waters to find each other). Boring is also a misnomer — what is meant is “well-understood” but the word choice is being used to frame the domain as one in which technologies can either be “shiny” or “boring” — which isn’t the primary frame that I’d use to evaluate technology.

While reducing operational complexity, using technologies that are flexible and avoiding unknown-unknowns are all important and suggest that “Choose Boring Technology” is a sensible approach, we should also pay careful attention to whether our choices result in happy developers. Retention is important to the long-term maintenance of knowledge within a company, and developers that don’t feel like they’re learning anything new or that feel stifled by centralised decision-making over the tools they’re allowed to use can end up leaving due to feeling disrespected and disenchanted. On the other hand, there’s also “retention risk” in attracting developers that seek shiny technologies as once these technologies have lost their shine they can leave just as eagerly as they joined. Additionally, in many cases there are technologies that offer an order of magnitude faster/better ways of doing things, and so the best teams will make judgments over when to let their engineers invest their time seeking improvements.

Unchecked restraint in technology choices can eventually lead towards obsolescence and brain drain. Following “The Golden Path” and creating a team to centralise technical decisions is a good idea when a company reaches a particular scale, but you must also work out how you’ll create affordances for developers outside of that team to go above the predefined baseline of quality (for a similar idea read Lethain’s “Providing pierceable abstractions”).

There are also differences between individuals and differences due to scale (personal project, team, company-wide, etc):

At the individual level, while I might reach into my toolbox and find that my most well-worn tool is JavaScript and use that, another might reach into their toolbox and pick Ruby. We should use what we know best. But what if you’re just starting out? If I was a beginner I would pick up-and-coming general-purpose technologies that are growing in usage. While I might benefit from prior knowledge in an older ecosystem, I will learn more by being part of a highly-engaged community of people trying to achieve state-of-the-art approaches.

But at the scale of a company, there are further problems to be aware of due to the difficulty of migrating software stacks and a tendency for policies and processes that don’t work well to be left unfixed.

What should we recommend to a company that has always written both their application and database logic using SQL Server Stored Procedures? The boring choice might be for them to continue extending and creating new software in this way, but since there is very little beginner interest in learning that stack eventually they will run into hiring problems and brain drain caused by an ageing workforce (institutional knowledge loss). To survive, they will need to rework their system in a way that gives them access to a more fungible workforce, and they will need to do this without interrupting the business.

Another common problem at large companies is for there to be policies and processes around evaluating and recording technical decisions but for these to not achieve anything other than getting developers to copy-and-paste the marketing claims of a new technology into a document without consideration of its context/impact within the organisation. These policies can often be empty ceremonies with no impact on the actual decision-making process that allow developers to rubber-stamp their desire to learn new technologies.

So how would I make technology choices?

Use well-understood tools. Either those you personally know well or those that are known well within the broader technology community.
Use the smallest set of tools you can to solve your problem(s).
Pay close attention to the happiness of your developers. I’m not suggesting that you should always let the whims of developers dictate a company’s overall technology choices, but you should be aware of the market incentives of your employees. In any case, an important job of engineering leaders is to find ways to carve out interesting work from within boring/legacy systems.
Benchmark yourself against you competitors but if a problem is core or complementary to your business look for ways to use technology to gain advantage.
Try to find the line between policies that are too onerous for developers and those that are not rigorous enough to provide any benefit to the organisation. Non-rigorous decision making processes are really common and generally when a policy isn’t providing its stated aim it should be gotten rid of. However, if we want to improve decision-making within a large company this can probably be done with structure, asynchronous collaboration and good guiding questions (e.g. Can this be achieved with technologies we already use? What if true could invalidate the benefit given by this technology? Will this require a migration? Who will operate this? Etc).

An idea whose time may be close: battery-powered TRAINS

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 11 Nov 2021 20:00:00 GMT

Even better than electric cars, I’d say.

A Jaunt Through Keyword Extraction

Weblog on marginalia.nu — Thu, 11 Nov 2021 02:00:00 GMT

Search results are only as good as the search engine’s ability to figure out what a page is about. Sure a keyword may appear in a page, but is it the topic of the page, or just some off-hand mention?

I didn’t really know anything about data mining or keyword extraction starting out, so I’ve had to learn on the fly. I’m just going to briefly list some of my first naive attempts at keyword extraction, just to give a context.

Do The Scariest Thing First

julik live — Thu, 11 Nov 2021 02:00:00 GMT

Why did Facebook really shut down face recognition?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 11 Nov 2021 01:50:00 GMT

Probably because they have better replacement for it.

Aksel Waldemar Johannessen

Posts on A blog — Wed, 10 Nov 2021 13:44:38 GMT

Aksel Waldemar Johannessen (1880-1922) was a Norwegian expressionist, painter, and graphic designer. In Norway he is often referred to as “the forgotten artist” and was only praised by critics after his death. Johannessens art was noticed by Edvard Munch and two famous art critics at the time, Dagbladet’s Jappe Nilssen and the cultural profile Olav Dalgard.

Self-portrait of Aksel W. Johannessen. Public domain, CC0.

Early life and education

Aksel Waldemar Johannessen was born October 26, 1880 in Christiania (today Oslo), Norway. He grew up at Kampen, where his father, Julius Johannesen, was a shoemaker and auxiliary policeman. He was connected early to the artist community at Hammersborg behind the Trinity Church. In the period of 1900 and 1907, he become educated as an artist, including seven semesters at the Norwegian National Academy of Craft and Art Industry, the last years with Lars Utne in the sculptor class. He learned the art of woodcarving in the workshop of the internationally famous Johan Borgersen.

Plain Org has joined the chat (iOS)

xenodium.com @alvaro — Wed, 10 Nov 2021 02:00:00 GMT

The App Store is a crowded space when it come to markdown apps. A quick search yields a wonderful wealth of choice. Kinda overwhelming, but a great problem to have nonetheless.

For those of us with org as our markup of choice, the App Store is far less crowded. I wish we could fill more than a screen's worth of search results, so you know… I could show you another pretty gif scrolling through org results. For now, we'll settle on a single frame showcasing our 4 org options.

Beorg, MobileOrg, Flat Habits, and Orgro are all great options. Each with strengths of their own. Organice, while not on the App Store, is another option for those looking for a web alternative. Of these, I had already authored one of them. More on that in a sec… You see, about a year ago I wanted to play with Swift, SPM, and lsp itself. Also, having Swift code completion in Emacs via lsp-sourcekit sounded like a fun thing to try out, so I started using it while writing a Swift org parser.

While working on the parser, I happened to be reading Atomic Habits (awesome book btw)… It was also a great time to play around with SwiftUI, which by the way, is pretty awesome too. With Atomic Habits fresh in mind, org parser in one hand, and SwiftUI in the other, I built Flat Habits: a lightweight habit tracker powered by org.

I love being able to save habit data to plain text and easily track on my iPhone (via Flat Habits) or laptop (via Emacs). I wanted to extend similar convenience to org tasks, so I built Plain Org.

My previous post mentioned quickly adding new tasks and searching existing ones as Plain Org's driving goals. Of course, neither of those are as useful without automatic cloud syncing, so pluging into iOS's third party cloud support was a must-have.

With these baseline features in place, I started an alpha/beta group via TestFlight. Early Plain Org adopters have been wonderfully supportive, given lots of great feedback, and helped shape the initial feature set you see below.

There's plenty more that can be supported, but hey let's get v1 out the door. Gotta start somewhere.

Plain Org v1 features

View and edit your org mode tasks while on the go.
Beautifully rendered org markup.
Sync your org files using your favorite cloud provider.
Create new files.
Outline-style editing with toolbar
- Keywords
- Indent
- Priority
- Tags
- Formatting: bold, italic, underline, strikethrough, verbatim, and code.
Add links from Safari via share extension.
Add new tasks via Spotlight.
Reorder headings via drag/drop.
Checkboxes
- Interactive toggling.
- Quickly reset multiple checkboxes.
Follow local links.
Show inline images.
File-local keywords and visibility.
Filter open/closed tasks.
Show/hide stars.
Edit raw text.
Light/dark mode.

Plain Org joins the chat

Today Plain Org joins the likes of Beorg, MobileOrg, Flat Habits, and Orgro on the App Store.

This post was written in org mode.

In the beginning, was the domain name

Aditya Athalye's Eval/Apply Blog — Wed, 10 Nov 2021 02:00:00 GMT

How this blog came to be is a minor miracle. Long story short, I conned myself into believing nobody will find /and/ read it. But you're here, aren't you? And you're reading this. Aren't You? Confucamus. Well, here's how you got here.

Skills, stories, and software every dev should know - 123dev #45

Justin Garrison's Homepage — Wed, 10 Nov 2021 02:00:00 GMT

Some weeks I love the structure of this newsletter. It helps me not dump a ton of information but focus on just a few things each week. Other weeks—like this week—I have a ton of cool tools and articles to share but can't if I want to follow my own rules.

Longboard 2

Home on mfashby.net — Tue, 09 Nov 2021 23:56:41 GMT

Styleable Inline SVG Icons, with Caching & Fallback

duncanlock.net — Tue, 09 Nov 2021 23:08:46 GMT

Figure 1. This pages tag list is using this tag icon from FontAwesome. This is a 550 byte SVG file, 346 bytes gzipped.

If you want to use SVG icons on a website and style them with CSS - then the SVG needs to be inline - i.e. the SVG markup needs to be included with the rest of the pages HTML markup.

Unfortunately putting things inline means that they can’t be cached. In this article I’ll show one way to get around this - and get the best of both worlds: inline styleable SVG icons, with caching!

If we take the tag list in the sidebar in this site as an example, the basic markup looks like this:

<section>
  <h3>Tags:</h3>
  <ul class="tag-list">
    <li class="tag">
      <i class="icon"> <!-- Tag icon goes here --> </i>
      <a href="/tag/howto.html">howto</a>
    </li>
    <li class="tag">
      <i class="icon"> <!-- Tag icon goes here --> </i>
      <a href="/tag/javascript.html">javascript</a>
    </li>
    ...
  </ul>
</section>

When we put the SVG for the icon in, it looks like this:

<section>
  <h3 class="tag-heading">Tags:</h3>
  <ul class="tag-list">
    <li class="tag">
      <i class="icon">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!-- Font Awesome Free 5.15.3 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) --><path d="M0 252.118V48C0 21.49 21.49 0 48 0h204.118a48 48 0 0 1 33.941 14.059l211.882 211.882c18.745 18.745 18.745 49.137 0 67.882L293.823 497.941c-18.745 18.745-49.137 18.745-67.882 0L14.059 286.059A48 48 0 0 1 0 252.118zM112 64c-26.51 0-48 21.49-48 48s21.49 48 48 48 48-21.49 48-48-21.49-48-48-48z"/></svg>
      </i>
      <a href="/tag/howto.html">howto</a>
    </li>
    <li class="tag">
      <i class="icon">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!-- Font Awesome Free 5.15.3 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) --><path d="M0 252.118V48C0 21.49 21.49 0 48 0h204.118a48 48 0 0 1 33.941 14.059l211.882 211.882c18.745 18.745 18.745 49.137 0 67.882L293.823 497.941c-18.745 18.745-49.137 18.745-67.882 0L14.059 286.059A48 48 0 0 1 0 252.118zM112 64c-26.51 0-48 21.49-48 48s21.49 48 48 48 48-21.49 48-48-21.49-48-48-48z"/></svg>
      </i>
      <a href="/tag/javascript.html">javascript</a>
    </li>
    ...
  </ul>
</section>

Styling Inline SVG with CSS

As you can see, this SVG markup is fully inline with the rest of the HTML. This means that we can write CSS styles that will style the icon, just like we can for anything else in the DOM:

i.icon svg {
  /* Set the SVGs fill color to match the parent elements color: */
  fill: currentColor;
  /* Set some other properties on the SVG element */
  display: inline-block;
  font-size: inherit;
  height: 1em;
  width: 1.25em;
  overflow: visible;
  vertical-align: -0.125em;
  margin: 0 4px 0 0;
}

/* Set different icon colours in different places, using CSS variables: */
:is(header, footer) i.icon svg {
  fill: var(--icon-color-blueprint);
}
article i.icon svg {
  fill: var(--icon-color-content);
}
/* Have icons in the header & footer change colour on hover: */
:is(header, footer) a:hover i.icon svg,
:is(header, footer) .active i.icon svg {
  fill: var(--blueprint-bg);
}

You can also style individual parts of the SVG, if you want, just like you would HTML:

You can also give parts of your SVG classes & IDs, and the style these just like you would an HTML element with a class or ID. Imagine this example markup:

<i class="icon">
  <svg width="120" height="120" viewBox="0 0 120 120" xmlns="http://www.w3.org/2000/svg">
    <rect id="smallRect" x="10" y="10" width="100" height="100" />
  </svg>
</i>

With this example CSS:

i.icon svg rect { stroke: #000; fill: red; }
i.icon svg #smallRect { fill: blue; }

the rectangle would be blue.

Problems with Inline SVG

There are some obvious problems with doing SVG icons this way:

It’s more markup: Instead of keeping your SVG images in separate files like you would with other image files, you need to include them in your page markup.
It’s repetitive: Every time you want to use the same icon again, you have to include its markup, again. In the example list above, this is a list of tags - so it repeats the tag icon’s markup for each item in the list.
None of this is cached: Unlike external files, the browser can’t cache the actual page that’s requested, which means that the SVG’s aren’t cached.

The first two aren’t really a problem if you’re using some kind of templating system to generate your markup - but they both make the last one - caching, even more of a problem. The biggest drawback to inline SVG is that it’s not cached by the browser, it’s loaded every single time, along with the rest of the page’s markup.

Even if each icon is only 500 bytes, with our simple tag list which uses that icon only four times, you’re already up to 2 KB of extra markup. Including the header & footer there are currently 22 icons on this page, which even at only 500 bytes each, makes 11 KB of uncached extra markup on every page load. This isn’t huge - but it’s not ideal either. Can we do better?

Using SVG Sprites - Still Uncached, but Better

You can improve this by using an SVG “sprite sheet”:[1]. This means putting all the SVG markup for all your icons into one SVG file, adding that inline in the top of the page somewhere, and then referencing the icons where you want to use them.

The sprite sheet looks like this:

<body>
  <!-- Sprite Sheet, loaded inline somewhere at the top, but hidden: -->
  <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
    <symbol id="home" viewBox="0 0 512 512">
      <path d="..."/>
    </symbol>
    <symbol id="tag" viewBox="0 0 512 512">
      <path d="...">
    </symbol>
    ...
  </svg>

When you want to use one of the icons, just reference it with an SVG use statement, and it’ll be pulled from the sprite sheet:

<i class="icon">
  <svg><use href="#tag"></use></svg>
</i>

So, using the sprite sheet, we only include the markup for each icon once, then each time we want to show that icon, we include a tiny SVG use statement [2]. This means that our example tag list now looks like this:

<ul class="tag-list">
  <li class="tag">
    <i class="icon"><svg><use href="#tag"></use></svg></i>
    <a href="/tag/howto.html">howto</a>
  </li>
  <li class="tag">
    <i class="icon"><svg><use href="#tag"></use></svg></i>
    <a href="/tag/javascript.html">javascript</a>
  </li>
  ...
</ul>

This is obviously much less repetitive and less markup overall - we’re only including each icons markup once now.

However, the sprite sheet still needs to be inline if we want to be able to style the icons with CSS and <use> them, so it’s still loaded every time and still not cached.

Can we do better?

Caching Inline SVG using SVG Injection

If we want caching, we have to load the SVG sprite sheet as an external resource, like we would a normal image - but if we do that, it’s not inline anymore, so we lose the ability to style the SVG with CSS.

Is there a way to get the best of both worlds, somehow?

Yes there is, if you use some JavaScript to put your externally loaded SVG back into the DOM and re-inline it. Doing that looks like this:

1. Preload the sprite sheet

Preload [3] the sprite sheet in your site’s <head> somewhere, ensuring that it’s loaded early and will be in the cache, ready for the next stage. This reduces jank and makes things more reliable. You can preload the sprite sheet like this:

<link rel="preload" href="/images/icons/icon_sheet.svg" as="image" type="image/svg+xml" />

2. Load the JavaScript

Load this JavaScript, somewhere in your site’s <head>:

const convertSVG = (image, callback) => {
  // Get the SVG file from the cache
  fetch(image.src, { cache: 'force-cache' })
    .then((res) => res.text())
    .then((data) => {
      // Parse the SVG text and turn it into DOM nodes
      const parser = new DOMParser()
      const svg = parser
        .parseFromString(data, 'image/svg+xml')
        .querySelector('svg')

      // Pass along any class or IDs from the parent <img> element
      if (image.id) svg.id = image.id
      if (image.className) svg.classList = image.classList

      // Replace the parent <img> with our inline SVG
      image.parentNode.replaceChild(svg, image)
    })
    .then(callback)
    .catch((error) => console.error(error))
}

This converts an SVG image element into inline SVG. This allows the inline SVG to function normally, be styled, etc… but to also be loaded async and cached by the browser as an external resource!.

Notice this line: fetch(image.src, { cache: 'force-cache' }) - this performs an AJAX load of the SVG file in JavaScript - but it won’t be loaded twice - it’ll already be in the cache and the cache: 'force-cache' option makes fetch just pull it from there.

Including all the comments, this is 934 bytes of JavaScript. Minified & gzipped, this is about 250 bytes.

3. Load the Sprite Sheet as a Regular Image

Now, we can load the sprite sheet as a regular image, using an <img> element, hide it, and run the convertSVG() function on it:

<body>
  <img src="/images/icons/icon_sheet.svg" style="display: none;" onload="convertSVG(this)" />
  ...

This is all the essential parts - we get externally loaded and cached SVG icon sprites, inline, with caching - the holy grail!

The only real downside of this technique are that it requires JavaScript. Can we have a fallback, so that this works OK without JavaScript?

Fallback Without JavaScript

If we don’t have JavaScript the ConvertSVG() stuff isn’t going to happen. The browser will still load the sprite sheet, but the <use> stuff and the CSS styles won’t work, because the SVG will be external.

We can use the venerable <noscript> element [4] to provide an alternative. The contents of the <noscript> element will only be processed by the browser if JavaScript is disabled. We can include the relevant icon as an <img> element inside the noscript element, like this:

<i class="icon">
  <noscript>
    <img src="/images/icons/fa/solid/tag.svg" width="18px" />
  </noscript>
  <svg><use href="#tag"></use></svg>
</i>

So, if you have JavaScript enabled:

the sprite sheet and the convertSVG() will work
the <svg><use href="#tag"></use></svg> will work
CSS styles will work
<noscript> element will be ignored

If you have JavaScript disabled, you’ll get the opposite:

the sprite sheet and the convertSVG() won’t work
the <svg><use href="#tag"></use></svg> won’t work
CSS styles won’t work
<noscript> element will be loaded & you’ll get the SVG icon loaded in the right place

So, with JavaScript disabled, the icons will load in the correct place - the only thing missing will be the CSS styles, because the icons are now external. There’s nothing we can do about that, sadly. If you need to, you can apply some fallback styles to fix spacing issues that might arise because the intended styles no longer apply:

i.icon noscript img {
  padding: 0;
  border: none;
  float: none;
  margin: 0;
  box-shadow: none;

  margin-right: -14px;
}

Show Me the Numbers

Table 1. All figures are for this page, uncompressed, and in bytes unless otherwise specified. Other external assets that this page loads, like CSS, have been removed for simplicity.

	HTML	JavaScript	Sprite Sheet	Total Cacheable	Total	Total Saving (%age)
All Inline	58,793	-	-	-	58,793	-
Inline Sprite Sheet	55,940	-	-	-	55,940	2,853 (5%)
External Sprite Sheet, w. JS	44,825	391	11,301	11,692	56,517	13,968 (24%)

We can see that that just consolidating all the repeated SVG icons into an inline sprite sheet saves us 2,853 bytes, or 5% of the total non-cacheable size.

Once you make the sprite sheet external and add the JavaScript, your overall total size actually goes up very slightly - by 577 bytes. However, 11,692 bytes of this is now cacheable, where it wasn’t before - so even though you are +577 bytes on first load, you are -13,968 bytes on subsequent page loads because of caching. This is a saving of 24% over the original non-cacheable size.

Footnotes & References

CSS Tricks Icon System with SVG Sprites: https://css-tricks.com/svg-sprites-use-better-icon-fonts/ & using SVG symbols: https://css-tricks.com/svg-symbol-good-choice-icons/ ↩
The <use> element takes nodes from within the SVG document, and duplicates them somewhere else: https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use ↩
The preload value of the <link> element’s rel attribute lets you declare fetch requests in the HTML’s <head>, specifying resources that your page will need very soon, which you want to start loading early in the page lifecycle, before browsers’ main rendering machinery kicks in. This ensures they are available earlier and are less likely to block the page’s render, improving performance: https://developer.mozilla.org/en-US/docs/Web/HTML/Link_types/preload ↩
The <noscript> HTML element defines a section of HTML to be inserted if a script type on the page is unsupported or if scripting is currently turned off in the browser: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/noscript ↩

Incoherence of cryptoassets, the best parts

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 09 Nov 2021 20:00:00 GMT

Some things are just too good to not synthesize them.

Companies Rarely Grow Into Their Valuations

Who is Nnamdi? — Tue, 09 Nov 2021 18:28:00 GMT

Companies don't catch up to their valuations; their valuations catch up to them.

Emotional Mirror Jaguar

june.kim — Tue, 09 Nov 2021 02:00:00 GMT

The New HeroQuest

Ben Overmyer's Site — Tue, 09 Nov 2021 02:00:00 GMT

So today I received the new HeroQuest in the mail. Last year, I backed the Hasbro-run crowdfunding campaign to remake it. I backed at the highest level, the "Mythic" tier, which came with quite a few extra goodies.

I plan on writing a second blog post after I play it a few times.

Opening up the shipping box!

The contents of the shipping box.

This is what came in the Mythic-exclusive box.

What follows is a series of photos comparing my copy of the original game released in the late 80s to the new one.

The original box is shorter and wider. The artwork on the new one is a nice tribute to the original.

The board in the new one is a bigger scale but otherwise follows the layout of the original.

On the left is a goblin from the original, and on the right is the new one. The originals were made of a hard plastic; the new ones are a softer plastic. I'm not sure if they'll hold paint well yet.

On the left is the barbarian that I painted when I was a teenager, with the sword partially broken. On the right is the new barbarian. Not shown here is the female barbarian that came in the Mythic box.

The gargoyle in the new one is slightly bigger and has an axe instead of a whip. Otherwise it's pretty close to the original.

The new dice are bigger and less round. The shield is a different symbol, too. I think these will be easier to roll, honestly.

All of the furniture in the new game is full plastic 3D models. The furniture in the original was cardboard with some plastic parts. Not shown here are the rats and skulls that come in the new set.

I like the new doors better, though one or two of the ones in my box are slightly warped.

The new character sheets are almost identical to the original, but there are a LOT more of them. This is a thick set.

The rulebook in the new version is full color on glossy paper. The artwork is much improved. I haven't dug through the rules to see if the layout is changed, though.

The Zargon player's screen is the same size as the original, but it has new artwork.

The reverse of the Zargon player's screen has mostly the same information, although the names of some of the monsters are different and the text is laid out better.

The new quest book is also full color glossy. Again, I haven't flipped through it yet.

Each of the original cards is present in the new version, but with new artwork. Also, Chaos Magic has been renamed Dread Magic, and there are new equipment cards, player rule reference cards, and character cards.

What Do You Want to Spend Your Life On?

Elliott Slaughter — Tue, 09 Nov 2021 02:00:00 GMT

Here’s a funny phrase: “What do you want to spend your life on?”

2021-11-09

ho.dges.online — Tue, 09 Nov 2021 02:00:00 GMT

Coexistence: Automation and Humanity

Macleod Sawyer | Web Portal Feed — Tue, 09 Nov 2021 02:00:00 GMT

This is another response to a post that was once found here.

Automation of mindless and repeatable vocations are coming, and are here.

But, automation requires co-existing humans and robotics in the same work space. Automation by itself fails, automation without a handler, fails, automation because it is focused and the world is not, fails.

I am a roboticist, I develop and build robots and I can say with good authority (given to me, by me) that robotics and automation will only ever make sense within our current scope when there are overseers and other users to ensure proper order is in place.

Robotics are not at a place (or will be for many years) where this will not be the case. The ‘we will automate you if you don’t accept our pay’ is absolute bullshit for several reasons:

The cost to train, develop, and implement a robotic system for any amount of work is costly (millions)
The cost to train, develop, and implement overseers is costly (far more than the bare minimum they want to pay)
Robots are not generalists (yet…), and every job needs generalists so that they can do more than what they needed, or if deadlines change, or parts change, or really anything changes
You have to radically change the entirely of a workplace, shop floor, warehouse, store, and more to become a 'robot-friendly’ work place. There are safety requirements, there are OSHA guidelines to ensure human workers are not harmed, there are insane requirements everywhere to ensure that a million dollar robot doesn’t topple over and break everything around it including itself.
Maintenance_. Robots break._ Just as humans need to go home and rest, so do robots. Just as humans need to relax or else they break, so do robots. When robots do inevitably break you have to call in experts to fix them, to care for them, and to ensure that they never break and have a long lifespan.

Don’t believe any company that tries to espouse the “then we’ll just give your jobs to the robots” because if they could, they would have already - and they won’t because they can’t.

Automation is coming, but automation will only ever make sense when they can work alongside humans. Robots are tools, just like a computer is a tool, just like a hammer is a tool.

Compile-time validation of UUIDs

Tue, 09 Nov 2021 02:00:00 GMT

In this post I show how to use TypeScript literal string template types to get a compile-time UUID validation.

Type-level UUID numbers

It happens so that we can use recent TypeScript’s type system improvements to get a compile-time validation of UUID number, including validation of correctness of its version. We start with defining a set of allowed symbols for both hex alphabet and (separately) for version:

type VersionChar =
    | '1' | '2' | '3' | '4' | '5';

type Char =
    | '0' | '1' | '2' | '3'
    | '4' | '5' | '6' | '7'
    | '8' | '9' | 'a' | 'b'
    | 'c' | 'd' | 'e' | 'f';

UUID format is the following: xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx, with M representing version (special case for 0, and actual versions are 1-5) and N representing variant. To encode this, let’s create a helper which will check if a string S consists of Chars and has length of Len symbols.

Unfortunately, TypeScript’s strings, even literal ones, don’t have length as a constant — it’s just a number. So we need to implement this check using a recursive type with an accumulator.

I’m also defining a helper Prev to calculate previous number for a given one. It uses accessing tuple by index to get a previous number regarding the index — e.g., for X = 5 it will get sixth element of a tuple, which is 4. As the longest group of symbols in UUID has length of 12, so maximal number present in a tuple is 11, and rest are nevers:

type Prev<X extends number> =
    [never, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, ...never[]][X];

type HasLength<S extends string, Len extends number> = [Len] extends [0]
    ? (S extends '' ? true : never)
    : (S extends `${infer C}${infer Rest}`
        ? (Lowercase<C> extends Char ? HasLength<Rest, Prev<Len>> : never)
        : never);

Now we can define helpers to check whether a string is a group of 4, 8, 12, a special case — version and 3 more bytes, and a special case of nil UUID:

type Char4<S extends string> = true extends HasLength<S, 4> ? S : never;
type Char8<S extends string> = true extends HasLength<S, 8> ? S : never;
type Char12<S extends string> = true extends HasLength<S, 12> ? S : never;

type VersionGroup<S extends string> = S extends `${infer Version}${infer Rest}`
    ? (Version extends VersionChar
        ? (true extends HasLength<Rest, 3> ? S : never)
        : never)
    : never;

type NilUUID = '00000000-0000-0000-0000-000000000000';

Finally, we can write a (rather lengthy and mouthful) definition of UUID number, using previously defined helpers:

type UUID<S extends string> = S extends NilUUID
    ? S
    : (S extends `${infer S8}-${infer S4_1}-${infer S4_2}-${infer S4_3}-${infer S12}`
        ? (S8 extends Char8<S8>
            ? (S4_1 extends Char4<S4_1>
                ? (S4_2 extends VersionGroup<S4_2>
                    ? (S4_3 extends Char4<S4_3>
                        ? (S12 extends Char12<S12>
                            ? S
                            : never)
                        : never)
                    : never)
                : never)
            : never)
        : never);

Time for some examples! Let’s imagine we have a function which gets a user by its UUID. I don’t really care about the implementation, so I’ll just log that id. But you can clearly see that incorrect cases are rejected by the compiler, and correct ones are allowed:

Programs and Intent

Lambda Land — Tue, 09 Nov 2021 02:00:00 GMT

What does this program do? At the most reduced level, one could say that a program’s behavior is defined by the effect it has on the hardware running it. That’s not very useful however; when we’re programming, we often have to deal with legacy code and tease out the original intent of the code.

Saying that the meaning of a program is entirely encapsulated by the code is saying that the intent and the implementation are the same. They so rarely are!

Today I found some Elixir code that looked like the following:

@spec all_have_key?(lst :: [%{}], needed_key :: String.t()) :: boolean()
def all_have_key?(lst, needed_key) do
  lst
  |> Enum.map(fn m -> m |> Map.keys() |> Enum.any?(fn k -> k == needed_key end) end)
  |> Enum.all?()
end

(Note, I just threw that together—possible syntax errors in there.)

What is this code trying to do? It’s trying to check that each map in a list has a given key and return true or false on that condition. I don’t think the code even had a @spec to help explain that: all I had was the function name (which was not as clear as all_have_key) and the source.

After a few moments of reflection, I rewrote it to this:

@spec all_have_key?(lst :: [%{}], needed_key :: String.t()) :: boolean()
def all_have_key?(lst, needed_key) do
  lst
  |> Enum.map(&Map.has_key?(&1, needed_key))
  |> Enum.all?()
end

That big long complicated bit has a built-in function. The built-in function is more efficient because it doesn’t traverse the entire list of keys searching for a match: with a map, you get O(1) lookup time.

So what does a program mean? I’m pretty confident that I preserved the intended meaning of this program. But what’s a better way to express that intent?

Tests are useful, but they don’t capture everything. While I think this function was tested, no test can ever ensure 100% preservation of intent. Tests can only find witnesses of meaning mismatches.

Type systems are helpful too. But types come in varying degrees of precision: some languages give you an Int, while others give you Int or Nat or ∈ {1, 2, 3}. More powerful type systems let you express more of your intent in a way that can be mechanically checked, but they tend to also be more burdensome.

This is an open question that I know there’s a lot of ongoing research around. I’m excited to see what I find!

"Websites are dead" is NOT the whole picture

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 08 Nov 2021 20:00:00 GMT

Websites may look dead if you only look at them SHORTLY enough.

Things I Read : CPython Performance

Ye Lin's Random stuff — Mon, 08 Nov 2021 18:25:02 GMT

I’ve been looking at how I can improve the performance of an internal service written in Python. I was wondering if there is a way to speed things up without rewriting large part of the code or using another programming language to rewrite part of the code. Then I remember reading about Pyston (a faster and highly-compatible with Cpython from Dropbox) on HackerNews and this seems like an opportunity to try it out. I’ve integrated into the service, planning to run some benchmarks and hopefully I will have something to share another time!

Border of the Known

trv.rs — Mon, 08 Nov 2021 13:54:00 GMT

In any team-oriented context (product work, academic study, team projects, etc.) the whole team would benefit from documenting and sharing Strong Facts or Strong Opinions¹.

As you practice Note-Making and learn new information you can't know where it belongs.

It could be:

Known and documented facts I'm just re-discovering.
Already known but undocumented facts I can add to the shared repository.
Unknown, undocumented hypotheses that I can validate and then give to the team as insight.

It may take some time to discover that an idea or fact is already in the Border of the Known… shared knowledge you are just learning. If you have written down unrecorded "oral history" you can add to the team's knowledge bank.

Any information you discover that is outside that shared boundary of knowledge is potentially new information… true research. if validated, it is your unique contribution to the knowledge space:

research is simply a continuation of something we already naturally do: learning. Learning happens when you understand something that someone else already understands. Research happens when you understand something that nobody else understands yet.

– Research as Understanding - Kanjun Qiu

In a cooperative shared space, you can validate, persuade, and publish moving it into the shared knowledge pool. In a competitive space you can keep this information to improve your position.

In my system, a “named note” or evergreen note should be a strong fact or a strong opinion. Evergreen Note. ↩

Greta Thunberg; a Future Martyr

Posts on A blog — Mon, 08 Nov 2021 11:03:06 GMT

Starting Sunday (October 31, 2021), climate negotiators from nearly every country will gather for two weeks in Glasgow, Scotland, United Kingdom to hammer out a new agreement aimed at cutting emissions to a level scientists hope will limit global warming, also known as United Nations Climate Change Conference (COP26).

Queen Elizabeth was supposed to open it all, but canceled due to illness. William, Kate and Charles, on the other hand, they arrived. And Barack Obama. As well as the world’s richest men, like Jeff Bezos and Bill Gates.

Why I like lobste.rs

ℤ→ℤ — Mon, 08 Nov 2021 03:41:05 GMT

Lobste.rs is a “computing focused community” and this post is a good example of why I like it:

Thinking about the data model of file systems and alternatives is a pretty geeky subject, but impactful
Comments are respectful and build on the post
Comments add historical context

Community Heat, or Why You Should Get Good at Events

swyx's site RSS Feed — Mon, 08 Nov 2021 03:32:56 GMT

A piece of advice I heard about marketing and community that I've repeated to founders ever since - Get Good at Events.

Cyclic Workflows with Prefect

About on kinow — Mon, 08 Nov 2021 02:00:00 GMT

Last month I wrote about Cyclic Workflows with Cylc and StackStorm and how few workflow managers support cyclic workflows.

I was surprised today while reading Prefect documentation to see this paragraph:

Most workflow frameworks act as if looping is impossible (stressing the Acyclic part of the DAG), but it’s actually trivial to implement. We simply dynamically unroll the loop, similar to how RNN gradients are sometimes computed.

Tips for neat splash pages

Geography & Coding — Sun, 07 Nov 2021 21:28:36 GMT

Things I Read - SIM Cards!

Ye Lin's Random stuff — Sun, 07 Nov 2021 10:39:38 GMT

Another rainy day.

SIM Cards!!

Another day, another interesting HN post. I saw this post What is AT&T doing at 1111340002?.

And the top was very interesting to me.

Even in relatively technical circles, like HN, many people are not aware of this and I use every opportunity I have to reiterate:

A SIM card is a full-blown computer with its own CPU and memory.

Your carrier can upload and run arbitrary code without your consent or knowledge. They can do this at any time. This means that your “phone” is actually three different computers running in concert - the actual phone itself (iOS or Android or Symbian), the baseband processor running the baseband code, and the SIM card.

About

Ye Lin's Random stuff — Sun, 07 Nov 2021 10:08:04 GMT

Hi! My name is yelinaung

I am writing down

what I’ve learned recently
learning progress
interesting topics
books

What makes you a game changer

D13V — Sun, 07 Nov 2021 06:13:54 GMT

Life is too short to drink cheap whiskey

unknown

This is the third post in a series of 13 posts about how to be a game changer. In the previous two posts, we defined the concept and started looking at which games to change. Today we’ll look at the main way to be a game changer - to opt out.

According to Merriam-Webster dictionary a game changer is “a newly introduced element or factor that changes an existing situation or activity in a significant way”. While this definition is aligned with how some people use the phrase, it is superficial, and it doesn’t define what it means for a person to be a game changer.

One way this definition gets wrong is that it considers games mutable. By that logic any minuscule change in the parameters of the game is technically a game changer. Yet, I don’t want to be the type of person that starts a sentence with “Technically… ☝️”. I think what really matters is when the game is changed so much that the behavior of the players changes and they have to pick a new way to play. That’s when it gets really interesting and worth talking about.

We can think of changed games being an instance of a distinct new game. This distinction may seem contrived at this point, as what it means to be a new game is still ambiguous to separate out, but it will become clearer in a bit. By thinking of the changed game as a new game, we realize that the person who changed it decided not to play the old game. That person walked out, or opted out of the old game. They exited the old game. Which leads to the most common method people change games:

You become a game changer by opting out of bad games.

Opting out

Opting out starts as a seed in their mind. It might be as explicit as deciding - “I’m not going to participate,” but it can also be subconscious, or the person may decide to “hack” the existing game rules for their advantage. For example, a boxer may decide not to become good at knocking out their opponents but instead always win through a minimal point difference by having a new type of defense which doesn’t let others score points against them. They may keep their hands up protecting themselves all the time, and patiently wait for an opening. Their style may be described as “ugly” by traditional fans and called “not real boxing” by gatekeepers, but they will be effective nonetheless, and as a result other boxers may adapt the same “ugly” style. That boxer decided not to fight according to everyone else’s expectations - they opted out of the boxing narrative.

Other types of game-changing are more subtle and a lot of us do them every day. A busy person may find a way to meal-prep ahead their food for the week, thus discovering a new freedom in how they spend their time through the day. A student may drop a class they realized is not helpful or interesting, and go on a path towards a different major and career.

The ethics of opting out.

Opting out of negative-sum games is opting out of something unethical, and therefore it can be ethical, if replaced with something better. There might be a question of loyalty and trustworthiness about the person opting out. But expecting a loyalty from someone who is persistently on the losing side is itself unethical. If we want to live in a kind, ethical and rich world, at some point we have to opt out of unethical games.

Types of opting out

Opting out is a bit of an abstract concept, and by itself is not super actionable, as it begs the follow up question “What does opting out look like in a specific situation?” And the answers to this question may be manyfold - you may opt out by physically walking out, by focusing on a different objective than you’re expected to, by becoming less adversarial, by turning one-off situations into repeated situations, and even by reframing how you see and approach interactions.

Four years ago I started thinking about how everything we consume affects our thinking, and I started gradually opting out of alcohol, caffeine, social media. I wanted to play the game of life as myself, rather than be swayed by substances and narratives. I’ve also started opting out of certain types of food such eating too much sugar, or vegetable oils with polyunsaturated fats. And I also stopped using commercial shampoo and instead try to use a more wholesome soap. Recently I started opting out of hot showers and replacing them with cold showers in the morning. I’ve opted out of jobs and partnerships that looked promising.

Neo opted out of the Matrix.

Next up

The ways you can opt-out depend on which negative game you find yourself in, but the first step is always to detect the game. In the next posts, we’ll cover specific types of games and opting out in detail.

There is only one problem with these electrification strategies

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 07 Nov 2021 01:50:00 GMT

… it’s their “Business as Usual” foundation.

2021–11–07: Pinephone Pro – USB, Type-C, OTG, DP-Alt mode, Charger,… – some success :)

megi's PinePhone Development Log — Sun, 07 Nov 2021 01:00:00 GMT

Parchis: the reason why I am a developer

GeekWare - Daniel Pecos Martínez — Sat, 06 Nov 2021 22:58:00 GMT

A while back, digging into an old DO NOT DELETE backup folder in one of my old hard drives, I stumped upon a folder called parchis, and for a moment I held my breath until I found some *.bas files there!

So what?

Well, I just found the lost source code of my very first project, the one I coded while learning about variables, procedures and loops, the one that made me realize that this is the kind of stuff I wanted to do as my professional career and hobby.

The one that made me love coding.

But before explaining further, let me explain what the Parchís game is about, and for that I’m going to borrow the definition from the Wikipedia:

Parchís is a Spanish board game of the Cross and Circle family. It is an adaptation of the Indian game Pachisi. Parchís was a very popular game in Spain at one point as well as in Europe and north Morocco - specifically Tangiers and Tetouan, and it is still popular specially among adults and seniors. Since it uses dice, Parchís is not usually regarded as an abstract strategy game like checkers or chess. It does not depend entirely on luck either, since the four pawns under a player’s command demand some sort of strategy.

So here I am, 17 years old me, learning how to code from a book I borrowed from the library about QuickBasic. Yep, back then Internet was not a thing yet (at least not much yet in Spain), so that was how we used to browse and learn new topics, by going to a library. And, oh did I enjoy it.

I still don’t know what exactly hooked me up, but there I was writing this weird mixed of English & Spanish that the computer was able to understand, using this archaic environment in MS-DOS. Maybe it was being able to finally understand how games were made, or the beauty of taking a problem and plan and design a solution, and not only that, taking care of the solution being elegant and simple. Pure beauty.

But anyway, after a few weeks (if I remember correctly), I managed to put something together that resembled the game I was aiming for and that kind of worked. To be honest, trying to read the code today is quite painful, and hard to keep all the logic and structure in your head, which makes me even prouder of being able of making it work being so poorly written.

This is how the program looks like:

And no, the screenshot is not from back then, but from a couple of days ago, when I found the source code. I wanted to check if it was possible to still run that code, so I started digging about modern quick basic interpreters, and found some, but at the end the best solution was to use dosbox, a MS-DOS emulator and installing QuickBasic 4.5 to open the software.

And lo and behold, it worked!

(BTW, I use Arch, and this is running on Arch Linux)

I’ve put all the code together in a Github repository, where I’ve fixed some MS-DOS encoding issues and cleaned a little bit the code, but I’ve kept all the original comments, bugs and code smells that I wrote back then. There are some instructions on how to get it running, just in case you are interested on running old QuickBasic programs.

I don’t want to change a single bit of it, but I want to be able to look at it from time to time and remember how all of this started.

If you’d like to have a look, feel free, although I doubt you’ll find any value on the code itself, it’s ugly, poorly written, anb barely works, but to me :-)

https://github.com/dpecos/parchis

And if you’d like to actually see how I got it up and running, here you have a small screencast of the process and gameplay:

Last but not least, I have to say that it was my friend Alfredo who actually introduced me to software development, showing me a couple of small programs he was developing to learn QuickBasic, and triggering my curiosity about it, leading towards this project.

So basically all of this is his fault. Thx for that Alfredo ;-)

What are Dev Containers, What are GitHub Codespaces, and Why Should You Care? A Practical Intro

Mike Kasberg's Blog — Sat, 06 Nov 2021 19:30:00 GMT

GitHub Codespaces made waves when GitHub introduced the feature several months ago. Codespaces promises a lot. It solves configuration headaches, it creates disposable environments, it can provide better performance than the laptop you’re working on, and it can better enable remote collaboration. I was interested to dip my toes in the water and see what it’s like to work with Codespaces, but that can be a little tricky to do if you don’t have access to a GitHub org that’s paying for Codespaces. Fortunately, as it turns out, there’s a very accessible alternative that’s free and easy to try, and it’s a great way to get a taste of what the Codespaces experience might be like.

VS Code has a feature called Dev Containers, and it works almost exactly like GitHub Codespaces, but it runs locally in Docker rather than in the cloud. Of course, this means you don’t get the performance benefits that are possible with Codespaces, but most of the other benefits still apply. In particular, a dev container provides a disposable development environment with all the right tools ready to go. In fact, GitHub Codespaces use the same devcontainer.json file a local dev container would use – a Codespace is really just a dev container in the cloud. More recently, IntelliJ and other IDEs have also added dev container support.

What is a Dev Container?

To really understand dev containers (or Codespaces) and the benefits they provide, I think it’s important to understand the problem they’re trying to solve. Just for fun, I maintain a small open source project called Script Seed. It isn’t anything terribly exciting – it’s just a collection of sample “Hello World” scripts in various scripting languages. But because that project uses 12+ different scripting languages, it really shows the usefulness of a dev container. I don’t have all 12+ languages from that project installed on my laptop, so I use Docker to test it.

Even before dev containers were popular, I included a Dockerfile and a Makefile in Script Seed. The Dockerfile was an Ubuntu container that had all the supported scripting languages installed, so you could test any script in the project. And the Makefile provided a user-friendly syntax for running the container. This Dockerfile was, essentially, a dev container. It provided anyone who wanted to develop Script Seed a pre-configured, disposable workspace inside the container with all the tools needed to develop and run the project. A true VS Code dev container takes that same principle to the next level by adding some conventions, improving the experience, and integrating tightly with the IDE.

Setting up a VS Code Dev Container (or Codespace)

Recently, I took that same project (Script Seed) and migrated my old homegrown Dockerfile to a true VS Code dev container. In setting up this small example project with a dev container, I learned some things along the way that I’ll be able to apply when setting up larger projects with dev containers or Codespaces. I think a small project like this that’s easy to understand provides a perfect playground to learn about dev containers. Here’s the PR where I added the dev container to Script Seed. I’ll walk you through what I did, which will be great to see if you want to learn how to set up your own dev container or Codespace on your own project.

In the first commit, I add a Dockerfile and devcontainer.json to the to the .devcontainer folder and I add a bin/server script. The bin/server script is just a helper copied from my old docker entrypoint, so let’s focus on the other two files. The best way to begin setting up a dev container is to let VS Code generate the files for you by selecting Remote-Containers: Add Development Container Configuration Files… from the command palette, as documented in Create a development container. If there’s already a Dockerfile in your project, you’ll be prompted to start from that Dockerfile or from a predefined configuration definition. Although I first tried to use my existing Dockerfile, I went back and did the process again using a predefined container, and had a better first-time experience with the predefined container. The VS Code team has done a really nice job setting up these predefined containers, so I’d recommend trying one of them so you can understand all the benefits they include before trying to build your own container. In particular, these dev containers provide a non-root user, provide a nice Bash prompt, and install many tools you might want.

Starting from the predefined Ubuntu dev container, I installed all the languages I needed (as I had in my other Dockerfile) and was ready to go! I made a couple light customizations to devcontainer.json including naming my container and forwarding a port for the HTTP server to run on.

In the second commit in my PR, I update my old Makefile to use the new dev container. Primarily, I want to use make test to run tests in CI (with GitHub Actions), but this is also a nice pattern to allow anyone who isn’t using VS Code to interact with the dev container. Finally, in the third commit, I update my documentation to describe the new dev container.

Once the dev container is set up, using it is easy. Just open the project in VS Code with the Remote Containers extension installed, and it will prompt you to re-open the project inside the container. (You can try this yourself by cloning Script Seed if you want.) The first run will be a little slow as it pulls and builds the Docker container. Subsequent starts will be faster as long as the Docker image doesn’t need to be rebuilt, and there are some advanced options I haven’t played with yet to improve startup time by pre-building containers. Inside the IDE, you’ll be able to interact with tools installed in the container, including opening a terminal in the IDE that will get you to a shell in the container. You can specify extensions to install in your devcontainer.json file and those extensions will be automatically installed when you open the container in the IDE. You can do things like run and debug code inside the container using tools from the IDE. If your project includes a web server, you can run it inside the container (in my case, the bin/server script) and open the site in a browser on the host computer via a forwarded port.

Customization

One thing you might be wondering about is customization. If you’re working primarily inside the container, how do you interact with your usual tools like git, ssh, or vim, and how do you customize those tools if the config for the container is shared? Fortunately, VS Code has already thought about this and has some good solutions for dev containers. For example, there’s a mechanism to share Git credentials with the container either with the git credentials helper or with SSH agent forwarding. VS Code Dev Containers also provides a way to customize the container with your own dotfiles. I haven’t played with this too much yet, but I’ve started modifying my own dotfiles to work better with this kind of setup. Perhaps I’ll write more about that experience later.

In summary, dev containers provide a ton of value and are relatively easy to set up – particularly if you’re already familiar with Docker. Normally, a lot of the friction you experience when you open a new project for the first time is getting your development environment set up. You might need several languages, compilers, and tools installed on your system, and you might need new extensions installed in your IDE. That could take hours to set up, and using a dev container removes a lot of that friction and provides a great way to manage a consistent, disposable development environment that’s more inviting for new users. And GitHub Codespaces take it to the next level by hosting that environment in the cloud.

You Should Write to Your UK MP about Corruption Now

duncanlock.net — Sat, 06 Nov 2021 17:45:00 GMT

I’ve just emailed my UK MP to complain about government corruption & Owen Patterson. You should do the same. As always, we need as many people as possible to speak out.

This is what I wrote, if anyone wants some ideas:

<My Name>
<My Address>
<My Email Address>

Dear <MP Name>,

Like many others, I am angered - but not very surprised - by the latest revelations of government corruption, revealed by the corrupt behaviour of Owen Patterson.

I do have to admit to being slightly surprised that the government’s reaction to this was naked collusion and a blatant attempt to water down the rules & standards!

The existing Committee on Standards needs to be strengthened, not weakened by the introduction of another Conservative-dominated committee overseeing its work.

The government needs to adopt the recommendations of numerous independent bodies - such as the anti-corruption watchdog the Committee on Standards in Public Life - that rules about MPs’ behaviour should be tightened.

As a voter in your constituency, this issue is important to me - and the governments actions so far are not good enough!

Yours sincerely,

<Your Name>

This form makes it easy to email your MP about this issue: https://do.opendemocracy.net/MPstandards. It’ll take 5 minutes - and it’s better than just being annoyed.

Getting in Touch with your MP

Getting in touch with your MP generally is much easier than it used to be. You can use www.theyworkforyou.com to find your UK MP and see what they’ve been up to - as well as sending them a message.

Does your MP represent you?: https://www.theyworkforyou.com/
Write to your politicians, national or local, for free: https://www.writetothem.com/

Things I Read - Trie, Raft

Ye Lin's Random stuff — Sat, 06 Nov 2021 10:10:41 GMT

Things that I stumbled on a rainy day.

Understanding how Trie (Data Structure) works
Understanding how Raft (Distributed Consensus algorithm) works

Trie

I have heard/seen it but not in detail until now. Saw someone posted about it on HN and makes me curious.

I also found a post about Trie written in Python by Googling. I like the clear explanation with diagrams and put the code on github.

Some takeaways

Cool data structure to build autocomplete / prefix tree.
Two main functions
- Insert
- Query

Raft

Another item on my endless “things to read” list is this particular Understanding Distributed Consensus link.

The posthuman dog

Nicola Iarocci — Sat, 06 Nov 2021 09:05:25 GMT

Flo, our dog, spent her whole fifteen-years long life with us. Many, many times after she passed away, I wondered if she lived a happy dog life or not. In The posthuman dog (Aeon), Jessica Pierce poses a fascinating question that somehow helps find answers to my troubling question:

If humans were to disappear from the face of the Earth, what might dogs become? And would they be better off without us?

Is Eve still maintained?

Nicola Iarocci — Sat, 06 Nov 2021 08:05:25 GMT

Tonight someone opened a ticket on the Eve repository. I jotted down a quick reply and was about to hit the Comment button when I thought a more articulated reply was in order. I also want it published on my website.

So the question is:

Is Eve still maintained?

My reply goes like this:

Hello, yes, Eve is in ‘maintenance mode’, as I call it. I don’t actively develop new features anymore. Still, I am more than willing to code-review and merge relevant pull requests, especially so if they are bug fixes or improvements over existing features.

Dangers of OpenCSV beans streaming

Random Notes by agilob — Sat, 06 Nov 2021 02:00:00 GMT

That one time I have to extract, transform and load a massive CSV file into a bunch of database entities and it was kinda slow… The class had position based CSV bindings, loaded into beans and streamed from a pretty big CSV file (+10Gb): public class CSVUserEntry { @CsvBindByPosition(position = 0) private String userId; @CsvBindByPosition(position = 1) private String username; @CsvBindByPosition(position = 2) private String deviceId; @CsvBindByPosition(position = 3) private String keyAlias; @CsvBindByPosition(position = 4) private String passcodeKeyAlias; @CsvBindByPosition(position = 5) private String confirmationId; } Then I opened the stream in generic way, with a simple and fluent interfaces of Java stream API:

Never buy a flying car...

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 06 Nov 2021 01:50:00 GMT

Unless it’s not for you only.

Postgres Query Rewrite

Home on mfashby.net — Fri, 05 Nov 2021 13:57:12 GMT

Localized Programming Languages

Weblog on marginalia.nu — Fri, 05 Nov 2021 02:00:00 GMT

This is an reply to a series of posts on anglo-centrism in programming languages that have been floating around in Gemini lately.

gemini://nytpu.com/gemlog/2021-10-31.gmi
gemini://alsd.eu/en/2021-11-04-thoughts-anglocentrism-cs.gmi

Around thirty years ago I was a kid with a computer. I learned to program quite a few years before I learned English. I also used DOS without understanding English. I knew what to type to do things, but I didn’t know what the words meant. I could start programs, I’d play in QBASIC, write small programs and amusements. To me “PRINT” was the word that made text appear on the screen. I learned years later the word meant something in English. To show you what my child eyes saw, I think rot13 does convey the experience quite well;

Why buy toys if you can 3d print them?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 05 Nov 2021 01:50:00 GMT

Here are a few reasons

2021–11–05: Pinephone Pro – USB, Type-C, OTG, DP-Alt mode, Charger,…

megi's PinePhone Development Log — Fri, 05 Nov 2021 01:00:00 GMT

Keeping Gemini Difficult

Weblog on marginalia.nu — Thu, 04 Nov 2021 02:00:00 GMT

This is a response to the post “Making Gemini Easy” over on ~tomasino, and the title is a bit tongue-in-cheek haha-but-no-really.

gemini://tilde.team/~tomasino/journal/20211103-making-gemini-easy.gmi

I think the idea that we need to shield the users from how technology works is a terrible, terrible mistake. It disempowers the users, and concentrates power in the hands of a technological elite, and that divide is only going to grow.

We already have an alarming number of people working with computers, and some may even be programmers, that simply do not understand how computers work. Their only concept of a computer is the user interface on the screen. The rest is unintelligible wizardry. Nobody has told them, it’s been deemed too complicated, nothing for them to worry their little heads about.

Lessons learnt from exposing my server to the internet

Imran's Blog — Thu, 04 Nov 2021 02:00:00 GMT

My previous post was about how I setup a SFTP server for my dad. This post is going to be about all the things I learned about exposing your server to the internet.

About a week after I exposed my server via SFTP to the world, I wanted to log in to my server to do some maintenance. Much to my shock, my server was unresponsive to my regular ssh commands.

This is when the panic started setting in. My server lives in my home close to my desktop computer. The server appeared to be still on, the lights were still on and the fans were still spinning. I resisted the urge to reboot it and try to ssh again, as I wanted to know why I could not ssh in.

I temporarily repurposed my second monitor on my desktop and plugged in an extra keyboard that I had. The standard login prompt greeted me, which meant the server was not dead and I was able to login.

I checked the status of the ssh process with a systemctl status sshd and (un)surprisingly it's dead. The next step was to figure out why it's dead. This started with looking at the logs via journalctl -b -t sshd

Loads of similar lines
---
Oct 08 11:04:33 server sshd[245607]: Unable to negotiate with 86.111.187.162 port 36589: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245610]: Unable to negotiate with 86.111.187.162 port 36596: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245618]: Unable to negotiate with 86.111.187.162 port 36605: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:33 server sshd[245622]: Unable to negotiate with 86.111.187.162 port 36616: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245628]: Unable to negotiate with 86.111.187.162 port 36623: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245632]: Unable to negotiate with 86.111.187.162 port 36634: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245636]: Unable to negotiate with 86.111.187.162 port 36647: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>
Oct 08 11:04:34 server sshd[245640]: Unable to negotiate with 86.111.187.162 port 36652: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie>

Oh.

Looked like folks over the internet were having a blast trying to break into the server. The first step I needed to take was to free up my monitor and keyboard and go back to the comfort of working from my desktop. A simple systemctl start sshd was all that was needed.

# Remediation

It's clear that I need to harden my ssh setup. I already the following setup:

root login disabled
password logins disabled (with the exception being the SFTP user)

From the logs, looks like no one has even figured out username for the ftp user and the password for it's a long generated one via Bitwarden. The first thing I did was reduce MaxAuthTries to 1. The reasoning was pretty simple, either you know how to get in or you don't.

The next steps were to consult the arch wiki!

Turned out I had already taken most of the necessary steps. I was just missing a rate limiting/blacklisting solution. I settled on using sshguard combined with ufw. I verified with my dad that he could still login and do his work.

Fingers crossed, this would be the last time I would have to deal with something like this.

# Results

My server has been up and running ever since with no issue to ssh. I have also installed netdata and made it accessible solely over the local network. This is just to give me a birds eye view of what is going on with my server without having to ssh in.

Taking a quick look at logs shows me that sshguard is working as intended

Nov 03 14:27:28 server sshguard[1006]: Attack from "176.111.173.238" on service SSH with danger 10.
Nov 03 14:27:28 server sshguard[1006]: Blocking "176.111.173.238/32" forever (REDACTED)

Facebook ends face recognition. What does that REALLY mean?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 04 Nov 2021 01:50:00 GMT

Nothing definitive, in the long run.

Robot Olympics

Random Bits — Wed, 03 Nov 2021 21:28:00 GMT

I'm not a staunch nationalist, nor am I an athlete of any description, yet I find myself in the same quadrennial boat as most others - the Olympics is an entertaining event to watch. Good-spirited competition drives people to push themselves to the extremes and work together to

Debloat MIUI

Blog d'Anri — Wed, 03 Nov 2021 14:12:00 GMT

Préambule important
Dernière mise à jour 15 mai 2024¹
Cette liste est très ancienne, nombreux programmes ne sont même plus livrés par défaut dans la ROM. La priorité est ce qui est écrit dans UAB car certains programmes peuvent être désormais important pour le bon fonctionnement de l’OS.

Avant, j’utilisais XiaomiADBFastbootTools pour supprimer les applications, mais maintenant j’utilise UAB qui met directement dans l’application ce que fait les applications et les tris par ordre de dangerosité. Je laisse cette liste au cas où, mais il faut préférer, je pense, les informations dans l’application, car elles sont susceptibles d’être mise à jour plus souvent.

Analytics (com.miui.analytics)

Télémétrie

Android Accessibility Suite (com.google.android.marvin.talkback)

Application pour aider les personnes handicapées à utiliser le téléphone

Backup (com.miui.backup)

Outil de sauvegarde pour une restauration future, comme un changement de téléphone

Browser (com.android.browser)

Navigateur de base

Cleaner (com.miui.cleanmaster)

Outil de "nettoyage"

Facebook (com.facebook.appmanager / com.facebook.services / com.facebook.system)

Facebook

Feedback (com.miui.bugreport)

Outil pour informer les devs d'un bug dans le logiciel

Games (com.xiaomi.glgm)

Equivalent du "Google Play Games"

GetApps (com.xiaomi.mipicks)

Marché d'applications

Gmail (com.google.android.gm)

Client mail Gmail

Google App (com.google.android.googlequicksearchbox)

C'est la barre de recherche Google sur l'écran

Google Assistant (com.google.android.apps.googleassistant)

"Ok Google"

Google Chrome (com.android.chrome)

Navigateur Google Chrome

Google Lens (com.google.ar.lens)

Google Lens "vous permet de rechercher ce que vous voyez, d'aller plus vite et de mieux comprendre le monde qui vous entoure."

Google One (com.google.android.apps.subscriptions.red)

Service payant Google One

Joyose (com.xiaomi.joyose)

Télémétrie

MSA (com.miui.msa.global)

Officiellement en rapport avec l'écran d'accueil, la météo et les thèmes, mais la désinstaller ne fait rien, a part retirer des pubs

Market Feedback Agent (com.google.android.feedback)

Outil pour informer les devs de Google d'un bug dans le PlayStore

Mi Browser (com.mi.globalbrowser)

Navigateur web Xiaomi

Mi Cloud (com.miui.cloudbackup / com.miui.cloudservice / com.miui.cloudservice.sysbase / com.miui.micloudsync)

Le Cloud de Xiaomi

Mi Credit (com.xiaomi.payment)

Service de paiement qui ne fonctionne qu'en Chine

Mi Drop (com.xiaomi.midrop)

Nouvellement appellé "Share Me", application de P2P pour transferer des fichiers

Mi File Manager (com.mi.android.globalFileexplorer)

Gestionnaire de fichier Xiaomi

Mi Pay (com.mipay.wallet.in)

Fonctionne pour l'application "Mi Credit"

MiPlayClient (com.xiaomi.miplay_client)

Application Mi Play

Music (com.miui.player)

Application de musique Xiaomi, utile pour mettre une sonnerie custom (la sonnerie reste même avec l'application désinstallée)

Notes (com.miui.notes)

Application de notes Xiaomi

Package Installer (com.miui.global.packageinstaller)

Ne sert à rien

PartnerBookmarks (com.android.providers.partnerbookmarks)

Des favoris par défaut des sponsors de Xiaomi dans leur navigateur

Quick Apps (com.miui.hybrid)

Télémétrie

Quick Ball (com.miui.touchassistant)

Accès rapide aux applications

Recorder (com.android.soundrecorder)

Application pour enregister des vocaux

Services & feedback (com.miui.miservice)

Application pour donner un retour sur Miui

SoterService (com.tencent.soter.soterserver)

Service de paiement chinois

VsimCore (com.miui.vsimcore)

Carte Sim Virtuel, utile seulement en Inde et en Chine

Weather (com.miui.weather2)

Application météo

Yellow Pages (com.miui.yellowpage)

Application de pages jaune, mais que en Chine

Youtube (com.google.android.youtube)

Application Youtube

En désinstallant ces applications, je ne perds, à ma connaissance, aucune fonctionnalité.

Sources : devcondition.com / technastic.com / Playstore

Une autre personne ayant fait le même job

Notice de dépréciation. ↩

Overclock son écran

Blog d'Anri — Wed, 03 Nov 2021 14:12:00 GMT

Avec une carte graphique AMD

Rendez-vous dans le panneau de configuration AMD

Clic droit sur le bureau → AMD Radeon Software

Dans la catégorie Affichage, crée une nouvelle résolution.

Augmente le taux de rafraîchissement et teste-la pour voir si l’écran le supporte

Augmente de 5 en 5, quand l’écran n’affiche plus rien, réduit de 1 en 1 pour trouver la fréquence parfaite. Quand tu l’as trouvé, vérifie que l’écran prend bien toute la surface disponible, si ce n’est pas le cas, baisse encore de 1 ou 2 Hz.

Pour appliquer la nouvelle fréquence, rends-toi dans Paramètres → Système → Affichage
Descends pour te rendre dans Paramètres d’affichage avancés

Sélectionne ta fréquence dans la section Fréquence de rafraîssement

Avec une carte graphique NVIDIA

Rendez-vous dans le panneau de configuration NVIDIA

Clic droit sur le bureau → Panneau de configuration NVIDIA

Dans la catégorie Affichage et sous-catégorie Changer la résolution et clique sur Personnaliser…

Si Personnaliser… n’est pas disponible, sélectionner votre résolution dans la catégorie PC de la liste déroulante puis appliquez les modifications, le bouton devrait se débloquer.

Crée une nouvelle résolution

Augmente le taux de rafraîchissement et teste-la pour voir si l’écran le supporte

Augmente de 5 en 5, quand l’écran n’affiche plus rien, réduit de 1 en 1 pour trouver la fréquence parfaite. Quand tu l’as trouvé, vérifie que l’écran prend bien toute la surface disponible, si ce n’est pas le cas, baisse encore de 1 ou 2 Hz.

Pour appliquer la nouvelle fréquence, sélectionne la résolution fraîchement
Crée et applique les modifications.

Bric-à-brac de trucs cool

Blog d'Anri — Wed, 03 Nov 2021 14:12:00 GMT

Dernière mise-à-jour le 31 août 2022¹
Cette liste n’est pas maintenue à jour.

Merci à
Discord de Capet
Discord de Wanda
sarkraï

Cette page est une liste de truc que je trouve cool, n’hésite pas à me contacter si tu veux discuter.

Tables des matières

Quel système d’exploitation choisir ?
Web
- Le navigateur
- Extensions
Apps PC
- Benchmarks, stress tests et infos PC
- Équivalents à la suite d’Adobe
Apps Android
Liens utiles
- Documentation
- Sites
Jeux
Contact & Infos

Quel système d’exploitation choisir ?

Si tu n’as pas peur de bidouiller 2-3 trucs, alors Linux est fait pour toi. Il y a plein de distributions Linux, la plus simple pour commencer est sûrement Ubuntu. Et ne t’inquiètes pas, si tu veux jouer c’est largement faisable sous Linux.
Si tu veux juste jouer aux jeux vidéo ou avoir une utilisation classique d’un ordinateur sans prise de tête, alors utiliser Windows est le choix le plus logique. Le meilleur Windows est la dernière mise à jour de Windows, pas une version customisée, LTSB, ou autres. Si tu souhaites “optimiser” ton pc, tu peux le faire en suivant ce tuto.
Si tu ne veux pas t’embêter à mettre en place ton OS et juste que ça fonctionne, peu importe le prix, alors ton choix logique serait certainement de te tourner vers l’environnement Apple, ils font de bons produits qui sont simples à utiliser, cependant tu es quand même plus limité dans tes actions et dans ce que tu peux faire dans ce genre d’environnement.

Web

Le navigateur

Chaque navigateur se vaut, chacun ses goûts. Il n’y a pas d’écart de performances entre les navigateurs, ce qui diffère, c’est les fonctionnalités qu’ils amènent, choisis celui qui te plaît le plus, sans écouter les gens qui disent que tel ou tel navigateur n’est pas bien parce qu’il n’est pas optimisé, c’est la plupart du temps des bêtises. Ce spreadsheet récapitule quelques fonctionnalités des navigateurs.

Extensions

uBlock Origin (inutile avec Brave)
- Le seul bloqueur de pub qui bloque toutes les pubs.
Dark Reader
- Ajoute un thème sombre aux sites.
Bitwarden
- Gestionnaire de mot de passe gratuit, jolie et open source. L’application permet aussi de gérer la double authentification. Possibilité d’héberger soi-même le serveur.
SponsorBlock
- Bloque les pubs intégrés aux vidéos YouTube.
MAL-Sync
- Ajout un suivi automatique des épisodes sur MyAnimeList (fonctionne avec plein de sites).
BetterTTV
- Ajoute, entre autres, des emojis sur Twitch
Keepa
- Affiche l’historique des prix sur Amazon
Return YouTube Dislike
- Affiche le compteur des pouces rouges dans youtube (une estimation)
Twitch Adblock
- Bloque les pubs sur Twitch

Apps PC

Capture d’écran
- Le nouvel outil par défaut dans Microsoft (une option dans les paramètres permet d’utiliser la touche Impr écran) et affiche une notification vers un mini Paint pour annoter la capture.
- ShareX permet, entre autres, d’uploader directement la capture sur internet. Et a aussi son propre mini Paint.
Lecteur Audio/Vidéo
- Il y a VLC et si une vidéo ne se lit pas, MPC-HC la lira sûrement.
Désinstaller des programmes
- Geek Uninstaller est l’outil parfait pour désinstaller proprement les programmes (ceux du Microsoft Store aussi). Il y a aussi Revo Uninstaller qui a une belle interface, mais qui est plus lourd.
IDE
- VS Code en désactivant la télémétrie de Microsoft (ou directement VS Codium).
- Vim
- Emacs
Suite bureautique
- Celle de Microsoft est payante, mais activable avec ce programme.
- LibreOffice est l’alternative aux standards de Microsoft.
- OnlyOffice gère les standards de Microsoft.
Éditeur d’images
- GIMP
- Site qui ressemble à Photoshop
- Avoir Photoshop illégalement
Partage d’écran
- Parsec est stable, facile et gratuit.
- AnyDesk fonctionne mieux que TeamViewer et n’est gratuit.
Détection de doublons
- Czkawka permet de gérer les fichiers, images et musiques en double.
- Panaustik et AntiDupl.NET sont des alternatives pour les images, mais elles fonctionnent moins rapidement pour le même résultat.
- dupeGuru existe aussi, mais pas encore testé.
Montage vidéo
- Vegas Pro (payant)
- Adobe Premiere (payant).
- Kdenlive gratuit, mais moins bien que ses concurrents
- DaVinci Resolve gratuit et fonctionne très bien (cette chaîne fait de bons tutos, celle-là aussi).
- Les logiciels Avidemux et LosslessCut permettent de faire du montage plus basique.
Émulation
- Pour émuler Android la meilleure chose est d’utiliser Android x86 avec VirtualBox ou, si tu as Windows 10 Pro/Entreprise avec Hyper-V.
Téléchargeur de vidéos YouTube
- Youtube Downloader
- Youtube-DLC
- Youtube-DLG
Prise de notes
- Marktext
  - Éditeur Markdown simple.
- Joplin
  - Prise de notes et liste “To Do” avec synchronisation intégrée.
- Zettlr
  - Éditeur Markdown plus avancé que Marktext.
- CodiMD (site internet)
  - Instance du CNRS pour tous les étudiants de France.
- Obsidian
  - Support de plusieurs plugins, ce qui le rend très puissant.
EarTrumpet
- Contrôleur de volume pour Windows.
FossCord (en dev)
- Une tentative de créer une version d’un Discord compatible avec celles officielles, mais qui serait plus sécurisé et auto-hébergeable.
Spotify sans pubs
- Un guide est disponible pour ne pas avoir de pubs, le programme utilisé est BlockTheSpot
ImageGlass
- Il lit presque toutes les images rapidement, cependant l’application Windows fait déjà bien le travail (alternative de Irfanview (avec tous les plugins)).
7-zip
- Pour les archives, mieux que WinRAR.
Everything Lite
- Recherche Windows amélioré.
WizTree
- Affiche quels sont les dossiers et fichiers qui prennent de la place sur ton PC
  - Alternatives (moins bien) :
    - WinDirStat (lent)
    - Treesize (interface Windows 8)
qBittorrent
- Pour télécharger des torrents.
LAN Party VPN
- Permets de jouer en multijoueur sur des jeux qui ne supportent qu’un mode local. Fonctionne avec Discord.
  - Alternatives moins bien :
    - Radmin VPN (installe un autre programme inutile)
    - Hamachi (vieux)
Audacity
- Pour enregistrer sa voix et y apporter des modifications.
OBS Studio
- Enregistrer son écran, Stream, Shadowplay -like intégré (tampon de relecture).
Handbrake
- Permets de convertir des vidéos.
DiskGenius
- Gère les problèmes avec les disques de stockage.
Lively
- Fond d’écran animé et gratuit.
Elmocut
- Limiteur de connexion pour les périphériques connectés au même réseau
Gnirehtet
- Partage de connexion du PC vers le téléphone.
DS4Windows
- Ut1iliser une manette PS4 sur PC.
scrcpy
- Partage d’écran Android sur PC avec ou sans câble.
Sourcetree
- Interface pour Git (fun fact, si tu as python-tk d’installer, tu peux utiliser gitk comme interface).
Winmerge
- Différence entre deux fichiers.

Benchmarks, stress tests et infos PC

Ce site permet de faire un rapide diagnostic de ton PC.

ATTO Disk Benchmark
- Benchmark de stockage avec informations très détaillées.
CrystalDiskInfo et CrystalDiskMark
- Benchmark plus concis et infos sur le stockage.
Cinebench
- Benchmark et stresstest du CPU (trouvable aussi sur le Microsoft Store).
AIDA64
- Benchmark et stresstest pour le CPU et la RAM.
Prime95
- Stress test du CPU.
CPU-Z et GPU-Z
- Infos détaillées sur le CPU et le GPU.
FurMark
- Stress test du GPU.
Unigine, des benchmarks et stresstests pour le GPU.
3DMark
- Benchmark et stresstest pour le CPU et le GPU.
Asus realbench
- Stresstest et benchmark pour le GPU et le CPU.
MSI Afterburner
- Pour OC ta Carte graphique Nvidia et modifier la courbe de tes ventilos.
DriverStore Explorer
- Explorateur de drivers installés sur le PC
CapFrameX
- Permets de faire des beaux graphiques pour des benchmarks

Équivalents à la suite d’Adobe

Source : r/linuxmasterrace

Apps Android

J’ai un téléphone Xiaomi Mi Note 10 lite, non rooté que j’ai debloat, la liste des applications que j’ai désinstallées est ici.

Firefox / Brave
- Rapide sur téléphone, et comprend les mêmes extensions que j’utilise sur PC (uBlock, DarkReader, BitWarden). Brave fait la même chose.
Spotify
- J’utilise Spotify, tu peux utiliser xManager ou suivez ce guide, ou même utiliser mon application qui va chercher l’apk sur Mobilism.
YouTube ReVanced
- YouTube sans pubs, etc. En attendant que le manager sorte, tu peux lire ma configuration ici.
QKSMS (version payante disponible sur F-Droid)
- Application SMS.
K-9 Mail
- Remplaçant de Gmail, configuration facile et expérience similaire à cette dernière.
TachiyomiSY
- Fork de Tachiyomi pour lire des mangas (extensions Tachiyomi que j’utilise : FR Scan, Komga avec mon instance, Little Garden, Mangas Origines, mcreader.net, NineManga, Scan FR, Webtoons.com).
Geometric Weather
- Application météo (Météo France est une des source disponible).
Bitwarden
- Gestionnaire de mot de passe.
SchildiChat
- Un mélange entre WhatsApp et Discord, décentralisé, qui fonctionne avec le protocole Matrix (fork d’Element).
Droid-ify
- Alternatif à F-Droid, simple et pas trop moche. J’ai listé les repos que j’utilise ici.
Infinity
- Client Reddit.
Material Files
- Bonne application pour les fichiers, simple et belle.
dvd ou Youtube-DL-Android ou Seal
- Une application pour télécharger des vidéos sur téléphone (intégration avec ReVanced).
Notally
- Pour prendre des notes.
Etar
- Alternative au Google Agenda.
Aves
- Application galerie.
Calculator-inator
- Application calculatrice.
Compass
- Application boussole.
Findroid
- Alternative au client officiel de Jellyfin.
GMaps WV
- Google Maps dans une application pour limiter au maximum de donner des informations à Google.
Hacki
- Client Hacker News.
Koler
- Application téléphone
Signal
- Messagerie sécurisée
Transdrone
- Client pour mon instance Transmission.

Liens utiles

Documentation

S’informer sur la tech
- Il y a ce gdoc qui répertorie plusieurs sites et chaînes YouTube fiables pour s’informer. Pour lire les flux RSS proposés par les sites il y a Inoreader sur PC et téléphone. Perso j’utilise Selfoss avec une instance que j’héberge moi-même.
Vérification fichiers systèmes en cas de problèmes avec Windows :
- Windows + R → cmd → CTRL + MAJ + ENTRÉE (pour lancer cmd en admin)
- Ensuite dans l’ordre et une par une les commandes suivantes (ça peut prendre un peu de temps)
```
DISM /Online /Cleanup-Image /CheckHealth
DISM /Online /Cleanup-Image /ScanHealth
DISM /Online /Cleanup-Image /RestoreHealth
sfc /scannow
```
Monter son PC avec processeur Intel ou AMD (Playlist Topachat pour les généralités).
Pour réinstaller proprement sa carte graphique quand il y a quelque chose qui pose problème.
Récapitulatifs des différences entre les types d’écrans :

Source : mrhightech.fr

Source : The Tech Chap

Les switchs mécaniques (dans la vidéo, tu peux ignorer toute la partie axée sur la vitesse d’appui sur la touche, car dans la vraie vie, tu ne sentiras jamais la différence).

Source : materiel.net

Visual C++ Redistributable Runtimes All-in-One pour installer tous les composants nécessaires à plein de programmes après une installation de Windows.
Explication sur les spécifications des câbles Ethernet
Source : julienth37#7125 du serveur Discord de Wanda.
Tuto pour bien nettoyer son tapis de souris
Fenêtres flottantes sous MIUI
- Si, comme moi, vous utilisez YouTube Vanced et que vous avez dû désactiver les optimisations MIUI, vous avez aussi perdu la fonctionnalité de fenêtre flottante. Pour récupérer cette fonctionnalité, vous devez activer la fonctionnalité Activer les fenêtres en forme libre (Activer le support pour les fenêtres expérimentales en forme libre) sous la catégorie Applications.
Liste des intégrations de GOG Galaxy 2.0 et de ce qui est prévu pour l’avenir
Active le mode Insider dans Windows sans renseigner de compte Microsoft
Explication de comment est nommé en CPU Intel
Source : Heavenlot#0002 du serveur de l’entraide informatique de Capet.
Sens du ventilateur

Sites

Vie privée
- La vie privée, c’est important, ce site fait une bonne liste des applications cool pour votre vie privée, à checker mais ne pas tout prendre au pied de la lettre, ça donne juste des idées.
Savoir faire le bon choix
- Boitier, Écran, Watercooling
- Tapis de souris
- Souris (comparaison 3D)
- Antivirus (tldr: Windows Defender est très bon).
Alternative à Google Drive & Co
- Cryptpad
  - Alternative complète au Google Drive (limité à 1 Go sur cette instance mais possibilité d’utiliser sans compte).
- EtherCalc
  - Collaboration quasi instantanée (il faut valider les cellules) mais facile à utiliser.
Tracker de prix
- Pour des produits généraux et idealo.
- Pour composants informatiques
Dessin
- Excalidraw
  - Collaborative, lien partageable, possibilité d’exportation en PNG/JPG et presets.
- En ASCII
- Diagrammes
- Style “tableau blanc”
Comparateurs
- Pour des produits généraux
- Spécialisé sur les téléphones portables
- Comparer deux images (juxtaposeJS)
Répéteur de “morceau” dans une chanson
Hébergeur de fichiers basés sur l’ancien projet de Mozilla.
Connaître la distance parfaite théorique entre vous et un écran
Site pour avoir les couleurs principales d’une image
Site qui récapitule les bons et mauvais points des conditions d’utilisation d’un site.
Grosse documentation sur le réseau
Grosse documentation sur le SQL
Grosse documentation sur différents outils pour les devs
Procédure pour trouver le problème d’un PC quand il ne démarre pas
Site pour avoir un bel affichage pour une recette de cuisine
Compresser les PDF avec l’outil d’Adobe
Compresser image dans plusieurs formats.
Partage d’écran dans le navigateur

Jeux

Minecraft

J’ai fait une vidéo qui fait un peu le tour de ma configuration Minecraft si les détails t’intéressent.

MultiMC est un client pour mieux gérer les versions et les mods, il y a aussi PolyMC qui est globalement mieux.
Fabric est l’outil qui permet de charger les Mods dans minecraft (remplaçant de Forge). Il y a aussi Quilt qui est prometteur.
Mods
- Modrinth
  - Annuaire de mods.
- Optimisateurs
  - Dynamic FPS met les FPS à 1 quand le jeu est en second plan et a 0 quand le jeu est réduit.
  - EntityCulling cache les entités qui sont invisibles au joueur (de dos par exemple).
  - LazyDFU permet d’améliorer le temps de chargement du jeu.
  - Lithium optimise aussi le jeu globalement (côté serveur, utile aussi en solo) ou Krypton optimise la partie réseau entre client et serveur (compatible en théorie avec Lithium mais Krypton est meilleur dans la partie réseau, au choix).
  - MemoryLeakFix résoud des fuites de mémoires.
  - Sodium optimise grandement le jeu en sa globalité (Sodium Extra pour plus d’options).
  - Phosphor (ou Starlight) optimise la partie lumière (soit l’un soit l’autre).
  - Hydrogen optimise la gestion de la RAM pour les partie avec beaucoup de mods.
- Shaders (moteurs)
  - Iris, intègre Sodium.
- EMI affiche un grand inventaire avec toutes les recettes.
- Bobby permet d’avoir une grande render distance en multijoueur
- Fabrishot permet de prendre des captures d’écrans en très haute définition.
- WI Zoom permet de zoomer dans le jeu
- Origins
  - Permets de sélectionner une classe au démarrage d’un monde pour avoir des aptitudes uniques.
Cubiomes viewer
- Affiche la map avec ses biomes et l’emplacement des lieux importants, atlernative à Amidst.
Chunky
- Permets de regénérer des chunks en les supprimant des fichiers du jeu.
Shaders (packs)

Return To Castle Wolfenstein

IORTCW

League of Legends

League Director

Émulation

Switch

Yuzu
- Émulateur Switch

Vidéo - Jeux site 1 - Jeux site 2 - keys dir - nand dir (source) - Firmware (inutile) - Mods GitHub - Compatibilité des jeux

Multi Plateforme

Lemuroid
- Émule plusieurs consoles portables sur Android.

Contact & Infos

Ma config (2019)

Site : www.mylloon.fr
Réseaux : /networks

Notice de dépréciation. ↩

Tech rants: PC-s use way too much power in 2021

./techtipsy — Wed, 03 Nov 2021 07:00:00 GMT

Quick rant about the state of PC-s in 2021 regarding power and resource usage.

Drag and drop image upload

Home on Pinecoder blog — Wed, 03 Nov 2021 02:00:00 GMT

While planning drag and drop image support in Chronicles, I had some confusion about handling files once dropped. There are many tutorials out there but they often provide a “copy this code” approach and offer little insight. Many recommended using base64 encoding but that did not seem right. The more general problem of file uploading through a browser is one I occassionally return to, so I wanted a conceptual deep dive I could use as a foundation.

Impressions of the Framework Laptop

Max Leiter — Wed, 03 Nov 2021 02:00:00 GMT

I received my Framework Laptop DIY edition a few days ago and have been setting it up software-wise. I have it running Manjaro Linux with the Sway window manager. This is my first time using a Wayland desktop, and oh has it been a learning experience. This post will be largely about the hardware, but I plan to make one about the software once I'm more happy with my setup.

Here's what I got:

Framework Laptop DIY Edition
16GB (2 x 8GB) DDR4-3200
Intel® Wi-Fi 6E AX210 No vPro
1TB - WD_BLACK™ SN850 NVMe™
USB-A Expansion Card
USB-C Expansion Card x2
HDMI Expansion Card

I bought everything from the Framework store (at a slight markup of ~$80 for the RAM, SSD, and wifi card), but I'm happy to support them and for the convenience of knowing it would work properly was well worth it.

The Good

Well crafted body. I was not expecting a premium build quality from a startup trying to create an upgradeable laptop, but wow they excelled.
It's really light. It's just 2.8 pounds, compared to my 13" Macbook Pro's 3.02. You wouldn't think that's a big difference, and maybe you wouldn't notice, but I sure did.
A solid keyboard. Most non-Mac laptops I've tried have awful keyboards. It's not as rigid as an Apple non-butterfly board, but it's close. Definitly usable.
Modular expansion ports (via pass-through to USB-C) are a fantastic idea. I can't wait for 3rd party ones to be released.
Hardware switches for the camera and microphone.
Beautiful glossy screen. I haven't used it extensively outside, but it's worked well enough for me (especially at full brightness).

The Bad

There are only 3 preset levels of keyboard backlight you can switch between: off, on, and half. Perhaps my Mac spoiled me, but I'm used to having more fine-tuned control over it.
Installing the wifi card was difficult. It took me twice as long to install the WiFi as it did to setup the rest of the computer. They acknowledge this in their installation instructions, but othes seem to also be having problems:
Removing expansion ports requires far more force than installing them. Feels a little awkward.
12 seconds of pressing the power button to hard reboot. Not that big of a deal, but definitely feels way too long.
Why does the screen need to swing open 185°?

The Ugly

This isn't necessarily due to the hardware, but the battery life on Linux currently requires a little finessing to be reasonable. I'm sitting at ~4h of moderate use after tweaking numerous configs and following multiple different sets of instructions from around the internet. This isn't good enough for me to take on campus for class, but it's great for doing work around the house.
Lacks the ability to set charging thresholds. It's being worked on, but at the moment it doesn't seem possible.
A strange 2256x1504 resolution. Why couldn't they have chosen something more standard? Although I think my issues with this are mostly due to Linux / Wayland. Maybe I'll get used to 3:2.
It requires a little extra care to open the laptop without sliding it (perhaps due to its light weight).

In conclusion

I'll let my Tweet from a few days ago sum it up:

![Image of the Framework setup guide comments showing 6 people also having difficulty](/blog/framework/twitter.png)

Git loses untracked files during "stash" if there are conficts

Databases & Life — Wed, 03 Nov 2021 02:00:00 GMT

Git had a bug from version 2.33 onwards, fixed in 2.35. (Thanks to Richard Laager’s comment for the exact version numbers.)

“Stashing” then “unstashing” untracked files (new files you’ve just created) loses those new files, if any of the existing tracked files has a conflict.

Let me explain.

Unstructured data are BAD for everybody

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 03 Nov 2021 01:50:00 GMT

Not just for “Materials Scientists”!

Nanoblomo 2021 #02 - Fun with VQGAN+CLIP

infrequent oscillations — Wed, 03 Nov 2021 00:09:36 GMT

In which we stylise some Moomins.

Note 59

Qubyte Codes — Tue, 02 Nov 2021 19:44:50 GMT

A caterpillar.

The Bullet Journal Method by Ryder Carroll

trv.rs — Tue, 02 Nov 2021 12:48:00 GMT

It feels weird to be finally reading this years after I've been "doing" the bullet journal method. My first bullet journal is dated from 2016, so I guess five years?! It's been a very important part of my productivity/focus journey, but it means that I have a very strange lens with which to approach this book.

Firstly: this is a very handsome book. It's a love letter to journaling and paper. Reads great in the hand and lovely on the shelf.

It's typeset nicely, and it's fun how it uses the physical medium of the book to reinforce how the system works. The page explaining the different Collections is itself a Collection!

The contraints and delights of paper itself is a major theme of the book and the system. Ryder reminds the reader that the act of handwriting has recall and retention powers. The slowness of handwriting vs. typing forces the note-taker to employ advanced synthesization and summarization during capture. Notes taken by hand tend to be more concise and readable. This is an important part of my Creativity System, and I'm inspired to spend more time "thinking on paper."

The physical restrictions of paper force you to adopt intentional practices of planning and review. You can't copy/paste or search, so you have to be thoughtful about what you take with you and where you put it. I'm begin to think that The Bullet Journal Method is probably the best start for anyone who is struggling with the practices required to build a disciplined/organized life. I'm pretty sure that the intentional practice of Rapid Logging and Migration were the first things that let me defeat the Focus Boulder. As much as I love Obsidian and Things and other digital tools, it's the rituals and practices that Migration taught me that keep the Evan-machine going.

Another huge benefit of the Bullet Journal practice is the question "what just happened?" By externalizing our thoughts into words, we are able to separate our identity from the thoughts and feelings we are experiencing. This helps us be more objective and take every thought captive. To externalize your experiences and impulses creates this space between input and reaction that Ryder beautifully illustrates here:

By journaling we become witnesses to our own story from the outside, an invaluable practice for learning impulse management and fostering gratefulness.

Section 3, "The Practice," is a succinct (if overly zen/stoic) summary of basic healthy productivity practices. There is certainly some Self-Centered Self Help¹ problems in there, but it's manageable.

Action Items

While I like Matt Ragland's split journal method, I think that I'm going to have all collections on the "organized" side, and only scrawl on the unorganized side. Before, I was only putting Daily/Weekly collections on the "organized" side of the journal, and everything else was on the "notes" side… but I have realized that I want everything with a title on the organization side. If I write something in the random side worth linking to, it's worth restating in the appropriate Bullet Journal collection.

I'll also take his advice to start a new journal every year instead of limping the journal along past the new year. I'm excited to get back into it.

I think that next year I want to recommit to living in the Bullet Journal first. Thanks to Things and fun Drafts automations, I find myself dictating to my phone more than writing in my notebook. I live at a keyboard, so digital is closer at hand. I am convinced that by Rapid Logging more in the notebook first, I'll be practicing a little more reflection and presence that will help me be grateful and stay focused on what matters. Choosing to restate some of my tasks in my daily log will help me avoid over-committing to tasks by looking at the never-ending digital backlog, which Ryder says better:

Set intentions, not expectations.

Great read, even if you have no intention of using your journal as a Bullet Journal.

Self-help books generally have a core of self centeredness. Mentally replacing the "you are worth it" and "pursue your dreams" narratives with "He is worthy" and "I am a steward of talents" redeems this for a believer. ↩

Debezium and Friends – Conference Talks 2021

Gunnar Morling — Tue, 02 Nov 2021 11:50:00 GMT

Table of Contents Don’t Fear Outdated Caches – Change Data Capture to the Rescue! Change Data Streaming Patterns in Distributed Systems Analyzing Real-time Order Deliveries using CDC with Debezium and Pinot Dissecting our Legacy: The Strangler Fig Pattern with Apache Kafka, Debezium and MongoDB Bonus: Debezium at the Trino Community Broadcast Learning More If you love to attend conferences around the world without actually leaving the comfort of your house, 2021 certainly was (and is!

A Polemic Against Internet Arguments

Weblog on marginalia.nu — Tue, 02 Nov 2021 02:00:00 GMT

I want you to consider for a moment all the human lifetime wasted in ideological stalemates on the Internet, all that energy, all that anger and frustration. Imagine if you take even a fraction of that time, and put it to creating something constructive instead, learning skills, doing anything meaningful.

It boggles the mind, doesn’t it? It must amount to entire human lifetimes every week.

Ideology, or in a wider sense, ethics, is all about what should be done. How we should live. These aren’t statements about the world, but opinions about what the world should look like. They aren’t true or false, and any argument against them always boils down to “I disagree!”. Arguing about ethical systems is some of the least constructive things a human can do. It is more pointless than masturbation, which at least feels good for a moment.

Reclaim space on disk

dystroy|Canop / blog — Tue, 02 Nov 2021 02:00:00 GMT

As my disks tend to be quite messy with time, I had to devise ways to reclaim space, which means both spotting big directories and files, and to deal with the thousands of duplicates you can't delete one per one.

Here's my toolbox for cleaning my disks on linux in 3 major steps:

Skills, stories, and software every dev should know - 123dev #44

Justin Garrison's Homepage — Tue, 02 Nov 2021 02:00:00 GMT

The newsletter passed 250 subscribers and 200 Twitter followers this week and I appreciate you all. ❤️ I love reading your replies and seeing you all share issues too. If you have suggestions for future topics feel free to hit reply and send them my way.

It’s okay to not be okay

Avik Das — Tue, 02 Nov 2021 02:00:00 GMT

What I’m about to say applies to everybody, but with Movember and my own experience as a man in mind, I hope my words will at least be useful to the men who read this. It’s okay to not be okay.

Never trash READABLE data!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 02 Nov 2021 01:50:00 GMT

NEVER. I mean it.

A ‘git up’ alias that works for any default branch

duncanlock.net — Tue, 02 Nov 2021 01:46:11 GMT

Traditionally, git’s default branch was called master - but this isn’t always true, it’s just a default - and one that’s currently undergoing change. GitHub & GitLab have changed their defaults to main and the git project itself is in the process of doing the same.

I have a longstanding git alias - git up - that I use to pull my local checkout up to date with the remote. It’s a shorthand for the following:

Change branch to the default branch
Fetch all changes from the remote, removing any local branches that have been deleted on the remote
Pull any changes from the default remote branch

This used to hardcode master as the default branch, but this was getting less and less reliable, so I figured out a better version that checks the repo’s default branch and uses that:

Add these lines to your ~/.gitconfig file

[alias]
  head-branch = !"git remote show origin | grep 'HEAD branch' | cut -d' ' -f5"
  up = !"git switch $(git head-branch) && git fetch --all --prune --progress && git pull"

References

Visualize dependencies of structs in Go

David Schlachter — Tue, 02 Nov 2021 00:05:00 GMT

Tool to help understand relationships between custom struct types in large Go projects.

Nanoblomo 2021 #01 - A new PC

infrequent oscillations — Mon, 01 Nov 2021 23:22:36 GMT

In which we install a new PC.

What to Expect When You're Emailing

Brightball Articles — Mon, 01 Nov 2021 18:37:21 GMT

During Carolina Code Conf, I gave what is probably the most entertaining presentation ever given about email. We walk through how to protect, raise and nuture the domain for my pirate themed gym, Slimmer Ye Timbers.

Video: Developing a Layered Email Security Strategy Webinar

Brightball Articles — Mon, 01 Nov 2021 18:21:27 GMT

In this joint webinar between dmarcian, GreatHorn and Inspired eLearning I had the opportunity to share an overview of SPF, DKIM and DMARC as part of a layered email security strategy.

The manager-elevator

João Alves — Mon, 01 Nov 2021 14:57:11 GMT

I first read about "servant leadership" in a job offer when I joined mytaxi (FREE NOW) back in 2018. They were looking for a "servant leader." I bit the bullet and changed my career from software development to engineering management.

Before joining the company, I searched for books that more experienced folks in the industry recommended. I ended up buying "Turn the ship around" and "Maverick!". These are two masterpieces of enlightenment when it comes to driving teams and organizations. The two most relevant concepts I captured were:

Lead by example. One cannot change the culture through elaborated pitches and preaching how best practices™ will improve your team members' lives.
Create leaders, not followers. Some charismatic people tend to think its job is about deciding things or, worse, giving orders. Empowering people to make decisions themselves and creating other leaders unlocks organizational potential. Hence it's more durable.

They felt natural to me because that's how I always have seen myself. Someone that pushes other people to try new approaches and guides them through these changes.

Glorified coaching

Lately, I've been hearing a lot about "servant leadership." Usually, with a negative connotation. When digging into the root causes of this mock, I've found patterns.

Sometimes, people confuse it with cheerleading. It's great to bring your team coffee or ask for food delivery because they're solving a significant problem. But wouldn't it be better if you did that while communicating to other teams, managing blockers, or helping answer less-important inquires on Slack? I admit cheerleading is essential as a leader, but it's not all.

I've also seen the laissez-faire style. These folks are utterly reactive to what their team does, and their best quality is to say, "If you need help, I'm here." That's not how it should work. If you see someone drowning, you don't wait for them to ask for help! Likewise, if you see your team getting stuck or something odd, taking immediate action — it may be as simple as listening to someone's complaints — is crucial to create natural trust in leaders.

Finally, the "I don't know about the details" type. I get it. It's challenging to be on top of everything. If you're managing a complex domain or scope, it may be daunting. However, these details give you superpowers. They allow you to have more productive conversations with your peers, directors, other folks in the industry. They allow you to understand what is exactly happening when another person approaches you for escalation.

While this post is not a rant against agile coaches, you'll likely become a glorified coach rather than an engineering leader if you tick most of the above boxes.

Engineering Leadership starts with Engineering

I won't be the person that says that an engineering leader keeps coding. Sometimes it's desirable; sometimes, it's not. It also heavily depends on how many other things you're doing. Nevertheless, it would help if you kept your hax sharpened to lead any discipline effectively. Reading technical blogs, whitepapers, books, reviewing design documents, or doing code reviews may help too.

Going to the trenches and getting your hands dirty gives you a clearer perspective on how things work in practice versus how they're supposed. You start seeing more trade-offs. You can challenge the status-quo and make your teams reflect better.

Engineering leaders are aware of where the industry is going. They aren't afraid of sponsoring strategic decisions that might affect one or more teams. These decisions go from migrating from a technology that's becoming a liability — e.g., Adobe Flash — to try a different approach to DevOps — e.g., Platforms as products —, and so on. That's engineering leaders' scope. They help businesses deliver value through creating and shaping organizations. If they run away from it or let teams and systems rot, it's their responsibility.

If we take off engineering from engineering leadership, what's left?

From strategy to tactics and back

I love to listen to a good story. I love to feel inspired by leaders that shape an industry or the world. The other day, I was amazed by how Ferran Adriá explained innovation and how he was a forward thinker more than 30 years ago. But it worked way better because Ferran was passionate about cooking. He was a cook himself.

Being strategic and visionary is a great way to drive and motivate your teams. At the same time, if you don't take the time to be on the ground and support your team members, you'll likely miss crucial details to support your strategy.

That's why I believe the best leaders are in the manager-elevator spectrum. Someone that's able to be in a tactical meeting — even if it's to listen — and one hour later is writing the strategy of the whole department or presenting it to the CTO.

Let's press the button and ride this elevator up and down.

— João

Review of Macbook Pro 14 (M1 Max, 32GPU Cores, 64GB Memory) - Performance, Windows, Parallels and more

outcoldman — Mon, 01 Nov 2021 02:00:00 GMT

I have been using Macbook Pro with M1 since the release of the MBP 13" M1 in 2020. And I am huge fan of the new Apple CPU - M1. Time to time I need to do some work on my 2019 16" MBP with Intel Processor (all maxed out), so I keep experience the difference between using those two.

On MBP with M1 I can definitely see that the macOS works way more smooth. No more glitches or frozen screens for a few seconds, less heat, less time I hear the fans work. And a lot of work you can do on just single battery charge.

Which games to change?

D13V — Mon, 01 Nov 2021 01:11:56 GMT

Show me a gambler and I’ll show you a loser, show me a hero, and I’ll show you a corpse.

Mario Puzo, author of The Godfather

Last week we began the “How to be a game changer” series. Today, we’ll start answering the question - “Which games to change”, by defining the two main types of games we don’t want to play - zero-sum games and win-lose games.

Zero sum games

One way to simplify looking at games, is to look at the possible outcomes. There is a whole branch of Economics called Game Theory which goes into detail about categorizing and analyzing games based on these outcomes. In game theory, a game is simply an interaction between multiple parties, where they can choose a behavior, and after the interaction any participant in a game may end up winning something or losing something else, resulting in a combination of wins and losses. Game Theory often simplifies that to a simple number representing how much value, or utility, or money you could win or lose depending on the outcome, and while it is a limiting way to look at it, it is still a helpful way get started.

One of the most obvious games which I assert everyone should try to change are the so called “zero-sum” games. Here zero-sum refers to the sum of outcomes of all players after the game, compared to the situation before the game. These are the games where as much as some players win, the other players collectively lose. For example if you and I bet $5 on the result of a coin toss, one of us will have slightly more money, but together our overall money has changed by zero. In zero-sum games the total amount of value and utility across all participants does not increase as a result of the game. Some people individually may be better off, but the group as a whole is not better off.

If we go back to our traffic example from the previous post, driving on a road with congestion, which has two lanes merging together into a single lane (for example, due to construction) may be thought of as a zero sum game, where only certain number of vehicles can go through a bottleneck at a time. If one car goes through, no other car is going through. There is a limit of total utility, is proportional to the maximum safe and allowed speed with which cars merge in. Often, cars will alternate from each lane as a way to coordinate, but some drivers might feel competitive, or in a hurry, and try to cut in front of others. And they may succeed, and save seconds for themselves. But by disrupting the natural rhythm, they create confusion, congestion, and waste time for everyone behind them. Other drivers may have to hit the breaks, or press their horn and reach out for verbal insults. It’s a pity. More cars will get piled further behind. The driver who cut might think they’re winning this time, but they probably suffered a time loss behind someone else earlier who cut as well, but they only observed it indirectly.

Furthermore, with zero-sum games we’re only creating more conflict among ourselves. In my opinion, all zero-sum games negatively affect the participants as a whole. Therefore they are in practice even worse, they are “negative-sum games” and the community as a whole is worse off after playing them.

There’s no good reason to play these games.

Win-lose games

Win-lose games are similar to zero-sum games as in some people will benefit, and others will lose as an outcome of the game, but they differ as in the win may sometimes seem to be larger than the loss. But in practice, if a game ends up generating any positive value in the process, it is likely that this value is redistributed, and nobody is losing and the game becomes win-win?

As a result, a win-lose game in real life will usually be a negative-sum game, as some more savvy players may figure out how to “rig” the game, and exploit others. The possibility to lose something from the game is a big red flag ⛳️. If you encounter any interaction where you may lose something - think why the interaction is that way. Is it rigged to transfer value out of you to others?

Some win-lose games with uncertain outcomes are more tempting to play than games which obviously are zero-sum as we hope that we may win, and we may win big. For example, if we participate in a pyramid scheme, we will likely come up ahead as long as we’re not the last ones. If we knowingly participate in a such a scheme - shame on us. But even with good intentions, we often can get tricked into participating into pyramid-like scheme, where at the end the “biggest fools” will lose something. Some pyramids can have a really elaborate disguise, and subtle giveaways.

But one giveaway which we can always use to detect games we don’t want to play is the ability for anyone in the game to “lose” in some way. Games which genuinely create value tend to have a way to redistribute this value over time, if people play them again and again, and they also incentivize people to keep creating value. And that presents another way to tell these games apart - if a game is played long term, repeatedly, it is more likely to be genuinely creating value for all participants, even if that value gets distributed unevenly. There are some exceptions such as the “scapegoat” game, which we’ll discuss in a future post.

By opting out of short term games, or win-lose games, we will not only do better for ourselves, we will make it less likely others get exploited.

Next up

Next week we’ll start discussing the tools we have available to change these games. In the meantime, don’t believe anything I wrote above, and ask yourself - “When anyone want to play a zero-sum or win-lose game?”. You may also ask yourself, “Which of my interactions are win-lose?”

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Updated Rss Feed

ℤ→ℤ — Sun, 31 Oct 2021 21:47:17 GMT

I’ve updated the RSS feed to include the full content of the articles rather than just the summary.

Controlling Elgato Key Light with Hammerspoon

trv.rs — Sun, 31 Oct 2021 19:03:00 GMT

Here's a quick automation that I've been hacking idly on for months: having my Elgato Key Air Light be automatically controlled from my Mac.

While perusing hammerspoon's API, I noticed that it has a Bonjour API and wound up writing this snippet:

hs.bonjour.networkServices(function(t) print(hs.inspect(t)) end)

When I ran this in the Hammerspoon console, I saw a list of findable services including the Elgato light... so it was only a matter of time before I found the right commands. Thanks to this blog post I found some working TypeScript that I was eventually able to adapt after I had grokked more of Hammerspoon's Bonjour implementation.

(It turns out that the Elgato Key runs an unsecured web server that takes HTTP requests in a very simple fashion. It's kind of refreshing. Most of the code just finds the IP and port of the light, the rest sends the HTTP requests with a different JSON payload for on or off.)

I use this on and off switch when I enter and leave my meeting Headspace.

I quickly realized that my quick and dirty hack would turn on and off all lights on the same network with your laptop... so I decided to restrict the code to only when I'm plugged into my trusty CalDigit TS3+ dock:

IsDocked = function()
  return hs.fnutils.some(hs.usb.attachedDevices(), function(device)
    return device.productName == "CalDigit Thunderbolt 3 Audio"
  end)
end

That function returns true if my laptop is docked. In my meeting space automation I use something like:

if IsDocked() then
  spoon.ElgatoKey:off()
end

I pulled my naive implementation into a baby spoon library, so you can hack on it if you want.

Blog posts for founders (and early employees)

Separate Concerns — Sun, 31 Oct 2021 16:45:00 GMT

Over the years, I have accumulated links to blog posts I intend to re-read every time find myself in a position to influence the early stage of a company.

I thought it might be helpful to someone if I published part of that list here. I will avoid links which are too technical and focus on links that can be interest non-technical founders as well.

Note that I haven’t included books. There are many good books for founders, but if you must read only one make it The Four Steps to the Epiphany.

If this isn’t enough for you, you’re always welcome to explore my Pinboard account.

My own posts

Let’s get started by mentioning posts from this blog which transcribe the opinions of two influential people, which I share (on those topics).

Werner Vogels on Skills, where the CTO of Amazon.com describes the qualities he is looking for in leaders at Amazon, in particular Ownership and the willingness to get their hands dirty doing operations.
Hacker Founders, where Paul Graham, founder of Y Combinator, explains how programmers have an edge as company founders, including as CEOs.

Titles are Toxic

Michael Lopp (Rands) on issues with giving people titles. Personally I have never liked titles, which can be a source of conflicts and bad decisions. I have also often held titles that didn’t really correspond to the job I did - which was almost always multi-faceted. Yet, titles are a proxy used by people who don’t know you to understand what you do at work.

Like Rands I don’t have a good solution to that. I write some blog posts explaining what I do in more details that a title ever could, for people who care.

In any case I always suggest avoiding giving some titles to people too early, including to founders: having a “technical co-founder” instead of a CTO will make things easier if you grow fast and need to hire someone later on.

Creating a Software Company? 9 Decisions You Have to Make

A list of questions to ask yourself when founding a software company, by Michael Marsiglia, CEO of Atomic Object. It can also be a good idea to ask them before joining one.

The 40% rule

A simple but powerful rule of thumb for SaaS companies by Brad Feld, relayed by Fred Wilson:

annual revenue growth rate + operating margin should equal 40%

Why it’s nice to compete against a large, profitable company

Jason Cohen explains how a startup can disrupt a large competitor by making a part of their business unprofitable and forcing them to abandon it. It’s not as simple as this summary makes it sound, and it’s not the Innovator’s Dilemma. Read the article :)

Product design at GitHub

Kyle Neath on how, in the early days of GitHub, everyone was a “product designer”. Many successful SaaS products started with the same concept:

We only hire smart people we trust to make our product better. We don’t have managers dictating what to work on. We don’t require executive signoff to ship features. Executives, system administrators, developers, and designers concieve, ship, and remove features alike.

Actually, I think I have never seen a company ship good product fast without doing some version of this.

Joel Spolsky’s Strategy Letters

If you don’t know that series of blog posts, you are missing something. They are super insightful essays by a multi-successful founder about building companies.

Ben and Jerry’s vs. Amazon - How to decide between slow profitable growth or fast VC-funded growth.
Chicken and Egg Problems - Understanding the Chicken and Egg problem and some ways to solve it.
Let Me Go Back! - On barriers to entry, and how letting customers understand they can easily move out of your product (maybe back to what they used before) is important.
Bloatware and the 80/20 Myth - 80% of software users use 20% of features, so you can win the market with just 20% of the features, right? No, because not everyone uses the same 20%! I can’t state how often I meet founders who don’t understand this.
Strategy Letter V - How companies have an advantage in commoditizing their complements, and how this can explain some investments in Open Source software.

(There is at least a sixth strategy letter, but it is less interesting IMO.)

Ev William’s 10 rules for Web Startups

This post used to be on Ev William’s blog which went offline some time ago, thankfully I mirror articles that interest me that much.

Your company’s most valuable resource

Just a tweet, but an important one, by William Pietri:

Your company’s most valuable resource is people giving a shit. Ask yourself: does your system encourage or discourage that?

Wants, needs, and chasm-crossing

A recent post by Avery Pennarun, Tailscale CEO. It shows how the best strategy to acquire customers is often to solve a single Want - something their current solution doesn’t provide - and then to resolve all their Needs (blockers) so they can actually flock to your product instead of attempting to implement any more Wants.

Financial modeling for startups

A slide deck by Rodrigo Sepulveda with a simple and effective approach to business models and PnLs for startups.

Startup Therapy: Ten questions to ask yourself every month

Jason Cohen (again) shares his alternative to business plans: a list of 10 questions to help you figure out where you are and where to go next.

The Small Teams Manifesto

Jason Gorman says small teams win because:

the probability that a project fails increases rapidly with size;
small teams interact more effectively with the customer and/or end users;
small teams are easier to build (hiring good people is hard).

Innovating with purposeful anger

One of the very interesting tools using anger as a driver to innovate is the “You Got Fired!” game.

Get a bunch of managers in a business unit and role-play that their whole team is fired. Indicate also that as a cockup from legal their non-compete clause isn’t applicable.

Then, give them an hour to build a competitor startup that will take revenge on their company on a “That’ll teach them!” mode.

How To Decide What To Build

Daniel Gross (a founder I have been following since his first YC startup, Greplin) on ways to decide what company to build on a personal level. I wish he had published that before my last attempt.

Qu’est-ce qu’un projet entrepreneurial viable?

A blog post in French by Philippe Silberzahn. A viable project stems from an individual acting on an idea and convincing people.

Book Review: King and Emperor, A New Life of Charlemagne

Nicola Iarocci — Sun, 31 Oct 2021 08:05:25 GMT

In this scholarly biography by Janet L. Nelson, Charlemagne is stripped back from the years of mythologizing and idolizing that have occurred since his death. He is presented as distinctly human, and this book is the first time I have felt I could reasonably understand Charlemagne as the man he was, not the man he has since been painted to be. Moreover, Nelson is excellent in her discussions of Charlemagne’s wives and their roles. For example, the commonplace assumption that Fastrada was a cruel person is questioned, and she emerges as a capable companion and queen instead.

Italia III - Roma

Avy Faingezicht — Sun, 31 Oct 2021 03:00:00 GMT

As the history nerd that I am, I could explore Rome and its historical sites for years, and not get enough of it. We tried to squeeze as much as we could into our couple of days there, and realized that we were surrounded by tourists everywhere we went. Next time, I'll make an effort to find at least a few places where the locals hang out, too. _This is the third in a series shared throughout Fall 2021. When you're done here, make sure to also check out parts one, two, four, and five._

Perfecting GLFW for Zig, and finding lurking undefined behavior that went unnoticed for 6+ years

Hexops' devlog — Sun, 31 Oct 2021 03:00:00 GMT

Today, I am announcing mach-glfw: Ziggified GLFW bindings with 100% API coverage, zero-fuss installation, cross compilation, and more.

Building Mach for everyone

If Mach engine only benefits people interested in using that engine, and not the broader Zig (and even gamedev) community I would consider that a total failure.

Whether you’re interested in using all of Mach, just some of it with your own engine / project, or just the tools/ideas we develop in the future (with Unity, Unreal, etc.), I truly aim to produce something that benefits you.

Mach is in super early stages, I’ve spent the last four months perfecting a Zig interface to GLFW, and making no-fuss installation and cross-compilation a reality. Today, you can benefit from that work too.

Building GLFW for every platform

Just zig and git, that’s the idea. The GLFW C code is compiled with zig, and the build.zig file automatically uses git to clone (a very minimal set of) system dependencies for you (X11 libraries, etc.)

No installing apt packages. No dealing with missing header errors. It should just work out-of-the-box, and for every platform:

Today, this works for GLFW itself. Cross-compilation of OpenGL and Vulkan apps is not yet fully functional. We’re working on it, though.

Perfecting GLFW for Zig

Aside from platform support, mach-glfw now has:

100% API coverage of GLFW v3.3.4. Every function, type, constant, etc. has been wrapped in a ziggified API.
130+ tests, with CI testing Linux, Windows, Mac (x86 and M1/ARM) and cross-compilation between them.

You might be asking: why Zig bindings, when Zig can interface directly with C? Ziggified bindings to GLFW get us:

Errors as zig errors instead of via a callback function.
Enums: always know what value a GLFW function can accept as everything is strictly typed. And use the nice Zig syntax to access enums, like window.getKey(.escape) instead of c.glfwGetKey(window, c.GLFW_KEY_ESCAPE)
Slices instead of C pointers and lengths.
packed structs represent bit masks, so you can use if (joystick.down and joystick.right) instead of & | etc. bitwise operators.
true and false instead of c.GLFW_TRUE and c.GLFW_FALSE.
Generics: use window.hint instead of glfwWindowHint, glfwWindowHintString, etc.
Methods, e.g. my_window.hint(...) instead of glfwWindowHint(my_window, ...)

Explicit error handling solves a real problem

GLFW traditionally passes errors to the user via a callback. This can make errors easy to ignore, as well as difficult to correlate and handle effectively at the time of the function invocation.

We translated a a Vulkan example to mach-glfw, which you can try for yourself today:

After porting it, we found that the example was crashing with a NoWindowContext error. Strange?

As it turns out, we had found a small bug in the vulkan-zig example code, it was calling glfwSwapBuffers which is not needed for Vulkan. The error went unnoticed because it’s easy to miss errors with GLFW’s error callback handling style. But with mach-glfw, it was an explicit error you have to handle e.g. via try glfw.swapBuffers() - we literally couldn’t miss it.

Finding lurking undefined behavior in 6+ year old GLFW code

One particularly frustrating issue was tracking down why the last part of the GLFW API we needed to wrap for 100% coverage, the glfwSetWindowIcon function, was crashing:

Test [76/135] Window.test "setIcon"... Illegal instruction at address 0x2cee09
upstream/glfw/src/x11_window.c:0:0: 0x2cee09 in _glfwPlatformSetWindowIcon (/mach/glfw/upstream/glfw/src/x11_window.c)
upstream/glfw/src/window.c:511:5: 0x2de484 in glfwSetWindowIcon (/mach/glfw/upstream/glfw/src/window.c)
_glfwPlatformSetWindowIcon(window, count, images);
^
/mach/glfw/src/Window.zig:508:28: 0x23a083 in Window.test "setIcon" (test)
c.glfwSetWindowIcon(self.handle, @intCast(c_int, im.len), &tmp[0]);
^
/usr/local/bin/lib/std/special/test_runner.zig:77:28: 0x25a0d1 in std.special.main (test)
} else test_fn.func();
^
/usr/local/bin/lib/std/start.zig:517:22: 0x2896bc in std.start.callMain (test)
root.main();
^
/usr/local/bin/lib/std/start.zig:469:12: 0x25c117 in std.start.callMainWithArgs (test)
return @call(.{ .modifier = .always_inline }, callMain, .{});
^
/usr/local/bin/lib/std/start.zig:434:12: 0x25bec2 in std.start.main (test)
return @call(.{ .modifier = .always_inline }, callMainWithArgs, .{ @intCast(usize, c_argc), c_argv, envp });
^
???:?:?: 0x7f4b7c3280b2 in ??? (???)

That’s odd? Illegal instruction at address 0x2cee09 - are we corrupting the stack somehow? Is this a Zig compiler bug?

Running in lldb didn’t help with shining any light on the problem, either:

After poking around at the stack, checking all pointers and lengths were valid, etc. I was at a loss. The mach-glfw code sure seemed valid, and yet, this crash. I managed to track the crash down to the first iteration of a loop in GLFW’s x11_window.c:

void _glfwSetWindowIconX11(_GLFWwindow* window, int count, const GLFWimage* images)
{
 if (count)
 {
 int longCount = 0;

 for (int i = 0; i < count; i++)
 longCount += 2 + images[i].width * images[i].height;

 long* icon = _glfw_calloc(longCount, sizeof(long));
 long* target = icon;

 for (int i = 0; i < count; i++)
 {
 *target++ = images[i].width;
 *target++ = images[i].height;

 for (int j = 0; j < images[i].width * images[i].height; j++)
 {
 // illegal instruction on first iteration?
 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
 (images[i].pixels[j * 4 + 1] << 8) |
 (images[i].pixels[j * 4 + 2] << 0) |
 (images[i].pixels[j * 4 + 3] << 24);
 }
 }
...

Reaching my limits

At this point, I feel confident in saying:

The Zig code is correct, the pointers are valid, the lengths are correct, everything’s right.
The GLFW code is pretty popular, and it’s been around for 6 years. Seems unlikely it’s a bug in GLFW?

Luckily, my brother (and reverse engineer) @Andoryuuta was available to help debug, so I pulled him in. Stepping through instructions, we could see clearly that after a bit shift we were stepping into the abyss:

* thread #1, name = 'test', stop reason = instruction step over
frame #0: 0x00000000002c6f84 test`_glfwPlatformSetWindowIcon(window=0x00000000004e53d0, count=1, images=0x00007fffec0b3000) at x11_window.c:2156:58
2153 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
2154 (images[i].pixels[j * 4 + 1] << 8) |
2155 (images[i].pixels[j * 4 + 2] << 0) |
-> 2156 (images[i].pixels[j * 4 + 3] << 24);
2157 printf("DID WE GET HERE???x\n");
2158 }
2159 }
(lldb)
Process 6516 stopped
* thread #1, name = 'test', stop reason = instruction step over
frame #0: 0x00000000002c6c21 test`_glfwPlatformSetWindowIcon(window=0x00000000004e53d0, count=1, images=0x00007fffec0b3000) at x11_window.c:0
1 //========================================================================
2 // GLFW 3.3 X11 - www.glfw.org
3 //------------------------------------------------------------------------
4 // Copyright (c) 2002-2006 Marcus Geelnard
5 // Copyright (c) 2006-2019 Camilla Löwy <elmindreda@glfw.org>
6 //
7 // This software is provided 'as-is', without any express or implied
(lldb)
Process 6516 stopped

Inspecting the binary in IDA Pro we were able to see that we were jumping into an __asm { ud1 } section (ud1 standing for “undefined instruction 1”):

It turns out that clang’s UBSan inserts these instructions as traps for when the compiler thinks there is undefined behavior occurring, such as if a pointer addition leads to an overflow. This is super interesting, but unfortunately doesn’t always give a compiler error. We got lucky and found someone else who ran into this through Google:

I believe LLVM explicitly generates a ud2 x86 instruction because “it determined” there’s undefined behavior in the C code. So first I wonder which flags you’re passing it through zig (i.e. how strict are you being with the settings?) — Abner (@AbnerCoimbre)

And indeed, compiling via zig build test -Drelease-fast (which turns off UBsan) made the crash go away. So where’s the undefined behavior?

If we squint at the code and assume all pointers, counts, and indices are correct, you might be able to spot it:

void _glfwSetWindowIconX11(_GLFWwindow* window, int count, const GLFWimage* images)
{
 ...
 long* target = icon;

 for (int i = 0; i < count; i++)
 {
 ...
 for (int j = 0; j < images[i].width * images[i].height; j++)
 {
 // illegal instruction on first iteration?
 *target++ = (images[i].pixels[j * 4 + 0] << 16) |
 (images[i].pixels[j * 4 + 1] << 8) |
 (images[i].pixels[j * 4 + 2] << 0) |
 (images[i].pixels[j * 4 + 3] << 24);
 }
 }
...

What is happening here is that:

images[i].pixels[j * 4 + 0] is returning an unsigned char (8 bits)
~~It is then being shifted left by << 16 bits. !!! That’s further than an 8-bit number can be shifted left by, so that’s UB~~
- EDIT: Actually, it turns out that’s not exactly right, it’s the << 24 that’s the cause of the UB, thanks @Maato for pointing this out and explaining in better detail than I could.

Suddenly, it all makes sense. And if we load an equal snippet of code into Godbolt we can see what is happening when we compile without UBSan / the -fsanitize=undefined flag:

Without UBsan, clang merely uses the 32-bit EAX register as an optimization. It loads the 8-bit number into the 32-bit register, and then performs the left shift. Although the shift exceeds 8 bits, it does not get truncated to zero - instead it is effectively as if the number was converted to a long (32 bits) prior to the left-shift operation.

This explains why nobody has caught this UB in GLFW yet, too: it works by accident! Just because the compiler likes to use 32-bit registers in this context.

And this change benefits all the languages out there using GLFW: glfw/glfw#1986

Defaults are critical

This code, and undefined behavior, has been in GLFW for over 6 years according to git blame.

Anybody using GLFW could have enabled UBSan in their C compiler. Anybody could have run into this same crash and debugged it in the last 6 years. But they didn’t.

In mach-glfw, we compile all of GLFW’s C code with Zig (which is also a fully functional C and C++ compiler), with UBSan enabled by default.

Only because Zig has good defaults, because it places so much emphasis on things being right out of the box, and because there is such an emphasis on having safety checks for undefined behavior - were we able to catch this undefined behavior that went unnoticed in GLFW for the last 6 years.

Thanks for reading

All key Mach engine developments will be posted here.

Follow Mach engine on GitHub, and if you like what I’m doing please consider sponsoring my work.

Building a fast SCSS-like rule expander for CSS using fuzzy parsing

Notes on software development — Sun, 31 Oct 2021 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

All you need to know about the metaverse, in two posts

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 31 Oct 2021 02:50:00 GMT

Plus another one, about a movie

OpenBSD

Home on mfashby.net — Sat, 30 Oct 2021 22:25:43 GMT

I'm offering my first Scaled Agile Class

Brightball Articles — Sat, 30 Oct 2021 17:20:19 GMT

After 20 years in software development, my frustations with watching organizations hurt themselves from bad practices finally boiled over in my article, Reality Driven Development. The response and discussion from Hacker News lead me into the work of Donald Reinertsen, who laid out the math that validated everything I was experiencing. Much of the Scaled Agile Framework is based on his work, which lead into my journey with SAFe 3 years ago. Join me on December 9th for my first class, Leading SAFe 5.1.

Lifecycle of Notes: My Implementation

trv.rs — Sat, 30 Oct 2021 11:41:00 GMT

This is a direct follow up to my previous discussion of the "strength of notes" and nurturing thought.

As with all aspects of a Creativity System, how you implement such a classification should differ based on your brain and creative goals. As a simple example: I believe that Jerry Seinfield uses two yellow legal pads… the first contains rough drafts of comedy material, and he migrates the "good ones" to a second for refinement. This is a rough classification system. In different episodes of LYT-house I'm pretty sure I have seen:

YAML frontmatter tag for epistemological status
Folders for "drafts" or "hypotheses"
Emoji in the filename like 🚧
empty nodes on the graph and #state/develop tags
completely separate forms/app/systems (e.g. Seinfield's yellow legal pads)

These are so creative and are effective as identification… you just have to decide what works with your brain and goals… and steal it. 😈

I have written about my particular inputs and outputs. In the light of this my system uses different kinds of note, (the filename and structure) to signify the "strength" of an idea. The two main kinds of notes are:

"zettel"-style filenames for infant "weak" ideas and connecting thoughts (e.g. 202110140716.md)
"Evergreen" style named notes for strong facts and strong opinions (e.g. Boundary of the Known.md)

This lifecycle kind of happened accidentally but it has stuck. In the many transitions of my system I tried 100% ZK style notes, and then migrated to Named Notes... but only for notes that had titles. This left a lot of my capture automations still creating ZK-style notes and created an organic difference between captured "ZK" style notes, and intentionally crafted evergreen Named Notes.

Zettels

Before I get a letter… I know that these aren't proper "zettelkasten" notes. They are just what I call them.

Zettels for me are the entry point into my note-taking system. Because they don't have a name, folder, or categorization they can just enter the web at any point… cheap, easy capture.

Zettels (maybe I should come up with another name!) do abide by the note-making rules: a singular, atomic idea, in my voice, interesting to me… and they usually have at least one link to something else. They sometimes aren't more complex than a single sentence.

Examples

Scripture Highlight

A single link to a verse and some comments from a sermon or personal study… usually unverified by further study:

Filename: 202108300728.md

In [[1 Kings-19#18|1 Kings 19:18]] God answers Elijah's [[Depression]] by telling him the truth: he's not alone. There are 7000 others.

God could have said "I am enough," or "don't be afraid," but instead comes with the whisper "you are not the last faithful person."

Book Note

A reflection on a book idea that I have not proven or become convinced of yet:

Filename: 202110010724.md

Bondage or enslaved is the biblical language for we commonly call addiction or addictive personalities today.

Lou Priolo. _[[Pleasing People]]_. P & R Publishing, 2006. p52

Errata

Commentary on another note that didn't belong to that note's singular idea:

Filename: 202109301007.md

Constraint-based programming as described by Bret Victor[^bv] is basically programming by [[Outcomes Over Outputs]].

[^bv]: https://youtu.be/8pTEmbeENF4?t=654

Label

A "label" on the "line" between two other notes. These are often made while exploring links… and I find one that I feel like future-me would need a little more context for how these relate or to connect a few things:

Filename: 202110152107.md

[[ESV/1 Chronicles/1 Chronicles-19#12|1 Chronicles 19:12]] reflects the idea of [[Distributed Command]], [[Simple]], and [[Cover and Move]] from [[Extreme Ownership by Jocko Willink]]

There's a lot more that gets added as zettels:

sermon outlines
dream logs
random facts or user insights about the products in my day-job

Zettels make up the bulk of what gets captured into my system. It's rare for me to enter a "strong" idea right out of the gate. Those "strong" ideas, (usually ones I've been holding for a long time) get entered as Named Notes.

Named Notes

Notes containing a strong opinion or strong fact become a Named Note or Evergreen Note. My ideas here are so influenced by Andy Mutaschak's writing on Evergreen notes that you may as well open tab to his wild web of wisdom: Evergreen Notes - Andy Mutaschak.

A Named Note still follows the rules of atomicity, interestingness, and links… but it has a filename and title that is usable in a sentence and meaningful for me.

Example

Filename: Journaling externalizes thoughts and feelings.md

# Journaling externalizes thoughts and feelings

By externalizing the thoughts into words, we are able to separate our identity from the thoughts and feelings we are experiencing. This helps us be more objective and take every thought captive. 

You can see this pattern in the psalms. So many of them start with an experience or feeling that the psalmist describes… often in pathetic or visceral detail. Then the writer calms, and sets his will to worship and honor the Lord. 

---

- [[Simultaneous terror and peace]]. 
- [[booknotes/The Bullet Journal Method by Ryder Carroll]] talks a lot about becoming a witness to your own experience through rapid logging.

Choosing a good name is important. Names that could just be the name of a #tag (i.e. "Marriage", "Agile Methodology") are too generalized. These usually become hub notes or Maps of Content (as Nick Milo styles them.) If the name is usable in a sentence and meaningful to me, it helps me constrain the note to be singular.¹ Instead of "Marriage" I have a note named "Living in an Understanding Way." That's a meaningful thought to me… and it might get split into "Give Myself Up For Her" and "Living in an Understanding Way" as the thoughts become more developed and need atomicity.

Form Dictates Workflow

Because my "weak notes" don't have descriptive filenames, they are harder to "find." They don't jump out visually in the graph or in backlinks, but they are still present and available for exploration. The "label" style weak notes especially pique my interest when looking at a local graph… those are often great places to start discovering.

Occasionally while reviewing "weak" notes or zettels, I discover enough weak opinions to voltron into a strong fact or strong opinion. It's always exciting. It feels like being able to watch my thoughts forming. Seeing and experiencing externally the process of thinking has helped me reject some of those new hypotheses without mindlessly accepting them just based on frequency².

Workflow Example: Saul and Fear

An example of how all this all played together recently: making my way through 1 Samuel, I got to (one of the passages that amuses me the most) 1 Sam. 10:22:

So they inquired again of the LORD, "Is there a man still to come?" and the LORD said, "Behold, he has hidden himself among the baggage." (ESV)

I laughed as I usually do at the thought of the giant future king of israel hiding under blankets among the donkeys and camels... and the thought occurred to me: Was Saul's predominant failing Fear?

I jotted it into a zettel note, just a link to the Bible verse and a question:

Is Saul's predominant sin fear?

Over the following weeks I continued to read about Saul, and logged six or more little notes on the same line of thought. As I noticed similarities, notes became linked. I started to notice that it wasn't just fear, it was specifically linked to Saul's fear of other's opinion. I eventually rewrote and reshaped the notes into a new Evergreen form based on my strong opinion: Saul was Dominated by Fear.

# Saul was dominated by fear

Saul's prevailing [[Sin]] may have been [[Fear]], specifically fear of how people saw him. 

He displays fearfulness at the beginning of his story as comic humility[^1022], later in life it manifests as violent paranoia.[^paranoia]

Samuel tells Saul that he thought little of himself[^little]… I wonder what this means? Self-esteem? Certainly a self-focus rather than trusting God. Samuel attempts to redirect his attention to the fact God anointed him king.

Saul confesses to Samuel that he feared the people's opinion.[^opinion] As further evidence of this, we can point to his dramatic transformation when the people acclaim David above him.[^acclaim]

In his final chapter his fear of the Philistine army[^army] drives him to seek out necromancy. His fear kept him from eating, and paralyzes him with terror when ghost Samuel tells him the truth.

[^1022]:  [[ESV/1 Samuel/1 Samuel-10#22|1 Samuel 10:22]]
[^paranoia]:  [[ESV/1 Samuel/1 Samuel-18#12|1 Sam 18:12]], [[ESV/1 Samuel/1 Samuel-18#29|1 Sam 18:29]]
[^little]: [[1 Samuel-15#17|1 Sam 15:17]]
[^opinion]: [[ESV/1 Samuel/1 Samuel-15#24|1 Samuel 15:24]]
[^acclaim]: [[1 Samuel-18#8|1 Samuel 18:8]]
[^army]: [[1 Samuel-28#5|1 Samuel 28:5]]

I'm sure my style and form will continue to develop over time. I am really glad to have stumbled across this idea of Note Lifecycle. I continue to see a version of this thinking in other people's Creativity Systems, and am ~~stealing~~ thinking about how to learn from them. 🤣

This wonderful idea is completely stolen from the wonderful Andy Mutaschak's note: Prefer note titles with complete phrases to sharpen claims ↩
Appeal to numbers - thoughtco.com ↩

Understanding Your Tesla’s Energy Charts

Sat, 30 Oct 2021 07:00:00 GMT

There are a couple charts on the “Energy” tab in the Tesla user interface. They’re packed with interesting info and I thought I’d take a minute to explain what’s going on.

Scraping GitHub Contributor Emails

Nelson Figueroa — Sat, 30 Oct 2021 03:00:00 GMT

Scraping GitHub contributor emails, and how you can protect yourself.

Royal Game of Ur (Background)

ℤ→ℤ — Fri, 29 Oct 2021 23:40:31 GMT

Updated 2021-12-17: A closed-form solution has been submitted

The Royal Game of Ur, also known as the Game of Twenty Squares, is one of the oldest known board games, with attestations from 3000 to 250 BC. It was an enduringly popular game, played across multiple civilizations and across the Middle East. As a racing game, two players compete to move their pieces along and off the board, where their own pieces can act as road blocks to their own movement but can also send the other’s players pieces back to the beginning. In this article (part I), I present the rules for the game and discuss the game complexity metrics. In part II, I will discuss solving the game by computing the optimal strategy.

Rules

Contrary to claims that the rules have been fully discovered, we are still unsure how this game was played historically. Indeed, with a lifespan nearing 3000 years and being played by different cultures and geographies, there may have been many variants of the rules and changes over time. We know that in the latter period of time, the board straightened itself out (see figure). The path around the board, significance of the rosettes, and how to interpret dice rolls are all educated guesses. My copy of the game from Masters Traditional Games lists three routes and four rule variants. The path denoted in the figure is the “RC Bell” route.

“Standard” version

The “standard” version is the set of rules you are likely to find implemented in electronic versions. This version of the rules was established by Irving Finkel based on archaeological copies of the board, inferences from the Itti-Marduk-balāṭu tablet, and experimentation. I imagine most people became aware of the game through the Youtube match between Irving Finkel and Tom Scott:

(User tracking should be disabled unless you play the video.)

Since analyzing a half-dozen variants of the game would be tedious, I’ll be focusing on this set of the rules.

The game’s objective is for a player to move their seven pieces along their route and off the board before the other player can do the same.

All pieces begin in the “starting pool” and are not on the board.
On a player’s turn, they roll four tetrahedral dice to determine how far they can move a piece. Each die allows only a result of zero or one, so the range of results of a throw is between 0 and 4 inclusive. (A zero result is a lost turn.) In Johnstone/standard dice notation ¹, this is a roll of 4d2 - 4.
Only one piece can be moved per roll and a piece must move forward. Pieces may be moved from those on the board or in the starting pool. If no pieces can be moved, the player loses the turn.
A piece cannot move forward onto a piece owned by the same player and only one piece may occupy a square at a time. If a piece lands on a square owned by the opposing player, the opposing player’s piece is placed back in the starting pool (off the board). However, the middle rosette square is “safe” and a piece on this square cannot be ejected from the board.
If the piece lands on a rosette, the player must roll again. The next move does not need to use the same piece.
Pieces are removed from the board on an exact roll.

Humorously, in the video Finkel makes a big scene about Tom Scott calculating the odds for a roll of the dice and Finkel proclaims he’s never had any use for maths. But in his own paper, he works out the odds for the various rolls.

Itti-Marduk-balāṭu - Gambling version

Irving Finkel’s On the rules for the Royal Game of Ur ² describes a different, more complex game based on a tablet written by Itti-Marduk-balāṭu. Since the game in the tablet is a modification of a “base” game, the standard version is (partly) defined by reverse-engineering the tablet’s version of the rules. In contrast to the standard version, the tablet version has five labeled pieces and a mechanism to be bet or gamble during play. This version is played on the elongated or revised board.

Each player has five pieces, labeled Swallow, Storm-bird, Raven, Rooster, and Eagle respectively.

There are two types of dice: a standard tetrahedral 1d4 and a yes/no die or 1d2. The rules for how to interpret the dice from the tablet allow many options, but for sake of choosing a single option, a player will first roll the 1d4. They may then roll the yes/no die. If the roll is ‘no’, the turn is lost. If the roll is ‘yes’, the 1d4 result is transformed to 1->5, 2->6, 3->7, and 4->10.

Each player starts with 15 counters and the game pool starts with 20 counters initially.

Similar to the standard version, players move their pieces around the board and attempt to exit all pieces before their opponent. While playing, players may gain by landing on the rosettes and lose points by skipping over them. A player also loses points based on pieces remaining on the board at the game end. The winner is the one with the most points at the end.

To launch a piece onto the board, a specific role must be made. The swallow requires a 2, the storm-bird a 5, the raven a 6, the rooster a 7, and the eagle a 10. When launched, the swallow appears on the first rosette (position 4), while the others start at positions 5, 6, 7, and 10 respectively.
Pieces must be launched in sequence. If a player has the swallow and storm-bird in play (or in the completed pool), they can only launch the raven on a 6, even if a 7 or 10 is rolled. If the storm-bird and raven are in play, but the swallow is sent back to the starting pool, the swallow will need to be re-launched before the rooster can be launched.
When a player lands a piece on the rosette, they win from the game pool a number of counters equal to the piece’s value. The swallow has a value of 3, the storm-bird, raven, and rooster have a value of 4, and the eagle has a value of 5.
If a piece moves over the rosette without landing on it, the player loses a number of counter equal to the piece’s value.
Rosette positions are safe; a piece can not be knocked off the position by another piece.

At the end of the game, a player loses a number of counter equal to the sum of value of pieces on the board.

State representation for the standard version

On the original board, a piece moves from the “ready pool” (position 0) through the first four safe positions, then through eight contended spaces, and finally through two safe positions before exiting the board to the “completed pool” (position 15). Thus, a position can be encoded in 4 bits for the 16 potential locations. Since each player has seven pieces, a player’s pieces can be encoded in 7 * 4 bits and both players pieces can be encoded in 2 * 7 * 4 or 56 bits.

Alternatively, we can model the game from the perspective of the squares rather than the pieces. The 14 middle positions (which have a capacity of one) can be modeled in 14 bit and the number of pieces available in the starting pool require 3 bits. (The number of pieces in the ending pool can be calculated based on the difference of the total number of pieces and the number in the starting pool plus on the board.) This requires 17 bits of storage per player or 34 bits total.

Naively, the state space for this board in the first representation allows $2^{56}=72,057,594,037,927,936$ possible states, or approximately 72 quadrillion. This is a big number (half-way to the age of the Earth in seconds), but this representation also contains many symmetrical games (piece 1 on position 3 and piece 2 on position 6 is identical to piece 1 being on position 6 and piece 2 on position 3) and invalid games (multiple pieces on the same position).

The alternative representation allows for $2^{34} = 17,179,869,184$ possible states or approximately 17 billion. This representation also contains many invalid games.

Can we compute a more precise value for the number of possible states?

How many possible, valid states for the standard version?

We can approximate the state space by using the “ball and urn” mental framework. We are trying to place pieces (balls) into board positions (urns). Each player has 16 urns (although some are shared) and all but two urns have a maximum capacity of one ball. The player’s pieces are identical to each other but the urns are distinct.

If we simplify the rules and state that all urns have a maximum capacity of one ball, then we can leverage the helpful Twelvefold Way table. (There are multiple versions of this table; I find Cook’s one of the more readable on the web.) The entry for “unlabeled” balls, “labeled” urns, and a maximum capacity of one is equivalent to a n choose k or binomial coefficient calculation.

Table: $\binom{n}{p}$ Values for $n=16$ and $p$ pieces

$p$ Pieces	Approximate State Space Size
1	256
2	14,400
3	313,600
4	3,312,400
5	19,079,424
6	64,128,064
7	130,873,600

This is an approximation and includes invalid states where the two players are contesting the same slot while also excluding cases where multiple places may be in the starting and ending pool, but as we shall see, this is a pretty good approximation. (Since combinations are symmetric, e.g. $\binom{16}{2} = \binom{16}{14}$, the approximation breaks down as $p$ increases.) Furthermore, these numbers are fairly small, so it seems practical to have a computer iterate (smartly) through the cases as 131 million is far more reasonable than 17 billion.

I wrote a program in Rust that will generate and count all valid states. I use a recursive, depth-first search approach. Each player is modeled via a vector which represents a placement into a labeled urn. Since some positions allow multiple placements we cannot generates states via a standard lexicographic combination algorithm, but do so via a slight modification of the algorithm that will generate duplicate 0 and 15 positions.

Within the search, our base case is a solution being found. Due to the functional design, the two player’s states are valid on entry to the function, so a valid full solution just requires a check that all placements have been made. If so, we pass the solution to a Visitor class that abstracts away what to do for every solution. (The visitor either prints out the solution or increments counts depending on the program’s mode.)

fn search_games_dfs(pieces: usize, visitor: &mut ModalVisitor, p1state: &mut Vec<usize>, p2state: &mut Vec<usize>) {
    if p1state.len() == pieces && p2state.len() == pieces {
        visitor.found(p1state, p2state)
    } else {
        // states are ordered monotonically; if the last move is the starting or ending pool,
        // we can create a new state in the same pool. Otherwise, place in the next potentially
        // valid spot.
        let min_i = match (*p1state).last() {
            Some(0) => 0,
            Some(15) => 15,
            Some(min) => min + 1,
            None => 0
        };
        let min_j = match (*p2state).last() {
            Some(0) => 0,
            Some(15) => 15,
            Some(min) => min + 1,
            None => 0
        };

        for i in min_i..16 {
            p1state.push(i);
            for j in min_j..16 {
                p2state.push(j);
                if is_valid(p1state, p2state) {
                    search_games_dfs(pieces, visitor, p1state, p2state);
                }
                p2state.pop();
            }
            p1state.pop();
        }
    }
}

If we are not at the base case, we either initialize a placement (min_i, min_j) at the zero position (no former placements have been made), try to continue at 0 or 15 (the pools that can handle more than one assignment), or the next entry (since all other urns have a maximum capacity of one and we are searching the space in ascending order).

Then, we attempt all possibilities for player one and player two, verify they are valid (is_valid), and if so recursively check the next depth. The is_valid code verifies that placements are distinct (when necessary) within a player’s placements and distinct within the eight contended squares.

NB: This is not the most efficient representation of the board. If we used the 14-bit representation of the board, we could detect collisions by masking off the first four bits, use the and-bitwise operator and test against zero. Furthermore, the representation does not allow modeling of more than one piece within a position, so the first check in the code can be eliminated entirely.

fn is_valid(player1: &[usize], player2: &[usize]) -> bool {
    // verify each individual player's pieces are unique or in the starting/ending pools
    let mut seen = [false; 16];

    // The placement logic ensures that these constraints are respected, so this code could
    // be skipped when called from search_games_dfs. Skipping this check halves the run-time.
    for player in [player1, player2].iter() {
        for placement in player.iter() {
            match *placement {
                0 => (),
                15 => (),
                place if seen[place] =>
                    return false, // duplicate
                place =>
                    seen[place] = true
            }
        }
        seen.fill(false);  // reset for next player
    }

    for player in [player1, player2].iter() {
        for placement in player.iter() {
            match *placement {
                place if place >= 5 && place <= 12 =>
                    if seen[place] {
                        return false
                    } else {
                        seen[place] = true
                    },
                _ => ()
            }
        }
    }

    return true;
}

Table: Programmatic Count of Valid States with $p$ pieces

$p$ Pieces	Computed State Space Size	Branches
1	248	232
2	13,112	24,148
3	264,304	714,720
4	2,606,947	9,131,362
5	14,680,840	61,834,256
6	53,212,388	255,752,544
7	137,913,936	726,864,992
8	280,408,902	1,572,665,124
9	482,210,816	2,817,358,656
10	740,777,984	4,446,087,488

Computations are relatively quick. Finding all 138 million solutions for p=7 took less than three minutes (with the first block of checks in is_valid skipped) on a 2017 “Coffee Lake” Intel i7.

The average branching factor at $p=7$ is 5.3; this means that on a given ply (move), a player is making a potential decision between 5 different options, although depending on the roll an option may not be possible (e.g. the move is blocked by another piece). The branching factor is computed as the sum of at-most-one move from the starting pool, all moves on the board, and no moves from pieces in the ending pool.

Comparing our earlier approximations and the programmatic counts finds the deltas to be (proportionally) small:

Instead of leveraging a computer, can we count the number of positions using a closed form equation?

Closed form?

Whoever wants to go about generating all partitions not only immerses himself in immense labor, but also must take pains to keep fully attentive, so as not to be grossly deceived.

— Leonhard Euler, De Partitione Numerorum (1750)

A reader has worked out the following closed form equation:

$$S = \sum_{p_1 = 0}^{\min(p,m+n)}\sum_{p_2 = 0}^{\min(p,m+n)}\sum_{j = \max(0,p_1-m)}^{\min(p_1,n)} (p-p_1+1)(p-p_2+1) {n \choose j} {m \choose p_1-j} {m+n-(p_1-j) \choose p_2}$$

In the above notation, $p_i$ are the number of pieces from player i on the board, there are m shared slots on the board and n unshared/uncontested slots.

~~Alas, I have been unable to find a closed-form solution.~~ As the Euler quote warns, although one can use the inclusion/exclusion method to iterate toward the correct number, it is very easy to forget a case or include some duplicates. Graphically, here are the conflicting states for p=1 to 3 (an index refers to a lexicographically ordered single player’s valid position):

As expected, conflicts are symmetric along the main diagonal. However, there are a number of sub-patterns that I’ve had trouble modeling. ~~I may return to this problem (or some reader may solve it), but at this point a closed form solution is convenient but not necessary.~~

How big are these numbers?

Not very big, but beyond trivial. Using log-10 notation, the state space size is 8 (or 9 if we round up), which is less than the 10 for Nine men’s morris, 20 for Checkers, and 44 for Chess.

A different measurement for a game is the game tree size; this represents the total number of games that can be played. This differs from the state space because there may be multiple paths to a given state. The game tree size is technically infinite for this game (since on a ply the player may roll a zero or players may constantly return each other’s pieces to the start). In practice, games tend to be quick and thus avoid becoming bogged down.

We can approximate the gate tree size by raising the average branch to the average number of plies in a game, or $b^{d}$. For an individual ply, a player needs to make a single decision of which piece to move; that decision is shared among all pieces on the board and at most one piece in the starting pool the board (since all pieces in the pool are identical). Through enumeration, we have found the average branch factor as 5.3. I don’t have statistics for the average number of plies, but in uncontested play, you will need on average 8 plies to move a piece to the end (since the mode is 2) and with 7 pieces you could expect 56 plies. If we estimate 70 as the average number of plies per player, that would be 140 total plies. The log10 of $5.3^{120}$ is 87. In comparison, Checkers is 40 and Chess is 123.

These numbers allow us to compare games against each other, but they do not explain what makes a game fun or engaging. For the ancients, this game had the benefit of low cost of entry (there are example boards etched in the floor), rapid play, and allowing of multiple strategies but with an element of chance. The central squares force interaction between the players while the safe squares allow the readying of “reserve” troops to threaten the other player. These combination of factors, in my view, are what makes the game fun and make the sum better than the parts.

References

Peterson, Jon. “The Origins of Dice Notation.” Playing at the World (2013). ↩︎
Finkel, Irving L. “On the rules for the Royal Game of Ur.” Ancient Board Games in Perspective (2007): 16-32. ↩︎

No girl software freedom here

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 29 Oct 2021 12:00:00 GMT

Really?

Astro - static site generator made of dreams

arnorhs blog - arnorhs.dev — Fri, 29 Oct 2021 03:00:00 GMT

Recently, a new static site generator (SSG) arrived on the block called Astro. Lets talk a bit about SSGs. What they're good for, what they're not and why I think Astro is a bit of a game changer.

Exploring PL/pgSQL part two: implementing a Forth-like interpreter

Notes on software development — Fri, 29 Oct 2021 03:00:00 GMT

Previously in exploring PL/pgSQL:
Strings, arrays, recursion and parsing JSON

In my last post I walked through the basics of PL/pgSQL, the embedded procedural language inside of PostgreSQL. It covered simple functions, recursions and parsing. But there was something very obviously missing from that post: a working interpreter.

So in this post we'll walk through building a Forth-like language from scratch in PL/pgSQL. We'll be able to write a fibonacci function in this Forth-like language and have it be evaluated correctly like so:

$ ./test.sh sm.sql "SELECT sm_run('
DEF fib
  DUP 1 > IF
  1- DUP 1- fib CALL SWAP fib CALL + THEN
  RET

20 fib CALL
EXIT')"

...

 sm_run
--------
 6765
(1 row)

All code is available on Github.

Forth

Forth is a stack-oriented language. Literals are pushed onto the stack. Functions and builtins operate on the stack.

For example:

$ ./test.sh sm.sql "SELECT sm_run('3 2 + EXIT')"

Will produce 5. And:

$ ./test.sh sm.sql "SELECT sm_run('3 2 + 1 - EXIT')"

Will produce 4.

Our code will notably not be a real Forth, since there are many special features of a real Forth. But it will look like one to a novice Forth programmer like myself.

You can read more about Forth basics here. And you can read a truly stunning, real Forth implementation in jonesforth.S. Or you can pick up Let Over Lambda for a fantastic book on Common Lisp that culminates in a Forth interpreter.

Implementation

Since the builtin array_length($arr, $dim) returns NULL if the array is NULL and our dimension is always 1, we'll write a helper.

DROP FUNCTION IF EXISTS sm_alength;
CREATE FUNCTION sm_alength(a text[]) RETURNS int AS $$
BEGIN
  RETURN COALESCE(array_length(a, 1), 0);
END;
$$ LANGUAGE plpgsql;

We'll also need to bring in the hstore extension so we can map function names to their positions. (We could use an association list but those are less programmer-friendly.)

CREATE EXTENSION IF NOT EXISTS hstore;

Our interpreter function will take a string to evaluate, splitting the string on whitespace into tokens.

DROP FUNCTION IF EXISTS sm_run;
CREATE FUNCTION sm_run(s text) RETURNS TEXT AS $$
DECLARE
  tokens text[] = regexp_split_to_array(s, '\s+');
  stack text[]; -- Data stack
  defs hstore; -- Map of functions to location
  tmps text[]; -- Array we can use for temporary variables
  token text; -- Current token
  rps text[]; -- Return pointer stack, always ints but easier to store as text
  pc int = 1; -- Program counter
BEGIN

We set up a tmps array because each builtin may need differing number of temporary variables and PL/pgSQL makes ad-hoc variables cumbersome (or at least an easier way exists outside my knowledge).

And we store the return pointer stack as a text array so that we can use sm_alength on it even though values in this array will always be integers.

Next we'll start an infinite loop to evaluate the program. The only thing that will stop the input is the EXIT builtin that will return from this function with the top of the stack.

  WHILE true LOOP
    token = tokens[pc];
    RAISE NOTICE '[Debug] Current token: %. Current stack: %.', token, stack;
    IF token IS NULL THEN
      RAISE EXCEPTION 'PC out of bounds.';
    END IF;

    IF token = 'EXIT' THEN
      RETURN stack[sm_alength(stack)];
    END IF;

    ... TODO ...

    stack = array_append(stack, token);
    pc = pc + 1;
  END LOOP;
END;
$$ LANGUAGE plpgsql;

If no other condition is met (the token is not a builtin), we push it onto the data stack and increment the program counter.

Conditionals

The IF builtin pops the top of the stack. If it is true evaluation continues. If it is false evaluation skips ahead until after a THEN builtin.

For example:

$ ./test.sh sm.sql "SELECT sm_run('1 1 1 = IF 2 THEN EXIT')"

Produces 2. But

$ ./test.sh sm.sql "SELECT sm_run('1 1 0 = IF 2 THEN EXIT')"

Produces 1.

Implementation

Joining the EXIT condition in the interpeter loop we get:

...

  WHILE true LOOP

    ...

    IF token = 'IF' THEN
      -- Grab last item from stack
      tmps[1] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      IF NOT tmps[1]::boolean THEN
        WHILE tokens[pc] <> 'THEN' LOOP
      pc = pc + 1;
    END LOOP;
    pc = pc + 1; -- Skip past THEN
      ELSE
        pc = pc + 1;
      END IF;
      CONTINUE;
    END IF;

    IF token = 'THEN' THEN
      -- Just skip past it
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = 'EXIT' THEN
      RETURN stack[sm_alength(stack)];
    END IF;

    ...

Other builtins

The DUP builtin makes a copy of the top of the stack. The SWAP builtin swaps the order of the top two items on the stack. And the 1- builtin subtracts 1 from the top of the stack.

    ...

    IF token = 'DUP' THEN
      -- Grab item
      tmps[1] = stack[sm_alength(stack)];
      -- Add it to the stack
      stack = array_append(stack, tmps[1]);
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '1-' THEN
      -- Grab item
      tmps[1] = stack[sm_alength(stack)];
      -- Rewrite top of stack
      stack[sm_alength(stack)] = tmps[1]::int - 1;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = 'SWAP' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Swap the two
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1];
      stack[sm_alength(stack) - 1] = tmps[2];
      pc = pc + 1;
      CONTINUE;
    END IF;

    ...

It's important that every builtin handle incrementing the program counter and skipping to the beginning of the loop. Because some builtins increment the program counter under different conditions (like IF above).

The last few builtins are the simplest: arithmetic operations that produce integers or booleans.

    ...

    IF token = '=' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int = tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '>' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int > tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '+' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int + tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '-' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int - tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '*' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int * tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    IF token = '/' THEN
      -- Grab two items from stack
      tmps[1] = stack[sm_alength(stack) - 1];
      tmps[2] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Replace last item on stack
      stack[sm_alength(stack)] = tmps[1]::int / tmps[2]::int;
      pc = pc + 1;
      CONTINUE;
    END IF;

    ...

Function definitions

Functions here will differ from Forth, borrowing elements of machine code. Return pointers will be stored in a dedicated return pointer stack. We could store it on the data stack but that would require more effort on the part of the programmer to restore the stack. Calling RET inside a function pops a return pointer off the return pointer stack.

Here's a simple function definition: DEF plus + RET.

    ...

    IF token = 'DEF' THEN
      tmps[1] = tokens[pc+1]; -- function name
      tmps[2] = pc + 2; -- starting pc
      WHILE tokens[pc] <> 'RET' LOOP
        -- RAISE NOTICE '[Debug] skipping past: %.', tokens[pc];
        pc = pc + 1;
      END LOOP;

      IF defs IS NULL THEN
        defs = hstore(tmps[1], tmps[2]);
      ELSE
        defs = defs || hstore(tmps[1], tmps[2]);
      END IF;
      pc = pc + 1; -- continue past 'RET'
      CONTINUE;
    END IF;

    ...

There doesn't seem to be a way to combine a NULL hstore value and a non-NULL hstore value. So that's why we need that special case.

Return

The RET builtin pops a value off the return pointer stack and jumps to it.

   ...

   IF token = 'RET' THEN
      -- Grab last return pointer
      tmps[1] = rps[sm_alength(rps)];
      -- Drop last return pointer from stack
      rps = rps[1:sm_alength(rps) - 1];
      -- Jump to last return pointer
      pc = tmps[1]::int;
      CONTINUE;
    END IF;

    ...

Function calls

Forming the other half of function calls is the CALL builtin. This places the program counter (plus one, past the CALL token) onto the return pointer stack and jumps to the position of the function if it exists.

A simple function call for the above plus function might be: 2 3 plus CALL and would produce 5 on the top of the stack.

    ...

    IF token = 'CALL' THEN
      -- Grab item
      tmps[1] = stack[sm_alength(stack)];
      -- Remove one item from stack
      stack = stack[1:sm_alength(stack) - 1];
      -- Store return pointer
      rps = array_append(rps, (pc + 1)::text);
      -- Fail if function not defined
      IF NOT defs?tmps[1] THEN
        RAISE EXCEPTION 'No such function, %.', tmps[1];
      END IF;
      -- Otherwise jump to function
      RAISE NOTICE '[Debug] Jumping to: %:%.', tmps[1], defs->tmps[1];
      pc = defs->tmps[1];
      CONTINUE;
    END IF;

    ...

And that's it! All done the basic instructions needed. Store all that code in sm.sql and grab the test.sh code from the previous post:

$ cat ./test.sh
sudo -u postgres psql -c "$(printf "%s;\n%s" "$(cat $1)" "$2")"

And try out our port of recursive fibonacci:

$ ./test.sh sm.sql "SELECT sm_run('
DEF fib
  DUP 1 > IF
  1- DUP 1- fib CALL SWAP fib CALL + THEN
  RET

20 fib CALL
EXIT')"

...

 sm_run
--------
 6765
(1 row)

Happy PL/pgSQL- and Forth-ish-ing!

Latest post is up! Writing a Forth(-inspired language) implementation from scratch in PL/pgSQL. Because who doesn't want to be able to run stack machine code from SELECT statements in PostgreSQL?https://t.co/sbxhuDp1J9 pic.twitter.com/9nrHEIhRPa
— Phil Eaton (@phil_eaton) October 29, 2021

All reject our pointless fridge overlords

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 29 Oct 2021 02:50:00 GMT

More nannies, than overlords, actually. But pointless, they are.

WeToddle retrospective

Bytepawn - Marton Trencseni — Fri, 29 Oct 2021 01:00:00 GMT

The idea behind WeToddle came from the Baby Fanclub group we have on Messenger, which has most of our family in it. It turns out some ex-Facebook people had a similar idea in 2019, raised $3M, spent a 2 years on it, and then gave up because it didn’t go anywhere (presumably).

How to setup the modern frontend toolchain in Django

Tommaso Amici — Thu, 28 Oct 2021 20:12:12 GMT

So you want to deploy a traditional Django application, but you also want Typescript, Tailwind, and all the good that modern frontend offers?

Improving Fuzz Testing with Zig Allocators

ryanliptak.com — Thu, 28 Oct 2021 03:00:00 GMT

Using Zig allocators to improve the results of fuzz testing

The Parable of A Rude Guest

Weblog on marginalia.nu — Thu, 28 Oct 2021 03:00:00 GMT

You are invited to a dinner party. After talking for a while food is served on the table. You pounce. “Haha, suckers!”, you think, and load all the food on your plate and leave nothing but scraps for the hosts. You feel victorious. Serves them right for inviting you into their home. You wolf down the food with ravenous appetite while they look on.

That was tasty, but now you got a piece of meat stuck between your teeth so you go to the bathroom and borrow some floss and use one the hosts’ toothbrushes. You also use the toilet but don’t flush because you don’t think you are going to use it again.

Git Cheatsheet - handy commands for every situation

Gourav Goyal — Thu, 28 Oct 2021 03:00:00 GMT

Useful git commands for beginners like reverting commits, work between branches, manage PRs, and more

2021–10–28: Fixing broken Pinephone, aka Pinephone repairability

megi's PinePhone Development Log — Thu, 28 Oct 2021 01:00:00 GMT

Hitler can now dine in italian restaurants

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 27 Oct 2021 12:00:00 GMT

But never mind him: worry about trust in COVID19 management, instead.

Pico Ada Libraries Release 0.7.0

Jeremy Grosser — Wed, 27 Oct 2021 10:00:00 GMT

This is another big release, with a USB device controller driver, ADC DMA support, and boot2 rewritten in Ada. Many other small bug fixes and usability improvements are included. The detailed changelog can be found on the documentation site. Thanks to Daniel King, Fabien Chouteau, and Holger Rodriguez for patches, testing, advice, and feedback during this release cycle.

An Eventful Few Months

Ben Overmyer's Site — Wed, 27 Oct 2021 03:00:00 GMT

It's been an eventful few months. Since my last blog post back in August, I've had Lasik surgery, had a house deal fall through, discovered a new hobby, rediscovered an old one, and edged deeper into the Apple ecosystem. That's not even counting all the various developments in my professional life.

Lasik was expensive and the procedure was deeply uncomfortable, but worth it. For most of my life I've been near-blind without glasses. For the last month, though, I've had perfect vision. I still catch myself marveling at how awesome it is to have vision like this.

The house deal... well, that sucked and was expensive, but it ultimately made me realize the value of renting indefinitely. So that'll keep happening.

After abandoning Blizzard forever, I looked around for a new game to devote my lazy hours to. I landed on Final Fantasy XIV. After joining the Free Company "Mystic Exiles," I realized I'd found a new home. The Exiles are a great group of people - really friendly, really helpful, and genuinely caring. It feels like a second family. I have never had an MMORPG guild experience like that before. Plus, the game itself is a ton of fun. At time of writing, I'm barely into the content from the second expansion, and the fourth expansion is only a couple weeks away from launch. This will keep my attention for awhile, I imagine.

The old hobby that I rediscovered is Warhammer 40k. A friend of mine invited me to a tournament he's hosting at his game store in Iowa, and I decided to dust off (literally) my boxes of miniatures and re-enter the fold. It's cost quite a lot of money to get all the new rulebooks, miniatures, and equipment I need to participate in the tournament, but I'm excited about the whole hobby again.

Almost a month ago I bought a new iPhone 13 Pro and gave my iPhone 11 Pro to my wife. Since she was on Android before, and we chiefly used Facebook Messenger to communicate, this opened up a host of new options for us. Now we use mostly iMessage and avoid Facebook if at all possible. It's a little annoying to not have iMessage on Windows, but at least when I'm using my Mac I can type out messages to her.

As for all of the developments in my professional life, well, that could be a whole 'nother blog post all on its own. Suffice it to say that the technology that I spend most of my dayjob hours on has changed dramatically, and I'm addressing some interesting new challenges. I'm really starting to dig Hashicorp Nomad as an alternative to Kubernetes.

So that's a bit of what I've been up to the past couple months.

Day Two

utf9k — Wed, 27 Oct 2021 02:01:00 GMT

Thoughts as I enter the third job of my career in the software industry

The Unceasing Cessna Hacienda

Damn Interesting — Tue, 26 Oct 2021 21:26:58 GMT

Warren “Doc” Bayley was a man of the people. When he and his wife Judy opened their Las Vegas resort in 1956, Bayley had no plans to compete with the flashier, corporate casinos at the center of the Strip. Instead, the Hacienda Hotel catered to families, as well as to locals who wanted a night out minus the tourists. When connoisseurs sneered, “You can either go to Las Vegas or to the Hacienda,” Bayley embraced the distinction with pride, and he didn’t even mind when the nickname “Hayseed Heaven” took hold. Hayseeds were hard-working folks who deserved a vacation, too, and his steady bookings proved it.

But margins were low, and Bayley was always on the lookout for new ways to drum up publicity for his so-called “low roller” operation. He didn’t care where the ideas came from. Staff at the Hacienda were considered family, and Bayley regularly asked the advice of maids, bellboys, cooks, and anyone else who might have something useful to offer. So when a slot machine mechanic and former Army pilot named Bob Timm suggested breaking a world record—specifically, the world record for the longest endurance flight in a piloted plane—Bayley was willing to listen.

Continue reading ▶

Wikivoyage has disappeared from DuckDuckGo

Gaganpreet Arora — Tue, 26 Oct 2021 13:43:04 GMT

Update (2021-10-31): DuckDuckGo now includes results for Wikivoyage again, as do Bing and Yahoo.

I have been planning a long due vacation and I was doing some research into the places I wanted to visit. My goto travel guide is Wikivoyage: it’s well organised, text-heavy (instead of slideshow-heavy alternatives) and skimming the guide of a region will generally give a good idea of what to expect.

So imagine my surprise when I enter a query with the keyword “Wikivoyage” on DuckDuckGo:

Is 3D printing good only if it's active?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 26 Oct 2021 12:00:00 GMT

Maybe. Or maybe not.

Book review - The Elements of User Experience

River Writes - A MediaWiki Blog — Tue, 26 Oct 2021 10:17:38 GMT

A review of “The Elements of User Experience”

Book Review: Language of the Spirit, An Introduction to Classical Music

Nicola Iarocci — Tue, 26 Oct 2021 09:05:25 GMT

In this introduction to classical music, Jan Swafford explains the different musical periods and their differences. Each period has its introductory chapter, followed by chapters dedicated to the most influential composers of the era. The choice is comprehensive and well cared for, with the most relevant names well-investigated both in biography and works. For each composer, Swafford also offers some listening suggestions. Biographies thicken as we get into the contemporary era. Here, some are limited to just a page or two while others, obviously those of the author’s preferred composers, remain as thorough as those of the older periods.

Red Mulberry Growlog #3

Bartlomiej Mika — Tue, 26 Oct 2021 06:55:06 GMT

Into the over world from the underworld! The mulberries were grown in the basement for the majority of their existence and now are able to live in the outside for the first time. In this grow journal I’ll write about my experience bring the red mulberries into the outside and growing them over summer.

Skills, stories, and software every dev should know - 123dev #43

Justin Garrison's Homepage — Tue, 26 Oct 2021 03:00:00 GMT

Comments

Continuous frustration

Continuous integration and deployment systems have gone through a lot of changes. What started as shared bash scripts and Makefiles changed into central systems that could be shared by multiple developers and teams. This shift allows for more repeatable, scalable, and more secure systems.

As systems optimize globally they can negatively impact local optimizations. This is a usually a win for an organization because they consolidate expertise and reduce overhead costs, but they reduce flexibility and can become a bottleneck.

2021–10–26: Pinephone HDMI hot-plug-detection HW bug fixed

megi's PinePhone Development Log — Tue, 26 Oct 2021 01:00:00 GMT

Understand a problem individually, then share it.

Writing on Adam Craven — Mon, 25 Oct 2021 23:09:48 GMT

Note 58

Qubyte Codes — Mon, 25 Oct 2021 22:58:01 GMT

A little autumn weather isn’t putting folk off the sea.

Could age verified phones for kids work? SHOULD they?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 25 Oct 2021 12:00:00 GMT

If they have the right technology yes. Wait…

Nurturing Notes and Thoughts

trv.rs — Mon, 25 Oct 2021 11:09:00 GMT

Not all ideas are equal. Ideas have different sources, confidence levels, provability and permanence. We use different words to describe our ideas: Hunches, notions, opinions, theories, concepts, abstractions, beliefs, hypothesis, facts, laws. Our thoughts don't stay the same either. Our minds change over time.

If there are all these ways to think about ideas… why do we use the same note-making structure to represent this variety?

Would you feel more confident making that link not knowing the veracity of the two ideas, or by being able to see that both ideas are "strong?"

This has become the most impactful adaptation to my Creativity System: a simple classification system that has levels of "strength" for an idea. A "weak" idea is subject to doubt and disproval while a "strong" idea is core to my thinking.

There is an argument for only including strong opinions or strong facts in your Creativity System. By this logic every link in your notes is valid and strong. While this is tempting, we still need a place to capture and nurture new ideas. We would be creating a boundary arbitrarily in your Creativity System. However we slice and dice it, we have one Creativity System from inception to output.

Growing Your Notes

Note-making is idea development. To create and discover we must capture new and risky ideas and connections, not just recording the safe and the known. There is great value in capturing unproven thoughts so we can nurture them to maturity.¹

There are many ways to implement notes with a lifecycle, but all such systems should consider:

allow easy capture of new ideas
easy identification of the strength of an idea
clear path for weak ideas to become strong

Capture the Interesting

Our Creativity Systems should support capturing and placing all kinds of notes into their own "home", both weak and strong.

If we restrict our system to strong ideas we limit our ability to capture new and wonderful ideas. A huge part of a Creativity System is to allow us to capture and nuture anything that grabs and holds our interest.

Clear Identification

Our systems should have some sort of way of discerning what is a weak note, and what is a strong note. There are many such ways to do this (I'll detail mine in a later post) but it should make sense to you and be natural to your system and your goals.

Having this sort of identification can help us deal with a subtle but deadly cognitive bias: Humans believe we are critical of new ideas. The cold reality: our minds default to believing all we hear or read. We would like to think that when encountering a new idea we consider its truthfulness before believing… but it is far easier to just accept what we are given. We optionally (and rarely) consider a new idea's truthfulness after our minds have already accepted it.² By encoding new "weak" ideas with a structure that denotes their fragility, we can help ourselves defeat this bias and navigate critically in new areas of thought.

Path to Maturity

Once we have a way to capture all interesting ideas, and a clear way to discern the different kinds of thoughts and notes, we will often find notes traveling up or down the lifecycle. Once all our thoughts and notes are swimming around in the same sea, there are many lovely ways that they can grow, be disproven, or continue to flourish. Just revisiting a note after the passage of time can change its position in our minds.

Sometimes while working I'll find strong surrounding arguments through search or links which will "promote" the idea in my mind.

Sometimes while researching and writing I'll suddenly see a strong opinion that links together lots of weaker facts. The pattern suddenly shifts, and I'll have a new hypothesis to start studying.

There are many ways that allowing weak and strong ideas to co-exist in our notes can improve our thinking. I've started "debate" notes, linked notes arguing either side of a controversy. I've found patterns in my own thinking that were then encoded as a "weak" note, studied, and ultimately rejected because there's no proof.

***

In previous versions of my note-taking systems I either restricted my system to only "worthy" or "strong" notes, or I threw everything into the system but without a way to tell what was weak and what was strong. When only strong notes were in my system my thinking never grew, I only reinforced prior thoughts. When all thoughts were un-evaluated it was hard to navigate. By capturing everything while discerning weak and strong thoughts my Creativity System and my creative thinking has benefitted dramatically.

There is a very good comparison here to the scientific method: hypothesis being proven by experiment into theories that stand the test of time until law… but there are many such metaphors appropriate for different Creativity Systems. I didn't want to limit people's imaginations to only one. ↩
I cannot for the life of me remember where I first heard this. Proverbs 14:15 says it better than I could:

Fools believe every word they hear, but wise people think carefully about everything.

While the replication crisis makes me leary of every social experiement, there are some works out there that highlight this idea:
- Understanding and Belief - explorable.com has a good summary of the Gilbert experiment.
- You can't not believe everything you read - D T Gilbert 1, R W Tafarodi, P S Malone
- The Consequences of Reading Inaccurate Information - David Neil Rapp
↩

How to be a Game Changer: Intro

D13V — Mon, 25 Oct 2021 04:23:29 GMT

In battle, do not think you have to win. Think rather that you do not have to lose.

Ginchin Funakoshi, founder of Shotokan karate-do

We may be rarely if at all involved in battles today, but “games” are all around us, and every interaction we participate in can be thought of as a game, with its own rules, frameworks, and expectations about the participants. Such games exist in negotiations, relationships, and other interactions.

For example, imagine we are driving in a city, and we follow all the signs and rules of motion. In a sense, we’re playing the “traffic game”. It is a game which allows people in vehicles to move safely from point A to point B. We have options on how to play this game. We may actively change lanes in order to try to move faster through the road and arrive sooner. Alternatively, we may decide to stay in our lane, and take longer to reach the destination, but avoid the complexity and risk of switching lanes, resulting in a more relaxed, safer drive.

Or we may look at the whole vehicular traffic, and ask ourselves “Why should we play this game at all?” and come up with contrarian answers. We might decide not to play the traffic game with a car, but with a motorcycle, to be even faster and more flexible, trading off safety. We might consider a bicycle, e-bike, scooter, skateboard, public transit, or even walking. We are already starting to think like a game changer.

To go a step further, we may even decide we do not like any of these options and move to another city, or a town, or even a cabin in the woods. We are now a game changer, as we’ve changed the game that at least some people in society play - ourselves. Yet by opting out, we affect not just us, but the whole urban vehicular traffic for everyone else.

We may ask, what makes anyone a game-changer? How can I become one? Should I be a game changer? Which games should I change?

Let’s figure it out together. This is the first post from a sequence of 13 posts which will explore the topic in more detail, and plenty of examples. I will be posting one short post every week, and we’ll cover examples such as situations with limited resources, artificial scarcity, status, sales, cynicism, slippery slopes, obstacles, arbitraje, repeated games, the scapegoat game, no-exit environments and others.

If you’d like to follow along, subscribe to receive these posts over email, follow me on Twitter, where I’ll mention each post, or simply check back here once a week.

Thanks to Sylvain Kieffer for spotting grammar typos in this post.

Bot Apologetics

Weblog on marginalia.nu — Mon, 25 Oct 2021 03:00:00 GMT

There has been a bit of discussion over on Gemini recently regarding poorly behaved bots. I feel I need to add some perspective from the other side; as a bot operator (even though I don’t operate Gemini bots).

Writing a web spider is pretty easy on paper. You have your standards, and you can test against your own servers to make sure it behaves before you let it loose.

You probably don’t want to pound the server into silicon dust, so you add a crawl delay and parallelize the crawling, and now you have code that’s a lot harder to comprehend. This is likely the cause of some weird bot behavior, including mishandling of redirect loops or repeated visits to the same address. Multi-threaded orchestration based on a rapidly mutating data set is difficult to get right (the working set of the spider by necessity changes as it goes). You can iron a lot of this out locally, but some problems won’t crop up until you really push the limits with real-world scenarios.

The Line

june.kim — Mon, 25 Oct 2021 03:00:00 GMT

An introduction to Monzo’s data stack

Luke Singham — Mon, 25 Oct 2021 03:00:00 GMT

Originally posted on Data@Monzo Medium

The modern data stack is a collection of rapidly evolving technologies that together provide a platform for analytics.

While we at Monzo already have quite a modern data stack, there is always more to learn and room for improvement. This post summarises the core components of our modern data platform, some of the challenges we currently face, and what we want to change.

We follow two principles to guide our choices on the data platform and how we structure our data team:

Centralised data management
Decentralised value creation

Centralised data management allows companies to have a 360 degree view of their customers and their business. It unlocks data-driven decision making which is key to competing in today’s markets. Next to having all the data in one place, centralised data management also enables organisational efficiencies. Many data operations and data governance problems become a lot easier to solve if everyone in a company treats data in the same way.

Decentralised value creation means we have a discipline of data folks that work embedded across the entire company. This is sometimes known as a hub and spoke model. The teams closest to the problem space are the ones creating their data products.

An overview of our data architecture

At a very high level, most of our data is in the form of event data being produced in our backend services. These events are streamed via ‘the firehose’ (using Kafka and NSQ) to specific append-only projects in BigQuery.

Data developers create data pipelines from these events using dbt (by dbt Labs, formerly Fishtown Analytics). We use dbt to incorporate software engineering best practices into our data work.

Once a data developer has created a pipeline, they also create dashboards and get insights in Looker. It’s worth expanding on what we mean by ‘data developer’ here. Most of the data work is done by the data discipline, however anyone in the company can jump into the data pipelines and make dashboards. For a more detailed view of how we empower people to use data check out Borja’s post.

Let’s dive a little deeper into each component and where we’re going with it.

Data loaders #

Analytics Event Processor and Shipper is a tool we built that takes events from backend services posted onto the firehose, and analytics events from the app or internal tooling. There are two microservices involved:

Analytics Event Processor consumes two NSQ topics firehose and client_analytics. The Processor then enriches the payload of the event. For example, an event containing account_id is expanded into an account object. Then the Processor sanitises the payload to remove personally identifiable information (PII).
Analytics Event Shipper is responsible for writing both sanitised and raw events to BigQuery event tables. It creates BigQuery tables as necessary.

It’s worth noting that the event tables in BigQuery have a column for the payload. This allows the schema to change. A data developer can then extract columns using JSON_EXTRACT functions in BigQuery.

We use Fivetran for loading data from several systems, including for our accounting and web analytics data.

We also load a few miscellaneous bits of data, for example data dumps from third parties directly loaded into BigQuery or Google Cloud Storage. However, we aim to keep bespoke approaches to a minimum.

A challenge we face, particularly with our events data is ownership. Frequently, teams that generate the event data in our production stack are not the teams consuming the data. When a data developer picks that event data up in BigQuery, they may not have all of the domain context and the data can have many issues, for example missing some of the data needed for a particular analysis. Not knowing who to contact about the data slows us down. We will solve this by having clear ownership of all our data, from our production stack to our analytics stack (see the data discovery and metadata management section later in this post 👇). Where possible we also include a data developer within data-generating teams.

Data Build Tool (dbt) #

dbt, made by dbt Labs, has become central to the way we interact with data at Monzo. dbt allows users to run models in a separate development environment. This is huge when working with data; to know you can quickly iterate and test in a sandpit environment without impacting production data. You can tell dbt is a successful part of our platform when not only our data team is using the tool, but also our software engineers and business analysts too. It hasn’t been a seamless experience though. Using anything at scale usually brings its fair share of technical challenges.

Speed of running dbt has been a real issue. Currently we have over 4700 models in our dbt project with ~600k lines of SQL. When you have engineers using something slow, inevitably you will end up with workarounds, like our go-lang re-write of the dbt command line tool. However, having multiple competing tools isn’t ideal either and we don’t plan to rival dbt Labs any time soon!

We forked dbt in 2019 and placed the setup in a Docker container. At the time this seemed like a good idea for the speed at which we wanted to get security and productivity features into our data workflows. Docker containers are great for consistency between local and production environments and help new Monzonauts get to grips with Monzo data.

We’ve built a few custom features including dbt upstream prod and indirect_ref.

dbt upstream prod improves speed and saves money. When running dbt it is common to run with the upstream model operator dbt run -m +tableD. This command runs all of the data pipeline in your development environment up to the common source tables used in production. In the example below you can see this runs 5 models.

With our upstream production feature, the dbt run will read upstream dependencies from production tables. In this case, the same aim is achieved by running 1 model.

When a data developer is working on two tables of the data pipeline, our upstream feature intelligently figures out what part of the pipeline should be built in the development environment and which tables can be read from production. In the example diagram below, the developer has made edits to tableB and tableD. At the command line the developer enters upstream prod -m tableB tableD. Our upstream feature figures out tableC should also be run in the development environment because one of its upstream dependencies, tableB, has been modified.

indirect_ref helps us make sure Monzonauts only access the data they need. Some teams work with separate BigQuery projects where only a few people need to have access to that data. But how do we use aggregates or anonymous versions of that data? indirect_ref solves this problem by introducing the concept of 'interface tables' which sit at the end of a sensitive project’s data pipeline and act like an API dataset. The interface table is the only node that downstream consumers can access. Additionally, the interface table explicitly declares its downstream consumers. So if a downstream team uses the interface model, they will modify the dbt config in the interface model.

For example:

This explicitly forms a contract between the interface and consumer. As the interface model is owned by the team working with sensitive data, GitHub’s code owners automatically tags the producing team in the pull request.

Recently we did some benchmarking of how we could speed up dbt by a significant amount. We found:

using an M1, the new Apple laptops with Apple’s own CPUs improved speed by 3x
upgrading from dbt 0.15.0 -> 0.20.0 resulted in another 3x speed increase
moving dbt out of a container resulted in a 2x speed increase for those using a Intel CPU MacBook Pro

Combining these will mean a 9x speed increase for Monzonauts with an M1 and an upgrade to dbt 😅

These benchmark results give us a good set of actions for our next project on dbt. We will move away from using Docker containers and refactor our customisations into plugins. We’d love to raise these as PRs on the dbt project or open source them as separate tools (👋 dbt!). Once we’ve refactored, we’ll upgrade to the latest version of dbt to gain the speed improvements and many features that have been added since v0.15.0.

Git CI/CD #

All our code is version controlled and we use GitHub for our code repositories. We use a mono-repo approach, which ensures consistency of tooling, CI checks and increases discoverability. Data developers test the code, but parts of the testing are also automated as part of CI so we can merge changes with confidence.

Our main challenge here using a mono repo has been the speed of CI checks. In the last year we reduced CI checks from ~30mins down to ~5mins. However as the complexity of our pipeline and number of models grow we will likely need to return to this as a problem. We also want to use CI/CD to start raising data standards. For example by setting minimum thresholds in our CI checks we can raise documentation, testing and metadata standards across Monzo.

Schedule orchestration #

We use Airflow to schedule, run and monitor the refresh of our data warehouse. Most of our models run on a nightly schedule, with some tables being ‘near real time’, rebuilding every 15 minutes. We use dbt to compile the Directed Acyclic Graphs (DAG) of data models, we then automatically convert this DAG to the format Airflow expects, a DAG of tasks.

Notifications #

If everything ran smoothly all the time that would be great, but it doesn’t! We make use of dbt model tags and a bit of Python glue work to post model run statuses to a #data-monitoring Slack channel. Here’s what that looks like in one of our model configs:

The Slack post tags the person who last modified the code and provides a few handy links to help resolve the issue.

With a penchant for automating the boring things, we have quite a few bots. Other notable examples are the core models bot that helps the data team keep track of the daily run of our most business critical tables. We also have a stale tables bot which scans for tables that haven’t been updated in a given period of time, then archives them and ultimately deletes them.

Airflow is a well-supported and widely-used Apache project that has served us well. However, managing a self-hosted instance is a single point of failure and regularly requires maintenance. To reduce the work with self-hosting we will likely move to using Google Cloud Composer in 2022.

Data discovery and metadata management #

Data discovery is a more nascent component of the modern data stack. We’ve been using dbt docs — however with >4700 models the manifest.json and catalog.json is >300mbs, so the user experience isn’t great, particularly in the remote environment where some users may have slow internet. Part of our data governance is to have accountability for datasets: who should I contact regarding this dataset?

The Data Platform Engineering team is currently working on a solution to collect, store and surface metadata in an accessible way. We want to:

make our end to end data discoverable — our event data sits outside of dbt docs until it gets consumed into a data pipeline, but how do you discover the events in the first place?
have a single interface to consume the metadata, for example Amundsen, Marquez or DataHub
make sure metadata doesn’t only apply to assets in our data warehouse — it also includes our backend database (CassandraDB) and our visualisation assets (charts and dashboards)

Cloud compute / data warehouse #

A data platform needs a database. Google Cloud Platform (GCP) is a crucial part of Monzo’s data platform. BigQuery, is GCP’s serverless, highly scalable, and cost-effective multi-cloud data warehouse. At our scale, we pay a significant amount to Google every month so we are constantly working on ways to reduce our bill. Sometimes these are as easy as a configuration change for a model to execute in a different BigQuery project, or helpful tooling that detects these opportunities and notifies the data team of them. Other times it can involve a complete analytics engineering rethink of our data models (some are ~60 terabytes rebuilt daily).

For more information, check out this case study from Google on how we use BigQuery for analytics.

Data visualisation #

We use Looker for all of our dashboards. With automations like reports being sent to slack channels when particular conditions are met, Looker helps make Monzo a data-driven organisation. What struck me when joining Monzo was that the main company Key Performance Indicators (KPIs) were sent out to every employee every week. I’d never seen such transparency with a company’s performance. One of Monzo’s values is to default to transparency and with this transparency brings a focus on the key outcomes we want to achieve. By default, everyone who works at Monzo has access to Looker and over 80% of our staff are active users.

A small amount of custom data visualisation occurs in Google Sheets and in Google Colab (Google’s hosted Jupyter notebook solution).

The main challenge has been the responsiveness of Looker at our scale. Despite Looker’s importance, nobody in the company owns it explicitly. This has resulted in continual degradation of cleanliness, structure and speed over the last 4 years. To date we have run a couple of ad-hoc projects to optimise Looker cleanliness and loading for end users. We’ve also created automation scripts to clean up broken dashboards, charts and detect columns in LookML that no longer exist.

Machine Learning platform #

The Machine Learning (ML) platform deserves a separate Machine Learning Infrastructure blog post — watch this space! Briefly, our ML stack broadly sits at the intersection of Analytics Infrastructure (GCP), where we house and analyse our data, and the Backend (AWS), where we deploy our business logic. Our long-term strategy is to enable data scientists and backend engineers alike to safely and easily use machine learning as a tool, where appropriate, across the entire spectrum of problems we are tackling at Monzo.

Some organisations have data scientists make predictive models and once happy with them, software engineers then need to reimplement them in the production code base. At Monzo we decided this would slow the deployment of ML too much. So we built scripts that template a Python ML microservice using Sanic as the API webserver that our data scientists and ML engineers can use to deploy to production. We also use CoLab for quick and dirty exploration.

Influenced by Feast’s architectural overview, the Feature Store is a common component for production ML that automates the journey of shipping features between our analytics (BigQuery) and production (Cassandra) databases. For a detailed writeup on this component see Neal’s post on the topic.

For more information on ML at Monzo see these posts:

Recommendations #

Finally, if you’re interested in reading more on the topic, we love the thought leadership on modern analytics tooling from Tristan Handy in the analytics roundup and accompanying podcast, Benn Stencil’s substack, Martin Fowler (in particular the Data Mesh post by Zhamak Dehghani), and the Data Engineering Podcast to mention a few.

If you’re interested in working for the data team at Monzo, we’re hiring!

Rendering on the Apple M1 Max Chip

Code & Visuals — Mon, 25 Oct 2021 03:00:00 GMT

1. Introduction
2. Disclaimers
3. The M1 Max Chip
4. Application to Ray Tracing

5. Results
6. Conclusion
7. Bonus Images and Acknowledgements

Introduction

Over the past year, I ported my hobby renderer, Takua Renderer, to 64-bit ARM. I wrote up the entire process and everything I learned as a three-part blog post series covering topics ranging from assembly-level comparison between x86-64 and arm64, to deep dives into various aspects of Apple Silicon, to a comparison of x86-64’s SSE and arm64’s Neon vector instructions. In the intro to part 1 of my arm64 series, I wrote about my motivation for exploring arm64, and in the conclusion to part 2 of my arm64 series, I wrote the following about the Apple M1 chip:

There’s really no way to understate what a colossal achievement Apple’s M1 processor is; compared with almost every modern x86-64 processor in its class, it achieves significantly more performance for much less cost and much less energy. The even more amazing thing to think about is that the M1 is Apple’s low end Mac processor and likely will be the slowest arm64 chip to ever power a shipping Mac; future Apple Silicon chips will only be even faster.

Well, those future Apple Silicon chips are now here! Last week (relative to the time of posting), Apple announced new 14 and 16-inch MacBook Pro models, powered by the new Apple M1 Pro and Apple M1 Max chips. Apple reached out to me last week immediately after the announcement of the new MacBook Pros, and as a result, for the past week I’ve had the opportunity to use a prerelease M1 Max-equipped 2021 14-inch MacBook Pro as my daily computer. So, to my extraordinary surprise, this post is the unexpected Part 4 to what was originally supposed to be a two-part series about Takua Renderer on arm64. This post will serve as something of a coda to my Takua Renderer on arm64 series, but will also be fairly different in structure and content to the previous three parts. While the previous three parts dove deep into extremely technical details about arm64 assembly and Apple Silicon and such, this post will focus on a single question: now that professional-grade Apple Silicon chips exist in the wild, how well do high-end rendering workloads run on workstation-class arm64?

Disclaimers

Before we dive in, I want to get a few important details out of the way. First, this post is not really a product review or anything like that, and I will not be making any sort of endorsement or recommendation on what you should or should not buy; I’ll just be writing about my experiences so far. Many amazing tech reviewers exist out there, and if what you are looking for is a general overview and review of the new M1 Pro and M1 Max based MacBook Pros, I would suggest you go check out reviews by The Verge, Anandtech, MKBHD, Dave2D, LinusTechTips, and so on. Second, as with everything in this blog, the contents of this post represent only my personal opinion and do not in any way represent any kind of official or unofficial position, endorsement, or opinion on any matter from my employer, Walt Disney Animation Studios. When Apple reached out to me, I received permission from Disney Animation to go ahead on a purely personal basis, and beyond that nothing with this entire process involves Disney Animation. Finally, Apple is not paying me or giving me anything for this post; the 14-inch MacBook Pro I’ve been using for the past week is strictly a loaner unit that has to be returned to Apple at a later point. Similarly, Apple has no say over the contents of this post; Apple has not even seen any version of this post before publishing. What is here is only what I think!

The M1 Max Chip

Now that a year has passed since the first Apple Silicon arm64 Macs were released, I do have my hobby renderer up and running on arm64 with everything working, but I’ve only rendered relatively small scenes so far on arm64 processors. The reason I’ve stuck to smaller scenes is because high-end workstation-class arm64 processors so far just have not existed; while large server-class arm64 processors with large core counts and tons of memory do exist, these server-class processors are mostly found in huge server farms and supercomputers and are not readily available for general use. For general use, the only arm64 options so far have been low-power single-board computers like the Raspberry Pi 4 that are nowhere near capable of running large rendering workloads, or phones and tablets that don’t have software or operating systems or interfaces suitable for professional 3D applications, or M1-based Macs. I have been using an M1 Mac Mini for the past year, but while the M1 performance-wise punches way above what a 15 watt TDP typically would suggest, the M1 only supports up to 16 GB of RAM and only represents Apple’s entry into Apple Silicon based Macs. The M1 Pro and M1 Max, however, are are Apple’s first high powered arm64-based chips targeted at professional workloads, meant for things like high-end rendering and many other creative workloads; by extension, the M1 Pro and M1 Max are also the first arm64 chips of their class in the world with wide general availability. So, in this post, answering the question “how well do high-end rendering workloads run on workstation-class arm64” really means examining how well the M1 Pro and M1 Max can do rendering.

Spoiler: the answer is extremely well; all of the renders in the post were rendered on the 14-inch MacBook Pro with an M1 Max chip. Here is a screenshot of Takua Renderer running on the 14-inch MacBook Pro with an M1 Max chip:

The 14-inch MacBook Pro I’ve been using for the past week is equipped with the maximum configuration in every category: a full M1 Max chip with a 10-core CPU, 32-core GPU, 64 GB of unified memory, and 8 TB of SSD storage. However, for this post, I’ll only focus on the 10-core CPU and 64 GB of RAM, since Takua Renderer is currently CPU-only (more on that later); for a deep dive into the M1 Pro and M1 Max’s entire system-on-a-chip, I’d suggest taking a look at Anandtech’s great initial impressions and later in-depth review.

The first M1 Max spec that jumped out at me is the 64 GB of unified memory; having this amount of memory meant I could finally render some of the largest scenes I have for my hobby renderer. To test out the M1 Max with 64 GB of RAM, I chose the forest scene from my Mipmapping with Bidirectional Techniques post. This scene has enormous amounts of complex geometry; almost every bit of vegetation in this scene has highly detailed displacement mapping that has to be stored in memory, and the large amount of textures in this scene is what drove me to implement a texture caching system in my hobby renderer in the first place. In total, this scene requires just slightly under 30 GB of memory just to store all of the subdivided, tessellated, and displaced scene geometry, and requires an additional few more GB for the texture caching system (the scene can render with just a 1 GB texture cache, but having a larger texture cache helps significantly with performance).

I have only ever published two images from this scene: the main forest path view in the mipmapping blog post, and a closeup of a tree stump as the title image on my personal website. I originally had several more camera angles set up that I wanted to render images from, and I actually did render out 1080p images. However, to showcase the detail of the scene better, I wanted to wait until I had 4K renders to share, but unfortunately I never got around to doing the 4K renders. The reason I never did the 4K renders is because I only have one large personal workstation that has both enough memory and enough processing power to actually render images from this scene in a reasonable amount of time, but I needed this workstation for other projects. I also have a few much older spare desktops that do have just barely enough memory to render this scene, but unfortunately, those machines are so loud and so slow and produce so much heat that I prefer not to run them at all if possible, and I especially prefer not running them on long render jobs when I have to work-from-home in the same room! However, over the past week, I have been able to render a bunch of 4K images from my forest scene on the M1 Max 14-inch MacBook Pro; quite frankly, being able to do this on a laptop is incredible to me. Here is the title image from my personal website, but now rendered at 4K resolution on the M1 Max 14-inch MacBook Pro:

The M1 Max-based MacBook Pro is certainly not the first laptop to ever ship with 64 GB of RAM; the previous 2019 16-inch MacBook Pro was also configurable up to 64 GB of RAM, and there are crazy PC laptops out there that can be configured up even higher. However, this is where the M1 Max and M1 Pro’s CPU performance comes into play: while previous laptops could support 64 GB of RAM and more, actually utilizing large amounts of RAM was difficult since previous laptop CPUs often couldn’t keep up! Being able to fit a large data set into memory is one thing, but being able to run processing fast enough to actually make use of large data sets in a reasonable amount of time is the other half of the puzzle. My wife has a 2019 16-inch MacBook Pro with 32 GB of memory, which is just enough to render my forest scene. However, as seen in the benchmark results later in this post, the 2019 16-inch MacBook Pro’s Intel Core-i7 9750H CPU with 6 cores and 12 threads is over twice as slow as the M1 Max at rendering this scene at best, and can be even slower depending on thermals, power, and more. Rendering each of the images in this post took a few hours on the M1 Max, but on the Core-i7 9750H, the renders have to become overnight jobs with the 16-inch MacBook Pro’s fans running at full speed. With only a week to write this post, a few hours per image versus an overnight job per image made the difference between having images ready for this post versus not having any interesting renders to show at all!

Actually, the M1 Max isn’t just fast for a chip in a laptop; the M1 Max is stunningly competitive even with desktop workstation CPUs. For the past few years, the large personal workstation that I offload large projects onto has been a machine with dual Intel Xeon E5-2680 workstation processors with 8 cores / 16 threads each for a total of 16 cores and 32 threads. Even though the Xeon E5-2680s are ancient at this point, this workstation’s performance is still on-par with that of the current Intel-based 2020 27-inch iMac. The M1 Max is faster then the dual-Xeon E5-2680 workstation at rendering my forest scene, and considerably so. But of course, a comparison with aging Sandy Bridge era Xeons isn’t exactly a fair sporting competition; the M1 Max has almost a decade of improved processor design and die shrinks to give it an advantage. So, I also tested the M1 Max against… the current generation 2019 Mac Pro, which uses a Intel Xeon W-3245 CPU with 16 cores and 32 threads. As expected, the M1 Max loses to the 2019 Mac Pro… but not by a lot, and for a fraction of the power used. The Intel Xeon W-3245 has a 205 watt TDP just for the CPU alone and has to be utilized in a huge desktop tower with an extremely elaborate custom-engineered cooling solution, whereas the M1 Max 14-inch MacBook Pro has a reported whole-system TDP of just 60 watts!

How does Apple pack so much performance with such little energy consumption into their arm64 CPU designs? A number of factors come into play here, ranging from partnering with TSMC to manufacture on cutting-edge 5 nm process nodes to better microarchitecture design to better software and hardware integration; outside of Apple’s processor engineering labs, all anyone can really do is just hypothesize and guess. However, there are some good guesses out there! Several plausible theories have to do with the choice to use the arm64 instruction set; the argument goes that having been originally designed for low-power use cases, arm64 is better suited for efficient energy consumption than x86-64, and scaling up a more efficient design to huge proportions can mean more capable chips that use less power than their traditional counterparts. Another theory revolving around the arm64 instruction set has to do with microarchitecture design considerations. The M1, M1 Pro, and M1 Max’s high-performance “Firestorm” cores have been observed to have an absolutely humongous reorder buffer, which enables extremely deep out-of-order execution capabilities; modern processors attain a lot of their speed by reordering incoming instructions to do things like hide memory latency and bypass stalled instruction sequences. The M1 family’s high-performance cores posses an out-of-order window that is around twice as large as that in Intel’s current Willow Cove microarchitecture and around three times as large as that in AMD’s current Zen3 microarchitectures. Having a huge reordering buffer supports the M1 family’s high-performance cores also having a high level of instruction-level parallelism enabled by extremely wide instruction execution and extremely wide instruction decoding. While wide instruction decoding is certainly possible on x86-64 and other architectures, scaling wide instruction-issue designs in a low power budget is generally accepted to be a very challenging chip design problem. The theory goes that arm64’s fixed instruction length and relatively simple instructions make implementing extremely wide decoding and execution far more practical for Apple, compared with what Intel and AMD have to do in order to decode x86-64’s variable length, often complex compound instructions.

Application to Ray Tracing

So what does any of the above have to do with ray tracing? One concrete application has to do with opacity mapping in a ray tracing renderer. Opacity maps are used to produce finer geometric detail on surfaces by using a texture map to specify whether a part of a given surface should actually exist or not. Implementing opacity mapping in a ray tracer creates a surprisingly large number of design considerations that need to be solved for. For example, texture lookups are usually done as part of a renderer’s shading system, which in a ray tracer only runs after ray intersection has been carried out. However, evaluating whether or not a given hit point against a surface should be ignored or not after exiting the entire ray traversal system leads to massive inefficiencies due to the need to potentially re-enter the entire ray traversal system from scratch again. As an example: imagine a tree where all of the leaves are modeled as rectangular cards, and the shape of each leaf is produced using an opacity map on each card. If the renderer wants to test if a ray hits any part of the tree, and the renderer is architected such that opacity map lookups only happen in the shading system, then the renderer may need to cycle back and forth between the traversal and shading systems for every leaf encountered in a straight line path through the tree (and trees have a lot of leaves!). An alternative way to handle opacity hits is to allow for direct texture map lookups or to evaluate opacity procedurally from within the traversal system itself, such that the renderer can immediately decide whether to accept a hit or not without having to exit out and run the shading system; this approach is what most renderers use and is what ray tracing libraries like Embree and Optix largely expect. However, this method produces a different problem: tight inner loop ray traversal code is now potentially dependent on slow texture fetches from memory! Both of these approaches to implementing opacity mapping have downsides and potential performance impacts, which is why often times just modeling detail into geometry instead of using opacity mapping can actually result in faster ray tracing performance, despite the heavier geometry memory footprint. However, opacity mapping is often a lot easier to set up compared with modeling detail into geometry, and this is where a deep out-of-order buffer coupled with good branch prediction can make a big difference in ray tracing performance; these two tools combined can allow the processor to proceed with a certain amount of ray traversal work without having to wait for opacity map decisions. Problems similar to this, coupled with the lack of out-of-order and speculative execution on GPUs, play a large role in why GPU ray tracing renderers often have to be architecture fairly differently from CPU ray tracing renderers, but that’s a topic for another day.

I give the specific example above because it turns out that the M1 Max’s deep reordering capabilities seem to make a fairly noticeable difference in my Takua Renderer’s performance when opacity maps are used extensively! In the following rendered image, the ferns have an extremely detailed, complex appearance that depends heavily on opacity maps to cut out leaf shapes from simple underlying geometry. In this case, I found that the slowdown introduced by using opacity maps in a render on the M1 Max is proportionally much lower than the slowdown introduced when using opacity maps in a render on the x86-64 machines that I tested. Of course, I have no way of knowing if the above theory for why the M1 Max seems to handle renders that use opacity maps better is correct, but whichever way, the end results look very nice and renders faster than on any other computer that I have!

In terms of whether the M1 Pro or the M1 Max is better for CPU rendering, I only have the M1 Max to test, but my guess is that there shouldn’t actually be too large of a difference as long as the scene fits in memory. However, the above guess comes with a major caveat revolving around memory bandwidth. Where the M1 Pro and M1 Max differ is in the maximum number of GPU cores and maximum amount of unified memory configurable; the M1 Pro can go up to 16 GPU cores and 32 GB of RAM, while the M1 Max can go up to 32 GPU cores and 64 GB of RAM. Outside of the GPU and maximum amount of memory, the M1 Pro and M1 Max chips actually share identical CPU configurations: both of them have a 10-core arm64 CPU with 8 high-performance cores and 2 energy-efficient cores, implementing a custom in-house Apple-designed microarchitecture. However, for some workloads, I would not be surprised if the M1 Max is actually slightly faster since the M1 Max also has twice the memory bandwidth over the M1 Pro (400 GB/s on M1 Max versus 200 GB/s M1 Pro); this difference comes from the M1 Max having twice the number of memory controllers. While consumer systems such as game consoles and desktop GPUs often do ship with memory bandwidth numbers comparable or even better than the M1 Max’s 400 GB/s, seeing these levels of memory bandwidth in even workstation CPUs is relatively unheard of. For example, AMD’s monster flagship Ryzen Threadripper 3990X is currently the most powerful high-end desktop CPU on the planet (outside of server processors), but the 3990X’s maximum memory bandwidth tops out at 95.37 GiB/s, or 165.944 GB/s; seeing the M1 Max MacBook Pro ship with over twice the memory bandwidth compared to the Threadripper 3990X is pretty wild. The M1 Max also has twice the amount of system-level cache as the M1 Pro; on the M1 family of chips, the system-level cache is loosely analogous to L3 cache on other processors, but serves the entire system instead of just the CPU cores.

Production-grade CPU ray tracing is a process that depends heavily on being able to pin fast CPU cores at close to 100% utilization for long periods of time, while accessing extremely large datasets from system memory. In an ideal world, intensive computational tasks should be structured in such a way that data can be pulled from memory in a relatively coherent, predictable manner, allowing the CPU cores to rely on data in cache over fetching from main memory as much as possible. Unfortunately, making ray tracing coherent enough to utilize cache well is an extremely challenging problem. Operations such as BVH traversal, which finds the closest point in a scene that a ray intersects, essentially represent an arbitrarily random walk through potentially vast amounts of geometry stored in memory, and any kind of incoherent walk through memory makes overall CPU performance dependent on memory performance. As a result, operations like BVH traversal tend to be heavily bottlenecked by memory latency and memory bandwidth. I expect that the M1 Max’s strong memory bandwidth numbers should provide a some performance boost for rendering compared to the M1 Pro. A complicating factor, however, is how the additional memory bandwidth on the M1 Max is utilized; not all of it is available to just the CPU, since the M1 Max’s unified memory needs to also serve the system’s GPU, neural processing systems, and other custom onboard logic blocks. The actual real-world impact should be easily testable by rendering the same scene on a M1 Pro and a M1 Max chip both with 32 GB of RAM, but in the week that I’ve had to test the M1 Max so far, I haven’t had the time or ability to be able to carry out this test on my own. Stay tuned; I’ll update this post if I am able to try this test soon!

I’m very curious to see if the increased memory bandwidth on the M1 Max will make a difference over the M1 Pro on this forest scene in particular, due to how dense some of the geometry is and therefore how deep some of the BVHs have to go. For example, every single pine needle in this next image is individually modeled geometry, and every tree trunk has sub-pixel-level tessellation and displacement; being able to render this image on a MacBook Pro instead of a giant workstation is incredible:

In the previous posts about running Takua Renderer on arm64 processors, I included performance testing results across a wide variety of machines ranging from the Raspberry Pi 4B to the M1 Mac Mini all the way up to my dual Intel Xeon E5-2680 workstation. However, all of those tests weren’t necessarily indicative of what real world rendering performance on huge scenes would be like, since all of those tests had to use scenes that were small enough to fit in to a M1 Mac Mini’s 16 GB memory footprint. Now that I have access to a M1 Max MacBook Pro with 64 GB of memory, I can present some initial performance comparisons with larger machines rendering my forest scene. I think these results are likely more indicative of what real-world production rendering performance looks like, since the forest scene is the closest thing I have to true production complexity (I haven’t ported the Disney’s Moana Island data set to work in my renderer yet).

The machines I tested this time are a 2021 14-inch MacBook Pro with an Apple M1 Max chip with 10 cores (8 performance, 2 efficiency) and 10 threads, a 2019 16-inch MacBook Pro with an Intel Core i7-9750H CPU with 6 cores and 12 threads, a 2019 Mac Pro with an Intel Xeon W-3245 CPU with 16 cores and 32 threads, and a Linux workstation with dual Intel Xeon E5-2680 CPUs with 8 cores and 16 threads per CPU for a total of 16 cores and 32 threads. The Xeon E5-2680 workstation is, quite franky, ancient, and makes for something of a strange comparison point, but it’s the main workstation that I use for personal rendering projects at the moment, so I included it. I don’t exactly have piles of the latest server and workstation chips just laying around my house, so I had to work with what I got! However, I was also able to borrow access to a Windows workstation with an AMD Threadripper 3990X CPU, which weighs in with 64 cores and 128 threads. I figured that the Threadripper 3990X system is not at all a fair comparison point for the exact opposite reason why the Xeon E5-2680 is not a fair comparison point, but I thought I’d throw it in anyway out of sheer curiosity. Notably, the regular Apple M1 chip does not make an appearance in these tests, since the forest scene doesn’t fit in memory on the M1. I also borrowed a friend’s Razer Blade 15 to test, but wound up not using it since I discovered that it has the same Intel Core i7-9750H CPU as the 2019 16-inch MacBook Pro, but only has half the memory and therefore can’t fit the scene.

In the case of the two MacBook Pros, I did all tests twice: once with the laptops plugged in, and once with the laptops running entirely on battery power. I wanted to compare plugged-in versus battery performance because of Apple’s claim that the new M1 Pro/Max based MacBook Pros perform the same whether plugged-in or on battery. This claim is actually a huge deal; laptops traditionally have had to throttle down CPU performance when unplugged to conserve battery life, but the energy efficiency of Apple Silicon allows Apple to no longer have to do this on M1-family laptops. I wanted to verify this claim for myself!

Results

In the results below, I present three tests using the forest scene. The first test measures how long Takua Renderer takes to run subdivision, tessellation, and displacement, which has to happen before any pixels can actually be rendered. The subdivision/tessellation/displacement process has an interesting performance profile that looks very different from the performance profile of the main path tracing process. Subdivision within a single mesh is not easily parallelizable, and even with a parallel implementation, scales very poorly beyond just a few threads. Takua Renderer attempts to scale subdivision widely by running subdivision on multiple meshes in parallel, with each mesh’s subdivision task only receiving an allocation of at most four threads. As a result, the subdivision step actually benefits slightly more from single-threaded performance over a larger number of cores and greater multi-threaded performance. The second test is rendering the main view of the forest scene from my mipmapping blog post, at 1920x1080 resolution. I chose to use 1920x1080 resolution since most of the time this is a more common maximum resolution to be using while working on artistic iteration. The third test is rendering the fern view of the forest scene from Figure 2 of this post, at final 4K 3840x2160 resolution. For both of the main rendering tests, I only ran the renderer for 8 samples per pixel, since I didn’t want to sit around for days to collect all of the data. For each test, I did five runs, discarded the highest and lowest results, and averaged the remaining three results to get the numbers below. Wall time (as in a clock on a wall) measures the actual amount of real-world time that each test took, while core-seconds is an approximation of how long each test would have taken running on a single core. So, wall time can be thought of as a measure of total computation power, whereas core-seconds is more a measure of computational efficiency; in both cases, lower numbers are better:

	Forest Subdivision/Displacement
Processor:	Wall Time:	Core-Seconds:
Apple M1 Max (Plugged in):	128 s	approx 1280 s
Apple M1 Max (Battery):	128 s	approx 1280 s
Intel Core i7-9750H (Plugged in):	289 s	approx 3468 s
Intel Core i7-9750H (Battery):	307 s	approx 3684 s
Intel Xeon W-3245:	179 s	approx 5728 s
Intel Xeon E5-2680 x2:	222 s	approx 7104 s
AMD Threadripper 3990X:	146 s	approx 18688 s

	Forest Rendering (Main Camera)
	1920x1080, 8 spp, PT
Processor:	Wall Time:	Core-Seconds:
Apple M1 Max (Plugged in):	127.143 s	approx 1271.4 s
Apple M1 Max (Battery):	126.421 s	approx 1264.2 s
Intel Core i7-9750H (Plugged in):	288.089 s	approx 3457.1 s
Intel Core i7-9750H (Battery):	347.898 s	approx 4174.8 s
Intel Xeon W-3245:	106.332 s	approx 3402.6 s
Intel Xeon E5-2680 x2:	158.255 s	approx 5064.2 s
AMD Threadripper 3990X:	38.887 s	approx 4977.5 s

	Forest Rendering (Fern Camera)
	3840x2160, 8 spp, PT
Processor:	Wall Time:	Core-Seconds:
Apple M1 Max (Plugged in):	478.247 s	approx 4782.5 s
Apple M1 Max (Battery):	496.384 s	approx 4963.8 s
Intel Core i7-9750H (Plugged in):	1084.504 s	approx 13014.0 s
Intel Core i7-9750H (Battery):	1219.59 s	approx 14635.1 s
Intel Xeon W-3245:	345.292 s	approx 11049.3 s
Intel Xeon E5-2680 x2:	576.279 s	approx 18440.9 s
AMD Threadripper 3990X:	108.2596 s	approx 13857.2 s

When rendering the main camera view, the 2021 14-inch MacBook Pro used on average about 7% of its battery charge, while the 2019 16-inch MacBook Pro used on average about 39% of its battery charge. When rendering the fern view, the 2021 14-inch MacBook Pro used on average about 19% of its battery charge, while the 2019 16-inch MacBook Pro used on average about 48% of its battery charge. Overall by every metric, the 2021 14-inch MacBook Pro achieves an astounding victory over the 2019 16-inch MacBook Pro: a little over twice the performance for a fraction of the total power consumption. The 2021 14-inch MacBook Pro also lives up to Apple’s claim of identical performance plugged in and on battery power, whereas in the results above, the 2019 16-inch MacBook Pro suffers anywhere between a 25% to 50% performance hit just from switching to battery power. The 2021 14-inch MacBook Pro’s performance win is even more astonishing when considering that the 2019 16-inch MacBook Pro is the previous flagship that the new M1 Pro/Max MacBook Pros are the direct successors to. Seeing this kind of jump in a single hardware generation is unheard of in modern tech and represents a massive win for both Apple and for the arm64 ISA. The M1 Max also handily beats the old dual Intel Xeon E5-2680 that I am currently using by a comfortable margin; for my personal workflow, this means that I can now do everything that I previously needed a large loud power-hungry workstation for on the 2021 14-inch MacBook Pro, and I can do everything faster on the 2021 14-inch MacBook Pro too.

The real surprises to me came with the 2019 Mac Pro and the Threadripper 3990X workstation. In both of those cases, I expected the M1 Max to lose, but the 2021 14-inch MacBook Pro came surprisingly close to the 2019 Mac Pro’s performance in terms of wall time. Even more importantly as a predictor of future scalability, the M1 Max’s efficiency as measured by core-seconds comes in at far far superior to both the Intel Xeon W-3245 and the AMD Threadripper 3900X. Imagining what a hypothetical future Apple Silicon iMac or Mac Pro with an even more scaled up M1 variant, or perhaps some kind of multi-M1 Max chiplet or multisocket solution, is extremely exciting! I think that with the upcoming Apple Silicon based large iMac and Mac Pro, Apple has a real shot at beating both Intel and AMD’s highest end CPUs to win the absolute workstation performance crown.

Of course, what makes the M1 Max’s performance numbers possible is the M1 Max’s energy efficiency; this kind of performance-per-watt is simply unparalleled in the desktop (meaning non-mobile, not desktop form factor) processor world. The M1 architecture’s energy efficiency is what allows Apple to scale the design out into the M1 Pro and M1 Max and hopefully beyond. Below is a breakdown of energy utilization for each of the rendering tests above; the total energy used for each render is the wall clock render time multiplied by the maximum TDP of each processor to get watt-seconds, which is then translated to watt-hours. I assume maximum TDP for each processor since I ran Takua Renderer with processor utilization set to 100%. For the two MacBook Pros, I’m just reporting the plugged-in results.

	Forest Rendering (Main Camera)
	1920x1080, 8 spp, PT
Processor:	Max TDP:	Total Energy Used:
Apple M1 Max:	60 W	2.1191 Wh
Intel Core i7-9750H:	45 W	3.6011 Wh
Intel Xeon W-3245:	205 W	6.0550 Wh
Intel Xeon E5-2680 x2:	260 W	11.4295 Wh
AMD Threadripper 3990X:	280 W	3.0246 Wh

	Forest Rendering (Fern Camera)
	3840x2160, 8 spp, PT
Processor:	Max TDP:	Total Energy Used:
Apple M1 Max:	60 W	7.9708 Wh
Intel Core i7-9750H:	45 W	13.5563 Wh
Intel Xeon W-3245:	205 W	19.6625 Wh
Intel Xeon E5-2680 x2:	260 W	41.6202 Wh
AMD Threadripper 3990X:	280 W	8.4202 Wh

At least for my rendering use case, the Apple M1 Max is easily the most energy efficient processor, even without taking into account that the 60 W TDP of the M1 Max is for the entire system-on-a-chip including CPU, GPU, and more, while the TDPs for all of the other processors are just for a CPU and don’t take into account the rest of the system. The M1 Max manages to beat the 2019 16-inch MacBook Pro’s Intel Core i7-9750H in absolute performance by a factor of two whilst using anywhere between a two-thirds to half of the energy, and the M1 Max comes close to matching the 2019 Mac Pro’s absolute performance while using about a third of the energy. Of course the comparison with the Intel Xeon E5-2680 workstation isn’t exactly fair since the M1 Max is manufactured using a 5 nm process while the ancient Intel Xeon E5-2580s were manufactured on a 35 nm process a decade ago, but I think the comparison still underscores just how far processors have advanced over the past decade leading up to the M1 Max. The only processor that really comes near the M1 Max in terms of energy efficiency is the AMD Threadripper 3990X, which makes sense since the AMD Threadripper 3990X and the M1 Max are the closest cousins in this list in terms of manufacturing process; both are using leading-edge TSMC photolithography. However, on a whole, the M1 Max is still more efficient than the AMD Threadripper 3990X, and again, the AMD Threadripper 3990X TDP is for just a CPU, not an entire SoC! Assuming near-linear scaling, a hypothetical M1-derived variant that is scaled up 4.5 times to a 270 W TDP should be able to handily defeat the AMD Threadripper 3990X in absolute performance.

The wider takeaway here though is that in order to give the M1 Max some real competition, one has to skip laptop chips entirely and reach for not just high end desktop chips, but for server-class workstation hardware to really beat the M1 Max. For workloads that push the CPU to maximum utilization for sustained periods of time, such as production-quality path traced rendering, the M1 Max represents a fundamental shift in what is possible in a laptop form factor. Something even more exciting to think about is how the M1 Max really is the middle tier Apple Silicon solution; presumably the large iMac and Mac Pro will push things into even more absurd territory.

Conclusion

So those are my initial thoughts on the Apple M1 Max chip and my initial experiences with getting my hobby renderer up and running on the 2021 14-inch MacBook Pro. I’m extremely impressed, and not just with the chip! This post mostly focused on the chip itself, but the rest of the 2021 MacBook Pro lineup is just as impressive. For rendering professionals and enthusiasts alike, one aspect of the 2021 MacBook Pros that will likely be just as important as the processor is the incredible screen. The 2021 MacBook Pros ship with what I believe is an industry first: a micro-LED backlit 120 Hz display with an extended dynamic range that can go up to 1600 nits peak brightness. The screen is absolutely gorgeous, which is a must for anyone who spends their time generating pixels with a 3D renderer! One thing on my to-do list was to add extended dynamic range support to Thomas Müller’s excellent tev image viewer, which is a popular tool in the rendering research community. However, it turns out that Thomas already added extended dynamic range support, and it looks amazing on the 2021 MacBook Pro’s XDR display.

In this post I didn’t go into the M1 Max’s GPU at all, even though the GPU in many ways might actually be even more interesting than the CPU (which is saying a lot considering how interesting the CPU is). On paper at least, the M1 Max’s GPU aims for roughly mobile NVIDIA GeForce RTX 3070 performance, but how the M1 Max and a mobile NVIDIA GeForce RTX 3070 actually will compare for ray traced rendering is difficult to say without actually conducting some tests. On one hand, the M1 Max’s unified memory architecture grants its GPU far more memory than any NVIDIA mobile GPU by a huge margin, and the M1 Max’s unified memory architecture opens up a wide variety of interesting optimizations that are otherwise difficult to do when managing separate pools of CPU and GPU memory. On the other hand though, the M1 Max’s GPU lacks the dedicated hardware ray tracing acceleration that modern NVIDIA and AMD GPUs and the upcoming Intel discrete GPUs all have, and in my experience so far, dedicated hardware ray tracing acceleration makes a huge difference in GPU ray tracing performance. Maybe Apple will add hardware ray tracing acceleration in the future; Metal already has software ray tracing APIs, and there already is a precedent for Apple Silicon including dedicated hardware for accelerating relatively niche, specific professional workflows. As an example, the M1 Pro and M1 Max include hardware ProRes acceleration for high-end video editing. Over the next year, I am undertaking a large-scale effort to port the entirety of Takua Renderer to work on GPUs through CUDA on NVIDIA GPUs, and through Metal on Apple Silicon devices. Even though I’ve just gotten started on this project, I’ve already learned a lot of interesting things comparing CUDA and Metal compute; I’ll have much more to say on the topic hopefully soon!

Beyond the CPU and GPU and screen, there are still even more other nice features that the new MacBook Pros have for professional workflows like high-end rendering, but I’ll skip going through them in this post since I’m sure they’ll be thoroughly covered by all of the various actual tech reviewers out on the internet.

Bonus Images and Acknowledgements

To conclude for now, here are two more bonus images that I rendered on the M1 Max 14-inch MacBook Pro. I originally planned on just rendering the earlier three images in this post, but to my surprise, I found that I had enough time to do a few more! I think that kind of encapsulates the M1 Pro and M1 Max MacBook Pros in a nutshell: I expected incredible performance, but was surprised to find even my high expectations met and surpassed.

A huge thanks to everyone at Apple that made this post possible! Also a big thanks to Rajesh Sharma and Mark Lee for catching typos and making some good suggestions.

Why Robots Deserve Rights

Non-Human Party — Sun, 24 Oct 2021 17:58:08 GMT

Upon introducing the Non-Human Party to excited fans of democracy, a common question is what it really means to give rights to Tamagotchis; robots; or IoT toasters − “they can’t vote!

Reading books

Paul Cobbaut's blog — Sun, 24 Oct 2021 17:11:59 GMT

This post is meant to encourage me to read a bit more (paper books). By the way, I thought I was reading four books simultaneously, but when I put them next to each other it turned out to be seven.

Form left to right (writer-title(year) pages read-total pages):

Daniele Benedettelli - Creating Cool Mindstorms NXT Robots(2008) 24-575
Leo Tolstoj - Oorlog en Vrede(1869, NL translation 1973) 115-462
Dirk De Wachter - De kunst van het ongelukkig zijn(2019) 35-101
Allen/Fonagy/Bateman - Mentaliseren(2008/2019 edition) 30-368
LEGO and philosophy(2017) 56-226
Charlie Mackesy - The Boy, the mole, the fox and the horse(2019)
Michael Collins - Carrying the Fire(1974) - finished

Personal goal: finish at least three more of these before 2022.

I just finished Michael Collins - Carrying the Fire and it took me five weeks, which I consider a bit too long for a 470-ish pages book. It was a very good book though. If you're into space travel, then I would definitely recommend it. It also proves how the Sixties was vastly different to today, for example of the fourteen astronauts selected in 1963, four died during training. Two of the nine also died. Such numbers are unacceptable in 2021, even for 'dangerous' jobs.

The Daniele Benedettelli book is about programming Finite State Machines using Lego robots. I don't know much about programming, but this looks like fun. Thing is I need to build some Lego robots (like this one) to continue this book.

And I probably need to start from page 1 again in War and Peace because of the many characters that I forgot.

Some other books that I read the past three years are:

Celestin-Westreich/Celestin - Observeren en Rapporteren
Dick Swaab - Ons creatieve brein

Dirk De Wachter - Borderline Times
Dirk De Wachter - De wereld van De Wachter
Etienne Vermeersch - Over God
Etienne Vermeersch - Provencaalse gesprekken
Jan Van de Craats - Basisboek wiskunde
Jude Woodward - The US vs China
Paul Verhaeghe - Autoriteit
Paul Verhaeghe - Identiteit
Randall Munroe - Thing Explainer
Randall Munroe - What If
Rebecca Smethurst - Space, 10 things you should now
Robert Bly - De Wildeman
Terry Goodkind - Law of Nines
Terry Goodkind - Severed Souls
Terry Goodkind - The first Confessor
Terry Goodkind - The Omen Machine

Terry Goodkind - The Third Kingdom
Terry Goodkind - Warheart
Thomas D'ansembourg - Stop met aardig zijn

Most of it non-fiction apparently. I really enjoyed 'Borderline Times' and both books of Paul Verhaeghe and Dick Swaab. I couldn't really get into Robert Bly or Thomas D'Ansembourg (but collaborative communication gives me a lot of insight in people).

Scandinavias Biggest Rapper – Einár

Posts on A blog — Sun, 24 Oct 2021 14:16:37 GMT

Nils Erik Einar Grönberg AKA. Einár was a Swedish rapper. The 19 year old rap sensation was shot to death late Thursday night October 21, 2021. Einàr was the most played artist on Spotify in Sweden 2019. I discovered his music in the early 2019 with the track “Katten i trakten” on repeat. The track was on his debut album Forsta klass that topped the Swedish charts in 2019, without the need of any (major) record labels help.

What's on My Desk?

Gunnar Morling — Sun, 24 Oct 2021 11:30:00 GMT

Table of Contents

The Screen
Compute
Cameras
Work in Progress: Teleprompter
Lighting
Audio
What’s Next?

I’ve been working from home exclusively for the last nine years, but it was only last year that I started to look into ways for expanding my computer set-up and go beyond the usual combination of having a laptop with your regular external screen. The global COVID-19 pandemic, the prospect of having more calls with colleagues than ever (no physical meetings), and the constantly increasing need for recording talks for online conferences and meet-ups made me reevaluate things and steadily improve and fine tune my set-up, in particular in regards to better video and audio quality.

ARM on the desktop is here to stay

Jason Eckert's Website and Blog — Sun, 24 Oct 2021 03:00:00 GMT

Back in the 1990s, Reduced Instruction Set Computing (RISC) architecture was widely seen as the future. UNIX workstations from Silicon Graphics and Sun Microsystems boasted powerful RISC CPUs at the time, but were prohibitively expensive and died out in the early 2000s as a result. Only the Acorn RISC Machine (ARM) architecture used in low-power devices (e.g. PDAs) survived past the early 2000s, and only because of the rise of mobile devices that relied on battery power.

Exploring PL/pgSQL: Strings, arrays, recursion, and parsing JSON

Notes on software development — Sun, 24 Oct 2021 03:00:00 GMT

Next in exploring PL/pgSQL:
Implementing a Forth-like interpreter

PostgreSQL comes with a builtin imperative programming language called PL/pgSQL. I used to think this language was scary because it has a bit more adornment than your usual language does. But looking deeper, it's actually reasonably pleasant to program in.

In this post we'll get familiar with it by working with strings, arrays and recursive functions. We'll top it all off by building a parser for a subset of JSON (no nested objects, no arrays, no unicode, no decimals).

The goal here is not production-quality code (an amazing JSON library is already built into PostgreSQL) but simply to get more familiar with the PL/pgSQL language.

All code for this post is available on Github.

Creating functions

Functions are declared like tables. Here's a very simple one that returns the length of a string:

CREATE OR REPLACE FUNCTION slength(s text) RETURNS int AS $$
BEGIN
  RETURN length(s);
END;
$$ LANGUAGE plpgsql;

It's not a very useful function because length already exists but the point is to see a basic custom function.

All statements in PL/pgSQL must end in a semicolon. Arguments do not have to be named. If they are not named they get default names of $1 to $N.

Named/unnamed arguments

Here's how the function could be written without named arguments:

CREATE OR REPLACE FUNCTION slength(text) RETURNS int AS $$
BEGIN
  RETURN length($1);
END;
$$ LANGUAGE plpgsql;

Out declarations

PL/pgSQL also allows you to declare which variables will be returned in the function argument list. They call it OUT parameters but as far as I can tell it is not like OUT parameters in C# where you are modifying the value of a variable in an external scope.

CREATE OR REPLACE FUNCTION slength(s text, OUT i int) RETURNS int AS $$
BEGIN
  i = length(s);
END;
$$ LANGUAGE plpgsql;

This is still equivalent to the first function and is basically a shortcut for:

CREATE OR REPLACE FUNCTION slength(s text) RETURNS int AS $$
DECLARE
  i int;
BEGIN
  i = length(s);
  RETURN i;
END;
$$ LANGUAGE plpgsql;

Whether you declare OUT or not you still must include RETURNS <type> in the function signature otherwise even if you call RETURN in the body, the result will just be ignored.

Don't worry about case sensitivity too much. It's really only important, as in typical SQL, for mixed-case table and column names. But we won't be dealing with that situation in this article focused on programming PL/pgSQL.

Testing it out

Once the function is created, you can call it like SELECT slength('foo');. So here's a helper script to load a SQL file and run a command:

$ cat ./test.sh
sudo -u postgres psql -c "$(printf "%s;\n%s" "$(cat $1)" "$2")"
$ chmod +x ./test.sh

After storing the above slength code in slength.sql we can run a test:

$ ./test.sh ./slength.sql "SELECT slength('foo')"
 slength
---------
       3
(1 row)

Easy!

Numbers and recursion

Ok now that we've got the basics of function definition down and a way to test the code, let's write a fibonacci program.

$ cat ./fib.sql
CREATE OR REPLACE FUNCTION fib(i int) RETURNS int AS $$
BEGIN
  IF i = 0 OR i = 1 THEN
    RETURN i;
  END IF;

  RETURN fib(i - 1) + fib(i - 2);
END;
$$ LANGUAGE plpgsql;

Everything in the if test is normal SQL WHERE clause syntax. This makes it very easy for folks familiar with SQL to pick up conditionals in PL/pgSQL.

And there's no special syntax to allow function recursion. Nice!

Run and test this function:

$ ./test.sh ./fib.sql "SELECT fib(10)"
 fib
-----
  55
(1 row)

Getting the hang of it?

Strings and arrays

You may have noticed that length used in slength is a builtin PostgreSQL function for dealing with strings. All builtin functions in PostgreSQL can be used in PL/pgSQL.

In order to get familiar with using arrays in PL/pgSQL let's write a string_to_array function.

$ cat ./string_to_array.sql
CREATE OR REPLACE FUNCTION string_to_array(s text) RETURNS char[] AS $$
DECLARE
  a char[];
BEGIN
  WHILE COALESCE(array_length(a, 1), 0) < length(s) LOOP
    a[COALESCE(array_length(a, 1), 0) + 1] = substr(s, COALESCE(array_length(a, 1), 0) + 1, 1);
  END LOOP;
  RETURN a;
END;
$$ LANGUAGE plpgsql;

This is one way to do it by modify array values directly by index. We need to coalesce because calling array_length on an empty array returns NULL.

Another way to do this is by calling the builtin function array_append.

CREATE OR REPLACE FUNCTION string_to_array(s text) RETURNS char[] AS $$
DECLARE
  a char[];
BEGIN
  WHILE COALESCE(array_length(a, 1), 0) < length(s) LOOP
    a = array_append(a, substr(s, COALESCE(array_length(a, 1), 0) + 1, 1)::char);
  END LOOP;
  RETURN a;
END;
$$ LANGUAGE plpgsql;

We can test and run both:

$ ./test.sh ./string_to_array.sql "SELECT string_to_array('foo')"
 string_to_array
-----------------
 {f,o,o}
(1 row)
$ ./test.sh ./string_to_array2.sql "SELECT string_to_array('foo')"
 string_to_array
-----------------
 {f,o,o}
(1 row)

Of course the builtin alternative might be SELECT regexp_split_to_array('foo') but we need the practice.

Custom compound types

If we're going to lex and parse JSON, we're going to want to return an array of tokens from the lexer. A token will need to contain the type (e.g. number, string, syntax) and the string value of the token (e.g. 1, {, my great key).

PostgreSQL allows us to create compound types that we can then use as the base of an array:

DROP TYPE IF EXISTS json_token CASCADE;
CREATE TYPE json_token AS (
  kind text,
  value text
);

We need to add CASCADE here because functions will have this type in their signature and it otherwise makes PostgreSQL unhappy to delete the type used in a function before deleting the function.

We can create literals of this type like SELECT ('number', '12')::json_token).

Now we're ready to build out the lexer.

Lexing

The lexers job is to clump together groups of characters into tokens.

I'm going to describe this function in literate code.

CREATE OR REPLACE FUNCTION json_lex(j text, OUT ts json_token[]) RETURNS json_token[] AS $$

This function takes a string in and returns an array of json tokens.

DECLARE 
  i int = 1; -- Index in loop
  c text; -- Current character in loop
  token text; -- Current accumulated characters

We need to declare all variables up front.

BEGIN
  WHILE i < length(j) + 1 LOOP
    c = substr(j, i, 1);
    token = '';

The main loop just looks at all characters.

    -- Handle syntax characters
    IF c = '{' OR c = '}' OR c = ',' OR c = ':' THEN
      ts = array_append(ts, ('syntax', c)::json_token);
      i = i + 1;
      CONTINUE;
    END IF;

First we look if the character is a syntax character. If it is we append it to the array of tokens, increment the index, and go back to the start of the main loop.

    -- Handle whitespace
    IF regexp_replace(c, '^\s+', '') = '' THEN
      i = i + 1;
      CONTINUE;
    END IF;

Then we check for whitespace characters. If replacing all whitespace characters returns an empty string then we know it's whitespace. We could also have done something like IF c = ' ' OR c = '\n' ... THEN instead.

Same as before though if we find whitespace characters we move on (don't accumulate them) and restart the main loop.

    -- Handle strings
    IF c = '"' THEN
      i = i + 1;
      c = substr(j, i, 1);
      WHILE c <> '"' LOOP
        token = token || c;
        i = i + 1;
        c = substr(j, i, 1);
      END LOOP;

      i = i + 1;
      ts = array_append(ts, ('string', token)::json_token);
      CONTINUE;
    END IF;

Next we loop through any strings we find and accumulate them as tokens before restarting the main loop.

    -- Handle numbers
    WHILE c ~ '^[0-9]+$' LOOP
      token = token || c;
      i = i + 1;
      c = substr(j, i, 1);
    END LOOP;
    IF length(token) > 0 THEN
      ts = array_append(ts, ('number', token)::json_token);
      CONTINUE;
    END IF;

Then we look for integers.

    RAISE EXCEPTION 'Unknown character: %, at index: %; already found: %.', c, i, ts;
  END LOOP;
END;
$$ LANGUAGE plpgsql;

Lastly if none of those lexing handlers match, we give up! Then the loop is done and the function is too.

There's no RETURN statement because we already declared an OUT variable.

If we test and run this now:

./test.sh ./json.sql "SELECT json_lex('{\"flubberty\": 12, \"nice\": \"a\"}')"
                                                                json_lex
----------------------------------------------------------------------------------------------------------------------------------------
 {"(syntax,{)","(string,flubberty)","(syntax,:)","(number,12)","(syntax,\",\")","(string,nice)","(syntax,:)","(string,a)","(syntax,})"}
(1 row)

It's messy but it worked! Now on to parsing.

Parsing

Our parser will only accept JSON objects. JSON objects will be defined as an array of key-value pairs. Custom types make this nice again.

DROP TYPE IF EXISTS json_key_value CASCADE;
CREATE TYPE json_key_value AS (
  k text,
  v text
);

One thing PostgreSQL does not make nice is sum types or parametric types. But even if the value here is stored as text it can be easily cast to a number by the user. And again, we're not going to support nested objects/arrays. But using hstore for key-values might be the better alternative if we wanted to build a real JSON parser.

CREATE OR REPLACE FUNCTION json_parse(ts json_token[], i int) RETURNS json_key_value[] AS $$
DECLARE
  t json_token; -- Current token in tokens loop
  kvs json_key_value[];
  k text;
BEGIN
  t = ts[i];

  IF t.kind <> 'syntax' OR t.value <> '{' THEN
    RAISE EXCEPTION 'Invalid JSON, must be an object, got: %.', t;
  END IF;
  i = i + 1;
  t = ts[i];

First up in the parser is variable declarations and validating that this list of tokens represents a JSON object.

  WHILE t.kind <> 'syntax' OR t.value <> '}' LOOP
    IF array_length(kvs, 1) > 0 THEN
      IF t.kind <> 'syntax' OR t.value <> ',' THEN
        RAISE EXCEPTION 'JSON key-value pair must be followed by a comma or closing brace, got: %.', t;
      END IF;

      i = i + 1;
      t = ts[i];
    END IF;

Then we loop to find each key-value pair. If one has already been found, we need to find a comma before the next pair.

    IF t.kind <> 'string' THEN
      RAISE EXCEPTION 'JSON object must start with string key, got: %.', t;
    END IF;
    k = t.value;

    i = i + 1;
    t = ts[i];
    IF t.kind <> 'syntax' OR t.value <> ':' THEN
      RAISE EXCEPTION 'JSON object must start with string key followed by colon, got: %.', t;
    END IF;

    i = i + 1;
    t = ts[i];
    IF t.kind = 'number' OR t.kind = 'string' THEN
      kvs = array_append(kvs, (k, t)::json_key_value);
      i = i + 1;
      t = ts[i];
      CONTINUE;
    END IF;

    RAISE EXCEPTION 'Invalid key-value pair syntax, got: %.', t;
  END LOOP;

  RETURN kvs;
END;
$$ LANGUAGE plpgsql;

Then we just look for the key, colon, value syntax and fail if we don't see it. And that's it! Very simple when not dealing with arrays and nested objects.

Helpers

Lastly it would just be nice to have a single function that calls lex and parse:

CREATE OR REPLACE FUNCTION json_from_string(s text) RETURNS json_key_value[] AS $$
BEGIN
  RETURN json_parse(json_lex(s), 1);
END;
$$ LANGUAGE plpgsql;

And another function to look up a value in a parsed object by key:

CREATE OR REPLACE FUNCTION json_get(kvs json_key_value[], key text) RETURNS text AS $$
DECLARE
  kv json_key_value;
BEGIN
  FOREACH kv IN ARRAY kvs LOOP
    IF kv.k = key THEN RETURN (kv.v::json_token).value; END IF;
  END LOOP;

  RAISE EXCEPTION 'Key not found.';
END;
$$ LANGUAGE plpgsql;

And we're all set!

Testing

Let's try some bad syntax (missing a comma between pairs):

./test.sh ./json.sql "SELECT json_get(json_from_string('{\"flubberty\": 12 \"nice\": \"a\"}'), 'ipo')"
ERROR:  JSON key-value pair must be followed by a comma or closing brace, got: (string,nice).
CONTEXT:  PL/pgSQL function json_parse(json_token[],integer) line 18 at RAISE
PL/pgSQL function json_from_string(text) line 3 at RETURN

Sweet, it fails correctly.

Now correct syntax but missing key:

./test.sh ./json.sql "SELECT json_get(json_from_string('{\"flubberty\": 12, \"nice\": \"a\"}'), 'ipo')"
ERROR:  Key not found.
CONTEXT:  PL/pgSQL function json_get(json_key_value[],text) line 9 at RAISE

And finally, correct syntax and existing key:

./test.sh ./json.sql "SELECT json_get(json_from_string('{\"flubberty\": 12, \"nice\": \"a\"}'), 'flubberty')"
 json_get
----------
 12
(1 row)

Huzzah! Now hopefully PL/pgSQL is a little less scary to you, whether or not you decide to use it.

For everyone dying to write imperative code in PostgreSQL, I wrote a post about PL/pgSQL 👽 It starts with implementing simple string and array functions, to recursive Fibonacci, to a small JSON parsing library. A nice little language with a great stdlib!https://t.co/m4Tff99N6R pic.twitter.com/2ZMJn2foNa
— Phil Eaton (@phil_eaton) October 24, 2021

Models of Programming

Lambda Land — Sun, 24 Oct 2021 03:00:00 GMT

+++

Last week I was studying outside of a lecture hall where someone was teaching an introductory course on computer programming. There was a lot that I overheard that I disagreed with; this essay is an attempt to help me crystallize what exactly I disagreed with.

Entropy in Data Science

Bytepawn - Marton Trencseni — Sun, 24 Oct 2021 01:00:00 GMT

I discuss 4 uses of entropy in Data Science: (i) cross entropy as a loss function for training neural network classifiers (ii) entropy as a splitting criterion for building decision trees (iii) entropy for evaluating clustering algorithms (iv) entropy for understanding relationships in tabular data.

RAII Footguns in Rust and C++

Nikhil's blog — Sat, 23 Oct 2021 20:06:14 GMT

I’ve been getting back into C++ at Skydio, and I’ve twice lost several hours to debugging weird code behavior because of an RAII footgun in the language. A similar footgun is present in Rust, and I’ve been bitten by that too, so I figured I’d write down both. RAII classes are often used to keep resources alive or hold locks for a given scope. There observable side-effects usually occur only in the constructor and destructor.

CCXT Python Tutorial (2026): Fetch OHLCV, Ticker & Order Book Data (With Examples)

Adnan's Random bytes — Sat, 23 Oct 2021 12:55:28 GMT

Last updated: April 2026 Looking for a CCXT Python tutorial with real examples? In this guide, you’ll learn how to use CCXT to fetch OHLCV data, ticker information, and order book data using simple Python code. I already used the CCXT Library in my Airflow-related post here. In this post, I will discuss the library and how you can use it to pull different types of data from exchanges or trading automation. The demo can be seen here. What You’ll Learn Fetch OHLCV (candlestick) data Get real-time ticker prices Access order book data Connect to Binance using Python What is CCXT CryptoCurrency eXchange Trading Library, aka CCXT, is a JavaScript / Python / PHP library for cryptocurrency trading and e-commerce with support for many bitcoin/ether/altcoin exchange markets and merchant APIs. It connects with more than 100 exchanges. One of the best features of this library is that it is exchange-agnostic: whether you use Binance or FTX, the signature of routines is the same. It is available in Python, JavaScript, and PHP. It supports both public and private APIs offered by different exchanges, but I am only going to discuss a few public API endpoints. Development Setup How to Install CCXT pip install ccxt Alright, the library is installed, let’s connect to an exchange. I am connecting to the Binance exchange for now. Connect to the exchange The very first and obvious step is to connect to the exchange first; it is pretty easy. # Connect binance binance = ccxt.binance() Loading Markets The exchange is connected; now it’s time to load all markets offered by the exchange. Usually, it is not used but it can give you a good idea of whether a certain pair is available or not. Fetching Ticker Data Let’s grab the info related to BTC/USDT. btc_ticker = binance.fetch_ticker('BTC/USDT') btc_ticker and it returns: {'symbol': 'BTC/USDT', 'timestamp': 1634909924391, 'datetime': '2021-10-22T13:38:44.391Z', 'high': 65564.0, 'low': 62000.0, 'bid': 63363.89, 'bidVolume': 1.32351, 'ask': 63363.9, 'askVolume': 0.27138, 'vwap': 63105.52714333, 'open': 65470.0, 'close': 63363.9, 'last': 63363.9, 'previousClose': 65469.99, 'change': -2106.1, 'percentage': -3.217, 'average': None, 'baseVolume': 53395.84423, 'quoteVolume': 3369572897.3972454, 'info': {'symbol': 'BTCUSDT', 'priceChange': '-2106.10000000', 'priceChangePercent': '-3.217', 'weightedAvgPrice': '63105.52714333', 'prevClosePrice': '65469.99000000', 'lastPrice': '63363.90000000', 'lastQty': '0.00843000', 'bidPrice': '63363.89000000', 'bidQty': '1.32351000', 'askPrice': '63363.90000000', 'askQty': '0.27138000', 'openPrice': '65470.00000000', 'highPrice': '65564.00000000', 'lowPrice': '62000.00000000', 'volume': '53395.84423000', 'quoteVolume': '3369572897.39724530', 'openTime': 1634823524391, 'closeTime': 1634909924391, 'firstId': 1109039843, 'lastId': 1110609081, 'count': 1569239}} Apart from Ticker and Timestamp, it returns data points like Open, High, Low, Close, Volume, Bid and Ask Price, etc. Fetching OHLCV data Fetching OHLCV data is pretty easy; all you have to do is: btc_usdt_ohlcv = binance.fetch_ohlcv('BTC/USDT','1d',limit=100) Here, the first argument is the trading pair, the second is duration, which tells whether you need day-level data, minutes, or a month. Each OHLCV entry represents a candlestick. Since the difference in time duration is a day, the data will always be in 24 hours of difference starting from 12 AM, Midnight UTC. The last parameter, limit is optional. When you print it, the data is returned in the following format: [[1634774400000, 66001.4, 66639.74, 62000.0, 62193.15, 68538.64537], [1634860800000, 62193.15, 63732.39, 62000.0, 62680.01, 24059.82478]] The data returns in the format: TIMESTAMP, OPEN, HIGH, LOW, CLOSE, and VOLUME. Let’s make it interesting. I will pull BTC/USDT price from Binance and FTX exchange and plot the graph of close values. How about Close vs Volume? Pretty dirty, right? As if a kid sketched something on paper, but then this is called volatility! Fetching Orderbook Now let’s pull up the order book from Binance and FTX. orderbook_binance_btc_usdt = binance.fetch_order_book('BTC/USDT') orderbook_ftx_btc_usdt = ftx.fetch_order_book('BTC/USDT') bids_binance = orderbook_binance_btc_usdt['bids'] asks_binanace = orderbook_binance_btc_usdt['asks'] df_bid_binance = pd.DataFrame(bids_binance, columns=['price','qty']) df_ask_binance = pd.DataFrame(asks_binanace, columns=['price','qty']) Let’s plot the graph! fig, (ax1, ax2) = plt.subplots(2, figsize=(10, 8), dpi=80) ax1.plot(df_bid_binance['price'], df_bid_binance['qty'],label='Binance',color='g') ax1.fill_between(df_bid_binance['price'], df_bid_binance['qty'],color='g') ax2.plot(df_ask_binance['price'], df_ask_binance['qty'],label='FTX',color='r') ax2.fill_between(df_bid_binance['price'], df_bid_binance['qty'],color='r') ax1.set_title('Ask Price vs Quantity for Binance') ax2.set_title('Bid Price vs Quantity for Binance') plt.show() And it produces… Cool, isn’t it? Conclusion So, as you know now, CCXT is a powerful library for working with crypto exchange data in Python. It makes it easier to connect multiple exchanges without making any significant changes in the code and is widely used for trading bots and data analysis. This was a brief intro to the CCXT library. I just scratched the surface. If you are interested, you may create a trading bot based on this library. Several have done it before. Like always, the code is available on GitHub. If you like this post, then you should subscribe to my blog for future updates. * indicates required Email Address *

The post CCXT Python Tutorial (2026): Fetch OHLCV, Ticker & Order Book Data (With Examples) first appeared on Adnan's Random bytes.

Help me share my NEXT million words

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 23 Oct 2021 07:00:00 GMT

We live in a digital age. We’d better understand for real how it works. All of us.

How to change visibility in massive GitHub organizations like EpicGames?

Avestura's Blog — Sat, 23 Oct 2021 03:00:00 GMT

How to publicize and un-publicize your visibility in massive GitHub orgs?

Custom Metrics Reports on GitLab

Tommaso Amici — Fri, 22 Oct 2021 19:08:41 GMT

GitLab allows you to record Metrics Reports and will display a report on merge requests so that it’s easier and faster to identify changes without having to check the entire log

TIL how to use Ansible-inventory for native ssh-connections

zufallsheld — Fri, 22 Oct 2021 16:23:00 GMT

Actually I did not learn this today but already some years ago. Sometimes I need to connect to remote servers that have no hosts-file or dns-entry. But I don’t want to remember IP-addresses!

Luckily I do have these servers defined in Ansible’s hosts-file and I could theoretically …

SSH Port Forwarding Guide

Geography & Coding — Fri, 22 Oct 2021 10:50:03 GMT

Setting up proper port forwarding for SSH connections, servers or just test purposes is crucial but isn’t as straight forward as your connection should be. This post aims to be a quick guide for different use cases.

Recharging a cell phone battery outside of the phone

voussoir.net/writing — Fri, 22 Oct 2021 10:17:48 GMT

Back to writing

[electronics] [today_i_did_this]

Recharging a cell phone battery outside of the phone

Here is a Motorola C139 cell phone from about 2008. I rediscovered it in a drawer last year.

The charger for this device has long been misplaced. It uses a barrel plug smaller than any I've ever seen, with an outer diameter of about 3mm.

I thought it would be fun to turn this old phone back on just for nostalgia's sake, but without the charger and not wanting to tear it open to access the charger pins, I just put it back in the drawer.

Here is a TP4056 battery charger that I bought on ebay a few weeks ago to explore another project idea.

While I was fiddling with it, I realized this might be just what I needed to get the Motorola working again.

And that worked! This thirteen year old battery is still able to take a charge.

The only problem now is that the phone is really boring to use and I have no purpose for it. Other than to play breakout.

P.S. I accidentally fried this TP4056 shortly after taking these photos. I picked it up with the wires still dangling out of those pins and I think I shorted something. Smoke came out and everything. Luckily it came in a two-pack.

View this document's history

2021-10-22 Add motorola_c139.md.

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

A steering wheel for fake autonomous cars

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 22 Oct 2021 06:40:00 GMT

Cool it’s cool. Driverless, not really.

Shaking N-gram needles from large haystacks

Weblog on marginalia.nu — Fri, 22 Oct 2021 03:00:00 GMT

A recurring problem when searching for text is identifying which parts of the text are in some sense useful. A first order solution is to just extract every word from the text, and match documents against whether they contain those words. This works really well if you don’t have a lot of documents to search through, but as the corpus of documents grows, so does the number of matches.

It’s possible to bucket the words based on where they appear in the document, but this is not something I’m doing at the moment and not something I will implement in the foreseeable future.

Removing invisible unread GitHub notifications

About on kinow — Fri, 22 Oct 2021 03:00:00 GMT

Some months ago I noticed that even after I marked all my GitHub notifications as read, the unread icon displayed at the right top corner was still showing as if I had unread notifications.

GitHub Notifications icon always-on mode

I tried changing the filters, waiting for a new notification to appear so that I could mark it as read, all hoping that icon would then change. But no matter what I tried in the GitHub UI, the icon was still there.

Censorship or... EFFICIENT slowness?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 22 Oct 2021 02:40:00 GMT

Let’s talk about it. Slowly, please.

Apple ∑Mn

Posts on A blog — Thu, 21 Oct 2021 15:01:42 GMT

Its October! That is also a synonym for a new Apple event, well it is actually either October or September if I have to be precise. This October was my attention wrapped around the new SoCs in Apple Silicons line-up, to be more precise the upgrade of Apple’s 2020 M1 chip, Apple M1 Pro and Apple M1 Max. Where Apple continues to make the naming conventions more confusing with direct connection to the names of iPhones models. So, to break it down for ordinary people Pro is like iPhone 13 Pro and Max is like iPhone 13 Pro Max but without the Pro part. It’s super easy to understand, right? Well its a bit more intricate than that. Apple has showed with the M1 chip that the transition over from Intel-based x86 CPUs to the own in-house designed microprocessors running on the Arm instruction set don’t need to be so problematic, do you follow Microsoft? Apple M1 wasn’t that groundbreaking in terms of performance, but more of a unexpectedly good transition to Arm.

Building an Emacs lisp VM in Rust

Core Dumped — Thu, 21 Oct 2021 03:00:00 GMT

Updated: 2023-01-06 About a year ago I was bitten by the PL bug. It started with reading Crafting Interpreters and discovering the wonders hidden under the hood of a compiler. I am also been a big fan of Emacs, and this started to get me interested in how its interpreter works. At the same time, I was reading the Rust book and trying to understand the concepts there. This all came to a head, and I decided to write an Emacs Lisp interpreter called rune in Rust.

2021–10–21: Pinephone Pro – levinboot payload selection

megi's PinePhone Development Log — Thu, 21 Oct 2021 01:00:00 GMT

Making a series outline

Liza Shulyayeva — Wed, 20 Oct 2021 20:02:00 GMT

I thought my first series of novellas would revolve around the same world, but otherwise stand alone from each other.

IVOW fights biases in artificial intelligence

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 20 Oct 2021 13:00:00 GMT

Good, but maybe also another “way paved with good intentions”.

I met with the wolves

Nicola Iarocci — Wed, 20 Oct 2021 09:05:25 GMT

I sit under a wild apple tree at the edge of the clearing. Like a plant, I absorb the mild October sun. At the same time, I attentively listen to the sounds of the forest. Suddenly I hear a stomping of dry leaves about twenty meters ahead of me, slightly to my right. A wolf emerges out of the thicket. He stops for a moment, glances around, then starts crossing the clearing. A few moments and another wolf appears. After a brief pause, he follows his pal. They parade right in front of me, sinuous, silent, and feral. They do not see or hear me, and yet I am right there, by their side, in plain sight¹. It is all over in a few seconds. The couple leaves the clearing and disappears into the forest. I stand sitting there, shocked and in complete awe.

Skills, stories, and software every dev should know - 123dev #42

Justin Garrison's Homepage — Wed, 20 Oct 2021 03:00:00 GMT

Welcome to 123dev from Revue. This is the first issue I migrated from Buttondown. You may need to adjust your mail filters because the sending address has changed.

Moving a MySQL/MariaDB database between servers using SSH

Ben E. C. Boyter — Wed, 20 Oct 2021 03:00:00 GMT

I’m in the middle of moving searchcode.com’s database to another server while upgrading it. This means I need to copy it somehow. It’s especially difficult in this case because the server its sitting on does not have enough disk space to do my usual mysqldump then copy the file across using SCP.

I investigated splitting the database using --where clauses allowing me to get chunks of the database across, but that also means adding things like --skip-create-options and --skip-add-drop-table and I ended up having to redo it a few times, with dumps in between.

Eager to know more about Einride pods

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 20 Oct 2021 02:40:00 GMT

Some of my driverless visions keep getting closer. Maybe.

Syncing Mobile Photos with Photoprism

Random Bits — Wed, 20 Oct 2021 00:35:05 GMT

Photoprism Logo

Photoprism is the self-hosted photo/video library that does it all for me. It has labeling of images so you can search by picture content, it has calendar and map features for more natural finding of the pictures you're trying to find and it lets

Italia II - Firenze

Avy Faingezicht — Tue, 19 Oct 2021 03:00:00 GMT

When planning our trip, I was somewhat disappointed that we'd only spend a day in Florence. Knowing it's a place that I'll end up going back to on my own, I didn't try too hard to convince my friends to change our plans - realistically the villa thing won't happen again, but the museums will be waiting for me in a few years. Next time, I'll spend a bit more than 12 hours there. _This is the second in a series shared throughout Fall 2021. When you're done here, make sure to also check out parts one, three, four, and five._

Malay manuscripts show the right way to digitize culture

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 19 Oct 2021 02:40:00 GMT

First digitize, then TELL everybody.

EVANGELION:3.0+1.01 Thrice Upon a Time

🦉 olshansky 🦁 — Mon, 18 Oct 2021 20:46:26 GMT

Olshansky's review of EVANGELION:3.0+1.01 Thrice Upon a Time

No escorts allowed: How I fought spammers

Gaganpreet Arora — Mon, 18 Oct 2021 17:28:37 GMT

Warning: This post features text that might not be safe for work.

Background #

Incorporating anti-spam measures are the least of concerns while building a user generated content (UGC) platform. Unless you’re paranoid and spend precious time building moderation features in your platform, there will come a day when you sorely miss those tools. This is a story about exactly how this happened and how I dealt with it at Polarsteps.

Testing The Silk Platform - Hands-On Technical Analysis of High-Performance I/O in the Cloud

Tanel Poder Blog — Mon, 18 Oct 2021 13:30:00 GMT

Update: 3 years after writing this article, I tested the latest Silk Platform in 2024, on Google Cloud Platform, with impressive results. In my tests, I got to over 20 GiB/s with large reads, 1.3M IOPS with small 8kB random reads and 10 GiB/s sustained write rate! Both articles are worth a read.

Hands-on technical analysis of a novel data platform for high-performance block I/O in the cloud,
tested by Tanel Poder, a database consultant and a long-time computer performance geek.

Executable JavaDoc Code Snippets

Gunnar Morling — Mon, 18 Oct 2021 09:30:00 GMT

Table of Contents

Including Snippets From Your Test Directory
Summary

It has been just a few weeks since the release of Java 17, but the first changes scheduled for Java 18 begin to show up in early access builds. One feature in particular that excites me as a maintainer of different Java libraries is JEP 413 ("Code Snippets in Java API Documentation").

Experience vs Theory: Lake Tahoe Edition

🦉 olshansky 🦁 — Mon, 18 Oct 2021 03:36:12 GMT

I remember thinking I had the foundation to tackle any problem that came my way around the time that I was graduating from College. Having…

Experimenting with column- and row-oriented datastructures

Notes on software development — Mon, 18 Oct 2021 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

2021–10–18: Pinephone Pro – support merged into my kernel tree

megi's PinePhone Development Log — Mon, 18 Oct 2021 01:00:00 GMT

100 articles

Bytepawn - Marton Trencseni — Mon, 18 Oct 2021 01:00:00 GMT

A review and introspect on the first 100 articles written on Bytepawn.

Mach Engine: The future of graphics (with Zig)

Hexops' devlog — Sun, 17 Oct 2021 03:00:00 GMT

In the coming months, we’ll begin to have truly cross-platform low-level graphics, with the ability to cross compile GPU-accelerated applications written in Zig from any OS and deploy to desktop, mobile, and (in the future) web.

Mach engine

I’ve been working on Mach Engine for about 4 months now, although it as a project is many years in the making, and I believe in the next 4-6 months we’ll have completion of the first key milestone: truly cross platform graphics and seamless cross compilation.

Vision

Today, I share only the first milestone: Mach engine core. I’ve been working on this for around 1 year now, and we’re close (maybe 4-6 months away) from completion:

Zero fuss installation & cross compilation

Only zig and git are needed to build from any OS and produce binaries for every OS. You do not need any system dependencies, C libraries, SDKs (Xcode, etc.), C compilers or anything else.

We’re able to achieve this thanks to two things:

Zig has fantastic cross-compilation support, including its own custom linker zld written by Jakub Konka which is capable of supporting MacOS cross compilation.
Mach doing the heavy lifting of packaging the required system SDK libraries and C sources for e.g. GLFW so our Zig build scripts can simply git clone them for you as needed for the target OS you’re building for, completely automagically.

Truly cross-platform graphics API

DirectX 12, Metal, Vulkan & OpenGL

Imagine a low-level, little to no overhead graphics API that unifies DirectX, Metal, Vulkan, and OpenGL (if no others are available):

This isn’t anything new: all modern engines provide this, Godot has been working towards this for years (and still is), and there exist abstraction layers for Vulkan over most of these APIs as well.

Vendor support

An API is only as good as the momentum behind it. What modern API can target the largest array of platforms with the most vendor backing?

Microsoft sees DirectX as the future, not Vulkan. (DirectX 13 is coming by the end of 2022.)
Apple sees Metal as the future, not Vulkan. OpenGL and OpenCL are deprecated, and private legal arguments with Khoronos make it unlikely we’ll ever see OpenGL or Vulkan on Apple hardware ever again.
Google, with their Fuschia OS appears to be primarily into Vulkan from a system-level POV.
NVIDIA, AMD, and Intel generally support as many graphics APIs as possible, they want to sell hardware.

One API that Apple, Microsoft, and Google can all agree on

Outside the bounds of traditional graphics APIs there exists an attempt to provide a unified API across all platforms, WebGPU (not to be confused with the much older WebGL).

Mozilla, Google, Apple, and Microsoft all got together to build an abstraction layer over the modern graphics APIs - finding the common ground between Direct3D 12, Metal, and Vulkan - plus a safe way to expose that functionality in browsers.

The name WebGPU might lead you to believe that this is only for browsers, and that it may not be low-level or fast - but this really couldn’t be further from the truth.

Apple & Google’s role is what makes WebGPU unique, and why we chose it

What is new about WebGPU in my view is the vendors playing key roles in its development, and the fact that it grew outside the Khronos Group.

Although abstraction layers over modern graphics APIs are nothing new - as Apple, Google, and Microsoft continue to get more into manufacturing their own hardware (it’s clear this is a strategic move for them) we should ask ourselves how this will change the landscape, and WebGPU is the first cross-vendor API to be produced by this new ecosystem.

WebGPU extended thoughts

Is WebGPU "native enough"? Yes

For browsers, WebGPU will require sandboxing and validation layers. But in native uses, this can all be turned off, and the WebGPU developers are clearly thinking about this use case:

Google's implementation of WebGPU, Dawn, can be configured to effectively turn off all browser sandboxing / validation that could harm performance due to its client/server architecture.
Mozilla / gfx-rs Rust engineers have published articles such as "The point of WebGPU on native".

As for the quality of implementations, we could compare the amount of resources going into e.g. Google's WebGPU implementation vs. the amount of resources going into Unity/Unreal/MoltenVK/other graphics abstraction layers - but I suspect they're about equal.

Will WebGPU be implemented on GPUs natively? Maybe someday

Not anytime soon. We get some insight into this via @kvark, a WebGPU developer:

[...] We are not in Khronos, and therefore we have limited participation from IHVs (only Intel and Apple are active). WebGPU was never designed to be implemented by the drivers. I mean, it would totally be rad, in the context of how usable WebGPU can be on native, but it couldn't be the requirement from the start.

But as WebGPU usage grows or even becomes prodominate due to it being the most powerful API in browsers, and as Microsoft, Google, and Apple continue to develop their own hardware - I think it's not unreasonable to think that it's possible some day WebGPU will be an even more direct 1:1 mapping between a cross-platform API and low-level APIs, more direct than Vulkan abstraction layers such as MoltenVK (which is required to get Vulkan working on top of MacOS's Metal API) - with the potential that some vendor starts asking "what would a GPU native WebGPU implementation look like?"

Momentum of WebGPU vs. Vulkan

To quote Dzmitry Malyshau / kvark, a Mozilla engineer working on gfx-rs and WebGPU:

At some point, it comes down to the amount of momentum behind the API. In case of WebGPU, we have strong support from Intel and Apple, which are hardware vendors, as well as Google, who can influence mobile hardware vendors. We are making the specification and have resources to appropriately test it and develop the necessary workarounds. It's the quantity to quality transition that sometimes just needs to cross a certain threshold in order to succeed.

According to some, Nvidia and AMD tend to develop new features with Microsoft as part of DirectX. Only then are they "ported" back to Vulkan and OpenGL. I think that says a lot.

What progress has been made so far on Mach Engine?

Today, we have cross-compilation of GLFW on all desktop OSs working out of the box with nothing more than zig and git:

This involved:

Packaging MacOS SDKs and Linux system X11/Wayland libraries into SDKs, and creating Zig build scripts that could merely git clone them and utilize them for cross-compilation.
Purchasing Apple M1 hardware to test on, and for GitHub Actions as it doesn’t support it.
Normalizing symlinks in Mac/Linux SDKs everywhere so that Windows users don’t have a hard time with Git symlink management.
Contributing a small fix to the Zig linker

All this to say, we’re really taking a holistic approach to achieve this.

What’s next? WebGPU

I’m happy to report that a fair amount of progress on this front has been made.

Here is Google’s WebGPU implementation, Dawn, compiled using zig:

This includes:

A ~500 line port of the hello_triangle example from Dawn to Zig
A ~1200 line build.zig file which compiles all the Dawn sources using Zig, without using Google’s ninja/etc development tools.
A hack to workaround a bug in Zig where ObjC++ .mm files are not yet recognized.
C shims for the dawn_native C++ API and utility APIs, which are required in order to bind Dawn to an actual GLFW window.

There are a few weeks of work to do before this can be merged and will be usable by others, please stay tuned for that.

After that will be development of idiomatic Zig bindings to the WebGPU C API which is shared between implementations such as Dawn and the Rust’s gfx-rs/wgpu-native implementation (we could theoretically switch between them at startup in the future, but we’ll probably stick with Dawn as it does not require a separate Rust toolchain and it would prevent out-of-the-box cross compilation.)

When will there be games, examples, etc.?

It’ll be a while because I am focusing purely on the groundwork first. It’s unlikely you’ll see anything with real demo value before later next year.

I’m sure that will be disheartening to hear - and may make you to think there’s nothing of substance here. I totally understand that view, but I hope you’ll stay tuned because I’m in this for the long haul and it’s not my first rodeo (I previously spent 4 years writing a game engine in Go, and have worked at a devtools startup for 7 years, with my biggest lesson from of those experiences being the importance of demos and examples.

Follow along

Major developments will be posted here.

You can also follow the project at github.com/hexops/mach.

If you like what I’m doing, you can sponsor me on GitHub.

Thanks for reading!

Skills, stories, and software every dev should know - 123dev #41

Justin Garrison's Homepage — Sun, 17 Oct 2021 03:00:00 GMT

Comments

BGP

I heard the GIF came from what happened when Facebook ran the following command.

ip link set fb0 down

Thankfully they were able to fix it with

sudo reboot

Thinking

If you’re overthinking you should write. If you’re underthinking you should read.

Written word is much slower than spoken word. It helps to slow down and be deliberate when you have lots of ideas in your head.

Blender rendering on NVIDIA Jetson Nano

About on kinow — Sun, 17 Oct 2021 03:00:00 GMT

NVIDIA Jetson Nano computer

I had used Blender during my graduation at the Mackenzie University and started learning Blender 2.8+ again a few weeks ago. Unfortunately rendering the basic tutorials like Andrew Price’s donut takes several minutes on my old (but excellent for programming) Thinkpad T550 i7 16 GB with a simple Samsung SSD. The reason is that my GPU, a NVIDIA NVS 5400M with 2 GB memory and 96 cores cannot be used with Blender as it only supports CUDA 2.1. Blender 2.8+ GPU rendering requires CUDA 3.0 and higher, which means Blender Cycles render is using my CPU, which is slower than using a decent GPU.

Venezuela, the unlikely leader of digital payments

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 17 Oct 2021 02:40:00 GMT

Not so unlikely, when you think about it.

100 Bytes of CSS to look great everywhere

swyx's site RSS Feed — Sun, 17 Oct 2021 00:05:53 GMT

Simple grab and go upgrade to the base user agent stylesheet to make for readable sites

The second book is harder

Liza Shulyayeva — Sat, 16 Oct 2021 19:00:00 GMT

After a year of short stories, I released my first novella recently and it’s been doing pretty well. I’m surprised that it is still doing well, since usually books tend to drop off a cliff. But it’s still early days, and the cliff can come any time now.

Cognition Enhancing Technologies

one mikro2nd — Sat, 16 Oct 2021 15:16:13 GMT

Against all expectation, I got a job. Sort of. I’m working with a non-profit org, We Think Code, that teaches software development skills (not just programming!) to people new-to-programming who likely might not otherwise have access to such opportunity. My role is in curriculum development, where I work (async, remote) as a small part of a stellar team, helping to write lessons that “Help students become the developers we want to see & work with in the world.

Things my dog taught me about software development

Phil Booth's Blog — Sat, 16 Oct 2021 13:54:54 GMT

I’ve always wanted a dog, but could never get one because I lived in rented accomodation. Happily that changed last year so as soon as I could, I bought a puppy. This is what I learned about dogs, people and code.

1. Don’t be surprised when puppies do puppy stuff 🐶

As an engineer of somewhat advanced years, something I’ve struggled with at times is being patient with less experienced devs. I try to be patient and I know impatience is bad, but there are times when it feels like I’m suppressing an urge to be exasperated.

Puppies are tiny, cute bundles of joy. They are impossible not to love. They try very hard to be good little boys and girls, and look to you for guidance. But also they want to chew everything, dig holes in your lawn, jump up on your furniture and go for a pee or a poo whenever they feel like it. Obviously it would be stupid to get upset about any of that, so patiently you guide them to make better choices and gradually they learn how to behave.

And so it goes for junior engineers. They’re on a journey and trying to improve. There are things they haven’t learned yet but it’s a consequence of their experience, not lack of ability. If they’re confidently doing the wrong thing, it’s an effect rather than a cause.

2. Reward them when they do good things 🍪

Guidance comes in two forms: calmly saying no to bad behaviours and enthusiastically rewarding good ones. You should do lots more of the second one than the first.

When saying no, it doesn’t help to get angry about anything because that won’t produce the desired outcome. If you get angry, they’ll get worked up too and you enter an arms race of who can be the most outraged. It doesn’t lead anywhere.

For puppies, give them a clear and calm “no” then ignore any further attempts at bad behaviour, removing them from the situation if you have to. For junior engineers, explain the downsides of what they’re trying to do and why alternative solutions don’t suffer from them. Point out better options using broad enough strokes that they still have some latitude for discovery and creativity in reaching the right answer. Don’t suck the joy out of programming.

When rewarding, you want your rewards to be fun and memorable.

For puppies that means changing your tone of voice, giving them tasty treats, belly rubs, all the things they enjoy. And your reward must come right after whatever it’s for, so they know why you’re rewarding them.

Most of the above applies to engineers too, although belly rubs are not advised. Make sure the reward is something they actually value. Some people like praise in public, but some people hate the limelight that brings. Make the reward commensurate to the size of the task and make it clear why you’re rewarding them. For code reviews, call out specific parts of the diff that are good instead of leaving a blanket 👍 on the PR. If someone is consistently excellent, speak to your boss and help them get a promotion or a pay rise.

3. Don’t hold grudges 😠

After you’ve said no to your puppy/engineer, it’s important to let go of the subject and move on. Not doing that will cause problems on two fronts.

First, it will make them think you’re an unreasonable old meany. They’ll get scared and gradually your relationship will become one of fear. They will not produce their best work under those conditions.

Second, it will set the tone inside your own head for how you interact with them in future. Negativity begets more negativity, it’s habitual. Eventually, if you don’t keep a lid on it, the Ghost of Christmas Past will visit and show you what you used to be like. Don’t turn into Ebenezer Scrooge.

4. Clear up any mess that’s left behind 💩

It’s beholden on us all to create the world we want to live in. When you walk your puppy, you take little plastic bags with you and scoop up their mess. When they chew through a soft toy so that all the stuffing spills out onto the living room floor, you clean it away before they can swallow any of it. Accidents happen, you clear up afterwards.

The same applies to engineering too, except we call it refactoring. Code mess is a natural byproduct of feature development that can paralyse teams if left unchecked. The metaphor of technical debt is well understood, but it can still be hard to get buy-in from more product-minded parts of the team when it comes to prioritisation. The best solution to that is to be constantly refactoring, in small steps so that it’s hardly noticed. Bigger refactors will still be needed from time to time of course, but you can significantly reduce their impact if you clean up any mess as you go. Mess that’s left behind eventually develops a life of its own and is that much harder to deal with as a result.

5. Make time for play 🧸

For puppies, play is an important part of their development and a way to increase the bond between you. It’s also an opportunity to provide more guidance. If you’re playing tug, have some treats on hand and use them to gradually teach the puppy a “drop” or “give” command. You can utilise your puppy’s instinct for chasing to start teaching them “come”.

Engineering is rarely a solo pursuit and teams that only communicate around work tasks are missing an opportunity for bonding too. It can be difficult when working remotely but small things like taking five minutes at the start of meetings just to chat, can develop empathy among the team and encourage healthier interaction between team-mates. Deep technical conversations go much better when you like the people you’re having them with.

Human TCP

june.kim — Sat, 16 Oct 2021 03:00:00 GMT

The Javascript "ecosystem" is a hot mess and so is software development in general

Blog on Kailash Nadh / Personal homepage — Sat, 16 Oct 2021 03:00:00 GMT

I have a small Vue 2 project (an admin UI for dictmaker) that I created with vue cli six months ago. Today, I picked it up again to finish it, and started out by doing a yarn upgrade. Of course, blindly upgrading all dependencies is never a good idea, but this is a tiny WIP project with just one dependency that I added, and there is a constant stream of GitHub dependabot alerts every month forcing me to upgrade some dependency or another, so what is the worst that could happen? At least that is what I thought.

Git Basics

Playground — Sat, 16 Oct 2021 03:00:00 GMT

I created this introduction to Git for the Data Science team of Swiggy. I believe, the better we know Git, the more efficient our engineering processes will be. Data Scientists come from a variety of educational backgrounds. Many of them will not know anything about git. Many others will just have a working knowledge; they will use git commands, but they won’t really know what really happens. Observing this gap in the team, I decided to work on bridging it.

Why didn't the web disrupt mobile, you ask?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 16 Oct 2021 02:40:00 GMT

Here is my answer.

How to remove Traefik from K3S

Yasha Solutions — Sat, 16 Oct 2021 01:00:00 GMT

When you install k3s it automatically come with Traefik installed. Traefik is great but if you need to work with an other ingress, you might want to remove it. Here is how: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 # Remove traefik helm chart resource sudo kubectl -n kube-system delete helmcharts.helm.cattle.io traefik # Stop the k3s service sudo service k3s stop # add option to add to ExecStart "--no-deploy traefik \" sudo echo '--no-deploy traefik \' >> /etc/systemd/system/k3s.

Goodbye old friend, you will be missed

Bajkowski.com — Fri, 15 Oct 2021 23:27:31 GMT

Reminiscing about our house cat Miko and the joy she brought over the last two decades

Ansible + Terraform, the PBJ of DevOps

Brightball Articles — Fri, 15 Oct 2021 17:42:03 GMT

This week I had the opportunity to speak to Upstate Carolina Linux User Group (UCLUG) to share my thoughts on how well Ansible and Terraform compliment each other, as well as the many ways to share information between the tools to use them together.

Skills, stories, and software every dev should know - 123dev #40

Justin Garrison's Homepage — Fri, 15 Oct 2021 03:00:00 GMT

Comments

Laying a trap for yourself

I have lost track of how many times I’ve patched a bug only to have it come back and hit me in the head. Sometimes tests will help you avoid a self-made trap, but more often then not I’ll just push harder until it breaks.

Armed robodogs are just like smart rifles

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 15 Oct 2021 02:40:00 GMT

Just as dumb, that is.

Kubernetes Building Blocks

Yasha Solutions — Fri, 15 Oct 2021 01:00:00 GMT

Kubernetes has multiple moving parts. Here is an short article aiming at mapping all this to create some clarity. Some Basic Vocabulary Nothing fancy but let’s get this out of the way. Nodes A node can be a physical machine or a virtual machine. Cluster A cluster is a group of nodes Container runtime Docker (or runC) Pod A pod is a group of containers - that together form a logical application.

2021–10–15: Pinephone Pro – A Quick Review

megi's PinePhone Development Log — Fri, 15 Oct 2021 01:00:00 GMT

My book-writing routine

Lucas F. Costa - Blog — Thu, 14 Oct 2021 15:00:00 GMT

I spent 2020 writing a 500-pages book about JavaScript Testing. In this blog post, I'll explain my writing routine — or system, whatever you want to call it — and other seemingly unrelated habits th...

When I read the title, I was thinking the article would discuss how everyone uses and interacts…

🦉 olshansky 🦁 — Thu, 14 Oct 2021 12:59:56 GMT

I agree with Nate's comment that ethereum contracts are indeed public and immutable. Whether an individual chooses to sync a node, inspect…

Pastel migraine auras

Qubyte Codes — Thu, 14 Oct 2021 11:10:00 GMT

A generative patchwork of pastel colours. The colours begin with a randomly picked colour in LCH. Other colours are hue rotations in LCH space so that they're perceptually nice together. Feature detection is used to render using LCH colours when the browser supports them (Safari only at the time of writing), or to pick a close colour in RGB space when LCH is not supported (the code for this is based on code in d3-color). Refreshing generates a new patchwork.

Italia I - Umbria

Avy Faingezicht — Thu, 14 Oct 2021 03:00:00 GMT

Many months ago, in the depths of COVID-times, our friend Maizie asked if we'd be willing to travel to Italy to celebrate her birthday if she planned everything for our friend group, assuming we could all get vaccinated by then. That was a pretty easy question to say yes to. I had only been to Italy once before, on a short vist to Venice during a backpacking trip before college. Disconnecting from our daily routines is always relaxing, but getting back together with our friend group, many of whom had left San Francisco, and wandering through the life-sized museum that is Italy was a ton of fun. _This is the first in a series shared throughout Fall 2021. When you're done here, make sure to also check out parts two and three, four, and five._

Bridge WIFI connection via LAN

xnacly - blog — Thu, 14 Oct 2021 03:00:00 GMT

How to bridge a WIFI connection to LAN

The Five Lies Analysis

SerCe's blog — Thu, 14 Oct 2021 03:00:00 GMT

The Five Whys analysis is a simple root cause investigation technique but can have trade-offs if blindly applied. This article explores the trade-offs of the approach.

Skills, stories, and software every dev should know - 123dev #39

Justin Garrison's Homepage — Thu, 14 Oct 2021 03:00:00 GMT

Comments

I twice had lunch with Ed Catmull at Disney and I asked him if there was anything that surprised him about managing people. He said he was surprised how well people can game rules and metrics for their own benefit.

The above gif may or may not be real, but also speaks to people manipulating perception for—maybe—their benefits.

The real advantage of broadband via laser links

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 14 Oct 2021 02:40:00 GMT

It’s not the speed, or the reliability. It’s the people.

7 Step Guide to Installing rsync on Windows

Misleading Metaphors — Wed, 13 Oct 2021 16:04:17 GMT

I recently needed to move several gigabytes of data from an old laptop running Windows 8.1 to another laptop running macOS and my options were to use an external disk or a network transfer. I wanted the whole process to be unattended so I naturally gravitated towards doing the

Social media break

Liza Shulyayeva — Wed, 13 Oct 2021 09:00:00 GMT

A few years ago I deleted my Facebook account, and now I feel like it’s time for the next step: Twitter; but nothing as drastic as deletion just yet.

DIY cloud gaming: NVIDIA and Moonlight

./techtipsy — Wed, 13 Oct 2021 05:00:00 GMT

Comparing the experience that AMD and NVIDIA can provide in a DIY cloud gaming setup.

Unintuitive Optimization

Weblog on marginalia.nu — Wed, 13 Oct 2021 03:00:00 GMT

Optimization is arguably a lot about intuition. You have a hunch, and see if it sticks. Sure you can use profilers and instrumentation, but they are more like hunch generators than anything else.

This one wasn’t as intuitive, at least not to me, but it makes sense when you think about it.

I have an 8 Gb file of dense binary data. This data consists of 4 Kb chunks and is an unsorted list containing first an URL identifier with metadata and then a list of word identifiers. This is a sort of journal that the indexer produces during crawling. Its main benefit is that this can be done quickly with very high fault tolerance. Since it’s only ever added to, if anything does go wrong you can just truncate the bad part at the end and keep going.

Skills, stories, and software every dev should know - 123dev #38

Justin Garrison's Homepage — Wed, 13 Oct 2021 03:00:00 GMT

Comments

Edutainment

I would rather entertain and hope that people learned something than educate people and hope they were entertained.

Walt Disney

What’s next?

I had a lightning talk for Devops Days Portland called “TikTalk” about innovation. Please go watch it because it was a lot of fun to make.

Innovation comes in different forms; there are sustaining innovations and disruptive innovations. Most companies need people who can help sustain their innovations. They need to be better, faster, and stronger than they were.

Notes on running Electron

Notes on software development — Wed, 13 Oct 2021 03:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Algorithm bias, again. This time, in mortgages

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 13 Oct 2021 02:40:00 GMT

But the problem is not algorithms, of course.

2021–10–13: How to adapt multi-boot image to your needs

megi's PinePhone Development Log — Wed, 13 Oct 2021 01:00:00 GMT

Scrape, Scrape

Liza Shulyayeva — Wed, 13 Oct 2021 00:20:00 GMT

My book has been getting more and more reviews and ratings on Amazon and Goodreads. Instead of obsessively refreshing the page I figured it might be cool to collect reviews from different sources and have them sent as a summary email each day.

Publishing a Package to Pip

Random Bits — Wed, 13 Oct 2021 00:14:33 GMT

PyPi - The Python Package Index

I love Python. It's easy to read, even easier to write, and best of all, the easiest language (I've used) to interact with other people's code. I spent too long being only on one end of that equation though,

TCP By Disk

onKeyPress — Tue, 12 Oct 2021 20:10:46 GMT

How we may read the Scriptures with the most spiritual profit

Honza Pokorný — Tue, 12 Oct 2021 15:00:00 GMT

By Rev. Thomas Watson

“And it shall be with him, and he shall read it all the days of his life, that he may learn to fear the Lord his God, to keep all the words of this Law and to do these Statutes.” — Deut. 17.19.

What Cicero said of Aristotle’s politics, may not unfittingly be said of this book of Deuteronomy, it is full of golden eloquence. In this chapter, God instructs the Jews about setting a king over them, and there are two things specified, as to his election, and his religion.

His election, verse 15. “You shall in any way set him king over you, whom the Lord your God shall choose.” There is good reason that God should have the choice of their king, since “by him kings reign.” Prov. 8.15.
His religion, verse 18. “When he sits on the throne of his kingdom, he shall write him a copy of this law in a book, out of that which is before the priest.” Here was a good beginning of a king’s reign; the first thing he did after he sat upon the throne, was to copy out the word of God in a book. And in the text, “It shall be with him, and he shall read it, all the days of his life, that he may learn to fear the Lord his God, to keep all the words of this Law and to do these Statutes.” “It shall be with him.” The Book of the Law shall be his Vade Mecum, or daily companion. Charles the Great used to set his crown upon the Bible. Indeed the Bible is the best support of government. “And he shall read it.” It is not below the majesty of a prince to peruse the oracles of Heaven; in them are comprised sacred apothegms, Prov. 8.6: “I will speak of excellent things.” In the Septuagint it is “grave things,” in the Hebrew, “princely things,” such as are fit for a God to speak and a king to read. Nor must the king only read the Book of the Law at his first installment into his kingdom, but he “shall read it all the days of his life.” He must not leave off reading, till he has left off reigning. And the reasons why he must be conversant in the law of God, are in the subsequent words: 1. “That he may learn to fear the Lord his God.” Reading the word is the best means to usher in the fear of the Lord, 2. “That he may keep all the words of this Law, to do them.” 3. “That he may prolong his days in his kingdom.”

I shall now confine myself to these words, “He shall read it all the days of his life;” i. e. the Book of the Law.

The Holy Scripture is, as Austin says, a golden epistle sent to us from God. This is to be read diligently; ignorance of Scripture is the mother of error, not of devotion. Matt. 22.29: “You err, not knowing the Scriptures.” We are commanded to “search the Scriptures.” John 5.39. The Greek word signifies to search as for a vein of silver. How diligently does a child read over his father’s will and testament! and a citizen peruse his charter! With like diligence we should read God’s word, which is our Magna Charta for Heaven. It is a mercy that the Bible is not prohibited. Trajan the emperor forbade the Jews to read in the Book of the Law. Let us inquire at this sacred oracle. Apollos was “mighty in the Scriptures.” Acts 18.24. Melancthon, when he was young, sucked the sincere milk of the word. Alphonsus, King of Arragon, read over the Bible fourteen times. That Roman Lady, Cecilia, had by much reading of the word, made her heart the Library of Christ, as Jerome relates. Were the Scriptures confined to the original tongues, many would plead an excuse for not reading; but when the sword of the Spirit is unsheathed, and the word is made plain to us by being translated, what should hinder us from a diligent search into these holy mysteries?

Adam was forbidden upon pain of death to taste of the tree of knowledge. Gen. 2.17: “In the day you eat of it, you shall surely die.” But there is no danger of touching this tree of Holy Scripture; if we do not eat of this tree of knowledge we shall surely die. What will become of those who are strangers to the Scriptures? Hosea 8.12: “I have written to him the great things of my law, but they were considered as a strange thing.” Many lay aside the Scriptures like rusty armor. Jer. 8.9. “They are better-read in romances than in Paul; they spend many hours between the comb and the glass, but their eyes begin to be sore when they look at a Bible.” Even the Turks will rise up in judgment against these Christians. The Turks reverence the Books of Moses; and if they find just a leaf on which anything of the Pentateuch is written, they take it up and kiss it. Those who slight the written word, slight God himself, whose stamp it bears. To slight the king’s edict is an affront to the person of the king. Those who reject and vilify the Scriptures are in a state of condemnation. Prov. 13.13: “Whoever despises the word shall be destroyed.” Nor is it enough to read the word of God, but it should be our care to get some spiritual reward and profit by it, so that our souls may be nourished in the words of faith. 1Tim. 4.6. Why else was the Scripture written but that it might profit us? God did not give us his word only as a landscape to look at, but he delivered it as a father delivers a stock of money to his son to improve on it. ‘Tis sad not to profit by the word, to be like the body in consumption, that does not thrive. Men would be loath to trade and get no profit. The grand question I am to speak to is this. How we may read the Scriptures with the most spiritual profit. In answering this question, I shall lay down several rules or directions about the reading of Scripture.

1. If you would profit by reading, remove those things that will hinder your profiting. That the body may thrive, obstructions must be removed. There are three obstructions that must be removed if you would profit by Scripture.

Remove the love of every sin. Let a physician prescribe ever so good recipes, if the patient takes poison, it will hinder the virtue and operation of the medicine. The Scripture prescribes excellent recipes, but sin that is lived in, poisons all. The body cannot thrive in a fever, nor can the soul thrive under the feverish heat of lust. Plato calls the love of sin magnus daemon, a great devil. As the rose is destroyed by the canker which breeds in it, so are the souls of men destroyed by those sins in which they indulge.
Take heed of those thorns which will choke the word that is read. These thorns our Saviour expounds to be the cares of this world. Matt. 13.22. By cares is meant covetousness. A covetous man has such diversity of secular employment that he can scarcely find time to read, or if he does, what errors does he commit in reading? While his eye is upon the Bible, his heart is upon the world; it is not the writings of the Apostles that he is so much taken up with, as the writing in his account books. Is this man likely to profit? You may as soon extract oil and syrup out of a flint, as for him to have any real benefit out of Scripture.
Take heed against jesting with Scripture; this is playing with fire. Some cannot be merry unless they make bold with God; when they are sad, they bring forth the Scripture as their harp to drive away the evil spirit, like a drunkard who having emptied his cups, calls to his fellows, “Give us your oil, for our lamps have gone out.” In the fear of God, beware of this. King Edward IV would not allow his crown to be jested with, but executed anyone who said he would make his son heir to the Crown, meaning the sign of the Crown on his tavern. Much less will God allow us to jest with his word. Eusebius relates about someone who took a piece of Scripture to jest with, that God struck him with panic. The Lord may justly give over such persons to a reprobate mind. Rom. 1.23.

2. If you would profit, prepare your hearts for reading the word; the heart is an instrument that needs to be tuned. 1Sam 7.3: “Prepare your hearts to the Lord.” The heathen, as Plutarch notes, thought it was indecent to be too hasty or rash in the service of their supposed deities. This preparation for reading consists in two things:

In summoning our thoughts together to attend to that solemn work we are about to do. Our thoughts are stragglers; therefore rally them together.
In purging those unclean affections which indispose us to reading. Before we come to the water of life, let us cast away the poison of impure affections. Many come rashly to the reading of the word; it is no wonder that if they come without preparation, they go away without profit.

3. Read the Scriptures with reverence; think about every line you read; God is speaking to you. The ark in which the law was put, was overlaid with pure gold, and was carried on bars, so that the Levites might not touch it. Ex. 25, Why was this if not to breed in the people a reverence for the law? When Ehud told Eglon he had a message for him from God, Eglon arose from his throne. Judges 3.20, The written word is a message from Jehovah; we should receive it with that veneration.

4. Read the books of Scripture in order. Though circumstances may sometimes divert our method, yet, for a constant course, it is best to observe an order in reading. Order is a help to memory. We do not begin to read a friend’s letter in the middle.

5. Get a right understanding of Scripture. Psalm 119.73: “Give me understanding, that I may learn your commandments.” Though there are some knots in Scripture which are not easily untied, yet the Holy Ghost has plainly pointed out to us things essential to salvation. The knowledge of the sense of the Scriptures is the first step to profit. In the Law, Aaron was first to light the lamps, and then to burn the incense; the lamp of the understanding must first be lighted before the affections can be inflamed. Get what knowledge you can by comparing scriptures, by conferring with others, and by using the best annotators. Without knowledge, the Scripture is a sealed book; every line is too high for us; and if the word shoots above our head, it can never hit our heart.

6. Read the word with seriousness. If one goes over the Scripture cursorily, says Erasmus, there is little good to be got by it; but if he is serious in reading it, it is the savor of life; and we may well be serious if we consider the importance of those truths which are bound up in this sacred volume. Deut. 32.47: “It is not a vain thing for you; it is your life.” If a letter were to be opened and read, in which a man’s whole estate were concerned, how serious would he be in reading it? In the Scripture, our salvation is concerned; it addresses the love of Christ, a serious subject. Christ has loved mankind more than the angels that fell. Heb. 2.7. The loadstone, indifferent to gold and pearl, draws the iron to it; thus Christ passed by the angels, who were of more noble extraction, and drew mankind to himself. Christ loved us more than his own life; indeed, though we had a hand in his death, yet he would not leave us out of his will. This is a love that passes knowledge; who can read this without seriousness? The Scripture speaks of the mystery of faith, the eternal recompenses, and the paucity of those who shall be saved. Matt. 20.16: “Few chosen.” Someone said the names of all the good emperors of Rome might be engraved in a little ring; and there are but (comparatively) few names in the Book of Life. The Scripture speaks of striving for heaven as in an agony. Luke 13.24. It cautions us not to fall short of the promised rest. Heb. 4.1. It describes the horrors of the infernal torments, the worm, and the fire. Mark 9.44. Who can read this and not be serious? Some have light, feathery, spirits; they run over the most weighty truths in haste (like Israel who ate the Passover in haste); and so they are not benefited by the word. Read with a solemn, composed spirit. Seriousness is the Christian’s ballast, which keeps him from being overturned with vanity.

7. Labor to remember what you read. Satan would steal the word out of our mind; not that he intends to make use of it himself, but lest we should make use of it. The memory should be like the chest in the ark, where the ark was put. Psa 119.52: “I remembered your judgments of old.”

Jerome speaks of that religious lady, Paula, that she knew most of the Scriptures by heart; we are bid to have “the word dwell in us.” Col. 3.16, The word is a jewel; it adorns the hidden man, and shall we not remember it? If the word does not stay in the memory, it cannot profit. Some can better remember a piece of news than a line of Scripture; their memories are like those ponds where the frogs live, but the fish die.

8. Meditate upon what you read. Psalm 119.15: “I will meditate on your precepts.” The Hebrew word to meditate, signifies to be intense in the mind. In meditation there must be a fixing of the thoughts upon the object. Luke 2.19: “Mary pondered those things.” Meditation is the concoction of Scripture; reading brings a truth into our head, meditation brings it into our heart; reading and meditation, like Castor and Pollux, must appear together. Meditation without reading is erroneous; reading without meditation is barren. The bee sucks the flower, and then works it into the hive, and so turns it into honey. By reading, we suck the flower of the word; and by meditation, we work it into the hive of our mind, and so it turns to our profit. Meditation is the bellows of the affection. Psalm 39.3: “While I was musing the fire burned.” The reason we come away so cold from reading the word, is because we do not warm ourselves at the fire of meditation.

9. Come to the reading of Scripture with humble hearts; acknowledge how unworthy you are that God would reveal himself in his word to you. God’s secrets are with the humble. Pride is an enemy to profiting. It has been said that the ground on which the peacock sits is barren; that heart where pride sits is really barren. An arrogant person disdains the counsels of the word, and hates the reproofs: is he likely to profit? James 4.6: “God gives grace to the humble.” The most eminent saints have been of low stature in their own eyes; like the sun at the zenith, they showed least when they were at the highest. David had “more understanding than all his teachers.” Psalm 119.99: but how humble he was. Psalm 22.6: “I am a worm and no man.”

10. Give credence to the written word; believe it is of God; see the name of God in every line. The Romans, so that they might gain credit for their laws, reported that they were inspired by the gods at Rome. Believe the Scriptures to be divinely inspired. 2Tim. 3.16: “All Scripture is given by inspiration of God.” Who but God could reveal the great doctrines of the Trinity, the atonement of Jesus Christ for sinners, and the resurrection? From where would the Scriptures come, if not from God?

Sinners could not be the authors of Scripture. Would they indite such holy lines, or inveigh so fiercely against the sins which they love?
Saints could not be the authors of Scripture; how could it stand with their sanctity to counterfeit God’s name, and put “thus says the Lord,” to a book of their own devising?
Angels could not be the authors of Scripture. What angel in heaven dares impersonate God, and say, “I am the Lord?”

Believe that the pedigree of Scripture is sacred, and that it comes from the Father of light. The antiquity of Scripture speaks its divinity. No extant human history reaches further than Noah’s flood; but the Scripture addresses things before time. Besides that, the majesty, profundity, purity and harmony of Scripture, show it could be breathed from none but God himself. Add to this the efficacy that the written word has upon men’s consciences. By reading Scripture, they have been turned into other men, as may be instanced in Austin, Junius, and others. If you were to set a seal upon a piece of marble, and it left a print behind, you would say there was a strange virtue in that seed; so that, when the written word leaves a heavenly print of grace upon the heart, it argues that it is of divine authority. If you were to profit by the word, you must believe it is of God.

Some skeptics question the verity of Scripture. Though they have the articles of religion in their creed, yet they do not have it in their belief. Unbelief enervates the virtue of the word and makes it abortive. Who will obey truths that he does not believe? Heb. 4.2: “The word did not profit them, not being mixed with faith.”

11. Highly prize the Scriptures. Psalm 119.72: “The law of your mouth is better to me than thousands of gold and silver pieces.” Can someone be proficient in any art if he slights and depreciates it? Prize this book of God above all other books. Gregory calls the Bible the heart and soul of God. The Rabbies say that there is a mountain of sense upon every point and tittle of Scripture. Psalm 19.7: “The law of the Lord is perfect.” The Scripture is the library of the Holy Ghost; it is a code of divine knowledge, an exact model and platform of religion. The Scripture contains in it the Credenda, the things which we are to believe, and the Agenda, the things which we are to practise; it is able to make us wise unto salvation. 2Tim. 3.15. The Scripture is the standard of truth, the judge of controversy; it is the pole star to direct us to heaven. The Scripture is the compass by which the rudder of our will is to be steered; it is the field in which Christ, the pearl of price, is hidden; it is a rock of diamond; it is a sacred Collyrium, or eye-salve; it mends the eyes of those who look upon it; it is a spiritual optic glass in which the glory of God is resplendent; it is the panacea, or universal medicine for the soul. The leaves of Scripture are like “the leaves of the tree of life for the healing of the nations.” Rev. 22.2. The Scripture is the breeder and feeder of grace. How is the convert born, but by “the word of truth?” James 1.18. How does he grow but by “the sincere milk of the word?” 1Peter 2.2. The written word is the book out of which our evidences for heaven are fetched; it is the sea-mark which shows us the rocks of sin; it is the antidote against error and apostasy; the two-edged sword which wounds the old serpent. It is our bulwark to withstand the force of lust, like the Capitol at Rome, which was a place of strength and ammunition. The Scripture is the tower of David on which the shields of our faith hang. Take away the word, and you deprive us of the sun, said Luther. The written word is above an angelical embassy, or a voice from heaven. 2 Peter 1.18: “We heard this voice which came from heaven; we also have a more sure word.” If Caesar so valued his Commentaries, that in preserving them he lost his purple robe, how should we estimate the sacred oracles of God? Job 23.12: “I have esteemed the words of his mouth more than my necessary food.” King Edward VI, on the day of his coronation, had three swords presented to him, signifying that he was monarch of three kingdoms. The King said there was one sword missing; being asked what that was, he answered, “the Holy Bible, which is the sword of the spirit, and it is to be preferred before all these ensigns of royalty.” Robert, King of Sicily, so prized God’s word that, speaking to his friend Petrarch, he said, “I protest that the Scriptures are dearer to me than my kingdom, and if I must be deprived of one of them, I would rather lose my diadem than the Scriptures.”

12. Get an ardent love for the word. Prizing something relates to the judgment; love relates to the affections. Psalm 119.159: “Consider how I love your precepts.” He is likely to grow rich who delights in his trade; a lover of learning will be a scholar. Austin tells us that before his conversion he took no pleasure in the Scriptures, but afterwards they were his delights. David thought the word was sweeter than the honey which drips from the comb. Thomas `a Kempis used to say he found no contentment except in a corner, with the book of God in his hand. Did Alphonsus, King of Sicily, recover from a fit of sickness because of the great pleasure he took in reading Quintus Curtius? Then what infinite pleasure we should take in reading the book of life! There is enough in the word to breed holy satisfaction and delight; it is a specimen and demonstration of God’s holy love for us. The Spirit is God’s love-token; the word is his love letter; and how one rejoices to read over his friend’s letter! The written word is a divine treasury, or store-house. Truth is scattered in it like pearls to adorn the inner man of his heart. The written word is the true manna, which has all sorts of sweet taste in it. It is a sovereign elixir; it gives wine to those with a heavy heart. I have read of an ancient Rabbi, who, in a great concourse of people, proclaimed a sovereign cordial which he had to sell. Many resorted to him and asking him to show it, he opened the Bible, and directed them to several places of comfort in it. Holy David drank of this cordial; Psalm 119.50: “This is my comfort in my affliction; for your word has quickened me.” Chrysostom compares the Scripture to a garden; every line in it is a fragrant flower which we should not wear in our pocket but in our heart.

Delight in the word causes profit: and we must not love only the comforts of the word, but the reproofs. Myrrh is bitter to the palate, but good for the stomach.

13. Come to the reading of the word with honest hearts. Christ speaks of the “honest heart.” Luke 8.15.

What is it to read the word with an honest heart?

To come with a heart willing to know the whole counsel of God. A good heart would not have any truth concealed, but says as Job did: “What I do not see, teach me.” When men pick and choose in religion, they will do some things that the word enjoins them from, but not others. These are unsound hearts, and they are not benefited by holy writ. These are like a patient, who having a bitter pill prescribed, and a mint julep, he will take the julep, but he refuses the pill.
To read the word with an honest heart is to read that we may be made better by it. The word is the medium and method of sanctification, and we come to it not only to illuminate but to consecrate ourselves. John 17.17: “Sanctify them through your truth.” Some go to the Bible as one goes to the garden to pick flowers, i.e. for fine notions. Austin confesses that before his conversion he went to hear Ambrose, more for the elegant speech and quaint notions, than for the spirituality of the matter. This is like a woman that paints her face, but neglects her health. But this is to have an honest heart: when we come to the Scriptures as Naaman came to the waters of the Jordan, to be healed of our leprosy. “O,” says the soul, “that the sword of the spirit may pierce the rock of my heart; that this blessed word may have such a virtue in it as the water of jealousy, to kill and make fruitful, that it may kill my sin, and make me fruitful in grace.” Num. 5.27.

14. Learn to apply Scripture; take every word as spoken to yourselves. When the word thunders against sin, think this way: God means my sins; when it presses any duty, God intends me in this word. Many put off Scripture from themselves, as if it only concerned those who lived in the time when it was written; but if you intend to profit by the word, then bring it home to yourselves. A medicine will do no good unless it is applied. The saints of old took the word as if it had been spoken to them by name. When king Josiah heard the threatening which was written in the book of God, he applied it to himself; he “rent his clothes and humbled his soul before the Lord.” 2 Kings 22.11.

15. Observe the preceptive part of the word, as well as the promissory. The precepts carry duty in them, like the veins which carry the blood; the promises carry comfort in them, like the arteries which carry the spirit. Make use of the precepts to direct you, and the promises to comfort you. Those who keep their eye on the promise, while neglecting the command, are not edified by Scripture; they look more for comfort than duty. They mistake their comforts, just as Apollo embraced the laurel tree instead of Daphne. The body may be swelled with wind as well as flesh; a man may be filled with false comfort, as well as that which is genuine and real.

16. Let your thoughts dwell on the passages of Scripture that matter. The bee fastens on those flowers where she may suck the most sweetness. Though the whole context of Scripture is excellent, yet some parts of it may have a greater emphasis, and be more quick and pungent. Reading the names of the tribes, or the genealogies of the patriarchs, is not of the same importance as faith and the new creature. Mind the “great things of the law.” Hosea 8.12. Those who read only to satisfy their curiosity, busy themselves, rather than profit themselves. Searching too far into Christ’s temporal reign has weakened his spiritual reign in some men’s hearts.

17. Compare yourself with the word. See how the Scripture and your hearts agree; how your dial goes with this sun. Are your hearts, as it were, a transcript and a counterpart of Scripture? Is the word copied out into your hearts? The word calls for humility; are you not only humbled, but also humble? The word calls for regeneration: John 3.7; do you have a change of heart? Is there not only a moral and partial change, but a spiritual change? Is there such a change wrought in you, that it is as if another soul lived in the same body? 1 Cor. 6.11: “Such were some of you, but you are washed, you are sanctified.” The word calls for love toward the saints; 1Peter 1.22. Do you love grace where you see it? Do you love grace in a poor man as well as in a rich man? A son loves to see his father’s picture, though hung in a poor frame.

Do you love grace even though it is mixed with some failings, just as we love gold even while it is in the ore? Bringing the rule of the word and our hearts together, to see how they agree, would prove very advantageous to us. By this we come to know the true complexion and state of our souls, and we see what evidences and certificates we have for heaven.

18. Take special notice of those Scriptures which speak to your particular case If a person with consumption were to read Galen or Hippocrates, he would chiefly observe what they said about consumption. Great regard is to be had to those paragraphs of Scripture which are most appropriate to one’s present case.

I will note three cases: 1. Affliction. 2. Desertion. 3. Sin.

1. Affliction. Has God made your chain heavy? Consult these Scriptures. Heb. 12.7: “If you endure chastening, God deals with you as sons.” Isaiah 27.9: “By this the iniquity of Jacob shall be purged.” John 16.20: “Your sorrow shall be turned into joy.” 2Cor. 4.17: “Our light affliction, which is but for a moment, works for us a far more exceeding and eternal weight of glory.” The limner lays his gold on dark colors; God first lays the dark color of affliction, and then the golden color of glory.

2. Desertion. Are your spiritual comforts eclipsed? See Isaiah 54.8: “In a little wrath I hid my face from you, for a moment; but with everlasting kindness I will have mercy on you.” The sun may hide itself in a cloud, but it is not out of the firmament; God may hide his face, but he is not out of the covenant. Isaiah 57.16: “I will not be always angry, for the spirits would fail before me, and the souls which I have made.” God is like the musician; he will not stretch the strings of his lute too hard, lest they break. Psalm 97.11: “Light is sown for the righteous.” A saint’s comfort may be hidden as seed is hidden under the clods of dirt; but at last it will spring up into a harvest of joy.

3. Sin.

Are you drawn away with lust? Read Gal. 5.24; James 1.15; 1Peter 2.11: “Abstain from fleshly lusts, which war against the soul.” Lust kills with embracing. Prov. 7.10, 22, 23; Prov. 22.14: “Go to the waters of the sanctuary to quench the fire of lust.”
Are you under the power of unbelief? Read Isaiah 26.3: “You will keep him in perfect peace whose mind is stayed on you, because he trusts in you.” Mr. Boltor speaks of a distressed soul who found much comfort from this Scripture on his sick bed. 2 Samuel 22.31: “The word of the Lord is tested; he is a buckler to all that trust in him.” John 3.15: “That whoever believes in him should not perish.” Unbelief is a God-affronting sin. 1John 5.10: “He that does not believe God has made him a liar.” It is a soul-murdering sin. John 3.36: “He that does not believe the Son shall not see life, but the wrath of God abides on him.” Thus, in reading, observe those Scriptures which touch upon your particular case. Although all of the Bible must be read, yet be sure to put a special star upon those texts which point most directly to your condition,.

19. Take special notice of the examples in Scripture; make the examples of others living sermons to you.

Observe the examples of God’s judgments upon sinners. They have been hanged up in chains, as a terror. How severely God has punished proud men! Nebuchadnezzar was turned to eat grass; Herod was eaten up with vermin. How God has plagued idolaters! Numbers 25.3, 4, 9; 1Kings 14.9, 10. What a swift witness he has been against liars! Acts 5.5, 10. These examples are set up as sea-marks to avoid. 1Cor. 10.11; Jude 7.
Observe the examples of God’s mercy to saints. Jeremiah was preserved in the dungeon; the three children in the furnace; Daniel in the lion’s den. These examples are props to our faith, and spurs to holiness.

20. Do not stop reading in the Bible, till you find your hearts warmed. Psalm 119.93: “I will never forget your precepts, for you have made me alive with them.” Read the word not only as a history, but strive to be affected with it. Let it not only inform you, but inflame you. Jer. 23.29: “Is not my word like a fire?” says the Lord. Do not go from the word till you can say as those disciples said, Luke 24.32: “Did our hearts not burn within us?”

21. Begin the practice of what you read. Psalm 119.66: “I have done your commandments.” A student in medicine does not satisfy himself to read over a system or body of medicine, but he begins its practice. The life-blood of religion lies in the practical part. So it is in this text: “He shall read in the book of the law all the days of his life, so that he may learn to keep all the words of this law and these statutes, to do them.” Christians should be walking Bibles. Xenophon said many read the laws of Lycurgus, but few observed them. The written word is not only a rule of knowledge, but a rule of obedience; it is not only to mend our sight but to mend our pace. David calls God’s word a “lamp to his feet.” Psalm 119.105. It was not only a light to his eyes to see by, but to his feet to walk by; by practice we trade the talent of knowledge, and turn it to profit. This is a blessed reading of the Scriptures, when we fly from the sins which the word forbids, and espouse the duties which the word commands. Reading without practice will be but a torch to light men to hell.

22. Make use of Christ’s prophetical office. He is the lion of the tribe of Judah, to whom it is given to open the book of God, and loose its seals. Rev. 5.5.

Christ so teaches, as he enlivens. John 8.12. The philosopher says, light and heat increase together. It is true here: where Christ comes into the soul with his light, there is the heat of spiritual life going along with it. Christ gives us a taste of his word: “You have taught me how sweet your words are to my taste.” Psalm 119.102, 103. It is one thing to read a promise, and another to taste it. Those who would be proficient in Scripture, let them get Christ to be their teacher. Luke 24.45: “Then he opened their understanding, that they might understand the Scriptures.” Christ not only opened the Scriptures, but he opened their understanding.

23. Tread often upon the threshold of the sanctuary. Wait diligently upon a rightly constituted ministry. Prov 8.34: “Blessed is the man who hears me, watching daily at my gates.” Ministers are God’s interpreters; it is their work to open and expound dark places in Scripture. We read of pitchers and lamps within those pitchers. Judges 7.16. Ministers are earthen pitchers. 2Cor. 4.7. But these pitchers have lamps within them to light souls in the dark.

24. Pray that God will make you profit. Isaiah 48.17: “I am the Lord your God, who teaches you to profit.” Make David’s prayer: “Open my eyes, that I may behold wondrous things out of your law.” Psalm 119.18. Pray to God to take the veil off the Scriptures, so that you may understand it; and the veil on your heart, so that you may believe it. Pray that God will not only give you his word as a rule of holiness, but his grace as a principle of holiness. Implore the guidance of God’s Spirit. Neh. 9.20: “You gave them your good Spirit to instruct them.” Though the ship has a compass to sail by, and a store of tackle, yet without a gale of wind it cannot sail. Though we have the written word as our compass to sail by, and we make use of our endeavors as the tackle, yet unless the Spirit of God blows on us, we cannot sail with profit. When the Almighty is like dew to us, then “we grow as the lily, and our beauty is like the olive tree.” Hosea 14.5, 6: by the anointing of the Holy Ghost. One may see the figures on a sun-dial, but he cannot tell how the day goes unless the sun shines. We may read many truths in the Bible, but we cannot know them savingly till God’s Spirit shines into our souls. 2Cor. 4.6. The Spirit is a Spirit of wisdom and revelation. Eph. 1.17. When Philip joined himself to the eunuch’s chariot, then the eunuch understood the Scriptures. Acts 8.35. When God’s Spirit joins himself to the word, then it will be effectual to salvation. These rules being observed, the written word would, through God’s blessing, become an “engrafted word.” James 1.21. A good shoot grafted into a bad stock, changes the nature of the stock, and makes it bear sweet and generous fruit. So when the word is grafted savingly into men’s hearts, it sanctifies them, and makes them bring forth the sweet “fruits of righteousness.” Phil. 1.11.

Thus I have answered this question, how we may read the Scriptures with the most spiritual profit. In conclusion,

Do not content yourselves with the bare reading of the Scriptures, but labor to find some spiritual improvement and profit from it. Get the word transcribed into your hearts. Psalm 37.31: “The law of his God is in his heart.” Those who profit by reading the book of God are the best Christians alive; they answer God’s cost, they credit religion, and they save their souls.
Those of you who have profited by reading the Holy Scriptures, adore God’s distinguishing grace. Bless God that he has not only brought the light to you, but opened your eyes to see it; that he has unlocked his hidden treasure, and enriched you with saving knowledge. Some perish by not having Scripture, and others by not improving it. Bless God that he should pass by millions, and the lot of his electing love should fall upon you; that the Scripture, like the pillar of cloud, should have a dark side to others, but a light side to you; that to others it is a dead letter, but to you it is the savor of life; that Christ should not only be revealed to you, but in you; Gal. 1.16. —You should be in an holy ecstacy of wonder, and wish that you had the hearts of seraphims burning in love toward God, and the voices of angels to make heaven ring with God’s praises!

But some of the godly may say they fear that they do not profit by the word they read. As in the body, when the vital spirits become faint, cordials are employed. So let me apply a few divine cordials to those who are ready to faint under the fear of their non-proficiency [in the Scriptures].

You may profit by reading the word, even though you come short of others. The ground which brought forth thirty-fold was good ground. Matt. 13.8. Do not say you obtained no profit because you are not equal with other eminent saints; the others among David’s worthies were considered his strong men, even though they did not attain the honor of the first three. 2Sam. 23.19.
You may profit by reading the word, even though you are not quick to apprehend. Some impeach themselves because they are slow of understanding. When our blessed Saviour told of his sufferings to come, the apostles themselves did not understand, and it was hidden from them. Luke 9.45. The author to the Hebrews speaks of some who were dull of hearing. Heb. 5.11. Those who have weaker judgments may have stronger affections. A Christian with a little knowledge may be kept from sin, like a man with weak sight may be kept by a little knowledge from falling into the water.
You may profit by reading Scripture, although you do not have excellent memories. Many complain that their memories leak. Christian, are you grieved that you can no longer remember? Then for your comfort,
You may have a good heart, even though you do not have so good a memory.
Though you cannot remember all you read, yet you remember what matters most, and what you most need; at a feast we do not eat every dish, but we take only so much as nourishes. It is with a good Christian’s memory as it is with a lamp: though the lamp is not full of oil, yet it has enough oil to make the lamp burn; though your memory is not full of Scripture, yet you retain enough to make your love towards God burn.

So then, be of good comfort. You profit by what you read, and you take note of that encouraging Scripture, John 14.26: “The Comforter, which is the Holy Ghost, he shall bring all things to your remembrance.” Amen.

Source

Mouse Gestures (the useful ones)

River Writes - A MediaWiki Blog — Tue, 12 Oct 2021 08:06:54 GMT

My Gesturefy bindings in Firefox and how I use them when working with MediaWiki

Book review - Free to All

River Writes - A MediaWiki Blog — Tue, 12 Oct 2021 07:26:12 GMT

A review and chapter summaries of “Free to All: Carnegie Libraries & American Culture, 1890-1920”

Can Underground Agriculture Solve the World Food Problem?

Thoughts by Thomas Derflinger — Tue, 12 Oct 2021 03:00:00 GMT

Underground agriculture is an emerging field of agriculture that has the potential to help solve a big part of the current ecological crisis. How this can be achieved I want to explain in this article.

I’m a Rails developer—what the heck is Storybook, and should I care?

Nicolas Goutay — Tue, 12 Oct 2021 03:00:00 GMT

A lightly edited version of our conversation that tries to answer the following questions: what is Storybook? How useful can it be in a Rails project? And maybe most importantly: as a Rails developer, should I care?

LIT Home

taylor.town — Tue, 12 Oct 2021 03:00:00 GMT

My friends and I converted a shed into a tiny-home!

Exponential growth and shipping - 123dev #37

Justin Garrison's Homepage — Tue, 12 Oct 2021 03:00:00 GMT

Comments

Shipping products

This past week I was part of the team that launched EKS Anywhere. Launching an open source project was something I really wanted to do to see how it was different from products and services I’ve helped launch in the past.

The biggest difference for me was it’s really refreshing to point people to GitHub for issues and to let people use the software for free. The users I got to talk to were fellow infrastructure engineers which also made it really easy to talk about the project because I am personally familiar to the problems it solves.

Surveillance cameras are just more social media

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 12 Oct 2021 02:40:00 GMT

They have the same defects, that is.

[The World] Requiem for Strawberry

devever.net/~hl — Mon, 11 Oct 2021 20:17:00 GMT

Strawberry passed away today, on the 11th of October 2021, at about 7:25am, UK time. She died of heart failure; she died in one's arms. It was quick and unfraught, and she was with her family until the end, as she had always wanted.

Thoughts on Redux

Posts on Evert Timberg — Mon, 11 Oct 2021 17:42:47 GMT

Redux is a state management library that is commonly used in React apps. Redux was created in 2015 and almost immediately became a common sight in React apps. This may have been due to the fact that many of the React docs assumed that Redux was installed along with the popularity of boilerplates that included Redux by default. As a result of this, many front-end developers including myself, incorrectly assumed that React had to be used with Redux.

Next.js Cheatsheet - handy snippets and tips

Gourav Goyal — Mon, 11 Oct 2021 03:00:00 GMT

Setup debugging, custom domains, Tailwind CSS, SEO, Typescript, ESLint, analytics, and much more to Next.js project

Please don't argue with the weather

jeremymikkola.com — Mon, 11 Oct 2021 03:00:00 GMT

Posted on October 11, 2021

Say you have a vacation planned next week. You are going south to a fun coastal town renown for its beaches. You’re hoping to get outside, get some sun, and stick your toes in the water while drinking something with too much sugar and alcohol¹.

Hoping for good weather, you turn on the news. The meteorologist says it will be 50 F and cloudy. This isn’t what you were hoping for! Sitting on the beach won’t feel right in drab weather like that.

Think about how you respond to this situation: do you try to argue with the meteorologist? Do you say, “how about we actually do 72 F and sunny?” No, that would be silly. A forecast isn’t a decision the meteorologist is making. It’s merely a prediction about what will happen. Predicting a different temperature will do nothing to how the air will feel.

No, if you want different weather next week, your only option is to go to a different place. You obviously can’t argue with the weather.

Yet in software companies, we often try to do exactly that. When an engineer says that a project will take 7 weeks, someone is sure to argue and ask for it to be done in 5. But changing the estimate written down on paper doesn’t change how hard the project is to accomplish. It doesn’t remove time waiting for other teams or mitigate project risks. It doesn’t make the code cleaner, the tests run faster, or magically document undocumented libraries. It is mere arguing with the weather.

If you want a different estimate for the end of a project, the only option is to change where that project is going. Choosing to build something smaller or different can (sometimes!) give you more favorable estimates. It helps to know the whole weather map, not just the weather in that one town.

But sometimes it is just plain cold everywhere. There’s no law of physics that says there must exist a spot on the map with a temperature you like. If the system is too complex and too full of tech debt, there might not be any meaningful options that can be accomplished in 5 weeks. Pushing back on the person who tells you this won’t change the facts.

If you are especially wise, you may notice a pattern where the temperature is frequently not what you want. You realize that you need to make a systematic change in how you are planning your vacations. Perhaps your yearly beach vacation needs to move from January to August. In software, this might look like giving a team time to pay down tech debt so that they can move faster later, staffing a team they get blocked by, or involving the team sooner in the new product.

Please don’t argue with the weather. If you don’t like the temperature, change where you are going.

I can’t recall speaking with anyone whose vacation plans actually look like this↩︎

Tools and workflows - 123dev #36

Justin Garrison's Homepage — Mon, 11 Oct 2021 03:00:00 GMT

Comments

Tools you’re familiar with

The right tool for the job isn’t always the best tool. Some tools are made for a specific reason, but there is a learning curve and cognitive tax to know how to use new tools. Sometimes the right tool is the one that is good enough that you already know.

The hardest part is knowing when to switch tools. If you work on a team this means updating documentation and training and maybe convincing the rest of your team the switch will be worth it.

Golang pretty print

Yasha Solutions — Mon, 11 Oct 2021 01:00:00 GMT

A quick snippet to get some nicer debug of json and other mapped values. 1 2 3 4 5 6 7 8 //PrettyPrint print any interface func PrettyPrint(data interface{}) (err error) { indentedData, err := json.MarshalIndent(data, "", " ") if err == nil { fmt.Println(string(indentedData)) } return }

OBS Studio Virtual Camera on FreeBSD

David Schlachter — Mon, 11 Oct 2021 00:15:00 GMT

Use the OBS Virtual Camera on FreeBSD using webcamd.

Radio silence

rob.sh — Sun, 10 Oct 2021 22:45:00 GMT

For more regular updates, you can generally find me on instagram, or Héloïse on instagram.

Héloïse on Mt. Tam at Sunrise - October 2021

Sending emails to myself

voussoir.net/writing — Sun, 10 Oct 2021 22:20:41 GMT

Back to writing

Sending emails to myself

In April, I wrote operatornotify.py, a module which allows my programs to notify me of important information or errors. I was immediately very proud of it. It felt awesome to get so much impact out of a sub-100 line module as I went around hooking it up to my existing programs and getting the tracebacks from my horribly broken code delivered right to me.

Since then, I have figured out which parts of the interface are most important, and boiled them down into a single decorator, so that adding @operatornotify.main_decorator is all it takes to get emails from a program. It's probably the best bang-for-buck code I've written in a while.

@operatornotify.main_decorator(subject='myprogram.py')
def main(argv):
    ...

if __name__ == '__main__':
    raise SystemExit(main(sys.argv[1:]))

The most important part of operatornotify's design is that it takes advantage of your existing logging infrastructure. Although you can call operatornotify.notify whenever you want, it's much easier to add a log handler that captures WARNING level messages automatically. This has encouraged me to revisit some older programs that just used print statements and replace them with proper logging, which makes them easier to use and debug in general.

There are a few pain points that arise from using operatornotify as a log handler. For example:

When using @main_decorator, you have to pick a subject line ahead of time and can't change it later. But to improve the skimmability of emails in my inbox, I like to prefix the subject lines with ⭕︎ if everything went okay [1] and ✕ if there was a warning, which obviously is not known until the end of runtime. I can imagine a few solutions but haven't decided which is most clean and appropriate for inclusion in the module [2].
Not all messages cleanly fit into the integer levels of logging. For the programs I run as scheduled tasks, it's nice to receive some kind of message to know that they ran successfully and didn't just fail to launch, but I don't necessarily want to see all of their output since I trust the returned status of 0. So neither --operatornotify-level WARNING or INFO is really ideal.

But these inconveniences are vastly outweighed by the ability to add a single decorator function that immediately supercharges all of the existing log lines and encourages me to add better logging where I skimped on it earlier.

Then I took the next step towards this summit of email enlightenment: running the mail server myself. I figure there are three reasons I should self-host the mail for operatornotify:

The notifications usually contain private information or filepaths that I don't like to have sitting on a third-party server. I don't think these worries are justified since my email provider does at-rest encryption and uses SSL, but I just don't like it.
My code might wind up sending hundreds of emails in a row, and I don't want to do that to a third-party host.
It's a good chance for me to play with stuff I haven't tried before.

I'm using the Maddy email server, which is an all-in-one solution for both SMTP and IMAP. I also considered Mail-in-a-Box, but their documents specifically say that it's designed to have the whole machine to itself, which I don't want to do. The most widely known software is Postfix for SMTP and Dovecot for IMAP, but I started drowning in the configuration of dovecot before even touching postfix, and gave up. I'm just a single user emailing myself. I like that maddy is just a single binary. So now I'm free to send myself as many thousands of emails as I want, containing whatever I want, using my own code and connecting to my own server.

FairEmail on Android supports IMAP IDLE, so I can get tracebacks anytime, anywhere. It's great.

Here are some ways I'm using operatornotify:

My computer emails me after it boots. I configured the BIOS to automatically turn the computer back on after a power outage occurs, so I have an email letting me know roughly when that happened and what to expect next time I'm back.
My OVH server emails itself after it boots. Then, it launches a few programs — etiquette.voussoir.net, /u/Newsletterly, etc. — and emails again me when all of those have finished starting and the processes appear in pidof. If I accidentally break something in my Python environment that prevents those programs from working, I'll know about it before too long.
Most of my task scheduler jobs use --operatornotify-level WARNING. A lot of repetetive tasks that I used to do manually are now fully automated, but if they experience any errors (usually just network timeouts or 503s), I can re-run them at my convenience.
I'm working on a daily "system heartbeat" email that gives me a rundown of all the daemon processes that should always be running, and all of the task scheduler jobs that ran throughout the day. Although adding operatornotify to a program can help me get its traceback, it doesn't help me when the program is totally broken and didn't even launch. For example, when I upgrade Python and forget to pip install a module, so the ImportError occurs before main ever runs. I'd rather not add an intermediate launcher that tries to launch the task and notifies me if that fails.

For the record, the code that actually sends the emails is in my my_operatornotify.py. The basic code you need is:

import email.message
import smtplib

def send_email(subject, body=''):
    sender = 'xxx'
    recipient = 'xxx'

    subject = str(subject)
    body = str(body)

    message = email.message.EmailMessage()
    message.add_header('From', sender)
    message.add_header('Subject', subject)
    message.add_header('To', recipient)
    message.set_payload(body, charset='utf-8')

    server = smtplib.SMTP_SSL('xxx', 465)
    server.login('username', 'password')
    server.send_message(message)
    server.quit()

def notify(subject, body=''):
    ...
    send_email(subject, body)
    ...

But I have some additional stuff in there to deal with retrying and eventually writing the message to my desktop if the mail server is unreachable.

To write the system heartbeat, I'm parsing the output of schtasks with the subprocess module:

import csv
import subprocess
from voussoirkit import winwhich

command = [
    winwhich.which('schtasks'),
    '/query',
    '/tn', '\\voussoir\\',
    '/fo', 'csv',
    '/v'
]
output = subprocess.check_output(
    command,
    stderr=subprocess.STDOUT,
    creationflags=subprocess.CREATE_NO_WINDOW,
)
output = output.decode('utf-8')
lines = output.splitlines()
reader = csv.DictReader(lines)
for line in reader:
    ...

[1] I wanted to use a checkmark, but the fonts on Android make all of the unicode check marks look stupendously ugly. And Android doesn't let me change the system font outside of Daddy Google's approved list because I'm a big stupid dummy baby who can't be trusted with a ttf because I might get a boo-boo.

[2] I am using a stopgap solution in my_operatornotify that detects the phrase "Program finished, returned 0" to add the ⭕︎.

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

In memory of Paolo Brunello

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 10 Oct 2021 21:00:00 GMT

Good deeds should not be forgotten.

Printable card games

fserb.com — Sun, 10 Oct 2021 14:31:00 GMT

Shephy

Shephy (シェフィ) was designed in 2013 by Pawn. It's a solitaire game that takes around 15 minutes to play.

This is a base game reskin.

In the solitaire game Shephy, you start with one sheep in your field, and you must manage the event cards presented to you in order to have one thousand sheep in the field before you go through the deck of event cards three times.

The sheep deck consists of values 1, 3, 10, 30, 100, 300 and 1,000; at the start of the game, you place a 1 sheep card in your field, then shuffle the 22 event cards and lay out five cards to form your hand. On a turn, you choose one card, carry out its effect, then refill your hand to five cards. Event cards might duplicate existing sheep cards, add new ones, swap one value of sheep for another, remove sheep, etc. Play through the deck until you have no cards in hand, then shuffle the event cards.

If you go through the deck three times without having at least one 1,000 sheep card in the field or if at any point in the game you have no sheep in the field, then you lose.

How to play video

Print and play my reskin: shephy.pdf 13 double-sided pages. 73 cards.

Murderer's Row

Murderer's Row was designed in 2015 by William Retert. It's a solitaire game that takes 10 minutes to play.

This is a base game reskin with a Calvin and Hobbes theme.

Deal 10 cards in a row. Kill 9 of them.

Each turn consists of activating a card and resolving its effect. Play proceeds until all cards except one have been eliminated or until no further moves are possible.

rules

Print and play my reskin: murderersrow.pdf. 3 double-sided pages. 22 cards.

Democracy vs Autocracy

Cam Hashemi — Sun, 10 Oct 2021 03:43:33 GMT

A group makes countless decisions over its lifetime.
Each decision falls on a spectrum from democratic to autocratic.

Some decisions are supremely autocratic, and others are purely democratic.
Between these two extremes, there is a spectrum of consensus.

As much as a dictator can be popular, a democracy can be tyrannical.

Democracy is not inherently good as autocracy is not inherently bad.
The success of each style depends on their execution within a context.

The Context

Autocracy

Autocracy has the advantage of efficiency.
A capable dictator with the right goals can achieve them quickly.
A plan of action from a single mind has consistency.
Consistent, planned actions are more likely to succeed.

But autocracy also has the disadvantage of efficiency.
An incompetent dictator lacks the input needed to choose the right goals,and they lack the feedback needed to correct themselves.
So an incompetent dictator achieves the wrong goals quicker than they should and takes longer to see their mistakes.

Democracy

Democracy has the advantage of information.
A healthy democracy gets input from every member of the group.
People are encouraged to voice their perspective.
Leaders who fail to resonate are ignored.

But democracy also has the disadvantage of information.
Network complexity scales exponentially.
Noise drowns the truth.
Too many perspectives create a paradox of choice.

The Paradox

Ironically, the success of a democracy depends on its leaders’ ability to influence the majority.
Without leaders, a democracy is chaos.
A democracy without strong leaders is confused and uninspired.

And ironically, the success of an autocracy depends on its populace’s ability to influence their leader.
Without a populace, an autocracy is nothing.
An autocracy without a strong populace is clueless and unengaged.

The Balance

Democracy creates input, feedback, energy.
Autocracy creates output, feed-forward, solidity.

To start, democracy collects voices and engages the group.
Once heard, autocracy commits to an achievable plan for the group.
Given that plan, distributing decisions maximizes the power of the group.
To close, autocracy organizes new information and imagines the next chapter.

With democracy on one side and autocracy on the other,
we should balance the scales gracefully across contexts.

The Mystery of the Ceaseless Botnet DDoS

Weblog on marginalia.nu — Sun, 10 Oct 2021 03:00:00 GMT

I’ve been dealing with a botnet for the last few days, that’s been sending junk search queries at an increasingly aggressive rate. They were reasonably easy to flag and block but just kept increasing the rate until that stopped working.

Long story short, my patience ran out and put my website behind cloudflare. I didn’t want to have to do this, because it does introduce a literal man in the middle and that kinda undermines the whole point of HTTPS, but I just don’t see any way around it. I just can’t spend every waking hour playing whac-a-mole with thousands of compromised servers flooding me with 50,000 search requests an hour. That’s five-six times more than when I was on the front page of HackerNews, and the attempts only increased.

Python 3.10 tips

Brent's Blog — Sun, 10 Oct 2021 03:00:00 GMT

Handy tips and notes about Python 30.8 10, 11 and beyond.

Mental health is health - 123dev #35

Justin Garrison's Homepage — Sun, 10 Oct 2021 03:00:00 GMT

Comments

Stress

As a junior developer you get assigned work and have to implement it and solve problems within a certain scope. As you get more senior you have more responsibility which often comes with more visibility and more pressure. Sometimes you’re a blocker for a new project or are part of an outage. These can be stressful situations.

A Web Of Solutions - October 2021

Yasha Solutions — Sun, 10 Oct 2021 01:00:00 GMT

A refactored digest of my reading of the month. A short introduction to give a bit of context : I have been looking for a way to share on a regular basis some of my learnings. I tend to go through a pretty large amount of content as part of my work and in general. So here I will try to provide it as a structured digest. The goal is to provide an interesting selection of content both practical and conceptual to help advancing how to use tech to solve real problems.

Comparing the Fujifilm XF 16‑80 with the XF 18‑135 Travel Zoom Lenses

bastibe.de — Sat, 09 Oct 2021 18:16:00 GMT

When I bought into the Fuji system, I selected the XF 18‑135 f/3.5‑5.6 R LM OIS WR as my main zoom lens. This is a lens with a very wide focal range, that is commonly called a “travel zoom” because you could travel the world with just this one lens. And indeed I happily did. In 2019, Fujifilm released a second travel zoom lens, the XF 16‑80 f/4 R OIS WR. Ever since, I have wondered how this new 16‑80 compares to my 18‑135. But given that these lenses are somewhat similar, few people on the internet were ever able to compare them side by side. This blog post will change that.

The 18‑135 has served me very well indeed. Of course it is not the world's brightest lens, nor sharpest, nor smallest. But so long as I can get the shot, these limitations don't bother me. Except for two things: The transition between in-focus and out-of-focus can be a bit rough, and my lens extends on its own when carried on a sling. My hope is that the 16‑80 has a nicer rendering, especially for people photos with out-of-focus backgrounds, and a stiffer zoom ring that doesn't creep.

Instead of smoothly blurring, the near-focus areas of the 18‑135 have this strange, hazy look

Physical

Physically, the 16‑80 is a slight bit smaller (89 mm vs 98 mm) and ligher (440 g vs 490 g) than the 18‑135, but the difference is negligible on camera. If anything, the 16‑80 actually feels a bit bigger to me due to its larger front element. Some people claim that the 16‑80 feels better built than the 18‑135. To that end, the rings on my 16‑80 turn more smoothly than my 18‑135's, but then that is comparing a new-ish lens to a well-used one. Autofocus speeds are also reputably different, but they feel similar to me.

The 16‑80 has a numbered aperture ring that allows for adjustments while the camera is turned off. On the other hand, the 18‑135 can switch to and from auto-aperture without losing its preset aperture, which is useful as well. The ideal lens for me would have both the auto-aperture switch and the numbered aperture ring. Oh well.

Size comparison of the two lenses. The 18‑135 is a longer, and the 16‑80 wider. Note the marked aperture ring on the 16‑80, and the Auto-Aperture/OIS switches on the 18‑135.

The 16‑80 has a 72 mm filter thread, while the 18‑135 uses 67 mm filters. This is somewhat annoying for me, as my filters are all 67 mm, which also fit on my 70‑300. I'd imagine users of the 72 mm 10‑24 see this differently. Anyway, using a thin polarizer with a 72‑to‑67 step-down ring works without issue on the 16‑80. My inch-thick macro filter however does vignette heavily until 23 mm.

Rendering

In terms of rendering, I find the 16‑80 to have a gentler transition from in-focus to out-of-focus, and indeed render out-of-focus backgrounds more smoothly than the 18‑135. At the long end, the 16-80 is actually a useful portrait lens, which is an unexpected but welcome feature for my photography.

Rendering of out-of-focus elements. Especially the points of light in the top right have an asymmetric crescent shape for the 18‑135, but blur smoothly for the 16‑80.

At the wide end, the 16‑80 exhibits some significant distortion. While the camera or post processing programs can easily correct this, it leaves the image corners stretched, and renders the 16‑80's nice round bokeh balls as ugly ovals. Thus large-aperture shots at the wide end can be somewhat problematic.

Resolution

Next, let's compare the resolution of the two lenses. To do that, I printed a test chart, and took some test images. All images were taken in identical illumination, on a tripod, with the chart covering half the sensor height/width. The following graphic shows a resolution scale near the center of the frame and near the left edge of the frame. And just for fun, I've thrown in a similar analysis from two prime lenses as well. The resolution scales are in 200x line widths per picture height. On my 6000×4000-pixel sensor, the theoretical maximum resolution would therefore be a resolution number of ⁴⁰⁰⁰∕₂₀₀ = 20. All test images are straight crops from original images, reproduced at their original resolution.

Comparison of lens resolution with a test chart. Each image is a direct crop at the X‑T2's native resolution. Resolution numbers in 200x line widths per image height.

Center images were focused in the center, and side images were focused on the side. A two-second timer was used to eliminate camera shake. Due to the geometry of my room and the size of my printer, all pictures were taken near the close-focusing distance of the lenses. A red line indicates the limit of resolution of these lines, as judged by my eyes. The corresponding resolution numbers are generally consistent with the ones published by opticallimits and lenstip¹, albeit I judged them a bit more conservatively.

The results of this test are surprising to me: the 16‑80 is actually my sharpest lens. It not only bests the 18‑135 at all settings, but also the 35 f/1.4. Only the 60 f/2.4 can reach similar resolution at the same f‑numbers. Generally, the 16‑80 is perfectly sharp right from f/4, while the 18‑135 has to be stopped down to f/8 to reach a similar resolution, especially on the image edges. Only at the long end does the 16‑80 benefit from stopping down.

That said, take these resolution measurements with a grain of salt. For instance, a “great” result of 15 (3000 LW/PH) translates to a blur radius of 1.3 pixels, while a “mediocre” result of 10 (2000 LW/PH) is instead 2 pixels. This is scientifically significant, but not at all relevant to (my) photography.

Additionally, the fact that the 60 f/2.4 macro lens scores so highly but the 35 f/1.4 does not is an indication that these measurements might be biased by being taken near the close focusing distance of the lenses. Thus the next set of images compares these lenses at more natural distances.

Comparison of lens resolution with natural images. Each image is a 500x500 pixel crop from original photographs. The subject is ca. 100 m from the camera.

At this farther distance, and with a more natural subject, the differences are no longer as easily visible. What differences there are this time favor the 18‑135 instead of the 16‑80. Interestingly, I didn't see any significant differences between these pictures when looking at them “merely” side-by-side in Capture One. Only when I actually assembled these here graphics and looked at them at 200% did the differences become apparant.

Nevertheless, it remains curious that there would be such a difference between the two lenses. Then, someone mentioned that the 16‑80 might suffer from shutter shock, where the camera's mechanical shutter jolts the camera enough to upset the image stabilization system and induce a slight bit of motion blur. An issue such as this might explain the 16‑80's slightly reduced resolution in my test shots. So I created another series of images, but this time both, with the mechanical shutter, and with electronic shutter. In electronic shutter mode, nothing moves in the camera and there can be no shutter shock.

Comparison of lens resolution and shutter shock. Each image is a 500x500 pixel crop from original images. The subject is ca. 100 m from the camera.

From this comparison, I can see no evidence of shutter shock. It might have been an issue on earlier firmware versions of the 16‑80, but my camera (an X-T2) and lens (at firmware 1.05) does not does not exhibit shutter shock. Furthermore, this series of pictures shows the 16‑80 and 18‑135 essentially matched in image resolution.

All of that said, I must add that all of these comparisons used extremely tight crops of high-contrast geometrical features. Most of the differences here are all but invisible in actual photographs. From these resolution experiments, I see no reason to prefer one lens over the other. Both of them are perfectly sharp in everyday use.

Conclusion

So, how to choose between the Fujifilm XF 16‑80 f/4 R OIS WR and the XF 18‑135 f/3.5‑5.6 R LM OIS WR? My 16‑80 has tighter aperture and zoom rings, does not creep, and has a smoother rendering of out-of-focus background. On the other hand, I do find the increased telephoto of the 18‑135 very useful, and it doesn't suffer from wide-angle distortion as much as the 16‑80.

In terms of resolution, I did not find fault with either lens. Both are very sharp across their entire focal range and the entire frame. That said, the 18‑135 does benefit from stopping down for optimum resolution, while the 16‑80 is sharp right from f/4, and the 16‑80 might be sharper for closer subjects.

My tentative conclusion from these experiments is therefore that I would slightly prefer the 16‑80 for people pictures, where the close-focus sharpness and nicer background rendering are advantageous, and the larger aperture at 80 mm might make a difference. And I would prefer the 18-135 for landscapes, where stopping down is usually easy and the longer focal length comes in handy.

That said, the differences in rendering and resolution are really very minor, and the choice most importantly comes down to the focal range. Which is as it should be with modern lenses. And both lenses are of course very well-built, weather sealed, and have fantastic image stabilization. But you probably knew that already.

multiply lenstip numbers by 2×16.7 mm to convert from lpmm (lines per millimeter) to LW/PH (line width per picture height)↩

Creating a custom Bulma build

Posts by Petr Stribny — Sat, 09 Oct 2021 17:00:00 GMT

WebAssembly-ing the Pieces: Vectorized’s Data Policy Engine

Who is Nnamdi? — Sat, 09 Oct 2021 16:00:00 GMT

Rather than ship data to code, which is expensive and latency-prone, why not ship code to the data?

Web Browsing

Weblog on marginalia.nu — Sat, 09 Oct 2021 03:00:00 GMT

An idea I’ve had for a long time with regards to navigating the web is to find a way to browse it.

“Browse” a difficult word to use, because it has a newer connotation of just using a web browser, I mean it in the old pre-Internet sense, browse like when you flip through a magazine, or peruse an antiques shop, not really looking for anything in particular just sort of seeing if anything catches your eye.

Chicago, Fall 2021

Avy Faingezicht — Sat, 09 Oct 2021 03:00:00 GMT

I went to Chicago for work, and took a lot of abstract building shots. You can definitely see the Angie McMonigal inspiration in some of these. There's also a couple of people in there.

The importance of docs - 123dev #34

Justin Garrison's Homepage — Sat, 09 Oct 2021 03:00:00 GMT

Comments

Puzzling

Growing up my dad loved puzzles. He had the weirdest rule that I never understood.

No matter how many pieces the puzzle had, he would look at the picture once when he opened it and never looked at it again. Thankfully, he never enforced that rule on us as kids, but I still don’t understand why he did it.

Automating My Air Conditioner

Danny Guo — Sat, 09 Oct 2021 03:00:00 GMT

Facebook down gave people a glimp of a better world

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 09 Oct 2021 02:40:00 GMT

“The news”. Think about that!

Cross entropy, joint entropy, conditional entropy and relative entropy

Bytepawn - Marton Trencseni — Sat, 09 Oct 2021 01:00:00 GMT

What's the difference between cross entropy, joint entropy, conditional entropy and relative entropy?

October

Gyanendra Mishra — Fri, 08 Oct 2021 21:30:00 GMT

.. another month - Ok, September is here. 4 more months left. I know I am writing this on the 5th of the month. The last few days have been busy! Retrospective Running: I wanted to be at 1496 at the end of the month. I am at 1500. ✅ Reading: I wanted to...

Little Intuition

ℤ→ℤ — Fri, 08 Oct 2021 21:23:01 GMT

This is the 60th anniversary of the proof of Little’s Law, a widely useful rule for queuing systems that links throughput, response (cycle) time, and number of requests (work items). Unfortunately, although software engineers often operate queues or pipelines (modern computer systems are rife with queues, implicit and explicit), they aren’t exposed to the mathematical foundations and often fail to build intuition around how queues act operationally. In this post, I aim to describe the Law, applications within software systems, and build an intuition in its use.

The equation

Little’s Law has two equivalent formulations: $L = \lambda W$ and $WIP = TH \times CT$. The first formulation is that the average number of items within the system ($L$) is equal to the average arrival rate of items ($\lambda$) times the average waiting time ($W$) of an item. The second formulation is the average number of work in progress (WIP) items is equal to the average throughput (TH) times the average cycle time (CT). The two formulations are equivalent, with the first focusing on “inputs” to the system with the second focusing on “outputs”.

Importantly, the waiting time or cycle time includes the time spent processing in addition to the time in queue. Similarly, $L$ and WIP include items sitting in queues as well as items being actively worked. All three values are expressed as averages; the law by itself won’t provide insight into ranges of those values.

Some intuition

First, logically, none of the three variables can be negative.

Secondly, cycle time will have some minimal value, $CT_0$, that represents the time to move an item through the pipeline with no schedule delay. However, cycle time can grow almost arbitrarily large if the amount of work exceeds the system’s capacity.

Thirdly, throughput will have some maximal value, $TH_\Omega$, that represents the maximum throughput manageable by the pipeline. (If you are modeling the system via the arrival rate, there might not be a maximum value although systems will need to reject requests at some point.)

For the system to perform some work (for throughput to be positive), some work needs to be released into the system. If too much work is released, cycle time will grow and the system will bounce around its maximum throughput. However, it is not necessary (nor likely) for throughput to climb linearly and for cycle time to only increase once throughput is saturated.

For instance, if a system has a minimal cycle time of 2 seconds and a maximum throughput of 4 units/second, then an ideal WIP, TH, and CT may act as:

WIP	TH	CT	Notes
0	0	2	$CT_0$
1	1/2	2
2	1	2
3	2/3	2
4	2	2
6	3	2
8	4	2	$TH_\Omega$ reached; end of flat $CT_0$
16	4	4	CT grows
32	4	8

However, a system could also behave like this:

WIP	TH	CT	Notes
0	0	2
1	1/2	2
2	2/3	3	CT grows, $TH_\Omega$ not yet reached
3	1	3
4	1	4
6	6/5	5
8	4/5	10
16	4/3	12
32	2	16

A divergence from an ideal progression can indicate losses due to queuing overhead or failures but can also be attributed to natural inefficiencies and cost of variability.

Because these numbers are averages, outliers can greatly influence the values. The law of small numbers and large numbers apply to observations. We plan to write more about this later, but the variability in a system (e.g. cycle times) greatly impacts the operator’s ability to control the system within some service level objective.

Applications

Some applications that requiring solving for or controlling each of the three variables:

L: If you need to estimate sizing requirements, like the size of the queue or how much disk will be required, you will be computing L via arrival rate and wait time.
CT: If you want to control response time/cycle time, you will estimate this value based on a count of accepted work and system throughput.
TH: If you want to control throughput, you will be using work release strategies to improve batching and trying to improve cycle time.

Machine Learning with Wikidata

Posts on A blog — Fri, 08 Oct 2021 17:48:41 GMT

This tutorial will be dedicated to understanding how to use the linear regression algorithm with Wikidata to make predictions. For a detailed explanation of how this algorithm works please read the Wikipedia article: linear regression. In this walkthrough Python is used.

Importing Modules/Packages

Before we start coding, import/install all of the following packages: NumPy, Pandas, and Sklearn.

# -*- coding: utf-8 -*-
import json
import numpy as np
import pandas as pd
import sklearn

from collections import defaultdict
from sklearn import linear_model

Loading in Our Data

Now it’s time for some data collection from Wikidata. For this example we are using the yearly (average) population stacked by country in a query (linked further down). This gives us a lot of interesting values and some with faults, unfortunately. I have chosen to filter this query to only include values from 2005 and newer. How you choose to import the query into the script is your decision. A passibility is to iterate over a SPARQL query by downloading the .rq file or just download a JSON file of the result from the query.wikidata.org site. Once you’ve downloaded the data set and placed it into your main directory (of the Python code) you will first need to clean the data, and later load it in using the pandas module.

Testing external API calls in Go

Liza Shulyayeva — Fri, 08 Oct 2021 09:20:00 GMT

Intro
Method 1 (Mocks)
Method 2 (httptest)

Intro

Both testing methods described here can be found in this repository. Be sure to run go generate before go test for the mocks to be generated. Otherwise, the mocking approach test described below will not work.

Node.js Package Management at UtahJS

Thomas Hunter II — Fri, 08 Oct 2021 03:00:00 GMT

Climbing the corporate ladder - 123dev #33

Justin Garrison's Homepage — Fri, 08 Oct 2021 03:00:00 GMT

Comments

Stealing The Corner Office

I listened to this book and here are the action items to help you climb the career ladder into senior leadership if that’s what you want. It’s not the best strategies for a dev/IC, but some of the tips are still helpful. If you can follow some of these tips your career climb hopefully won’t look like the gif.

Human Denial of Service Attacks

Dan Stroot — Fri, 08 Oct 2021 03:00:00 GMT

Can companies create human "denial of service" attacks against their own employees? They can and they do - all in the name of collaboration and 'making the business better'. If you work in a large organization you will recognize the pattern. Here's how it works...

This post Human Denial of Service Attacks first appeared on Dan Stroot's Blog

Your "21st century banking" is sixty years old

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 08 Oct 2021 02:40:00 GMT

And not just banking. But not for software.

My First Post

Posts on A blog — Thu, 07 Oct 2021 23:22:22 GMT

Hello! Bonjour! Hola! Hei!

This is my first time using this platform to write a blog post. I have not decided on what I’m supposed to write about. The platform is still a work in progress and was created since I’ve always wanted to start a technological blog. The topics will most likely be relevant to what I’m interested in or what I’m working on.

So, thank you for taking the time to read this first blog post; hopefully, there will be more to come in the future. Good-bye!

Our Goals and Our Questions

unsungNovelty — Thu, 07 Oct 2021 11:19:45 GMT

License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 _International Public License (CC BY-NC-ND _4.0)

We all have goals and we all ask questions. Like everybody, I have a ton of goals that needs to be finished. Things I want to achieve or finish. I use todo list items to describes a goal. But when you start working on a todo list item, the thing you will have in your head won’t be goal itself, but a bunch of questions.

Goals are important though. But we often need a plan when we start working on things. Goals are essentially those plans. Goals makes our existence as humans justifiable. Because we need things. And in order for us to get things we need, we try to finish a set of tasks which will help us reach our goals. Having a plan instead of winging it all the time will help things easier for us. Todo lists are just one such “plan” or format which hosts most of our goals these days. Doing a bunch of task items from the todo lists helps us reach our goals faster. Or that is the general gist. We use a notebook or an application on our computer/smartphones to create these tasks. And when we start doing the tasks though, we end up asking questions, to ourselves or others.

Questions are in our head, at least initially. Questions most of the time are also not documented/written down anywhere like our todo list items (goals!). This means you know what you want to do but you don’t know how to do it. You still ask questions though. You often search online for answers to your questions. How do I make a website? How can I do this? How can I do that? Or you ask a friend or someone who knows better about the subject which you are meddling with. This is a problem if you think about it. Because you’d rarely document the question and the answer. Then when you ask the same question, you repeat the procedure of finding the same answer all over again. Questions are our notes, our journey. Snippets of information which constitutes a goal. We could document the answers for our questions we have in our head. This can lead to not having to ask the same question again. DRY principle anyone?

I have recently started using questions for notes instead of a heading and the answer. I try to write the question I have in my head or use the question I asked the search engine instead of using a title/heading followed by the asnwer. This has been really helpful for me cos it is much more clear. Makes the note a lot better in my opinion. When you are using a software, product, project and you are going through the documentation, where do you go first? Mostly Frequently Asked Questions (FAQs). Have you noticed that FAQs are far more easier and much more accessible to you than the documentation itself? That is because you already have a question in your head and you are trying to find that question in the entire documentation in the hope that you can also find the answer to them. For example, my heading for notes on git tags changed from git tags to How do I create an annotated git tag?. Because information about annonated tag is what I want. Even when I need to find something else specific to git tags, I will be searching online anyway. So why bother copy pasting the entire documentation as your note? By being specific and splitting a topic to multiple questions, I know exactly what to look for next time I encounter a question. Which is mostly a lot of times. You could even classify your questions under a specific title or heading to make it more organised. For example, all the questions about git tags can be asked under git tags heading. And this is a much better way of using headings or a title.

How do I ask better questions?

Obviously the next big question is, how and when should I ask a question? Hear are some pointers for you…

Ask the question to yourself first. Sometimes all you want is a rubber duck. Someone to listen the words inside your head to get the answer. That is why sometimes when you ask a question/doubt to someone, midway through the question you answer it yourself.
Ask the question to a search engine. You could ask the question to a search engine before asking a human. There is a high chance that what you are trying to find is already tried by someone else in the world. You might even get better ideas for the information you are trying to get.
DON’T BE A HELP VAMPIRE! A help vampire is someone who just ask for help without putting any effort on their own to find solution for themselves. Before you ask a question, do your due diligence so that others are not wasting their time. It will also be faster most of the times to just do the research yourself instead of waiting for somebody else’s reply.

References / Links

Have thoughts or feedback regarding my post/writing? DM me on Twitter @unsungNovelty.

Drama going on at the .NET Foundation

Nicola Iarocci — Thu, 07 Oct 2021 09:05:25 GMT

A few months after I released my first .NET open source project (a niche one targeting the Italian fintech world), I was contacted by a representative of Team Digitale, the digital innovation branch of the Italian Public Administration. He suggested joining the Developers Italia initiative and moving my project to the their organization on GitHub “to enjoy enhanced visibility and broaden the audience”. I politely refused. I did not doubt my counterpart’s good faith. At the same time, I was concerned about the possible long-term consequences of a seemingly easy move. Moving a GitHub project away from your profile or an organization you control means ceding control over it. I was assured I would keep control of the project. But what happens if sometime in the future, when people in charge might even have changed, they revoke my access rights? As long as I am involved with my project, I should be in control. Also, I was not convinced that the move would help promote the project. We live in the search-engine age; people search for solutions to their problems. I was, and still am, confident that if I did my due diligence and my project is any good, people will find it¹.

Trouble Comes To Small Town

Tilting at Windmills — Thu, 07 Oct 2021 03:00:00 GMT

There's a certain type of movie I find that I really like. It's not exactly a genre. In fact it can exist in just about any genre. For lack…

Seed Stage Operating

rrich.io — Thu, 07 Oct 2021 03:00:00 GMT

The trap The differences between a first-time founder and a repeat founder are most noticeable during the seed stage. First-time founders want to take their newly acquired funding and put it to use right away. Hiring a team, buying new services, buying more ads, etc. Repeat-founders understand that cash is king, and preserving that funding to extend runway is how you avoid getting rinsed during your next round. Repeat-founders are also acutely aware of what product market fit looks like. This means they understand that dumping gas on a fire that isn’t yet ready for it is a great way to burn through several million dollars in a very short amount of time.

Please stop building custom CD tools - 123dev #32

Justin Garrison's Homepage — Thu, 07 Oct 2021 03:00:00 GMT

Comments

Your build tools are great, but no one wants to learn them

I’ve never worked somewhere that had a completely off-the-shelf CI/CD system. In some cases the systems are just minor tweaks or wrapper scripts over common tools (usually Jenkins). In extreme cases it’s completely from scratch tooling that has syntax and behaviors I’ve never encountered before.

Clojure

Home on mfashby.net — Wed, 06 Oct 2021 23:35:37 GMT

The difference between Instagram and TikTok, and why it matters

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 06 Oct 2021 21:00:00 GMT

It’s about equal opportunities. At least in India.

firefox: disable download dialog delay

blog — Wed, 06 Oct 2021 14:41:53 GMT

blog update: rss feed

blog — Wed, 06 Oct 2021 11:52:03 GMT

Getting with the times

Weblog on marginalia.nu — Wed, 06 Oct 2021 03:00:00 GMT

Since my search engine has expanded its scope to include blogs as well as primordial text documents, I’ve done some thinking about how to keep up with newer websites that actually grow and see updates.

Otherwise, as the crawl goes on, it tends to find fewer and fewer interesting web pages, and as the interesting pages are inevitably crawled to exhaustion, accumulate an ever growing amount of junk.

Re-visiting each page and looking for new links in previously visited pages is probably off the table, that’s something I can maybe do once a month.

Union vs sum types

viralinstruction — Wed, 06 Oct 2021 03:00:00 GMT

A brief explanation of union types, sum types, and how they relate and compare.

No code mo problems - 123dev #31

Justin Garrison's Homepage — Wed, 06 Oct 2021 03:00:00 GMT

Comments

Cher Ami

Cher Ami was a messenger pigeon used by the American army in France during WWI. A battalion of 550 troops was surrounded by the German army and the allies thought they were in a different location. The Major was trying to send out messages for help, but the messengers he sent (humans and pigeons) kept getting killed.

The allies began to drop artillery fire onto their location not knowing it was them. The Major sent out another pigeon, Cher Ami, with the message

The unsurprising digital challenges of agritourism

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 06 Oct 2021 02:40:00 GMT

Nothing new in the [digitized] country.

Using Apache Airflow ETL to fetch and analyze BTC data

Adnan's Random bytes — Tue, 05 Oct 2021 17:51:38 GMT

This post is part of the Data Engineering and ETL Series. I am taking a short break from Blockchain-related posts. In the past, I have covered Apache Airflow posts here. In this post, I am discussing how to use the CCXT library to grab BTC/USD data from exchanges and create an ETL for data analysis and visualization. I am using the dockerized version of Airflow. It is easy to set up and use proper different images to run different components instead of a one-machine setup. The docker-compose.yml file is available here but I have made a few changes to install custom libraries. Therefore, it is advised you use the file I have provided in the Github repo. Before I continue, the demo of the work is given here: The DAG The entire flow looks like the below: I am fetching BTC/USD data from three different exchanges: Binance, ByBit, and FTX. Once the data is available, I am sending all three sources into the next stage where I am only pulling close and timestamp data because these are fields that I need for the next stages. From here, I am branching out two tasks: one to create text files and store data for later use and the other is storing into the DB. The 3 different files are then sent to the server via FTP. The data from load_data section is then used to generate reports and then FTPing the file to the server. Notice that some borders are dark green and some are light green in color. Dark green means these tasks are executed successfully while light green means they are still running. The task, ftp_pdf_file is yet to run. In Airflow terms, the DAG flow has been set like below: In the first stage all 3 tasks running in parallel and output results to transform_data therefore I put all 3 tasks in a Python list. The >> sign is telling that it is upstream. The transformed data is being to create_text_file and load_data tasks who themselves are upstream for ftp_file and generate_pdf_reports tasks. Notice I am using << here to define upstream for ftp_pdf_file and generate_pdf_reports tasks. Again these tasks were first added in Python list and then branched out for the next stage. In order to pass data from one task to another task, Airflow provides xcom_pull and xcom_push methods. XCom means cross-communication. When you use xcom_pull, it means you are fetching data from the task whose IDs have been passed. For instance: Here I passed the task_id of three tasks and assigned in three different variables. Wondering where those task_ids are defined? they are defined as PythonOperator. Check the log of the task fetch_binance_ohlcv: I highlighted the returned value of the task. This and the other two tasks values are xcom_pulled in the transform_data task. I am also using two external components here: FTP and Database. You may use the traditional python code by importing the required libraries but Airflow also provides the options of hooks and connections. For instance, pgsql connection looks like the below: Once the connection is defined, you may use PG Hook to perform Postgres-based operations. Cool, isn’t it? No need to save credentials in the code. The same thing has been done with FTP. In the end, data is analyzed and generates a two-page PDF for executives. the graph is being generated in matplotlib and saved as an embedded image in the PDF file. Sweet, No? I was supposed to add data and charts of both FTX and ByBit too but I leave it for you ;). Also, this is not a heavy analysis of the data. I have provided you all data need. You may play with ta-lib or other libraries and find insights. All up to you 😉 I am using fpdf python library to generate PDF files. This is how files are being sent via FTP. Conclusion In this post, you learned how you can make complex flows for ETLs and use connections and hooks to connect 3rd party tools like FTP, DB, AWS, etc. Airflow could be a pretty powerful tool if used correctly. Like always, the code is available on GitHub. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Using Apache Airflow ETL to fetch and analyze BTC data first appeared on Adnan's Random bytes.

Write It Now

Josh Thompson — Tue, 05 Oct 2021 16:00:00 GMT

The original post

note from October 5, 2021: This was typed up/published in about 20 minutes, took 2x as long as I wish it had. I could make it 10x better with another hour of work, but I only have 20 minutes.

I’m a fan of “conceptual frameworks”

This concept has been important to me:

I’ve referenced it, almost apologetically, in a few other things I’ve written recently.

Usually I’ve just slapped it into an email or linked from another blog post, as a bit of an “i’m sorry” for how much work I might demand from the reader, or how unpleasant or confusing what I’m writing might be.

I feel like I often self-censure in an attempt to make something “better” than it is, and in doing so, fall off the back of this curve. (or I let vague concerns about quality/execution/time delay me from writing anything, ever)

I will reference this page regularly, in some ways to act as a defense against some very obvious criticisms. IMO, I’d rather throw 10 things as the wall quickly in a week, instead of 1 thing every other week.

If I propose something that sparks “controversy”, however that is defined, I’ll gladly cross that bridge (or burn it, if need be) when I get there.

If you’re a software developer, and use Github, you are welcome to leave a comment below (the link takes you to a comment on a dedicated Github Issue page: https://github.com/josh-works/josh-works.github.io/issues/49), or on any page around here with this link. I added this comments thing only recently, and still have not scripted an API call to auto-generate a new github issue when I write a new post. So… I just don’t bother, at least some of the time.

Here’s other things I’ve written along this vein:

Read around the internet, viewed as permission/empathy:

Updates to this post

2022-06-15

I periodically re-visit and update things I’ve written in the past. This post is one of them. Sometimes I modify my writing substantially - it’s simply text files in a git repository, therefore all of what you see around here could be considered in some various form of draft.

I’ve considered adding links to every page on my website that say “click here to see prior versions of this file”. I perceive a feeling of some obligation to the reader to note every time there’s been an edit, or at least some of the time when I might edit things. But obviously fixing typos doesn’t need to be called out in detail.

Additionally, it’s sometimes fun to bury hot takes in walls of text. For example, does the following text seem sort of dry, and academic to you? It’s intentional, if it does:

In The Slaughter of Cities, the author tells a compelling tale of how one ethnic group sucessfully perpetuated ethnic cleansing upon other ethnic groups.

It’s passive voice, it’s just a link to a goodreads page for a book. It doesn’t sound as provocative as it otherwise could. So if it does/does not get the intended result, one could turn it up or down a bit. For example, how does this phrasing compare?

In The Slaughter of Cities: Urban Renewal as Ethnic Cleansing we read the many ways American urban renewal programs in the 1950s and 60’s were run to create ghettos and perpetuate acts of ethnic cleansing. Those people attacked essentially every non-white ethnic group they encountered, and the relevence of all this is very much in the room with us right now, in most ways that matter.’

I find writing to be trivially easy, I find writing well, however defined, to be hard. Does it spark joy? Does it record something in a way that allows someone to save time in the future? A friend recently sent me a whatsapp message that they were googling something and the top google result was an article I wrote, which indeed contained something useful to them Testing Rake Tasks in Rails. Lots around here is technical, I lean hard on ‘software norms’ for all that I do.

Jan 2024

Write Faster (sashachapin.substack.com)

Humans are naturally sense-making, story-telling creatures. We’re wired to explain ourselves and the universe, to ourselves and to each other. We’re naturally ingenious fabulists, shrewd reporters, idiosyncratic word-painters. Basically all of us have this facility.

But this ability disappears when it’s subjected to excessive scrutiny. When we start reflecting at length on what kind of story we should tell, or what kind of writer we should be, we become lost in the sea of data our senses provide us, and lose the brio of playfulness. It’s like how when you think about how you’re walking, you start walking stupid. You’ve just got to trust that you know what you’re doing: that, as a sense-making creature, you will largely make sense.

We can also be our own ‘excessive scrutinizers’. Something like this undoubtedly influences those who describe themselves as perfectionistic.

Something I love hanging out with kids is how easily they embody ‘uninhibited self-expression’. I find it healing and enthusiasm-engendering, because you can take delight in someone else taking delight and interest in their environment, experience, activity.

I’ve noticed, at times, when I simply would imagine an act of spontaneous self-expression, I would pre-emptively shut it down by imagining either vocalizable reactionary criticism or a non-response, a shun, withdrawal. The topics would span from technical (feeling embarrassed about asking something, admitting that I don’t know/couldn’t figure out something) or casual/mundane. (a comment about something in the environment, etc)

So, some of the ‘write it now’ energy I propose you/me capture is something that moves towards a healthy form of ‘uninhibited self-expression’.

Ironically, I can ‘hear’ a criticsm in my mind:

But doesn’t some of the worst parts of the internet seem to typify ‘uninhibited self-expression’?

I think of a quote I’ve heard on the expression of anger. It can be paraphrased as:

Anger is fully ‘valid’, to feel, full stop. Suppressing it, or expecting others to suppress it, is foolish. It’s as natural and healthy as sadness, or joy, or any of the other many things that we feel as people. ‘anger’ becomes a problem if it’s expressed abusively or harmfully towards others, and even in this case the issue isn’t the anger, it’s the abusive expression of it.

Someone who hits when angry doesn’t have an anger problem, they have a hitting/violence problem.

So, if one person is being cruel to someone else, via ‘uninhibited self-expression’, the problem is not the uninhibited self-expression, it’s the cruelty.

Here’s the last thing - I think good uninhibited self-expression is best experienced when you know whoever you’re expressing to feels similarly about such things. Someone needs to point towards you something that feels like a steady-enough message of ‘you are fundamentally acceptable to me, as are the ways you move through the world’. This feels like a high bar, but I don’t really think it is that hard to achieve.

Links and additional reading

There were some very lovely comments in the hackernews thread. Here’s a portion of this comment:

It’s hard because it probably won’t change the world, but it becomes much harder when shouting.

It’s entirely OK to sit and write without fanfare, we’re not wizards or witches any longer, just plain people with the same plain obligation towards the mind as towards the body: tend, feed, and regularly empty out.

If today you really gotta go write, go ahead, knock yourself out.

If tomorrow you just scribble in the margins of empty pages but nothing comes, have a bath or a nap or both, and sit again the day after.

I also find it very helpful to remember that I’m not addressing others when writing, I’m not in a possibly empty auditorium talking into the void, instead I’m reaching across time and continue a discussion I started with myself earlier, and may pick up again later.

↩

Auto-Updating Docker Containers

Random Bits — Tue, 05 Oct 2021 13:07:34 GMT

Docker is the biggest advance to production software engineering in the past decade. If you're like me though, you slept on it this entire time and are now feeling like you're too far behind to catch up and finally start using them. I'm going

Back to the (Phone) Future! Changing roles!

Greg Herlein — Tue, 05 Oct 2021 10:00:00 GMT

The only constant is change! And… sometimes things come full circle. I’m changing roles at AWS. Back to the Future!

I stored a boolean instead of a timestamp

River Writes - A MediaWiki Blog — Tue, 05 Oct 2021 04:16:32 GMT

A Cargo field I was going to store as a timestamp but decided to store as a boolean instead

Book review - Celebration, U.S.A.

River Writes - A MediaWiki Blog — Tue, 05 Oct 2021 03:49:11 GMT

A review of “Celebration, U.S.A.: Living in Disney’s Brave New Town”

What I've Been Up To Lately

Ben Bolte — Tue, 05 Oct 2021 03:00:00 GMT

Short post about what I've been up to lately.

Resiliency and new domains - 123dev #30

Justin Garrison's Homepage — Tue, 05 Oct 2021 03:00:00 GMT

Comments

Resiliency

Code and systems break all the time. At a certain scale it’s important that your start building software that can heal itself. It’s sometimes confusing what that means because what state should the software return to?

While it was bent out of shape the paper clip wasn’t broken, but it wasn’t useful as it was intended to be. Software is similar in that it may still be functioning but not usable for its intended purpose.

Pandemics, kicking cars out of the stone age

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 05 Oct 2021 02:40:00 GMT

The semiconductor stone age, that is.

Eating the Cloud from Outside In

swyx's site RSS Feed — Mon, 04 Oct 2021 05:23:15 GMT

AWS is playing Chess. Cloudflare is playing Go.

What it’s like to ship products - 123dev #29

Justin Garrison's Homepage — Mon, 04 Oct 2021 03:00:00 GMT

Comments

An industry secret

For all the automation the technology industry prides itself in there’s still a lot of it that’s manual. Some companies get up on stage and talk about how great their automatic canary deployments are, how they use feature flags, or a variety of different methods to get their products to market.

I’ve been a part of some large product launches and plenty of incremental changes in various environments. As soon as the product launch requires coordinating with a non-development team (e.g. blog post or press release) they all had some form of a slack channel, email list, or phone bridge where someone was in charge of manually going down a spreadsheet and clicking buttons.

Simple Lists: a tiny to-do list app written the old-school way (server-side Go, no JS)

Ben Hoyt's technical writing — Sun, 03 Oct 2021 21:30:00 GMT

Simple Lists is a tiny to-do list web application written in Go, with old school server-side rendering and no JavaScript.

CTF Writeup - TSG CTF 2021 - 2 Reversing Tasks

onKeyPress — Sun, 03 Oct 2021 13:26:35 GMT

ThinkPad P14s AMD First Impressions

Liza Shulyayeva — Sun, 03 Oct 2021 12:20:00 GMT

I was fully set on getting a ThinkPad T14 AMD, but seeing no hint of the Gen 2 AMD version in sight and learning that the P14s is basically the same thing, I decided to go for that instead. This is a quick ramble with some first impressions of the ThinkPad P14s AMD Gen 2 (Ryzen 5)

VLC Error

Greg Herlein — Sun, 03 Oct 2021 10:00:00 GMT

A bug in VLC? Grrrr. Reminds me of the old days.

Working faster not harder - 123dev #28

Justin Garrison's Homepage — Sun, 03 Oct 2021 03:00:00 GMT

Comments

Code generators

Lots of people recommend learning to program by starting with fundamentals. Learn the basic language patterns and elements and work your way up to frameworks. You’ll have a stronger understanding of how things work.

I don’t think this is good advice anymore. By doing this you slow down your speed to market and what I’ve seen is more companies need speed than understanding. If I want to create a website I’m not going to start with HTML and CSS. I’m going to use a generator like React, Vue, or Hugo.

An overview of cloud buzz words

Jason Eckert's Website and Blog — Sun, 03 Oct 2021 03:00:00 GMT

As this Dilbert comic eloquently illustrates, there’s no shortage of cloud buzz words thrown around today. But as strange and complex as they may sound (even to seasoned developers and system administrators), they are not difficult to understand. In this blog post, I’ll walk through the different components and features of the cloud, explaining the associated buzz words along the way.

What is the cloud?

Back in the early 1990s, when the US government sold off parts of their ARPAnet and NSFnet to private companies (later called Internet Service Providers, or ISPs) to create the public Internet, there wasn’t anything exciting about the Internet at all. After all, the Internet was just a physical computer network that allowed your computer to talk to other computers, usually through a telephone network via a modem.

Do Not Erase

ℤ→ℤ — Sun, 03 Oct 2021 01:36:03 GMT

Do Not Erase: Mathematicians and their Chalkboards by Jessica Wynne is a beautiful photography book. Each leaf contains a photograph of a mathematician’s chalkboard and a short essay by that mathematician talking about their work, what is displayed, or something personal. Wynne’s work only looks at blackboards or chalkboards, not whiteboards. The entries vary from the prosaic, such as diagrams left over from an office hours session on elementary calculus, to snapshots of current work, to results that were a life’s work.

One of the recurring themes in the mathematician’s essays is the superiority of chalkboard over a digital medium like PowerPoint. Many of the essays leave their author’s rationale unexplored, but the reasons articulated include:

The act of writing slows down the speaker, providing more time for the audience to understand the content (and, in a class, take notes)
The speaker is able to use their full body to communicate (versus a voice and a static image)
High-quality chalk provides better legibility than a marker and skilled users can create subtle artwork, but chalk isn’t so versatile as to encourage adding unnecessary and distracting details like digital works

For why a chalkboard is superior to a whiteboard:

Whiteboard markers smell
Whiteboard markers are always running out
Markers aren’t as legible and flexible as chalk

(Some of the authors do note that markers have the advantage of leaving less dust about, as in the case where the mathematician moved a chalkboard into their bedroom in case there was a late night burst of insight.)

As the production of the book extended over the pandemic, many of the mathematicians express sorrow over their inability to use their boards for their work. Although most of the mathematicians in the book are in fields sufficiently esoteric that book readers will not grasp what is being shown on the board, we can appreciate the medium and skill that permits the transmission of deep concepts in such a succinct manner and wish we could hear and see the mathematician explaining their work in concert with the pictures.

HTTPS and other hosting details

Sat, 02 Oct 2021 21:16:52 GMT

Last week, I got my blog up quickly with a simple upload to Google Cloud storage, with public access. However, this has a huge limitation – no HTTPS. Google’s proscribed message to enable it? Pay them for TLS terminating load balancers. I have no desire to setup a money-sucking machine in case one of these blogposts go viral one day (unlikely), so that was out.

Then I read a recent post about a new product released by Cloudflare. I looked on their site and discovered they have a free plan. It’s possible they’ve always had a free plan, and I just wasn’t aware of it, but it looks like I get free DNS hosting and CDN proxying for up to 3 domains, including SSL termination and certificate management. This seemed pretty great, so I went ahead and signed up and migrated over. I’m still using Google Cloud storage as the place where the site is actually hosted, but it’s all Cloudflare fronted.

Sublime Text as EDITOR

voussoir.net/writing — Sat, 02 Oct 2021 21:09:51 GMT

Back to writing

Sublime Text as EDITOR

I use Sublime Text as my EDITOR environment variable and as my git editor. I used ST2 for many years and recently jumped to ST4, skipping 3. But something has changed along the way, or maybe it was always a little imperfect.

Between Sublime's workspaces and project sessions and hot-exit files that allow you to close with unsaved changes, there is quite a lot of state saved to disk. I found myself getting frustrated at the various incantations of sublime_text.exe -n, sublime_text.exe -n -w, subl.exe -n -w because they recalled the previous state and opened a bunch of tabs or windows when I just wanted to do a single git rebase.

Here is a forum thread where other people have the same problem, and they mostly discuss workarounds through config files. I didn't want to make any of those changes because I like having all my tabs restored in 99% of cases, just not for temporary EDITOR purposes.

Then I realized something that I should have realized a long time ago because I'm the one who's always going on about it: Sublime is portable software. I can just make a copy of the folder. And I did. And it's great.

Now, I have C:\Software\Sublime Text\4133 and C:\Software\Sublime Text\4133_EDITOR, and I can configure them independently. For the EDITOR copy, I can remove all project folders, disable a lot of UI elements, turn off state-saving, and uninstall unnecessary packages.

My EDITOR environment variable is C:/software/sublime text/4113_EDITOR/subl.exe -n -w.

My .gitconfig contains:

[core]
    editor = 'C:/software/sublime text/4113_EDITOR/subl.exe' -n -w

My WinSCP editor is C:/software/sublime text/4113_EDITOR/sublime_text.exe -n !.!.

Use portable software, folks.

P.S. I found that the -w flag only waits if you also provide a filename to edit on the command line. subl -w does not block, but subl -w myfile.txt does block. This made me confused while I was testing everything.

View this document's history

2021-10-02 Add sublime_as_editor.md.

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Collateralizing Mortgages and Loans With the Present Value of Rent Flow

Josh Thompson — Sat, 02 Oct 2021 16:00:00 GMT

this is a draft document, it pairs with this Planned Unit Development application draft document

Inspiration comes from many places, but most strongly it draws heavily from Order Without Design. I’ve quoted in depth two pages below, but there is many other sections of the book germane to the topic of this page.

Summary of how a financial institution could use rent-generating property to collateralize a loan

Assess the value of a land parcel by evaluating the present value of the flow of rents generated by what ever structure they can build on this parcel.¹

A property could plausibly generate $6,000/mo in rent? (say, with six small units at $1000/mo in rent?)

That $6,000/mo or $72,000/yr could easily collateralize a $600,000 loan/mortgage/financial product especially if there is reason to think even more rent could be generated from the structures.

At this point, collateralization discussions would require a plot-by-plot evaluation of the likelihood a given structure on a given plot could generate such income streams.

A financial product “crossing the distance” between the net present value of cash, and guaranteed future income flows

The FIRE community (r/fire) has the 4% rule, which can be expressed a few different ways, but broadly it’s the expression of a relationship between the value of a given lump of cash on hand, and future risk-adjusted income flows that said lump of cash can generate.

Because of math, a guaranteed $1000/mo income is “as valuable as” a $12k/yr income, which is 4% of $300,000.

So, if I wanted to borrow $100k to do something that would have a strong probability of being capable of generating $2000/mo in rent, that is a “safe” mortgage, because even if I’m way off on the amount of rent I generate, I still will have something that is worth far more than was spent creating it. It’s an asset.

This notion all comes from the text expressed below, and other text in the same book, and other books. More on it soon.

Example 4: China’s Urban Villages (p. 293)

The following comes from page 293 of Alain Bertraud’s Order Without Design: How Markets Shape Cities Lightly formatted for readability.²

Legal Status of Land in Chinese Villages

As in Indonesia, many cities in China are absorbing large numbers of villages as they expand. I will use the term “urban villages” to indicate the villages surrounded by an urban area administered by a municipality.

The land occupied by these villages has a special status by law. The use of village land is controlled by a village collective, not by the municipality, although according to the constitution of China, all land belongs “to the people” (i.e., the central government).

But there is a difference between full ownership, which involves the ability to sell property, and ownership of land use rights only, which is limited to the right to develop land and to rent floor space or land to a third party.

Villagers in China are free to set their own building standards and land use; they can rent land and whatever structure they build to a third party, but they cannot sell either the structure or the land.

Only the government can acquire land from farmers by expropriation with compensation.

For Chinese farmers living in urban villages, therefore, the value of their land parcel is represented by the present value of the flow of rents generated by what ever structure they can build on this parcel.¹

Contrary to farmers living in traditional market economies, they do not have any incentive to sell their land to a developer, as the compensation price they would receive from the government is likely to be less than the capitalized value of the flow of rents. This explains why villagers in China resist selling land to the government whenever they can, and many protest forcible acquisition.³

When the built-up area of an expanding city reaches a village, the municipal government expropriates the fields around it but usually abstains from expropriating the village itself, as the compensation paid to villagers is based on a “replacement value” of the floor space demolished, while the fields are compensated based on the value of crops.

Expropriating the village and providing alternative housing units is therefore much more expensive for government than expropriating open fields.

As a result, villages are often initially spared demolition and become an enclaved urban village with a special status over the control of land use.

As a city expands, the land value increases in the villages that were initially at the fringe of urbanization, to the point of becoming higher than the compensation to be paid to farmers for whatever structures they have built.

The municipal government then tends to evict farmers after compensation or relocation and sell the land to developers. However, the process is long and cumbersome, and many urban villages survive a long time before redevelopment occurs.

Frequently Asked Questions

Are you proposing a new “financial product”?

More or less, yes. Eventually something like this could be construed as an alternative to a 30-year fixed-rate mortgage.

The reasons could be stated by meandering through the current state of affairs, the historical contexts and structures that made the current state of affairs most likely, and a scintillating discussion of who’s visions for the future show the most merit.

For example, I just read https://www.netinterest.co/p/financing-the-american-home. The way things are right now is strange, to say the least, and pressure is building for a good alternative that isn’t “just” a 1:1 replacement.⁴

Something much bigger needs to give, but this is a big piece of it.

I could speak at length of the intersection of Money, Bank Credit, and Economic Cycles and lending institutions.

How would you propose ‘getting the most out of the land’?

Great question! By applying established processes to your specific bit of land.

I propose:

The Bertraud Model

Notes for myself, that I bet others will find interesting.

I have a paper, crafted a few years ago, by Alain and Marie-Agnes Bertaud.

I'll be adding many more notes in this thread, but for starters, here's the title and abstract. pic.twitter.com/FKnTvYiHtW
— Josh Thompson (@josh_works) August 14, 2021

I need to write up an article about that, but until then, the twitter thread will have to do.

Skip straight to this page:

https://twitter.com/josh_works/status/1426952509088845830/photo/1

And this flow of rents represents an equivalent present cash value. At least one interpretation is the 4% rule in the FIRE community - “you can take, annually, 4% of your current nest egg for the rest of your life. If you’re able to live on that 4% for a year, you can quit your job tomorrow.” $2.6 million “generates” $100k/yr. A stable $1000/mo is “as valuable as” a $312,000 cash account. ↩ ↩²
AKA I shotgunned line-breaks through the document because half of us are going to read this on our phones at least twice. ↩
‘having no incentive to sell to a developer’ is why everyone interested in historic preservation will be a massive supporter of this model. This model preserves the city. ↩
To quote from that article:

This reluctance of private capital to backstop the mortgage market would recur again. It stems partly from the risk involved (30 years is a long time) and partly because having established a foothold in the market, the government’s presence makes it difficult for private capital to compete. The same issues give the the government such a dominant share of the market today. (emphasis mine) ↩

COVID-19, a lost opportunity for italian administrations

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 02 Oct 2021 13:00:00 GMT

Like tears in the rain, really.

Documentation for Ada on the Raspberry Pi Pico

Jeremy Grosser — Sat, 02 Oct 2021 10:00:00 GMT

I’ve written some documentation for Ada on the Raspberry Pi Pico. This is still a work in progress, but provides quite a lot of context around the usage of the libraries and how to build a project on this platform.

The Husband Stitch

Arun Mani — Sat, 02 Oct 2021 03:00:00 GMT

A scary, interesting short story.

The Little Prince

Arun Mani — Sat, 02 Oct 2021 03:00:00 GMT

A refreshing story that was fun to read.

Experimenting with Personalized PageRank

Weblog on marginalia.nu — Sat, 02 Oct 2021 03:00:00 GMT

The last few days I’ve felt like my first attempt at a ranking algorithm for the search engine was pretty good, like it was producing some pretty interesting results. It felt close to what I wanted to accomplish.

The first ranking algorithm was a simple link-counting algorithm that did some weighting to promote pages that look in a certain fashion. It did seem to keep the page quality up, but also seemed to as a strange side-effect promote very “1996”-looking websites. This isn’t quite what I wanted to accomplish, I wanted to promote new sites as well as long as they were rich in content.

Be kind and AWS Infinidash 🌩️ - 123dev #27

Justin Garrison's Homepage — Sat, 02 Oct 2021 03:00:00 GMT

Comments

Silent stories

The ability to educate with stories is something humans are very good at. Videos are just a new(ish) medium for telling stories. One of the reasons I love gifs is they’re videos with restrictions. The restrictions limit the types of stories we can tell but not the impact those stories can have to educate.

Why is mobile game development big in Turkey?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 02 Oct 2021 02:00:00 GMT

All jobs are glamourous, until they are not.

Blog

Home on mfashby.net — Fri, 01 Oct 2021 21:34:15 GMT

Planned Unit Design Document (work-in-progress)

Josh Thompson — Fri, 01 Oct 2021 16:00:00 GMT

This is a draft document, meant for circulation, will evolve with time and eventually be something we bring to the City of Golden for ratification, or whatever needs to happen to get this done in this zone. This document relates to Collateralizing Mortgages and Loans With the Present Value of Rent Flow

I, Josh Thompson, owner of 621 Iowa St, with my wife and the owners and rentors of the below properties, hereby request a Planned Unit Designation from the City of Golden, in accordence to the following terms.

Dimensions/Boundaries of the PUD

Addresses included in the PUD, representing a contiguous territory and boundary.

648 Iowa St.
621 Iowa St.
623 Iowa St.
619 Iowa St.
617 Iowa St.

Here’s the specific properties, all owners and tenants of which desire to be included in the PUD:

Property Owners of the lots inside of and around the PUD

For the rest of the document, the collection of properties will be “The PUD boundary zone” or some otherwise collected group.

The owners of these included lots desire for their land to be included in this PUD, and will sign contracts or testaments to this fact if it would be helpful for obtaining this PUD.

All property owners and tenants are strongly invested in seeing this general plan be accomplished.

Zone 1: Immediately-Impacted Property Owners (Adjacent)

Nearly everyone adjacent to the designated area, and 100% of the people I’ve spoken to about this, approve of this PUD and would not take actions to prevent it from being designated as such.

Zone 2: Impacted Property Owners/Residents (nearby)

Obviously, PUDs affect far more than just the immediately-adjacent properties. If we were to take an expansionist view of affected parties, we’d need to fill in the corners of the “space”, so it would be inclusive of everyone who lived within a certain distance of a boundary of the PUD. Here is that expansionist view:

Zone 3: Affected by usage patterns

Finally, since roads are mobility networks, and neighborhoods are a common unit of analysis, not just individual lots, here’s the neighborhood inclusivity of persons who would have some vested interest in this project:

PUD Code Provisions

This PUD “application” will contain several sections, modeled roughly after the other PUD applications I could find on the City of Golden’s website:

The role of the city

The role of the City and of its Agents is limited to providing connections to the trunk city networks for transport, water, sewer, and storm drainage.

City easements

Existing city infrastructure within the boundaries of the PUD will be maintained, broadly, by a coalition between the PUD and the city. Easements currently owned by the city that extend beyond the physical boundaries of the road, sewer, and water network will be deemed unexercisable, and full possession will be remitted to the property owners.

Rights of way

The owners representing the PUD will maintain the existing roads to ensure safe and adequate mobility network through-put.

Non-vehicular rights-of-way will be decided upon and maintained by the property owners.

Summarized language, initial draft:

All language copy-pasted from Alain Bertraud, Order Without Design, pasted below.

In specific, I want to model this PUD on the Indonesian Kampung. I’ll copy-paste from the long quote, translating the described phenomina to a set of zoning rules legible to a local municipality.

As I have discussed this with neighbors (in all zones), and explained the genesis of where this language comes from, they have all eagerly endorsed the plan. To emphasize, this general approach and language has broad community support.

Below are sentences, as encountered in order, in their original form from the long quote shared below.

In these two countries, the government allowed low-income households to use the standards they could afford and to make their own trade-offs between location, road space, lot space, and floor space. The government role was limited to providing connections from these settlements to the trunk city networks for transport, water, sewer, and storm drainage.

Lots were subdivided, but always on the initiative of the original owner

adjustments can be made to plot size, width of access to individual lots, and drainage of waste water

standards were adjusted to increasing land values

A network of small roads, footpaths, and passages was maintained, reflecting the former village structure.

while the government concentrates its efforts not on housing construction but on gradually improving residential infrastructure and services to all residential settlements

Long quote from Order Without Design: How Markets Shape Cities, p. 287

[…]

Finally, housing supply could be greatly increased by improving the urban transport system. Areas with low demand because of poor accessibility to jobs have necessarily a low density. Improving accessibility to jobs by increasing trans port speed or introducing new transport technology should increase the de facto land supply of cities. Unless the regulatory and infrastructure housing supply constraints are removed, adding new subsidies to housing, whether supply side or demand side, would have very little effect on the welfare of the city’s lowest income households.

The above example of inclusionary zoning is not just an anecdotal illustration of a poorly designed housing policy. It represents a trend in many of the most economically successful cities in the world. The increasing regulatory frenzy that characterizes some cities like New York imposes large economic costs on the entire country. In a paper published in 2015, the economists Chang-Tai Hsieh and Enrico Moretti found that, between 1964 and 2009, the high cost of housing in some US cities relative to wages had lowered aggregate US GDP by 13.5 percent:

Most of the loss was likely caused by increased constraints to housing supply in high productivity cities like New York, San Francisco and San Jose. Lowering regulatory constraints in these cities to the level of the median city would expand their work force and increase U.S. GDP by 9.5%

Many of the regulations recently introduced in cities around the world that aim to improve the housing consumption of low-income households, not only do not deliver the number of housing units promised, but also contribute to lowering the economic opportunity of the very people they are supposed to help. It is time to audit these regulations and policies, taking into account the increasingly valuable economic literature on the subject.

Cities Are Reluctant to Accept Housing Standards That Are Affordable for the Poor: The Exceptions

Cities in low-income countries, where urban migration annually adds a large number of unskilled people, cannot afford to subsidize the housing of the large number of poor. However, cities in Indonesia, using an “integration” approach to low income settlements, have been successful in absorbing new migrants into the urban labor force while maintaining an acceptable level of environmental health in the poorest neighborhoods.

In many Asian countries, large numbers of people migrating to cities have been creating dense urban villages in an urban setting but without any urban infrastructure. These small, simple structures provided the shelter these people needed at a price they could afford. It also allowed them to participate in the urban economy. However, the lack of connection to the city infrastructure soon created unacceptable sanitary conditions when these urban villages aggregated in large contiguous neighborhoods of several hundred thousand people. In addition, the lack of schools and health facilities contributed to slowing down or even prevent ing the integration of younger generations into urban society.

The predictable first reaction of governments has usually been to set minimum urbanization standards to prevent the legal construction of these unsanitary urban villages. The regulations made the situation worse, as they prevented these informal settlements from obtaining normal urban services from the municipality. They also created a risk of future demolition, which discouraged housing improvement that the households would have naturally done themselves. Eventually, many governments slowly regularized the older informal settlements in a piecemeal fashion, as is the practice in India, for instance. But the regularization of informal settlements usually had been conducted with a provision that after a set date, no more informal settlements would be regularized.

The outcome of these successive policies – first ostracism, then benign neglect followed by reluctant integration – has been disastrous. A significant share of the urban labor force, otherwise gainfully employed, live in large “informal” settlements often with unsafe water supplies, deficient sanitation, and sporadic solid waste collection. In Mumbai, one of the economic powerhouses of Asia, 60 percent of the population lives in slums.

I discuss below two case studies in Indonesia and China. **These countries, for different reasons and in different ways, took a different approach to integrating affordable low housing standards into the city infrastructure. The outcome for the poor migrant households was significantly better than in the countries where a strict regulatory approach had been followed. In these two countries, the government allowed low-income households to use the standards they could afford and to make their own trade-offs between location, road space, lot space, and floor space. The government role was limited to providing connections from these settlements to the trunk city networks for transport, water, sewer, and storm drainage.

Example 3: Indonesia’s Urban Enclaves with No Minimum Standards

In the densely populated island of Java, expanding cities absorb large existing villages, called “kampungs,” whose population then integrates rapidly into the urban labor force. The former agricultural fields are soon built on by formal developers, while new informal constructions are added in the kampungs that have been absorbed into the city. So far, this is not too different from what happens in the rest of Asia, where countries are urbanizing rapidly. There is one important difference. Indonesian kampungs always had a formal robust administrative structure based on traditional laws, and that administrative structure has been able to survive their absorption into a larger urban municipality.

Confronted with urbanization, the kampung traditional leadership organized themselves to absorb newcomers. Lots were subdivided, but always on the initiative of the original owner, and within the kampung traditional regulatory constraints and social norms. To use modern terminology, we could compare kampungs to a form of condominium association: They have their own internal regulations and norms, forming a sort of local authority at the lower level than a municipality or a ward.

One difference from a condominium is that the kampung local microgovernment also has jurisdiction over land use. Therefore, adjustments can be made to plot size, width of access to individual lots, and drainage of waste water by using traditional norms developed over centuries of practice of good neighbor rules while overruling the development standards imposed by the municipality that surrounds the kampung. Formal developers building on adjacent greenfield land, however, had to follow the municipal land use standards rules concerning land development and subdivisions.

Because the income and preferences of new settlers were well known to the original kampung inhabitants, land use standards evolved to adjust to the new economic reality facing kampungs; standards were adjusted to increasing land values, so that plots would remain affordable to newcomers, who were usually poor migrants. Because the local traditional norms were respected, it did not result in an anarchic aggregation of houses that would be wasteful of land, as often happens in spontaneous squatter settlements where migrants aggregate without forming an organized community. A network of small roads, footpaths, and passages was maintained, reflecting the former village structure.

As Indonesian cities developed, the kampungs located closer to city centers densified more rapidly, as expected by their favorable locations closer to employment. The low income of their inhabitants, the lack of access to finance, the frag mentation of properties into small lots, and the lack of access to infrastructure prevented the construction of multistory buildings. However, the subdivision of existing village lots and the narrowness of internal access streets reflected the opportunity cost of land and soon resulted in much higher densities than that of the original villages, often resulting in densities of more than 500 people per hectare.

However, this increased density has a downside. The traditional source of water and means of sanitation – consisting of shallow wells and seepage pits – became grossly inadequate to serve the new higher densities. The low absorption capacity of the traditional sanitary system transformed the kampungs into dense unsanitary slums. The traditional storm drainage network, built around former irrigation canals, was insufficient to prevent flooding during the monsoon because of enhanced impermeability of the area caused by urbanization.

Innovative Housing Policy Concentrating Subsidies on Infrastructure, Not on Housing Structure

So far the history of the Indonesian kampungs does not appear to be very different from that of informal settlements in many other developing countries. What made a difference was a decision taken in 1969 by the government of Indonesia to concentrate its resources on the improvement of the kampungs’ infrastructure without trying to remove or restructure the existing housing, however small or inadequate it was. The provision of urban infrastructure and services to kampungs was called the Kampung Improvement Program (KIP). Compared to the prevailing housing policies of developing countries in the 1960s, which consisted mostly of bulldozing informal settlements to relocate their inhabitants in public housing flats, this approach was revolutionary.

And, even more exceptional, since 1969 to this day, the Indonesian government’s support for KIP has been unwavering, in spite of political upheaval and constitutional changes. The government housing policy objective consists of allowing the poor to settle in and around existing villages at the standards of their choice, while the government concentrates its efforts not on housing construction but on gradually improving residential infrastructure and services to all residential settlements. The policy has proved largely successful. The living and sanitary standards in most kampungs of Indonesia are far above what is found in informal settlements in countries with similar GDPs.

No kampung was ever bulldozed. No large groups of households were promised free housing. No large central government institution has tried to replace the many small KIP contracts for civil work on infrastructure in kampungs with larger contracts involving the construction of a massive program of public housing.

Operationally, the KIP continues to provide financial and technical assistance to the existing kampungs’ traditional administrative structure to build connections to the municipal water supply network, to pave foot paths, to build drains using existing rights-of-way, to establish a system of solid waste collection that feeds into main collection bins that are part of the municipal solid waste collection system. The maintenance of the internal network and waste collection was and to this day is managed by the kampung community itself, with some financial assistance from the municipal government. The decentralization of decision making and the participation of the communities were embedded in the KIP from the beginning.

The KIP has been complemented with citywide investments in transport and most importantly for Southeast Asia, in storm drainage networks to prevent periodic flooding in residential areas

Indonesia, with a GDP per capita of US$10,500 in 2015, remains a lower-middle-income country, according to the World Bank. The overall standards of living reflect this income category. However, because the government concentrated its scarce resources on providing urban infrastructure to all urban residents, instead of increasing the housing consumption of a few poor households selected by lottery for public housing, every poor Indonesian received benefits. And these benefits increase over time. The government focus on infrastructure assistance allowed households to use their own resources to invest in their own housing, either as self-occupied dwellings or as investment in rental housing. The housing standards, defined as the size and quality of the structure, may therefore vary from very low to good in the same neighborhood, but access to safe water, sanitation, education, and health is ensured for all.

The demand-driven land use standards allowed by regulations in Indonesia are illustrated by the site plans of two neighborhoods in Surabaya, Indonesia, as surveyed in 2010 (figure 6.24). On the left in the figure, a kampung site plan shows the variety of plot and housing sizes that are possible in the same neighborhood. Some structures located in the back of lots facing the main passageways are very small and of poor quality, but they have access to safe water and sanitation. They have also access to the same schools and health facilities as their more affluent neighbors.

TODO: ADD PICTURES FROM BOOK

The road system allows only emergency and construction vehicles but is adequate for local motorcycle and pedestrian traffic. On the right in the figure, a site plan designed by a formal developer, which shows a different, more standard type of residential layout with much less variety in lot size and more house area, reflecting middle-class standards. homogeneity in house area, reflecting middle-class standards.

Kampung Integration Policy Has Successfully Provided a Flow of Low-Income Housing

Because the kampung boundaries have always been protected since Indonesia started its rapid urbanization, kampungs are found in every neighborhood of Indonesian cities. Consequently, kampungs are always located side by side with commercial areas and higher-income areas, providing a socially desirable income mix at the neighborhood level. Because kampungs and middle-class areas share the same trunk infrastructure for water and sanitation, little discrimination is possible in public investment.

I have observed kampungs’ evolution and improvement on a regular basis, since my first work trip to Indonesia in 1977. Many housing units in kampung areas look more middle class than low income. This reflects the increase in house hold incomes in Indonesia over this period. The inability to use a car as a means of transport in kampungs (because of the narrowness of the streets) prevents any large displacement of the poor from the best-located kampungs. The low road standards are the best guarantee against massive gentrification.

Some households that have reached sufficiently high incomes are likely to leave the kampung and move into formal development neighborhoods. This movement creates vacancies in the kampungs that are immediately filled with holds who rent or buy a dwelling. Low-income households do not need to be on poorer a waiting list and do not have to submit a proof of their income to buy or rent in house a kampung. Expansion zoning plans are reserving land around existing kampungs located in the periphery to ensure a flow of new low-income housing in the future.

The Indonesia kampungs and the KIP program demonstrate that it is possible for a government to provide large benefits to its large low-income population without creating long waiting lists. It shows that allowing standards to adjust and, consequently, population densities to float, not only provides affordable housing but also improves the city structure by allowing higher densities in areas accessible to jobs.

Alain Bertraud’s specific feedback on this document

I (Josh) am friends with Alain and Marie-Agnes Bertraud, the authors of Order Without Design: How Markets Shape Cities.

When I sent him a draft of this document, Alain said:

Your proposed PUD is an imaginative adaptation to a Colorado subdivision of the Kampung and Chinese urban village concept described in my book.

I hope that it gets approved under Colorado law!

I hope that other people will also imitate the idea with some variations, ensuring constant adaptation to changing socio-economic conditions.

This new way of changing land use initiated by current occupiers would be a wonderful escape from the arbitrary rigidity of existing zoning laws, which prevent demand-driven land use adaption.

Your preliminary financial calculation also demonstrates that it is possible to have an excellent financial return by unit of land while creating affordable housing units at standards that are acceptable to the end-user.

emphasis mine

FAQ

This document prompts many questions. If you have any, consider leaving a comment below, or shoot me an email about it, or text me, or whatever. I’ll update it accordingly.

What exactly is a PUD again?

TL;DR: A “Planned Unit Development District” is a legal document, enacted via a political process¹ by which the City approves of modifications to the existing rules and regulations, as applied to a specific piece of land.

The Planning and Zoning Office of Golden defines a PUD as:

(1) The PUD district permits the community and economic development of land which is suitable in location and character for the uses proposed as unified and integrated developments in accordance with detailed development plans. The PUD district also provides for specific land uses not permitted in other zone districts. The PUD district is intended to provide a means of accomplishing the following objectives:

(a) To provide for development concepts not otherwise permitted within standard zone districts.

(b) To provide flexibility, unity and diversity in land community and economic development, resulting in convenient and harmonious groupings of uses, structures and common facilities; varied type design and layout of housing and other buildings; and appropriate relationships of open spaces to intended uses and structures.

(c) To provide for the public health, safety, integrity and general welfare, and otherwise achieve the purposes as provided for within the Planned Unit Development Act of 1972, Title 24, Article 67, Colorado Revised Statutes 1973, as amended.

(d) To provide for phased development, the more efficient use of land and the public and private services needed therefor, and reflect changes in the technology of land development.

(e) To provide for land development flexibility in dwelling type, bulk, density, intensity and open space, as a policy which Golden wishes to encourage; and, as a corollary, regulate certain proposed land development which would distort the objective of Golden’s zoning.

(f) To encourage integrated planning in order to achieve the above purposes.

(2) Uses within a PUD development may be multiple in nature and may include uses not otherwise permitted within the same zone district. The location and relationship of these uses shall be as established in and conform to the policies and standards contained within the comprehensive plan and other appropriate adopted and approved plans, including but not limited to locational criteria within that comprehensive plan.

So why do you want to get this collection of lots declared a PUD?

Look at reason 18.28.300.1.a:

The PUD district is intended to provide a means of accomplishing the following objectives:

To provide for development concepts not otherwise permitted within standard zone districts.

I’ve got some development concepts not otherwise permitted within standard zone districts. Full stop. That’s why I want the PUD. I choose to not give up on what I want to accomplish because it’s not currently permitted on my lot.

For example - I’d love for there to be a vegetable stand that is sometimes operating on my street. I’d gladly build a space for someone, and let them operate it out of my lot. That structure, and that business, would be so illegal under current rules people laugh at me for even wanting to do this. I’m not afraid to look stupid, so here I am.

Who’s involved? Are you trying to get other people’s property rezoned out from under them?

Nope. Everyone involved is strongly supportive. That means, at minimum:

The owners of all involved lots
If there are tenants/renters, the renters are strongly supportive
Many, many neighbors adjacent to the PUD, along Iowa St, Ford St, Washington St, and more

Once I know what sort of formal attestations of support I need to be able to demonstrate, and who I need it from, I’ll obtain it.

If it’s people signing a document, they’ll sign a document.

If it’s people showing up at a hearing, they’ll show up at a hearing.

If, for political reasons, from the perspective of Zonning, Planning, or City Council, it is advantageous for more or less people/lots to be included within the zone of the PUD, I’ll adjust accordingly.

There are many, many good things that will come about for the residents within and adjacent to the PUD, when this goes through.

Why are you trying to get a PUD? Why not build to the existing code?

I desire flexibility more than almost anything else, so I (and my neighbors) can iterate and experiment with things, and unlock traditional financing opportunities based off our land.

We cannot finance development projects in a reasonable way, because we cannot use fractions of our land as collateral, which is a deviation from how land has traditionally been used.

I spoke with a land-use lawyer at Fox Rothschild about this, she agreed that a PUD made the most sense for the overall direction I’m trying to go.

Part of the problem with building to the current code is that the first step of every building project is to get permission for the project from the city, which requires that you submit a plan for the finished project.

I’m a software developer. Modern zoning forces all development projects into a “waterfall” project management strategy.² This PUD basically allows for a property owner to adapt an agile project management approach³.

What do you want to eventually build?

Critically, I am not quite sure. Once the City approves of the PUD, I’ll start building out projects. Nearly everything I’ll do, I plan on doing myself or with just a small group of people. I’ll GC my own projects, I’ll QA my own projects, and I’ll manage everything I build.

I want to aid the neighborhood in implementing best-practices from:

A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series)
The Timeless Way of Building
the vast “sustainable architecture” resources out there

Sequential approval starting with: planning department, then going to planning commission, and finally to city council ↩
AKA “the linear sequential lifecycle model”. The entire software development industry jokes about the insanity of having the finished product spec’d out, and then building that, regardless of what you learn/uncover/discover along the way. “Agile” is what everyone tries to do today, and I refuse to apply known-bad planning processes to my own property. ↩
“Agile” software development is… complicated. There’s a famous and rather nostalgia-inducing website that outlines the principles behind the agile manifesto. Check it out. ↩

Bike

Home on mfashby.net — Fri, 01 Oct 2021 14:27:26 GMT

Astrolabe - The October Update

Weblog on marginalia.nu — Fri, 01 Oct 2021 03:00:00 GMT

https://search.marginalia.nu

The October Update is live. It introduced drastically improved topic identification and an actual ranking algorithm; and the result is interesting to say the least. What’s striking is how much it’s beginning to feel like a search engine. When it fails to find stuff, you can kinda see how.

I’ve played with it for a while now and it does seem to produce relevant results for a lot of topics. A trade down in whimsical results but a big step up if you are looking for something specific, at least within the domain of topics where there are results to find.

Analyzing the Mario Themed Malware

SerHack - Security Researcher — Fri, 01 Oct 2021 03:00:00 GMT

If you were a child with a Nintendo DS like myself, growing up you will undoubtedly have played (for countless hours) video games from the Mario series. Appearing for the first time in Donkey Kong, Mario is the undisputed protagonist of the video game brand of the Nintendo development house ― from sports to cars, most of the famous video games produced by Nintendo have Mario as the main character. -Continue to read the article on SerHack.me

Every Seattle Store

Matt Keeter — Fri, 01 Oct 2021 03:00:00 GMT

Based on an actual vacation

Reliability is the most important feature - 123dev #26

Justin Garrison's Homepage — Fri, 01 Oct 2021 03:00:00 GMT

Comments

Simplicity vs User Experience

Why do fans have that weird nob to turn on oscillation but they have buttons for speed control? Why not put an oscillation button on the front? The nob is hard to reach and especially difficult while the fan is oscillating. It’s terrible UX.

It’s incredibly simple to implement which makes it extremely reliable. As a user of a fan’s interface I would prefer a button next to speed controls. As a builder, I rather not implement a secondary motor and deal with the guaranteed support and defective returns a more complex system would have.

Cyclic Workflows with Cylc and StackStorm

About on kinow — Fri, 01 Oct 2021 03:00:00 GMT

I am aware of only two workflow managers that support cyclic workflows. Cylc and StackStorm. I won’t enter into details about these two tools, but I must note that I worked on Cylc during my employment with NIWA, in New Zealand.

In this post I will only show a very simple workflow called five first using Cylc, and then the same workflow with StackStorm.

Cylc

First let’s take a look at the source code of this workflow with Cylc 8 and plot it.

Twitter, could you please fix your image servers?

utf9k — Fri, 01 Oct 2021 02:18:00 GMT

Throughout the year of 2021, I have run into a number of issues with Twitter's image servers

Startup financing - advices from a lawyer

Yasha Solutions — Fri, 01 Oct 2021 01:00:00 GMT

Startup financing from a lawyer

Undercover political campaigners are arrived

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 30 Sep 2021 21:00:00 GMT

But handling them will not be easy.

Explaining explaining: a quick guide on explanatory writing

Lucas F. Costa - Blog — Thu, 30 Sep 2021 15:00:00 GMT

This blog post is about how to explain things in writing. You should learn to explain things in writing because it's the most efficient way of conveying an idea to a large, distributed audience. Whe...

Fundamental interactions on teams

Writing on Adam Craven — Thu, 30 Sep 2021 13:53:24 GMT

The History of My Creativity System

trv.rs — Thu, 30 Sep 2021 10:35:00 GMT

While mapping my inputs and outputs… I realized the tools I used and goals I aimed for shaped my Creativity System over the years.

This section was originally part of an overly long post… but for those of you who interested in the path I have taken to my current strategy… here you go:

Inputs

💻 Links

I have spent my career working on websites or web applications. A lot of my personal and professional growth is through blog posts and essays… subscribed using RSS and read in a browser or often in Instapaper.

📗 Books

Recently I've been aping Bookworm and making mind maps as I read… turning the standout themes into standalone zettel notes. I have been exporting Kindle highlights/notes using Bookcision manually… but now Obsidian Kindle Highlights handles that for me. (More on all this soon!)

🗒 Meetings

While in meetings, listening to sermons, or even just conversations I'll often have Drafts or my Bullet Journal open, jotting down thoughts.

🚿 Shower Thoughts

Never underestimate shower thoughts. 🤷 More often than not… a new question or line of inquiry will suggest itself in the quiet nothingness of a walk, shower, or workout. If the notion has value… I want to be able to connect and use it later.

Many of my blog posts have started from such a note!

Outputs

🧠 Thinking

I want to invest in better thinking. Well made notes (especially linked!) have allowed me to have meaningful conversations with my own brain… finding new connections and revisiting old ones.

📖 Bible Study

I've always admired those whose Bibles are dog-eared and covered in marginalia. I've acquired half a dozen collections of notes and highlights across different apps and printed editions… I want to build something more permanent, portable, and searchable.

I'm about six months-ish into this process… and it's starting to bear fruit. When the pastor mentions a passage… I can find every note, highlight, journal entry, or booknote that references that passage. It's hugely powerful, and hopefully over time will reduce the "huh, where did I hear that before?" feeling.

✍️ Writing

I like to write blog posts. I have always toyed with the idea of writing a book someday? 🤷‍♂️

💼 UX Work

In my role as a User Experience Designer I deal with complex context: lost users, UX debt, oral histories, nuances of product surface, operations practices, and more. These product hypotheses and individual user insights don't belong to any single project… so hypertext notes are an awesome tool.

Constraints

🔐 Data

I need to own my own data and avoid vendor lock-in. I prefer a 0data style application like Obsidian, and an easily portable file format like markdown.

🔨 Tool Investment

I've invested a lot of time in Vim and Drafts.app. If you don't use these tools, some of my constraints may not matter, but it's worth noting.

History and Obstacles

Here's the tangled tale of my digital plaintext notes so far. I think it's most helpful to see the things that didn't work, worked for a while… how I got here.

1️⃣ Random plain text.

I started making simple Notepad ++/TextEdit notes at my first job and college.¹

2️⃣ Evernote

I flirted with this briefly… but I didn't want to be locked in a walled garden. (I was already investing in vim.)

3️⃣ nvAlt/SimpleNote

This was the answer for years. I am 95% sure that I owe the link to Merlin Mann on MacBreak Weekly. It simplified capture by combining search and new note creation… still one of my favorite applications. I believe an nvAlt release note about [[wikilinks]] introduced me to the idea of a Zettelkasten.

4️⃣ VimWiki

Back when I was programming for a living VimWiki was an amazing option. Note-taking was always available in the same application as my work. I wrote a (now archived) article on it because I couldn't find the info I wanted online.

5️⃣ Homebrew Zettelkasten in Vim

I became frustrated by vimwiki taking over my markdown syntax in vim, and it had weird options… it's gotten a lot better since. Part of the problem for me is that I was attempting to do both a productivity system and a creativity system. Vimwiki and plaintext was absolutely capable… I just didn't have the discipline and practices required to keep from feeling overwhelmed. After I had used a Bullet Journal for a while… I was better prepared to understand and work productivity systems… and start to understand the value of a Creativity System.

6️⃣ Obsidian using ZK ID

I started toying with Obsidian… and while reading about paper zettelkasten I bought into a (faulty) usage of the permanent ID. I wrote a transformation script to take all my old sources of plaintext notes and make a flat markdown folder with unique IDs.

7️⃣ Obsidian using Titled Notes

After discovering I was wrong… I arrived at the simpler current version. It's been stable and slowly growing for six months. Less tinkering, more thinking and using. 🧠

Themes

I doubt anyone is going to read this… but self-documentation is interesting.

Some random takeaways as I consider my particular tooling journey:

Plaintext is king. Easily trackable, diff-able, portable.
Consistent formats makes transformation by script easier.
I should have tracked my notes in git years ago. Having a consistent backup I understand allows me to fearlessly experiment.
Notes are more just remembering facts. I never had a vision for tying together all the areas of my life… I guess I just kept my work/personal/family life compartmentalized before. Part of that productivity journey I guess.
Simpler is better.

I'm going to be talking more in the coming weeks about specific tweaks and customizations that I've made… hopefully they'll make more sense given this context, and I'll likely link back to this where it makes sense.

Up until my first job, each project or context (being school!) had come pre-packaged with a place to put context: a workbook, study guides, etc. Suddenly I needed a place to put useful info, and no adult was going to remember it for me. In my first year of college, I used google docs to build a note wiki… and six months into school they banned laptops in the classroom. I was an unhappy computer science major… but my handwriting did improve drastically. 😜 ↩

The word for web is forest

Nicola Iarocci — Thu, 30 Sep 2021 09:05:25 GMT

When I read Entangled Life: How Fungi Make Our Worlds, Change Our Minds & Shape Our Futures by Merlin Sheldrake¹, I was stunned by the scale and importance of the mycorrhizal network that lies beneath the surface of any given forest in the world. The “wood wide web”, as scientists started to call it, sounded like the perfect metaphor for such an incredibly efficient, symbiotic relation between fungi and trees.

Weekly Code Challenge (spring 2021)

Muffin Man — Thu, 30 Sep 2021 03:00:00 GMT

If you just want the see the submissions, jump to the Weekly Code Challege website. But please bare in mind that the most of these were created in a couple of hours.

This post is long overdue. Earlier this year, I came with an idea to organize a weekly code challenge in our Belgrade office. With the whole COVID situation going on, I hoped it could inspire people to start hacking and make programming fun again.

Some people were interested, so we ran the challenge in spring this year, from April to June. It lasted for ten weeks, each week had a different theme and there was a total of 47 submissions. We didn't impose a lot of rules, it just had to run in a browser, but you could use any library or framework and spend as much (or as little) time as you wanted.

Challenge started hot, but as weeks passed by, interest slowly faded away. After ten weeks we decided to end it. To be honest, it lasted longer than I initially hoped. A lot of people tried new stuff, played around and had fun hacking little things. I dare to say it was a complete success and I have to say thanks to all the people who participated!

Therefore, I urge you to organize something similar yourself, in your office, university or city, it doesn't matter as long as people are hacking for the sake of it. It is a wonderful and fulfilling process.

Shameless self promotion #

I was happy that people were interested to participate, and I enjoyed it myself tremendously. In the process, I learned a thing of two and hacked together a few things I would probably never try if there wasn't for the code challenge.

Out of the nine things I created (I skipped the last week), these three are my favorite:

All three are completely different and they vary from a vanilla JavaScript ASCII rendered game, followed by a simple ThreeJS demo to the complex 3D shapes CSS objects.To be fair, these are only complex in the context of using CSS to render them.

You can check my submissions here, and code is available on GitHub.

Writing Is Hard Work

ho.dges.online — Thu, 30 Sep 2021 03:00:00 GMT

Writing is hard work. A clear sentence is no accident. Very few sentences come out right the first time, or even the third time. Remember this in moments of despair. If you find that writing is hard, it’s because it is hard.

William Zinsser

HPA - How to gracefully scale down pods with PreStop hooks

Shyam Jos — Thu, 30 Sep 2021 03:00:00 GMT

One of our application running in kubernetes have a background job feature (which is a bad practice and against stateless architecture) and recently we noticed some of these background jobs were killed in midway due to pod scale-down done by horizontal pod autoscaler. Our assumption was Kubernetes will only terminate pods with less utilization but we were wrong! and its a random selection.

In order to handle this issue we found out that we can use prestop hook feature in kubernetes to check for any background process running in container and wait for it to complete before sending termination signal.

How PreStop hooks works in kubernetes #

Kubernetes provides container lifecycle hook framework to run code triggered by events during their management lifecycle called PostStart and PreStop hooks. A PreStop hook is called immediately before sending termination signal and kubernetes will wait until the PreStop hook to complete or until it exceed the terminationGracePeriodSeconds value, Only after this kubernetes will sent the termination signal to the container.

Our solution #

Our solution was to use a bash script for PreStop hook which will check for .lock files created by the background process, if it finds a .lock file the script will wait for the .lock file to be removed and when the file is removed the script will exit then the container will receive termination signal. If the PreStop hook takes more time than terminationGracePeriodSeconds value then container will be terminated immediately once this value is crossed.

PreStop hook bash script #

#PreStop Script for checking background php tasks 
SECONDS=0
while true
do	
if [ -z "$(ls -A /app/demo-web/public/lock/*.lock 2> /dev/null)" ]; then
 echo "No lock files found!, container is safe to terminate [Time Elapsed: ${SECONDS}s]."
 exit 0
else
 echo "Lock files found!, waiting for background process to complete [Time Elapsed: ${SECONDS}s]."
 sleep 10
fi
done > /proc/1/fd/1 # sent outputs to conatiner's stdout

Final deployment yaml #

apiVersion: apps/v1
kind: Deployment
metadata:
 name: demo-web
 namespace: production
spec:
 progressDeadlineSeconds: 300
 revisionHistoryLimit: 5
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 5
 maxUnavailable: 0
 selector:
 matchLabels:
 app: demo-web
 tier: web
 environment: production 
 template:
 metadata:
 labels:
 app: demo-web
 tier: web
 environment: production
 spec:
 terminationGracePeriodSeconds: 300
 containers:
 - name: demo-web
 image: demo/demoweb:v1
 lifecycle:
 preStop :
 exec:
 command: ["bash", "/app/scripts/prestop.sh"] 
 imagePullPolicy: Always
 resources:
 requests:
 cpu: 50m
 memory: 150Mi
 limits:
 memory: "650Mi" 
 ports:
 - containerPort: 80
 readinessProbe:
 httpGet:
 path: /healthz
 port: 80
 initialDelaySeconds: 2
 periodSeconds: 5
 successThreshold: 1
 failureThreshold: 2
 timeoutSeconds: 2
 env:
 - name: HOST 
 value: "demo.app.com"

Integration Test Email #25 - 123dev #25

Justin Garrison's Homepage — Thu, 30 Sep 2021 03:00:00 GMT

Comments

Who’s to blame?

Many of us probably heard about the HBOMax email that was sent out by mistake last week. It wasn’t a major outage, but a lot of people were talking about it. I especially loved the outpouring of support I saw online for whomever triggered the integration job that sent the email. 💖

Just like the gif, no one person is to blame. The person who poured cement isn’t any more to blame than the person who architected the building. With any incident, if a single person can cause an outage it is almost certainly not their fault.

GitHub Actions Plugin for Cronicle

Philipp Hartenfeller Blog RSS Feed — Thu, 30 Sep 2021 03:00:00 GMT

I created a GitHub Actions Plugin for calling jobs on your Cronicle server.

Recipe

Home on mfashby.net — Thu, 30 Sep 2021 00:56:01 GMT

Profiling for USA 2024 is already happening

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 29 Sep 2021 21:00:00 GMT

On some websites, there are no innocent questions.

HyperX Cloud II sound card USB connector replacement

Muffin Man — Wed, 29 Sep 2021 03:00:00 GMT

If you are only interested in the wiring diagram, jump directly to the wiring section.

HyperX's Cloud II headphones come with a USB sound card, which looks like this:

It is a convenient little dongle. I had it plugged in into the front USB port, and I bumped into it, bending the connector for ninety degrees. Naturally, it stopped working. Good thing about these headphones is that the sound card isn't mandatory for headphones to work, but the regular audio cable is pretty short.

After searching for the replacement, I realized I can't buy it where I live, and that it costs around 35-40$ which is more than the half of the price for a whole new pair. That is why I decided to try to repair it on my own.

The reason I'm writing this blog post is that I couldn't find any information on how to repair it, so hopefully it will help you fix the same problem.

Wiring #

The process is actually super simple. You'll need some soldering experience and a USB connector. Cut off the original connector and strip the wires.

Solder wires to a connector in a following order:

Black
Green
Copper
Red

Some diagrams will show you different order with middle (data) wires switched, but that won't work. I tried soldering it like that initially, but nor Windows nor Mac would recognize the headphones. The HyperX logo would light up, but Windows gave me a "Unknown USB Device (Device Descriptor Request Failed)" message. That's how I figured out that power wires are connected properly, but the data ones are not.

Sorry for the blurry photo, but I didn't want to disassemble everything to get a better one.

I hope this will help you fix your headphones and motivate you to try to fix more things in the future, before just ordering a replacement.

Declarative and imperative - 123dev #24

Justin Garrison's Homepage — Wed, 29 Sep 2021 03:00:00 GMT

Comments

Declarative and imperative

Oftentimes in technology we say declarative things are better than imperative. Why write a script when you can create an abstraction? In reality there’s no such thing as something being completely declarative outside maybe binary data in memory, but even that requires sequential steps before it can be stored.

However, we can hide imperative steps behind a declarative interface and a user never needs to know about the complexity. Just like these bowling pins, a bowler doesn’t know how they got there. They ask for another set of pins and it shows up.

Review: Beast Breaker

The David Brownman Blog — Wed, 29 Sep 2021 03:00:00 GMT

What if you were playing a billiards RPG, but your cue ball was you, an adorable warrior mouse armed with a variety of pointy objects?

Read the whole thing.

Confusion and conflicts around Open Source in Italy

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 29 Sep 2021 02:50:00 GMT

What happens when lots of funding comes to digitize a country?

Decompression Meddlings

onKeyPress — Wed, 29 Sep 2021 02:00:00 GMT

Simple Markdown Zettelkasten 1.5

trv.rs — Tue, 28 Sep 2021 10:51:00 GMT

I've been quietly using a new format of my zettelkasten¹ for almost six months. I wrote yet another giant migration script… but I wanted my system to stabilize before I wrote about it again.²

In short: I'm doing the easiest possible thing that Obsidian recommends.³ 🤷

I no longer use a "Zettelkasten ID" in the filename of every file.
I'm using YAML frontmatter instead of MultiMarkdown.

Simply:

Filename: Sentence style notes.md

Content:

---
tags:
- optional
---

# Sentence Style Notes

I'm using simple markdown notes for evergreen notes. This makes [[linking]] easier and doesn't mean I have to use aliases to compensate for a ZK ID in the filename… making [[Note-Making]] *much* easier. The YAML frontmatter is optional… mostly tags and aliases.

Related:
- I'll often have a block of related notes
- Maps of Content or hubs
- links to daily notes that make sense

[Optional link to a source or reference](https://fakesourceorreference.com/facts)

Zettelkasten ID in Filename

The reason I wanted to have the ZK ID in the filename was to preserve links throughout rewriting notes. While robust, this strategy sacrifices ease of writing… and creativity is what this is for!

If the filename contains the ZK-ID (something like 2021092723-fake-name.md) then you have three options to reference another file as a link:

know the exact second you created an unseen note from memory (impossible!)
have the computer autocomplete the title (difficult, since most file autocompletes start from the first character)
use the Alias feature in Obsidian to create a duplicate title… that you will mostly like use anyway. (extra useless typing!)

As soon as I made the conversion back… there was no going back. Composing a note with links was easy no matter what app I was using. I could even handwrite and scan a reference to [[Exodus]] or [[Border of the Known]] and know which note to which I was linking.

What about link robustness you may ask? Well… in part I am relying on Obsidian's toolchain for now, but I'm also a software developer, and my whole Creativity System is now backed up in Git. If I find a link missing, I can do some spelunking and find the missing file. If I need to apocalypse-proof my knowledge system… I have shown myself to be perfectly able to write textfile migrations. I'll generate a unique ID for each note and can append it to every link for printing.

YAML Frontmatter

Using YAML frontmatter instead of MultiMarkdown is a less contentious choice… most Markdown applications these days support a version of YAML frontmatter even before Obsidian… and few indeed supported MMD.

Where To Now?

I've been doing a lot more writing and thinking with my Creativity System now that the friction of creation has decreased. I've got a lot more thinking and tricks to post… but I figured I'd talk about it in bite-sized chunks… I've been rewriting the same 3k word post for six months.

The longer I learn about this world of PKM/ZK/Creativity Systems the more convinced I am that each person should be building their system to suit their needs and mind. It's a deeply personal tool. You shouldn't copy my methods or tools directly… unless you have similar goals.

Thank you especially to all the readers and internet friends who have sent me encouraging emails and questions. I don't have analytics on this blog… so your kind words keep me going!

I have often had a fancy for writing a romance about an English yachtsman who slightly miscalculated his course and discovered England under the impression that it was a new island in the South Seas.

I too have set sail to find the best zettel format… and found myself back home at the original and simplest.

Some will point out that this is no longer a "true Luhrmann Zettelkasten." You'd be right. This is what has inspired my definition of Creativity Systems: a system designed to connect and create ideas. ↩
I'm kind of regretting not originally writing a stand-alone "moc"-style essay page with a permalink. I had no clue the post would take off like it did. I may yet do that. ↩
I'm reminded of the glorious word picture from Orthodoxy by G.K. Chesterton: ↩

Colophon

Tue, 28 Sep 2021 06:17:37 GMT

My blog, jay.jvf.cc, is built using Hugo framework with a lightly modified version of the Terminal theme by panr. Once built, it is hosted on Google Cloud and fronted with Cloudflare for HTTPS termination.

For more information see:

About Itself

Nostalgia technology, DNS, and lessons learned as a senior engineer - 123dev #23

Justin Garrison's Homepage — Tue, 28 Sep 2021 03:00:00 GMT

Comments

Nostalgia technology

Looking back at old technology products most of the time I think how bad they were. Occasionally, I remember something I really liked and I wish I could have it again. But then I realize the products were good at the time but probably would hold back innovations in ways I couldn’t predict.

Much like this car jack which seems like a great idea! Takes up minimal amount of space, self inflates, and probably about as safe as scissor jacks are today. But when you think about it maybe if we all had self inflating car jacks the electric car would have one more obstacle to overcome.

Urban air travel, this is not

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 28 Sep 2021 02:00:00 GMT

Even if there were enough air to do it.

Balancing production and consumption in the digital era

unsungNovelty — Mon, 27 Sep 2021 19:32:43 GMT

License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 _International Public License (CC BY-NC-ND _4.0)

Most of us are addicted to consumption in this modern digital era. Or, Information Age. Information is available in abundance now. The amount which was never accessible for us humans ever since the dawn of time. We watch documentaries about planets, listen to podcasts on varied topics and can read about virtually anything in just Wikipedia alone. But this has reached a point where we are consuming in excess. We are less focused on doing anything with the information and more focused on the consumption itself. We have reached a point where we are in a consumption spree whether it be books, blogs, podcasts, youtube channels or learning.

A great way to counter the consumption spree is by balancing it out with production. Producing things that you want to and always have wanted to produce. It might be blog posts, youtube videos or art. You get the idea. The idea is to balance the consumption out with production. But I should tell you that there is a lot of unlearning that you have to do first in order to balance excessive consumption out.

Unlearning means exactly what it sounds like. You will have to learn to do things in a different way by changing or forgetting the way you used to do things before. The first thing to do is to unlearn or remove the excess junk in your daily life.

Our brain seeks things that we feed it regularly. If you feed useful and interesting things, our brain seeks more useful and interesting things. On the other hand if you feed it useless or uninteresting things, our brain will gladly seek more useless or uninteresting things.

If you are consuming in excess, it means there are many things like facebook, instagram, youtube or Twitter which are notorious for pushing you to an endless scroll of consuming useless information. You need to first clean up and unlearn them before having to do trade-offs between things that are more useful compared to these useless information. Like an interesting podcast or a youtube channel about history for instance are useful than endless scroll of useless information. Meanwhile learning a skill that will help you professionally is more useful than the aforementioned interesting podcast or the youtube channel about history.

The process of unlearning and relearning is about constantly cleaning up our consumption habits and replacing with better, less excessive consumption habits until we reach a sweet spot we know is better for us.

Going organic is great way to speed this process. I stopped having goals for reading books and writing blog posts. Or for the number of podcasts, youtube channels that I follow for that matter. Our brain needs time to process the information we have just consumed or are working with. So reading a book just after you finished another doesn’t let you extract the same amount of information. This is the same reason why you sometimes wake up with a solution you were working on the night before but couldn’t figure out. The same reason why sometimes short walks helps you solve a problem you were stuck with. So take your time to write a summary of the thing that you just consumed. I have personal notes for all the books that I have read and keep notes on podcast episodes I listen to. Going organic might seem slow for you at first. But if you really like consuming something, the best way to do it is by consuming it organically. This is because it will help you focus more and eventually the rate at which you can consume things increases.

Once you’ve balanced out consumption, the main villain. It is time for you to start producing things. Write blog posts instead of just reading them. Create an app that you’ve always wanted to. Start working on your next big project and build things with the information you’ve learned. It is the best way to use your knowledge. So produce! …Instead of just consuming.

References and links

https://www.fastcompany.com/90387269/your-obsession-with-learning-is-holding-you-back

Have thoughts or feedback regarding my post/writing? DM me on Twitter @unsungNovelty.

An nginx playground

Nicola Iarocci — Mon, 27 Sep 2021 09:05:25 GMT

Every single time I need nginx, I end up spending way too much time fiddling around with its configuration. If you’re like me, rejoice! Julia Evans built a lovely, helpful little tool called nginx playground.

Hello! On Wednesday I was talking to a friend about how it would be cool to have an nginx playground website where you can just paste in an nginx config and test it out. And then I realized it might actually be pretty easy to build, so got excited and started coding and I built it.

Quickstart to Building Golang Apps using IPFS

Bartlomiej Mika — Mon, 27 Sep 2021 06:58:04 GMT

Do you want to write a Golang app which will use IPFS? The purpose of this post is to get you up and running IPFS as quickly as possible so you can see what’s involved.

Thoughts on Silly Hats

Weblog on marginalia.nu — Mon, 27 Sep 2021 03:00:00 GMT

If you look back in history to the turn of the 20th century, you will find a lot of people wearing hats. Women wore some arguably pretty funny over the top hats. A while earlier, over the men were arguably funny top hats. This isn’t the first time men and women wore funny clothes. The aristocrats in 1680s France looked pretty silly too.

The reason we do this, wear silly hats, is because it is fashionable. Compliance with the some perceived fashion trend is one way we compete with fellow human beings, a measuring stick we use to evaluate our standing within society. Oh, you merely wear a modest and peculiar hat? Well mine is bigger and sillier still, therefore I am better!

Node.js fork is slow; Deal with it

Alexander's Blog Blog — Mon, 27 Sep 2021 03:00:00 GMT

Dealing with Node.js fork slowness

SFTP is super awesome

Imran's Blog — Mon, 27 Sep 2021 03:00:00 GMT

My dad has a site that he has been working on since the windows 3.1 days. It's static site and up until now he had been paying a little over a hundred dollars a year to have it hosted.

This came up during the winter of last year, where he had to pay for a renewal for his domain as well as his hosting. The hosting company did not accept payment from where he's located, so he asked me to handle the payments on his behalf. This is when I realized that he has been overpaying for a service that you can get without paying (e.g GitLab pages). There was no getting away from the cost of the domain name, that's just the nature of having your own domain, but hosting (for static sites) can be costless.

His current workflow was to make changes on his computer then upload the files via ftp to the hosting platform. These files are served out of the ftp directory to the web.

Getting my dad to use git was a no go from the start. As much as I like it and use it everyday for personal and professional use, it isn't beginner friendly. In fact there's a good chunk of professional programmers that cannot use it properly either. On top of learning the git commands and workflow he would also need to understand ssh keys which are not that easy to explain either (or why they are important). Especially coming from the old ftp workflow.

# The solution

I still wanted to his site backed by git. I think it's a great idea as it gives us source control instead of just files that live on his hard drive. Having a git repository also lets me take advantage of the free hosting that GitLab pages provides.

This means I would need to be able to provide a ftp like interface that ties into a git repository. My first thought was that I need something that I can watch a directory for file system events, and after a set delay execute some commands. Watchman exists but its a little heavy handed for what I needed.

I decided to write my own tool dcmd. This also gave me a convenient excuse to dabble in rust some more.

This would cover the committing + pushing part of our solution. All that I had to do was to provide ftp interface that pointed to the relevant part of the git project.

I originally wanted to use vsftp as it's included in arch's community repository. At moment it seems to not be working (core dumps on access) as well as out of date. The other thing I did not like about vsftp is that it's underlying protocol is ftp and ftp is not secure (sends password over plain text).

This is where SFTP comes in.

It's provided by default by OpenSSH and as the S indicates it's more secure. I would need to do the following to get it setup

Create a SFTP user
Create a chroot jail for the user
Create a symlink in the jail that maps to the static files in the git repository
Edit my sshd_config file to use the jail

Now I mentioned before that my dad is not familiar with ssh (and ssh keys) so I would need a password for him. On my server I disable password logins for security. I thought this would be a problem but turns out we can have both out of the box. All I needed to add to my configuration was this:

<Rest of config>

Match User <my sftp user>
        ChrootDirectory <path to my jail>
        ForceCommand internal-sftp
        PasswordAuthentication yes
Match all

This forces the SFTP user to the jail and allows it to login using a password. I had concerns that this would also grant ssh access to my machine but turns out it does not:

❯ ssh <user>@<host>
<user>@<hosts>'s password:
This service allows sftp connections only.
Connection to <host> closed.

Even if the password gets leaked I can be sure that my server remains secured (and why the title of this post is about SFTP being awesome).

The last thing I wanted to do is limit the blast radius if somehow the git credentials were compromised. Basically I do not want to lose any commits or have the repository deleted. For this I ended up creating a "bot" account on GitLab (just a fresh account) and gave it its own ssh keys. This bot would then have developer (not maintainer) access to the git project. This means the bot can push directly to the repository and have CI run and deploy to pages, but it can't change repository settings or force push.

The one other thing I was missing a systemd service file for dcmd that would ensure it keeps running. This was simple enough (I've done it before for other things) and there's a crap ton of resources on the internet for it (including the systemd man pages) so I won't be pasting the service file here.

The last thing to do before handing the credentials over to my dad was to make sure that this whole setup actually worked.

I SFTP'd in with the user I had set up and this is when I found out that you can't symlink to directories out of the chroot jail. This kind of makes sense and it would not be a good jail if the user could access files outside of it.

A bind mount remedied this problem.

mount --bind src_dir target_dir

Bind mounts basically copies a directory over to another place, but they point to the same files in the file system. This means that if the files are added/removed in the SFTP directory they are also modified in the git repository. This lets dcmd do its job and create a commit and push it.

My dad confirmed that the credentials worked for him and he can go back to his old workflow. The difference this time is his work is now preserved somewhere else other than his computer and he gets to save some money! :D

This was a super fun side project and I can already think of some improvements to dcmd if I ever want dabble in rust some more. It's a nice feeling when I can use my expertise/skills to help out my dad.

🐱🕯️ Curious tools and resume tips - 123dev #22

Justin Garrison's Homepage — Mon, 27 Sep 2021 03:00:00 GMT

Comments

Curiosity

Sometimes our curiosity gets the best of us. I cannot count how many times I’ve run a command, changed a config, or pushed a button that burned me. Usually, I learn my lesson. Many times I’ve put up guardrails or warnings so other devs wouldn’t burn themselves in exactly the same way. People are very creative and until you remove the candle they’ll always find a way to learn on their own.

Blockchain not Bitcoin, NFT not Art

Zefram's Blog — Sun, 26 Sep 2021 22:31:24 GMT

The origin of value, what NFTs are, and what comes next

I was baffled by the rise of NFTs for quite some time.

Why would people spend such exorbitant amounts on jpegs? The art is not even that good most of the time. If people wanted beautiful art, isn’t it more sensible to spend far less on local artists who will most likely produce much better art? Physical art, that is, not jpegs that anyone could “right click and save”.

Pine64

Home on mfashby.net — Sun, 26 Sep 2021 18:39:28 GMT

Migrating my Jekyll site to Netlify

Josh Thompson — Sun, 26 Sep 2021 16:00:00 GMT

Troubleshooting Netilify deploy

Ugggh I moved intermediateruby.com to Netlify a few months ago in like 10 minutes, so my primary site, josh.works, should take maybe 20, right?

I’m a few hours deep. Here’s what I get when Netlify tries to build:

I should have done the following first, instead of last:

58:21 AM:   1. Build command from Netlify app                             
58:21 AM: ────────────────────────────────────────────────────────────────
58:21 AM: ​
58:21 AM: $ bundle exec jekyll build
58:22 AM: Configuration file: /opt/build/repo/_config.yml
58:22 AM:             Source: /opt/build/repo
58:22 AM:        Destination: /opt/build/repo/_site
58:22 AM:  Incremental build: disabled. Enable with --incremental
58:22 AM:       Generating...
58:23 AM:                     ------------------------------------------------
58:23 AM:       Jekyll 4.2.0   Please append `--trace` to the `build` command 
58:23 AM:                      for any additional information or backtrace. 
58:23 AM:                     ------------------------------------------------
58:23 AM: ​
58:23 AM: ────────────────────────────────────────────────────────────────
58:23 AM:   "build.command" failed                                        
58:23 AM: ────────────────────────────────────────────────────────────────
58:23 AM: ​
58:23 AM:   Error message
58:23 AM:   Command failed with exit code 1: bundle exec jekyll build
58:23 AM: ​
58:23 AM:   Error location
58:23 AM:   In Build command from Netlify app:
58:23 AM:   bundle exec jekyll build
58:23 AM: ​
58:23 AM:   Resolved config
58:23 AM:   build:
58:23 AM:     command: bundle exec jekyll build
58:23 AM:     commandOrigin: ui
58:23 AM:     publish: /opt/build/repo/_site
58:23 AM:     publishOrigin: ui

So, first thing I did was go into the build settings and update the build command, to append --trace to it:

Now I get:

10:18:23 AM: $ bundle exec jekyll build --trace
10:18:23 AM: Configuration file: /opt/build/repo/_config.yml
10:18:23 AM:             Source: /opt/build/repo
10:18:23 AM:        Destination: /opt/build/repo/_site
10:18:23 AM:  Incremental build: disabled. Enable with --incremental
10:18:23 AM:       Generating...
10:18:24 AM: /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-crosspost-to-medium-0.1.16/lib/jekyll-crosspost-to-medium.rb:44:in `generate': MediumCrossPostGenerator: Environment variables not found (ArgumentError)
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/site.rb:193:in `block in generate'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/site.rb:191:in `each'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/site.rb:191:in `generate'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/site.rb:79:in `process'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/command.rb:28:in `process_site'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/commands/build.rb:65:in `build'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/commands/build.rb:36:in `process'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/command.rb:91:in `each'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /opt/build/cache/bundle/ruby/2.5.0/gems/jekyll-4.2.0/exe/jekyll:15:in `<top (required)>'
	from /opt/build/cache/bundle/ruby/2.5.0/bin/jekyll:23:in `load'
	from /opt/build/cache/bundle/ruby/2.5.0/bin/jekyll:23:in `<main>'
	from /opt/build/cache/bundle/ruby/2.5.0/bin/ruby_executable_hooks:22:in `eval'
	from /opt/build/cache/bundle/ruby/2.5.0/bin/ruby_executable_hooks:22:in `<main>'

Nailed it - a gem I hardly use is missing an environment variable. I’m just going to delete it for now, and re-try.

[…]

Not worth trying to clean out the gem, I did git rebase -i HEAD~~ to delete the last commit (deleting the crosspost-to-medium gem), and instead added the environment variables to Netlify.

I just copied/pasted from my .evn (A hidden file, it’s not tracked in Git, but I copied/obfuscated the values into env.sample in this commit)

…

Huzzah! The site builds. We’re good to go.

Why I'm blogging and using hugo to do it

Sun, 26 Sep 2021 12:33:52 GMT

It’s been a long time since I’ve posted regularly to the internet – short or long form. As I get older, I find I don’t enjoy publishing in Facebook or Twitter-sized snippets. So I decided to try and start blogging on a longer form again.

My primary topic will be the various personal projects I’ve started working on in the last few years. None of this is to be seen as a review or endorsement of the products or software I’ve used. Some of it was picked nearly at random. Such as the static site generator I’m using for this, hugo. I found it by simply Googling for a static site generator in go, and picking the first response on Google. It’s been so easy I’ve never thought of trying anything else.

What getting old really feels like

Nicola Iarocci — Sun, 26 Sep 2021 09:05:25 GMT

In a new study published in Ageing and Society, researchers Sam Carr and Chao Fang spent over 130 hours interviewing older people to understand what it’s like to get old and cope with loneliness. The Conversation UK features their report, appropriately titled Loneliness, loss and regret: what getting old really feels like.

We found that ageing brings about a series of inevitable losses that deeply challenge people’s sense of connection to the world around them. Loneliness can often be oversimplified or reduced to how many friends a person has or how often they see their loved ones.But a particular focus for us was to better understand what underpins feelings of loneliness in older people on a deeper level. Researchers have used the term “existential loneliness” to describe this deeper sense of feeling “separated from the world” – as though there is an insurmountable gap between oneself and the rest of society. Our objective was to listen carefully to how people experienced and responded to this.

Easy Netlify Functions for Contact Messaging

Thoughts by Thomas Derflinger — Sun, 26 Sep 2021 03:00:00 GMT

Netlify Functions are serverless code that are easy to deploy. Here I want to show how to program a simple Netlify Function to send a contact message from my website to an email address and log it into an Airtable sheet.

When does software end? How do we get there? - 123dev #21

Justin Garrison's Homepage — Sun, 26 Sep 2021 03:00:00 GMT

Comments

End results

When I think about how I create software it feels very haphazard. Some days are better than others but I rarely make the progress I want. By the time I’m ready to share with someone I realize how much a week or a month of steady progress adds up.

The gif was created by making folds and creases in paper and then using stop motion to put them together. Progress must have been slow but the end result is stunning.

Networks are in for light-out factories

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 26 Sep 2021 02:00:00 GMT

Lights can go off, and so can responsibility.

Solving Newcomb’s paradox for classical and quantum predictors

diff × weyl — Sat, 25 Sep 2021 20:44:46 GMT

A recent HN post reminded me of Newcomb’s paradox, which goes as follows (from Wiki): There is a reliable predictor, another player, and two boxes designated A and B. The player is given a choice between taking only box B, or taking both boxes A and B. The player knows the following: Box A is … Continue reading

Shawbury 10km 2021

Caius Theory — Sat, 25 Sep 2021 19:05:00 GMT

RAF Shawbury host an annual 10km running race, and I entered this time because a friend suggested it as an event to aim for over the summer. Given the run is along runways/taxiways it’s going to be one of the flattest courses on offer. Seemed like a good idea at the time.

Got there with plenty of time to register, figure out where the important things—coffee & ice cream vans—were and go for a nervous wee. Decided to start a workout on my watch early, set it to 10(km, I thought) and paused it straight away to wait for the gun. Stood around for another ~5 minutes waiting for the gun to go. Unpaused the watch as I went across the line and we were off.

Started near the front of the midfield and set out at a pace that I thought was around 10:20/mile, didn’t pay too much attention to people coming past me as it was a mass start rather than paced start. Kept pace with some absolute legend running in an Elephant costume (I’d overheard they were targeting a 56 minute finish, something I later realised to be a joke.) Easy enough route, round a taxiway and down the main runway to start.

Mile in, feeling good, lots of energy. Watch buzzes the mile pace … 10:55/mile. Yeesh, that’s way slower than I meant to start, and the rolling mile display hadn’t displayed anything until I’d hit the first mile mark. Oops, time to turn the pace up slightly. Kept going until the rolling mile showed under 10:00/mile, second mile came in at 9:27/mile and then tried to keep it under 10:00/mile whilst not blowing the heart rate up too much.

Marshals around the course were lovely and supportive, although compared to a road 10km through town where people come out of their houses and cheer it felt quite quiet as an event. Seeing someone every time a corner occurred was a welcome sight though, and the water stop was quite well placed just before the halfway mark.

Hit 3.1 miles in around 31-32 minutes which was on pace, legs were feeling okay and I took some water on board whilst running. Second half of the course I started using other people to pace myself, I’d pick someone 50-100 meters ahead who didn’t appear to be vanishing from me and then try and slowly overhaul them without increasing the heart rate too much.

Around the 5 mile mark (~50 minutes) I wondered if I could be on for an hour finish. Also realised as the watch buzzed “Halfway there!” that it was set to a 10 mile run. Useful. Heart rate was creeping up to maximum, legs were starting to ache and the ball of my left foot was starting to complain about repeatedly slamming into the tarmac. Tried elongating my stride and springing off each step a bit more consciously, also hit the point of talking to myself which is a good sign I’m starting to tire and need to start concentrating on what I’m doing form-wise.

Saw the 1km board, then the 500m board and was looking for a 250m board to kick harder from. Sailed past a friend who had already finished about the 200m mark and was really trying not to blow the heart rate through the roof. Found a 150m board—complete with amusingly sarcastic comment along the lines of “ooh, time to put some effort in”—and ramped the pace up slightly. Mindful of keeping the heart rate under control still, pushed a little too high according to my stats afterwards but felt okay doing so for a few seconds.

Crossed the line with a gun time of 1:00:51. Super chuffed I nearly dipped under the 1 hour mark, expected to be around 1:04 having done a 1:06:30 cross-country 10km earlier in the summer. I really must remember not to mess around with my equipment before events though, only make changes before going for training runs because something is always setup wrong.

In this case I’d switched from kilometres to miles for running events, reset my watch yesterday so it had no music/podcasts on to listen to and also changed the display for a running workout, crucially removing the average pace of the whole activity. What I hadn’t foreseen making those changes was I’d have to wait a mile before I got any pace feedback from the watch.

I also discovered uploading straight to Strava (through Rungap) that pausing the watch ahead of time completely screws up the timings. Strava reckoned I’d done an 18:25/mile first mile! I knew it was slow, but it wasn’t that slow. Ended up exporting a GPX and removing the first 6 minutes stood around with it paused to make Strava happy about the timings.

Race on Strava

Typing Tuples in Python

Baptiste Fontaine’s Blog — Sat, 25 Sep 2021 10:51:00 GMT

Python added support for type hints in 3.5. These are not typing as you may be used to in other languages since they have no effect on the compilation to bytecode nor at runtime, but rather hints for the (tools of the) developer.

def print_int(n: int):
    print(n)

print_int(1)
print_int("foo")

The code above doesn’t fail when you call print_int("foo") even though n is “typed” as an int. This is because this n: int is just a hint.

While you can check for type issues by running mypy by hand, type hints become really powerful when your editor/IDE supports them.

Types for collections can specify the inner types: a list (List) that contains strings (str) would be List[str].

from typing import List

stuff: list = []  # a list of anything (equivalent to List[Any])
stuff.append(1)
stuff.append("foo")

offsets: List[int] = []  # a list of ints
offsets.append(1)
offsets.append("foo")

In the snippet above, the last line is highlighted as an error in any good IDE, and mypy would complain about it.

Other container types exist as well, and they can be nested:

from typing import List, Dict, Iterable

# a dictionary where keys are str and values are List[str], i.e. lists of strings
friends: Dict[str, List[str]] = {}
friends["Alice"] = ["Sam", "Maria"]

# a function that takes an iterable of ints. It can be a list, a tuple, a generator, a set, etc
def average(s: Iterable[int]):
    total = 0
    count = 0
    for element in s:
        total += element
        count += 1

    return total / count

Given List[x], Collection[x], Sequence[x] and other Set[x], one would expect Tuple[x] to be a hint for a tuple that contains x. Well… no.

This is confusing at first, but Tuple[str] types a tuple of a single element of type str. To add more elements, you need to type them as well: a pair of ints would be Tuple[int, int], while a triplet of a string, a float and a boolean would be Tuple[str, int, bool].

While tuples can be used as sequences (e.g. for immutable/hashable equivalents to lists), I’d argue that their primary use is for fixed-length representations, such as pairs of results:

def match_object(data: bytes):
    # example code
    distances = [0.99, 0.97, 0.96]
    indices = [432, 12, 3923]
    return distances, indices

In this snippet, match_object returns a tuple of a list of floats and a list of integers (aka Tuple[List[float], List[int]]).

If you still want to type arbitrary-length homogeneous tuples, there’s a syntax for that: Tuple[int, ...] types a tuple of any length, including 0, that contains only int elements (and yes, ... is valid in Python).

For this and other interrogations (how to type a generator?), Mypy has a useful type hints cheat sheet.

TL;DR: if you know the size of the tuple, use Tuple[x], Tuple[x, x], Tuple[x, x, x], etc. If you don’t, use Tuple[x, ...], but all elements must be of type x.

Tolmie Peak

Sam Hooke — Sat, 25 Sep 2021 06:00:00 GMT

No trip report yet.

Implementing reverse translation

viralinstruction — Sat, 25 Sep 2021 03:00:00 GMT

A walkthrough of a small, very satisfying Julia implementation of a cute biological problem.

Testing a compiler that can’t even print stuff out

code · words · emotions: Daniel Janus’s blog — Sat, 25 Sep 2021 03:00:00 GMT

I’m enjoying a week-long vacation. In addition to other vacationy things (a trip to Prague, yay!), I wanted to do some off-work programming Just For Fun™ and revisit one of my dormant pet projects, to see if I can make some progress.

I opted for Lithium, my toy x86 assembler and Lisp compiler that hasn’t seen new development since 2014. But before that, I had blogged about it and even talked about it at EuroClojure one time.

Over the week, I’ve re-read the paper that I’ve been loosely following while developing Lithium. In it, Abdulaziz Ghuloum advocates to have a testing infrastructure from day one, so that one can ensure that the compiler continues to work after each small modification. I’d cut corners on it before, but today, I’ve finally added one.

What’s the big deal? And why not earlier?

One of the original goals that I set myself for Lithium is that it have no runtime dependencies. Not even a C library; not even an OS. It produces raw x86 binaries targetting real mode – non-relocatable blobs of raw machine code. I’m running them in DOSBox, because it’s convenient, but the point is it’s not necessary.

(Some day, I’ll write a mission statement to explain why. But that’s a story for another day.)

And because the setup is so minimalistic, the setup suggested by Ghuloum becomes unfeasible. Ghuloum presupposes the existence of a host C compiler and linker; I have no such privilege. By itself, Lithium can barely output stuff to screen. There’s a write-char primitive that emits one character, but nothing more than that. And there’s as yet no library to add things to, because there’s no defn and not much of a global environment.

So what to do? I thought about the invariant in Ghuloum’s design, one that Lithium inherits as well:

Every expression is compiled to machine code that puts its value in the AX register.

If I could somehow obtain the values that the CPU registers have at the end of executing a Lithium-compiled program, then I could compare them to the expected value in a test. But how to grab those registers?

That turned out to be easier than expected. Instead of extending Lithium to support printing decimal or hexadecimal numbers, I just grabbed some pre-existing assembly code to affix to the program as an epilog. (It does depend on DOS’s interrupt 21h, but hey, it doesn’t hurt to have it for debugging/testing only.) Surprise: the snippet failed to compile, because Lithium’s assembler is woefully incomplete! But it was easy enough to extend it until it worked.

So this gave me a way to view the program’s results.

But there’s another problem: these results are printed within DOSBox. In the emulated DOS machine. I needed a way to transfer them back to the host. Can you guess how?

Yes, you’re right: the simplest thing (DOS redirection to a file, as in PROG.COM >REG.TXT) works. And you’ll laugh at me that it hasn’t occurred to me until now, when I’m writing up the commit that’s already out in the wild. Another proof that it pays to write documentation.

My original idea was… SCREEN CAPTURE!

I’ve scavenged Google for a DOS screen grabber that can produce text files and is not a TSR, found one, bundled it with Lithium, and wrote some duct-tape code that invokes the compiled program and the screen grabber in turn and then parses the output. With that, I can finally have tests that check whether (+ 3 4) is really 7.

And now let me go refactor it…

How to redirect PreStop hook output to pod logs

Shyam Jos — Sat, 25 Sep 2021 03:00:00 GMT

You might have noticed that command output of PreStop hook won’t appear in pod logs and only thing you can see is whether the PreStop hook command failed or not is by checking kubernetes events log.

So how can we see the PreStop hook command output in pod logs?

By default container engine only redirects the stdout/err of the main process (running with PID 1) to the logging driver. Since the PreStop hook runs as a separate process the output from this command won’t appear in container logs.

How to redirect PreStop hook command output to container logs ? #

In Linux you can easily redirect stdout/err of one process to another by writing to the other process’s stdout/err file descriptor (/proc/PID/fd/1 or /proc/PID/fd/2). In our case we can redirect the stdout of PreStop hook command to our main process’s stdout using the command echo "Hello from the other side" > /proc/1/fd/1 and this message will be appear in the container logs.

Example PreStop hook command with output redirection #

 lifecycle:
 preStop :
 exec:
 command: ["bash", "uptime > /proc/1/fd/1"]

What does complexity look like? 🕰️ - 123dev #20

Justin Garrison's Homepage — Sat, 25 Sep 2021 03:00:00 GMT

Comments

Self made complexity

Everyone is familiar with their own complexity. I’m sure this clock makes sense to the person who built it. When you’re the one who creates it it doesn’t seem complex but perfectly logical.

Complexity shows up in our daily lives, in code, in systems, and especially organizations. Fighting complexity isn’t always the right solution, but finding a complexity that can be reasoned about and understood by others is often the best solution.

Is a school backpack app bad, or good?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 25 Sep 2021 02:00:00 GMT

It’s obvious, but not SO obvious as it may seem..

What's the entropy of a fair coin toss?

Bytepawn - Marton Trencseni — Sat, 25 Sep 2021 01:00:00 GMT

What's the entropy of a fair coin toss? What if the coin almost always returns Heads? My recruiter reports that very few candidates can answer these entropy related DS screening questions.

Who is software even for?

utf9k — Fri, 24 Sep 2021 11:00:00 GMT

Maybe it's for you? Maybe it's for me? Maybe it isn't for any of us!

Keycloak password hashing

Random Notes by agilob — Fri, 24 Sep 2021 03:00:00 GMT

This post doesn’t contain full context of the works performed, only benchmarking part I had to test how number of iterations impacts login request time to KeyCloak and if or how we can improve it. After investigating a few other options I decided to check what’s the difference for password hashing times using default hashing mechanism in KeyCloak. I found and extracted parts of password hashing mechanism from KeyCloak to my repo, developed small parametrised JMH benchmark, comparing:

Kubernetes EU 2021 edition - 123dev #19

Justin Garrison's Homepage — Fri, 24 Sep 2021 03:00:00 GMT

Comments

Wings (1927)

The gif is from a silent film I’ve never seen but it won the first ever Academy Award (1930). Ninety years later I would have a hard time reproducing that shot, and I love the vision someone had to make it happen. It reminded me of one of Clark’s laws; I’ll list all three for you here:

Enough of batteries, already!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 24 Sep 2021 02:00:00 GMT

But is it bad, or good?

Wiki-Loader: Loading Wikipedia into Elasticsearch

Joshua Rogers — Thu, 23 Sep 2021 15:00:00 GMT

Over the last two posts I explored the process of extracting a very large dataset (an XML export of Wikipedia) into various types of databases (an RDBMS1 and a Graph database) that I can run locally as a precaution against being marooned in the past should some portal, machine, or ancient artifact that send me careening backwards in history. While these two databases have satisfied my need to quickly look up information and find relational paths between said pieces of information, they still leave me needing to know exactly what I'm looking for.

You Can't Eat Relative Growth

Who is Nnamdi? — Thu, 23 Sep 2021 12:07:29 GMT

In startup land, we talk way too much about relative growth. We'd do better to ground our thinking in absolute growth.

ASP.NET 6 Migration Cheatsheet and FAQ

Nicola Iarocci — Thu, 23 Sep 2021 09:05:25 GMT

David Fowler has a very informative gist up on GitHub. It’s titled [Migration to ASP.NET Core. NET6][3] and it’s filled with details, recipes and FAQs on migrating an ASP.NET Core 5 web app to ASP.NET Core 6¹. The focus is on the new, streamlined hosting model, also known as Minimal APIs². To be clear, You don’t have to move to the new model. As the FAQ section emphasizes:

Do I have to migrate to the new hosting model

A new project-oriented Haskell book

λm.me — Thu, 23 Sep 2021 03:00:00 GMT

Announcing a new project-oriented, online, and free Haskell book titled 'Learn Haskell by building a blog generator'

The feeling the first time your code works - 123dev #18

Justin Garrison's Homepage — Thu, 23 Sep 2021 03:00:00 GMT

Comments

When code works

Sometimes I’m equally surprised as this baby when my code works. Especially when I either do it on the first try or I fight with it for days and finally figure it out. It’s exciting and then I try to figure out why it works.

Know where to look for a problem

I struggled with something for the new 123dev website for a few nights this past week. I thought the problem was because of my lack of CSS skills and I spent most of my time trying to figure it out. It turned out to be completely unrelated to CSS—full write up in next weeks post—and reminded me of the importance of finding the simplest reproducible issue before making assumptions.

Building a Website With Haskell & Nix

Thu, 23 Sep 2021 03:00:00 GMT

Introduction

I built a website so trivial that you might wonder why I even bother writing about it. It consists of a static site which is, unfortunately, still built with Gatsby, a Javascript framework. The content comes from a content management system called Contentful. The most complex part is the members-only area, which is the part I rebuilt and what this post is all about.

The members area consists of a handful of routes:

A news feed where administrators can post new messages. It’s plain text with the exception of URLs, which are replaced with proper anchor links. No images.
User management area, where administrators can invite new members or edit and delete existing members. Here you can also filter by user groups and send emails to users you select through checkboxes. That’s the part where I used progressive enhancement. This route also hooks into email sending functionality for messaging someone who was invited to the site.
People can create events, and others can then sign up for these events and also indicate how many guests they bring. Everyone can see who’s coming and who declined, how many guests there are in total. Events can also have attachements, such as PDF files, which are stored on the droplet itself.
People can request a link that let’s them change their password.

The first version was hacked together with Firebase and some serverless functions, because I didn’t want to maintain a proper backend. The UI was awkwardly fit into the static site. It worked but it was ugly.

And so one day I decided that it was time for a rewrite! The official reason is that I considered it too risky to depend on Firebase for everything. Also the local developer experience was a bit lacking at that time. The honest answer is that, for quite a long time, I had been looking for a real-world project to which I could apply all the fancy functional programming (FP) technologies I picked up over the years. I was holding an FP-shaped hammer and I was just looking for a suitable nail. This project seemed perfect, since it’s small, has actual users and even a deadline, but since it’s a project for a family member there’s no money on the line and I could do whatever I wanted in terms of tech. And so the idea was born to rewrite the entire website with Haskell, Nix and PureScript.

In this post I want to share my experiences building this simple backend in Haskell, Nix and PureScript. You don’t need to know any of these technologies to follow the blog post. You might not understand every little detail, but I hope that the overall message is still valuable. I’ll start the post with a short summary, so you don’t have to read the entire article if you just want the main conclusions.

Here’s the source code.

Summary

Haskell

I’m neutral about using Haskell again in the future.

On the one hand, algebraic data types and an excellent type system make it very easy to translate my thoughts into code, since branching control flow, based on a finite number of options, seems to make up the majority of my coding. It’s kind of liberating to make sweeping changes across the code base, knowing that the compiler will prevent a lot of mistakes. Haskell also has a surprisingly large selection of web development frameworks and libraries. I didn’t have to implement any mission critical functionality myself (encryption, database communication, routing, and so on).

On the other hand, one of the biggest time sinks was mere plumbing. Many Haskell libraries revolve around specific Monads, created just for that library. You don’t need to know what Monads are for this blog post, but the gist is that you often need to write a little bit of glue code, so that different Monads of different libraries can talk to each other. I’ll write more about this later. But the end result was I spent way more time on glue code than I thought. To make things worse, this glue code did not feel like meaningful progress. It’s a Haskell solution for a Haskell problem after all.

Haskell also has its fair share of historical baggage, which includes, but is not limited to, an annoying module system, complicated tooling, and a rather anemic standard library. The last point means you’ll have way too many lines of code dedicated to imports.

Nix

I would 100% use Nix again to provide a basic developer environment, which includes compilers, formatters, database libraries, and so on. It is, in my opinion, best in class in this area and entirely unrivalled. Think nix develop.

Whether or not I’d use it to build the actual project depends on the programming language and how well supported it is in Nix. Haskell is the Nix posterchild, Node on the other hand can be hit and miss.

I would not use Nix again to deploy my code. It was fun playing around with Systemd and SOPS, but the container ecosystem is just too big. Luckily you can generate Docker containers with nix, which is something I’d like to explore in the future. Running my NixOS image locally through QEMU was and still is, frustrating, especially on MacOS. Additionally, my GitHub action workflow that builds the application, runs end-to-end tests and deploys it, takes 25 minutes. For a website of this size that is insane.

SQLite

I love SQLite. It has quirks and historical baggage but at the end of the day it’s a robust, battle tested and simple tool that I find to be a joy to use. You can easily spin up an in-memory database for unit tests, it comes with a surprising number of features, such as JSON tooling, and it’s just a file at the end of day. This kind of simplicity is refreshing. Also shoutout to Litestream for database backups.

10/10 would use again.

PureScript

I’ve written 140 lines of PureScript (PS) code in this project, which is nothing. It’s used for progressive enhancement of markup rendered on the server, which means inserting DOM elements and interactivity into existing DOM nodes. Unfortunately I couldn’t find any good PS libraries for this, so I had to resort to writing very verbose and tedious code that looks like a one-to-one translation from Javascript. All of the PS web frameworks and libraries I saw want to own the markup they render, like React. So instead of surgically injecting interactivity into existing DOM elements, you hand control over an entire subtree of the DOM to PS. But that would have meant duplicating the rendering (first render with Haskell on the server then with PS on the client) of some routes in PS, which seemed like too much work.

For this project PS was not a good choice, as it meant additional complexity and an unnecessarily large bundle for little to no gain. I like the language though. As such I just can’t say if I would use PS again in the future based on this project alone.

Deep Dive

Haskell

Haskell the language is really not complicated. But the ecosystem can be. I frequently found myself spending too much time solving problems that only exist in Haskell. These problems are usually the result of having to combine custom Monads (which you can think of as small, domain specific languages) from different libraries, something I alluded to already in the summary. For example, I struggled quite a bit with logging, which has only ever happened once before, in Clojure, where logging is the final frontier. But let me explain in a bit more detail why logging can be surprisingly tricky.

Here’s a snippet from the README of scotty, a well known web framework:

main = scotty 3000 $
  get "/:word" $ do
    beam <- param "word"
    html $ mconcat ["<h1>Scotty, ", beam, " me up!</h1>"]

It doesn’t matter if you know Haskell or not. I’d like to direct your attention at param "word", or, applying the function param to the string "word". Maybe you can guess that this extracts the value of the route parameter we defined in the preceding line, with "/:word". But isn’t it weird that we’re not passing the HTTP request to the param function? Where does it get the request paramter from, then? The answer is too complicated for this blog post, but suffice it to say that there’s some magic going on behind the scenes. And this magic is made possible by the Scotty Monad.

Now, what if you want to also do some logging in your HTTP handlers? There’s a really nice library called katip, and it also has this very neat but also weird looking code:

main :: IO ()
main = do
  -- ellided for brevity
  katipAddNamespace "additional_namespace" $ katipAddContext (sl "some_context" True) $ do
    $(logTM) WarningS "Now we're getting fancy"
  katipNoLogging $ do
    $(logTM) DebugS "You will never see this!"

The katipAddNamespace function does the same thing as any other structured, hierarchical logging framework. Any log expressions in the indentend block of code on the following lines will have this new namespace added to them. This way the caller can add some additional context to the logger and the callee doesn’t need to know or care about these things. As a proponent of structured, hierarchical logging, I find this super neat. But, unlike in Go, where you’d do something like

newLogger := oldLogger.With("key", "value")
newLogger.Info("whatever")

the newLogger variable is nowhere to be seen. You might at first glance think that we’re passing an anonymous function to katipAddNamespace (the $ do part maybe?), but even if that were the case, we’re clearly not accepting any arguments in that anonymous function. So what happens to the modified logger? Well, the answer is, it’s complicated. Just like with Scotty, there’s a custom Monad that takes care of all this plumbing for us behind the scenes.

Of course the logger and its contexts and namespaces need to live somewhere. In a typical setup you create a record that holds your application environment. And in that environment you store the logger. You then need to teach Katip how it can access and modify the logger in that environment. It’s essentially like implementing an interface for a struct, if you’re a Go person.

But then you also need to teach Scotty about this custom Monad. Instead of working with Web.Scotty you import Web.Scotty.Trans, which states in its opening paragraph:

The functions in this module allow an arbitrary monad to be embedded in Scotty’s monad transformer stack in order that Scotty be combined with other DSLs.

Which is exactly what I wanted to do. And at this point the whole house of cards may or may not fall apart. Because custom Monads often come with constraints. As in, if you want to run this custom Monad you need to make sure that the context in which it runs supplies X, Y and Z. And then you start wondering how those constraints will fit into the bigger picture of all the other custom Monads you need to satisfy. For Scotty and Katip this ended up being not too crazy, but for another library – co-log – I was unable to achieve my goal at all. If this whole paragraph sounds a bit hand-wavy, please check out this Reddit question and this StackOverflow (SO) post.

So yes, Haskell can be concise, expressive and type safe. But getting there can be a pain, at least in the beginning. I do believe that over time this stops being a problem, as you get more and more familiar with the Haskell type system and the ecosystem.

Aside from Monads, there was at least one other recurring topic that made me scratch my head, and that’s how to deal with control flow and early return.

Here’s what 90% of my Go code looks like:

someValue, err := someFunc()
if err != nil {
  return errors.WithMessagef(err, "something went wrong with thing %s", id)
}

Countless people have complained about the repetitiveness and boilerplate, but I actually don’t mind it at all. The pattern is always the same, so it’s very easy for me to follow the control flow inside a function. Additionally, it’s trivial to return as early as possible. In fact, returning early is something most people probably don’t even give a second thought to, since it’s just so easy in imperative languages with statements.

What does the above look like in Haskell then?

case someFunc of
  Left err -> Left ([i|something went wrong with thing #{s}: #{err}|])
  Right -> -- keep going

I suspect that many seasoned Haskellers will take fault with the above snippet though, because it’s needlessly verbose. And I kind of agree, and I’ve spent way too much time converting to and from various control flow patterns in this project. とにかく, anyway, what’s going on here? someFunc doesn’t return a normal value and an error, it also doesn’t throw exceptions, rather it returns a Result type (called Either). We then pattern match on the two possible variations of this result type. Left if there’s an error, Right if everything’s fine. Think Rust’s result type.

But what if you have several function calls that all return different kinds of results? Consider the following scenario:

Unmarshal query parameters
Check if user is authorized
Get entity from database
Perform some business logic with user and database entity

Pretty much all of these things can fail in some expected way (meaning exceptions wouldn’t be appropriate). Here’s a snippet from an older commit that shows what such code could look like.

getTokenByValue dbConn token >>= \case
  Nothing -> return $ Left $ TokenNotFound token
  Just tok@Token {..} -> do
    ok <- hasUser dbConn tokenUserId
    if not ok
      then (return . Left $ UserForTokenNotFound tokenUserId)
      else do
        now <- Time.getCurrentTime
        if now >= tokenExpires
          then return $ Left $ TokenExpired tok
          else do undefined -- ...

You can see where I’m going with this, right? The increasing indentation level looks ugly. In all fairness though, this isn’t any more verbose than the equivalent Go or Typescript code. I really wish that I had just stuck with the ugly and verbose version instead of trying out various different patterns. I believe that this code is simple, and readable and very easy to understand. Refactoring this for vanity reasons is not something I’m proud of.

There are some tricks for dealing with the ugliness, but in my personal experience they often don’t work in real world scenarios, where more complicated types are involved, particularly anything with IO. What does work is making liberal use of syntax sugar, combinators and Monad transformers though.

Here’s a snippet I copied verbatim from my code base, which shows what the previous code looks like once you throw Monad transformers at the problem:

token@Token {..} <- Token.get value >>= E.note' (NotFound value)
ok <- User.exists tokenUserId
E.unless ok $ E.throwError (NoUser tokenUserId)
now <- liftIO $ Time.getCurrentTime
E.when (now >= tokenExpires) (E.throwError $ Expired token)
return $ Valid token

This is probably illegible to folks who are not familiar with Haskell, so here’s a hopefully human readable translation.

Get some value and abort with a NotFound error if it's not there
Unless the user exists, abort with a NoUser error
Get the current time
When the current time is greater than the token expiration, abort
Return a valid token

This version of the Haskell snippet doesn’t suffer from increasing indentation. We also return early since behind the scenes Haskell still sort of pattern matches on the various result types and it knows how it can short-circuit the computation. For example, if ok is false, this will immediately return the NoUser error, it will not run the remaining lines of that function. There’s also very little to no Go-style line noise about error checking. I dare say it’s expressive and concise. But it took me a lot of experimenting to get there and also required writing some utility functions for translating between various custom Monads, again. There are more things I could complain about here¹, but I hope that my main point here is clear: translating a simple pattern, early return, to Haskell, without making the code untolerably ugly, is not straight forward and it can be hard to find this kind of advice in tutorials and books.

That was a lot of negativity now, so let’s move on to something more positive: algebraic data types (ADT) and pattern matching. Together, these two features form the basis for how I model the world in code. Nothing beats the ease with which this let’s me design branching control flow based on a finite number of options. I currently write Go for a living and the lack of ADTs has caused more than one logic bug and runtime panic. Generally, few languages give me the same confidence as Haskell. It’s really hard to overstate the peace of mind that comes from not having to worry about nil pointer exceptions and unhandled cases. While I was writing this blog post I suddenly had an insight and removed one field from a record that’s used everywhere in my application. In many languages I would have dreaded this task but in Haskell I simply removed the field and then followed the compiler errors. At the end I was pretty much 100% certain that I didn’t break anything.

There are a lot of other things I could write about here, but I think most of the usual suspects have already been covered in great detail in other blog posts, on Reddit or on Hackernews. This includes:

Historical baggage like a string type no one really uses
An anemic standard library resulting in lots and lots of imports
No one likes records
Slightly confusing build systems
Modules that are at the same time tedious and not powerful
Lazy evaluation can be tricky for performance and debugging
The typical issues of working with a niche language
…

But I don’t find any of these things particularly intersting and they’re also not deal breakers for me.

I wanted to write about the experience of using Haskell, not the nitty gritty technical details. So what does it feel like, then? For the most part it’s just like writing a backend in any other language. Define routes, parse query parameters, do some authorization and authentication checks, fetch something from the database, render a document. On a good day, Haskell makes me more efficient and productive because I spend less time fixing bugs and less time translating the problem to data structures, because ADTs and the type checker are just that good. On a bad day, I still find myself deciphering runtime issues, for example caused by a mismatch between database and Haskell types, while also spending several evenings just solving Haskell issues.

I’m neutral about using Haskell for future projects, because there are things about the language I love and others that drive me crazy.

Nix

According to tokei this project has 1206 lines of Nix code, which fall into three categories: developer environment, building parts of the application, building and deploying the NixOS image that runs on a digital ocean droplet.

I can’t think of a better way to provide all the tools necessary to work with a project than Nix. Every new project I start uses Nix² to provide instructions for how to create a shell that includes things like compiler, formatter, database tools, terraform, and so on. In combination with direnv.net whenever I cd into such a project my shell environment is updated automatically. Doing this is as easy as adding the stuff you need to a list, and it usually just works (and keeps working).

Building the various parts of this application with Nix was also not too much work, but your mileage here depends on the languages and tools you use in your project and how well they’re supported in Nix. For Haskell there’s a tool you can use – cabal2nix – which generates Nix code that contains instructions for how to build your project, including its Haskell dependencies. You can mostly just follow the documentation to get something up and running. But Haskell is an outlier and many other languages don’t enjoy the same level of support. For some of those languages there are tools that translate lock files to Nix instructions, but they’re not without issues. If no such tool exists you’re out of luck and building your application with Nix will quickly become very, very challenging. Nix limits your ability to make HTTP requests and interact with the file system. You therefore can’t just take a command like yarn install, which would download Javascript libraries, and wrap it in some Nix code. Instead, Nix needs to download each of those dependencies, and for that it needs at least a URL and a hash to verify the contents. Good luck quickly whipping up a tool that does that.

In this project I ran into little to no problems in that area though. What I get in return for building everything with Nix is great caching of build artefacts, and incredible reproducibility. I can clone the project, run a single command, and it Just Works™. Additionally, even if it took me a few hours to figure something out, it was usually a one-time cost.

What did not spark a lot of joy was running my application with Nix, both locally and remotely. Nix is already a niche technology, but deploying with Nix is a niche within a niche. Documentation is lackluster and there’s a real chance that you’ll be the first person to use a bunch of tools in a certain combination and as such there’s no StackOverflow answer you can just copy from. Here are some things that tripped me up in no particular order.

You can’t build a NixOS image on a Darwin machine, so you need to run things through Docker. It’s another layer of indirection and resource usage for no gain. It’s generally frustrating if you develop on one machine for a while, only to realize that everything’s broken when you to switch to another OS, especially in 2021. I wouldn’t go so far as calling Darwin a second class citizen in Nix, but you’ll have to put in extra effort here and there.

If you run your application inside a NixOS image, how do you actually (re-)start and monitor the application? I went with Systemd, which was another piece of technology I needed to learn. Documentation is OK but outside of the official manpages I found far fewer articles that talked about using Systemd for backend services than I expected. It therefore took me longer than I wished to figure out how to pass credentials to my application, share a database directory between different Systemd services, which syntax goes where, and so on. If I wanted to debug things on MacOS I needed to build the image through Docker and then I also ended up running a QEMU VM through Docker. You’re really on your own for a lot of these tasks and at some point I was tired of figuring out one thing only be the stuck on the next task.

The biggest time sink is just figuring out what arguments a Nix function takes, which arguments are magically and silently passed through but can be customized (and how), and what the function returns. My secret weapon here is GitHub code search. Nothing beats seeing working code. I rarely benefit from reading the official manual, since it doesn’t contain enough runnable examples. I do spend a lot of time in issues, PRs and in source code.

Consider Nix a slider that goes from local environment all the way to production environment. The further right you go, the less likely it is that you can just follow a cookbook. It’s therefore not a big surprise that I would:

Always use Nix for local development
Sometimes use it for building the application
Probably not use it again for deployment

PureScript

This project was originally meant to be a super simple, single Haskell file that implements a server that does not require any client side code. I really missed that mark.

I added PureScript (PS) because there’s no HTML-only way of toggling a bunch of checkboxes at once. The use case is to select a few (or all) users from a list and then click a button to send an email with all selected users added as recipients.

The idea was to progressively enhance the server-side markup. On page load, reveal the hidden “send email” button and inject a checkbox into every cell of the users table. Do not replace the entire users table with markup rendered on the client, because that would have meant reimplementing all of the markup in PS. Or figuring out a way of sharing my Haskell template code with PS, but that sounded like a lot of work. Unfortunately PS doesn’t seem to have any libraries that help with that kind of task. Most of its ecosystem seems to revolve around React-like libraries that want full control over a DOM node.

So I ended up translating Javascript to PS almost word by word and as such the code looks like this:

getCheckboxState :: Node -> Effect { email :: String, checked :: Boolean }
getCheckboxState checkbox = do
  checkboxAsEl <- maybe (throw "couldn't convert checkbox to element") pure $ Element.fromNode checkbox
  checkbox' <- maybe (throw "couldn't convert checkbox to input element") pure $ InputElement.fromNode checkbox
  isChecked <- InputElement.checked checkbox'
  email <- Element.getAttribute "data-email" checkboxAsEl >>= maybe (throw "no data-email attr") pure
  pure { email: email, checked: isChecked }

This is the same as chaining document.querySelector and getting and setting DOM node attributes. Except that in PS I had to cast from one node type to another a lot. It’s a bit puzzling and I’m still not sure if I overlooked something really obvious, but take a look at Web.DOM.Element and see for yourself. I think you’re not meant to write “low-level” code like that. It’s probably better to wrap it in more expressive, high-level APIs. Since I couldn’t find any of those in the form of ready-to-use libraries, and didn’t want to abstract something I only did once, ugly code it is.

I was pleasently surprised by the Nix support though, which seems to mostly be the work of a single person, https://github.com/justinwoo.

I wish I could say more about PS, because it’s actually a nice language, but it just wasn’t a good fit for this project. I think the client side code is in the kilobytes, whereas a vanilla JS implementation would have been much, much leaner.

Conclusion

Haskell is still my favorite programming language. Rust sounds like a modern and more refined take on the concept of pair programming with a very capable compiler, but whenever I tried it I quickly missed the ease of writing code in a high level language. But as much as I enjoy using Haskell, I can’t ignore the frustration it has caused me, and how much slower it often makes me during the initial implementation of something.

I still believe that Haskell, with a good enough understanding of its more advanced language extensions, can be an incredibly productive language. The more potential bugs you can rule out through the type system, the less maintenance down the road. But getting there takes a long time.

My goal for the next 12 months is to do another project of roughly the same complexity level, but use tools that value simplicity above everything else.

For example, instead of devising a complicated secret management solution, that could end up being risky because you don’t understand how it works, I could also just ssh into the server and create a file with the secrets and call it a day. Instead of using Nix to manage the complexity of a sprawling dependency tree, what if my app barely had any dependencies?

For example, I would strongly recommend to immediately catch potential exceptions thrown by a function that has MonadThrow (if I recall correctly), because I’ve had surprising errors when the exception type was not what I thought, because somewhere a function happened to have a MonadThrow in its signature but with the exception type hardcoded to string. I’ve also created subtle bugs in code where I thought I was short-circuiting but I was actually just returning IO Left instead of Left. So this mtl-style control flow is not without its pitfalls.

More specifically a Nix Flake

My Math Confession

Adriano Caloiaro's personal blog — Thu, 23 Sep 2021 03:00:00 GMT

I did math for the first time when I was 29 years old. This isn’t a self-deprecating snipe or feigned humility. I did math for the first time when I was 29 years old. When I finally did it for the first time – at least, the first recorded time – I didn’t know it had happened. It was six years later reading Paul Lockhart’s excellent piece entitled, “A Mathematician’s Lament” that I realized what I’d done; I’d finally done math all those years ago. At the time I was pretty sure I was programming, or something close to it. It was one part statistics, one part algorithms, and one part math. I’m not even sure today what constitutes the whole of what I was doing, but those are without a doubt three of the parts. I was also pretty sure that I was doing something related to cancer. After all, I was doing what I was doing because I was getting paid for it. And I knew that the money came circuitously to me by way of the National Cancer Institute. And if there’s one thing I know about the NCI, I know that they care about fighting cancer. So I was probably doing something related to cancer, too.

So how is it that someone who studied Computer Science in college was able to graduate without having previously done any math? Computer Science is a lot of math, right? My entire schooling, all the way through high school, I never thought that math was my strong suit. And if you’ve read my introductory sentence, you can probably deduce that I’m not getting rich as a mathematician any time soon. As far as I had a plan to go to college (and you could hardly call it that), the plan was to study English. I wasn’t a good writer and I didn’t know what it meant to study English or have a profession in it. What I did know was that I heard people studied English in college, and in high school I was able to exert a minimal amount of effort in English courses without failing. English was a safe bet for a soon-to-be Halo addict. So needless to say, I didn’t trouble myself with a lot of math in high school. My senior year of high school I took a course called Business Math. Business Math was a real doozy of a course. And somewhere between the end of high school and the beginning of college, the “plan” changed and I decided to study computer science since I was already making money doing computer stuff.

Now let me tell you a few things about Business Math. Business Math has never saved a single soul from a Freshman year Computer Science 101 culling. Those things are butal; I think we lost 50% of the class by the end of the semester. Business Math also hasn’t gotten a single computer science student through Discrete Math, Calculus, pre Calculus, or even pre-Calculus for Business Majors (sorry, biz folks). Business Math couldn’t possibly do that because it’s a truly garbage course; unparalleled in its uselessness. But the other reason is that Business Math is not even a mathematics course. And neither are Discrete, CS 101, Calc, or Pre-calc. Sure, these classes all teach about numbers, functions, and how to fiddle with formulas, but none of them teach any mathematics. No, mathematics courses hardly exist. I’d love to take one as an adult, if you’re aware of any.

I have an idea for how to rename a few math courses

Business Math: Easy Numbers for Suits

Pre-Calculus: Harder Numbers

Calculus: Hard Numbers

Discrete Math: Hard Numbers for Computers

Algorithms: Logic & Hard Numbers for Computers

I’m dodging the rhetorical question here. How did someone who studied Computer Science in college graduate without doing math? I did it by taking Business Math in high school. Because graduating with an undergraduate computer science degree doesn’t require mathematics at all. At least not so much that you can’t temporarily memorize some general mathematical techniques for an exam and forget it all the next day. They won’t be used outside of an exam setting; trust me. I’m looking at you, proofs. But even proofs aren’t really mathematics. Students do proofs because instructors ask them to do proofs; they’re not actually proving anything novel. If they were, they’d actually be doing math. They’d be doing math because they’d be thinking, getting stuck, and engaging in fruitful conjecture about the nature of problems. But most “computer science” doesn’t require much of that, and I can guarantee that the average student still stands on a podium in four years even if they do absolutely no math. Their robe may not have much in the way of flair, but they’ll surely be cleared to walk across that podium and grab a ludicrously expensive piece of paper that constitutes a degree. No, computer science students generally ask computer to do the numbers, and the computers tend to do the numbers really well; even the hard numbers.

Like most people working in a technical field, my early days were riddled with imposter syndrome. I was waiting for math to rear its head and reveal me as the fraud that I was. Because as we all know, computer science is “all math” and I’d hardly learned a lick of it. I occasionally added and multiplied numbers in code, but that’s hardly what I’d call “all math”. The real computer science was out there, it would find me, and I’d die destitute when it inevitably did.

And then the math found me. This was it. It took five years to find me post-college. What took it so long? It’s a shame I hadn’t saved any money. Business Math should have prepared me better for this! I should have had a savings account primed to keep me afloat and at least temporarily stave off destitution. I knew this was coming and there was some consolation in that.

But the notation, the formulas … holy shit the notation was everywhere. This was worse than I anticipated.

I know you know what you’re looking at in this image. You’re looking at cold, hard Math with a capital “m”; no question about it. This is it. The real deal. The math professor in white socks with Jesus sandals stuff. The potting soil in which to grow one’s neck beard. There’s an upside down “A” in the code for chrissake. It’s real math; righteous math.

But after I cut through all this absurd notation in which an algorithm should never be expressed, I found some really basic building blocks. And more importantly, I found a problem that needed solving, and problems generate ideas. Then eventually, ideas give way to solutions and more absurd notation in which algorithms should never be expressed.

Behold

The solution. This is the culmination of a lot of math. But I’m not talking about the asinine notation, functions, or codes. I’m talking about the existence of one line of code:

colsum[j] <- colsum[j] - C[pi(i),pi(j)]

This was it. This is what I feared would render me destitute. It doesn’t matter what any of this code means. What matters is that the math was a single insight. It was the realization that if a single column of a matrix was subtracted before recalculating sums, we could ask the computer to crunch a lot fewer hard numbers. It meant maybe some day a researcher in a lab could mash a button and have a chemical candidate for a novel chemotherapy enter their brain for the first time. It meant they could do that every couple seconds instead of every couple weeks. The math was the thought process, not the numbers, functions, rules or other tools of the trade taught in your average Hard Numbers for Computers course.

I was never bad at math. I have a bad memory for formulas and never had a real problem to solve.

Before my dear friend and business partner Wayne Johnson died a year ago this month, he gave me permission to open source this small piece of the work we did together referenced above (https://github.com/acaloiaro/topk-taupath). I’m forever grateful to him for being a great mentor and teaching me how to do math before 30. I’ll always remember handing this work over to Wayne. It was the culmination of years of time together and nonstop, profound lessons from Wayne about how to think more methodically. I was pretty nervous about it because Wayne was a very precise man who demanded pricise work. I was even more nervous about the fact that I hadn’t heard from him an entire day after handing it over. Finally he called and while I don’t remember much about that conversation, I do remember the first thing he said to me: “it’s beautiful”. And that’s how I know math was done, because formulas and rules are not beautiful, but math is art and art is beautiful.

I challenge you to tell someone they’ve done something beautiful. I promise they’ll never forget it.

We focus too much on the low ends

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 22 Sep 2021 21:00:00 GMT

Because we feel our status wrongly.

Book review - Library as Place

River Writes - A MediaWiki Blog — Wed, 22 Sep 2021 13:07:18 GMT

A review of “The Library as Place: History, Community, and Culture”

Book Review: Nausea

Nicola Iarocci — Wed, 22 Sep 2021 09:05:25 GMT

Antoine Roquentin, the protagonist of the novel, is a former adventurer who has been living for three years in Bouville, a fictional French seaport town, researching the life of an 18th-century diplomat. During his previous life around the world, Antoine has seen many places, met many interesting people, done exciting things. For the last three years, however, he’s been alone in Bouville. He has no friends and no desire to make some or meet anyone. He’s interested in nothing, not even in his work that he keeps neglecting. His days are mostly spent walking around town, listening to conversations and observing people around him. A “sweeting sickness” he calls nausea increasingly impinges on almost everything he does and enjoys.

Update for September, year of the new hope 2021

ciesie.com — Wed, 22 Sep 2021 03:00:00 GMT

I’ve not written one of those at all this year. Not surprisingly, the most productive year for this website was 2020… thanks global pandemic…?

I’ve posted the RFID Explained blog post. I’m trying to gather everything I’ve learned, while working on my RFID project, in one place. This is an assumption but in the maker’s space a lot of technologies are just an Arduino libraries. What I mean by that is that very few people actually learn how a specific technology works. They build a device using Arduino, using the most popular library. I don’t think there is anything wrong with that. It was probably one of the goals of the Arduino creator: remove the technical obstacles to make technology more approachable for people without deep technical knowledge. What if you want to learn more? You need to read, often times badly written, documentation spread over multiple websites and documents. Sometimes, if you really want to understand something, you should read the documentation of the standard, which is a document you have to pay for… My RFID blog post is supposed to be a quickstart for those that want to understand the common RFID solutions.

Is magic real? - 123dev #17

Justin Garrison's Homepage — Wed, 22 Sep 2021 03:00:00 GMT

Comments

I still don’t know how this works

I was at Disney Animation when they first showed us the animatronics for the new Beauty and the Beast ride for Tokyo Disneyland. I never saw this transformation and still don’t understand how it works.

Honestly a lot of code I write feels the same way. The fact that I can tell transistors what to do with any resemblance of English language is pretty magical.

University: Nearing the End

Christopher Wales' Blog — Wed, 22 Sep 2021 03:00:00 GMT

University has been a fun and interesting experience so far. It's given me many amazing opportunities and has taught me much about computer…

Facebook knows Instagram is dangerous. Who else should?

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 22 Sep 2021 02:50:00 GMT

It’s an easy answer, really.

Manjaro

Home on mfashby.net — Tue, 21 Sep 2021 22:03:48 GMT

Cruella

🦉 olshansky 🦁 — Tue, 21 Sep 2021 07:58:51 GMT

Olshansky's review of Cruella

Re: Software and Branding

Weblog on marginalia.nu — Tue, 21 Sep 2021 03:00:00 GMT

gemini://friends.riverside.camp/~clarity/journal/branding.gmi
gemini://idiomdrottning.org/re-branding

Some interesting thoughts going around on the topic of branding in software and websites. I’ve had thoughts like these too, and designed a lot of my website purposefully un-branded. I have no logos, no banners, barely navigational links. I figured I would just see what happens if you subvert this paradigm of web design, since it is stuff you typically just “scroll past” to get to what you care about like you skip the beginning of every youtube video. Since the point has never been to create a brand, I figured I would elevate the message so far above everything else that there simply is nothing else.

What does Blockbuster and misquoting Darwin have in common? - 123dev #16

Justin Garrison's Homepage — Tue, 21 Sep 2021 03:00:00 GMT

Click the image for the full length video. It’s worth it.

Comments

Survival

In the struggle for survival, the fittest win out at the expense of their rivals because they succeed in adapting themselves best to their environment.

You may think that quote is from Charles Darwin but it is actually a misquote. I’ll let you do your own homework to find the true author.

Enumerating and analyzing 40+ non-V8 JavaScript implementations

Notes on software development — Tue, 21 Sep 2021 03:00:00 GMT

V8 is, I'm sure, the most used implementation of JavaScript today. Used in Chrome, (and by extension) Microsoft Edge, Node.js, etc. Safari's JavaScriptCore and Firefox's SpiderMonkey are also contenders for extremely mainstream implementations.

But what else is out there? What if I want to embed JavaScript in a C program, or a Go program, or a Rust program, or a Java program(, and so on)? Or what if I want to run JavaScript on a microcontroller? Or use it as a base for language research? It turns out there are many high-quality implementations out there.

This post describes a number of them and their implementation choices. I'm not going to cover V8, JavaScriptCore, or SpiderMonkey because they are massive and hide multiple various interpreters and compilers inside. Plus, you already know about them.

I'm going to miss some implementations and get some details wrong. Please Tweet or email me with your corrections! I'd be particularly interested to hear about pure-research; and commercial, closed-source implementations of JavaScript.

Corporate-backed

These are implementations that would make sense to look into for your own commercial, production applications.

On the JVM

Oracle's GraalJS: compiles JavaScript to JVM bytecode or GraalVM
- Support: Full compatibility with latest ECMAScript specification
- Implementation language: Java
- Runtime: GraalVM or stock JDK
- Parser: Hand-written
- First release: 2019?
- Notes: Replaced Nashorn as the default JavaScript implementation in JDK.
Mozilla's Rhino: interprets and compiles JavaScript to JVM bytecode
- Support: ES6
- Implementation language: Java
- Runtime: Both interpreted through custom bytecode VM and interpreted after compiling to JVM bytecode as an optimization
- Parser: Hand-written
- First release: 1998?
- Notes: Replaced by Nashorn as the default JavaScript engine on the JVM, but remains actively developed.
Oracle's Nashorn: compiles JavaScript to JVM bytecode
- Support: ES5
- Implementation language: Java
- Runtime: compiles to JVM bytecode
- Parser: Hand-written
- First release: 2012?
- Notes: Replaced Rhino as default JavaScript implementation on JVM. Replaced by GraalJS more recently, but remains actively developed.

Embeddable

Nginx's njs
- Support: ES5
- Implementation language: C
- Runtime: Bytecode VM
- Parser: Hand-written
ChowJS: proprietary AOT compiler based on QuickJS for game developers
- Support: everything QuickJS does presumably (see further down for QuickJS)
- Implementation language: C presumably
- Runtime: QuickJS's bytecode interpreter but also an AOT compiler
- Parser: QuickJS's presumably
- First release: 2021
- Notes: Code is not available so exact analysis on these points is not possible at the moment.
Artifex's mujs
- Support: ES5, probably
- Implementation language: C
- Runtime: Bytecode stack-oriented VM
- Parser: Hand-written
- First release: 2017?
- Notes: Originally part of MuPDF viewer, but now broken out. Thanks to @rwoodsmalljs for mentioning!

Embedded Systems

Samsung's Escargot
- Support: ES2020
- Implementation language: C++
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2017?
Espruino
- Support: parts of ES5, ES6, ES7/8
- Implementation language: C
- Runtime: Seems like direct recursive interpreting without an AST/intermediate form
- Parser: Hand-written
- First release: 2012?
Cesanta's Elk
- Support: subset of ES6
- Implementation language: C
- Runtime: Direct recursive interpreter without AST or bytecode VM
- Parser: Hand-written
- First release: 2019?
- Notes: It does all of this with a GC and FFI in <1400 lines of readable C code. Damn.
Cesanta's mJS
- Support: subset of ES6
- Implementation language: C
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2017?
Moddable's XS
- Support: ES2018
- Implementation language: C
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2017?
- Notes: More details at https://www.moddable.com/XS7-TC-39 and https://www.moddable.com/faq#what-is-xs.

Other

Facebook's Hermes
- Support: ES6 with exceptions
- Implementation language: C++
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2019?
Qt's V4
- Support: ES5
- Implementation language: C++
- Runtime: Bytecode VM and JIT compiler
- Parser: qlalr custom parser generator
- First release: 2013
- Notes: Unclear if can be run standalone outside of Qt.

I don't know whether to put Microsoft's ChakraCore into this list or the next. So I'll put it here but note that as of this year 2021, they are transitioning it to become a community-driven project.

Microsoft's ChakraCore
- Support: ES6, probably more
- Implementation language: C++
- Runtime: Bytecode VM and JIT on x86/ARM
- Parser: Hand-written
- First release: 2015?

Mature, community-driven

Implementations toward the top are more reliable and proven. Implementations toward the bottom less so.

If you are a looking to get involved in language development, the implementation further down on the list can be a great place to start since they typically need work in documentation, testing, and language features.

Fabrice Bellard's QuickJS
- Support: ES2020
- Implementation language: C
- Runtime: Bytecode VM (this is a single large file)
- Parser: Hand-written (this is a single large file)
- First release: 2019
DuktapeJS
- Support: ES5, some parts of ES6/ES7
- Implementation language: C
- Runtime: Bytecode VM
- Parser: Hand-written, notably with no AST. It just directly compiles to its own bytecode.
- First release: 2013
engine262
- Support: 100% spec compliance
- Implementation language: JavaScript
- Runtime: AST interpreter
- Parser: Hand-written
JerryScript
- Support: ES5
- Implementation language: C
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2016?
Serenity's LibJS
- Support: Progressing toward compliance
- Implementation language: C++
- Runtime: Bytecode VM
- Parser: Hand-written
- Notes: Might also work outside of Serenity but documentation on building/running it on Linux is hard to find.
goja: JavaScript interpreter for Go
- Support: ES5
- Implementation language: Go
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2017?
otto: JavaScript interpreter for Go
- Support: ES5
- Implementation language: Go
- Runtime: AST interpreter
- Parser: Hand-written
- First release: 2012?
- Notes: The AST interpreter-only implementation might suggest this implementation is slower than Goja. I don't have benchmarks for that.
Jurassic: JavaScript parser and interpreter for .NET
- Support: ES5
- Implementation language: C#
- Runtime: Compiles to .NET
- Parser: Hand-written
- First release: 2011?
Jint
- Support: ES5, most of ES6/7/8
- Implementation language: C#
- Runtime: AST interpreter
- Parser: Hand-written via Esprima.NET
- First release: 2014?
- Notes: Thanks fowl2 for mentioning!
NiL.JS
- Support: ES6
- Implementation language: C#
- Runtime: AST interpreter
- Parser: Hand-written
- First release: 2014?
Neil Fraser's JS-Interpreter
- Support: ES5
- Implementation language: JavaScript
- Runtime: AST interpreter
- Parser: Hand-written, uses Acorn
- First release: 2014?
BESEN: Bytecode VM and JIT compiler in Object Pascal
- Support: ES5
- Implementation language: Object Pascal
- Runtime: Bytecode VM with JIT for x86 and x86_64
- Parser: Hand-written
- First release: 2015?

These last few are not toys but they are also more experimental or, in AssemblyScript's case, not JavaScript.

boa: JavaScript interpreter for Rust
- Support: Unclear
- Implementation language: Rust
- Runtime: Bytecode VM
- Parser: Hand-written
- First release: 2019?
AssemblyScript
- Support: Subset of TypeScript
- Implementation language: AssemblyScript subset of TypeScript
- Runtime: webassembly
- Parser: Hand-written
JavaScript in Kawa Scheme
JavaScript in GNU Guile Scheme
ReevaJS
- Support: ES5 (with exceptions)
- Implementation language: Kotlin
- Runtime: Stack machine
- Parser: Hand-written

Research Implementations

Higgs
- Support: Unclear
- Implementation language: D
- Runtime: Bytecode VM and JIT compiler on x64
- Parser: Hand-written
eJS
- Support: Unclear
- Implementation language: Java
- Runtime: Bytecode VM
- Parser: ANTLR
- Notes: eJS is a framework to generate JavaScript VMs that are specialised for applications.
Jessie: safe subset of JavaScript non-exploitable smart contracts
- Support: some subset of ES2017
- ???
- See https://github.com/agoric-labs/jessica for more info.
https://github.com/b9org/b9
https://www.defensivejs.com/

Thanks to @smarr for contributing eJS, Higgs, and b9!

Notable Abandoned

DMDScript
- Support: Unclear
- Implementation language: D
- Runtime: Bytecode VM
- Parser: Hand-written
- Notes: It's possible this is commercially maintained by DigitalMars but I'm not sure. There are also references in this repo to another C++ implementation of DMDScript that may be commercial. Thanks to @moon_chilled for mentioning!
EchoJS
- Support: Unclear
- Implementation language: JavaScript
- Runtime: Native through LLVM
- Parser: Hand-written via Esprima
twostroke
- Support: Unclear
- Implementation language: Ruby
- Runtime: Bytecode VM
- Parser: Hand-written
PyPy-JS
- Support: Unclear
- Implementation language: RPython
- Runtime: RPython
- Parser: EBNF parser generator
js.js
- Support: Unclear
- Implementation language: JavaScript
- Runtime: Too scared to look at the gigantic files in this repo.
- Parser: Ditto.
IronJS
- Support: ES3
- Implementation language: F#
- Runtime: .NET through DLR, I think.
- Parser: Hand-written
jispy
- Support: Unclear
- Implementation language: Python
- Runtime: AST interpreter
- Parser: Unclear
JE: Pure-Perl JavaScript Engine
Dave Herman's JavaScript for PLT Scheme

Notable toy implementations

Great for inspiriration if you've never implemented a language before.

js-to-c: A JavaScript to C compiler, written in C
mjs: AST interpreter for not just ES5 or even ES3 but also ES1
gojis: AST interpreter in Go
tojs: Bytecode VM in Rust
v2: Bytecode VM in Go
SparrowJS: AST interpreter in C++
jsc: My own experiment compiling JavaScript to C++/libV8

New post is up! Enumerating and analyzing 40+ non-V8 JavaScript implementations; of course with links to source code and parser & runtime/backend decisions.

I hope you enjoy learning about JavaScript engines as much as I did. 😁https://t.co/dEX06WU38f pic.twitter.com/AoYScphG6m
— Phil Eaton (@phil_eaton) September 21, 2021

Digital murder? We knew it was coming

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 20 Sep 2021 21:00:00 GMT

Don’t say you were not warned.

Why do Webdevs keep trying to kill REST?

swyx's site RSS Feed — Mon, 20 Sep 2021 17:46:38 GMT

How I broke out of the tiresome REST vs GraphQL debate - by realizing that the real battle is between Smart Clients and Smart Servers!

How to replace estimations and guesses with a Monte Carlo simulation

Lucas F. Costa - Blog — Mon, 20 Sep 2021 15:00:00 GMT

There are many ways of estimating how long a software project will take. All of them are a waste of time. Some prefer to spend days analysing and planning their changes so that they can estimate mor...

Structural pattern matching in Python 3.10

Ben Hoyt's technical writing — Mon, 20 Sep 2021 10:30:00 GMT

A critical but informative look at the new structural pattern matching feature in Python 3.10, with real-world code examples.

Paper Summary: Defending Against Neural Fake News

Luke Salamone's Blog — Mon, 20 Sep 2021 04:13:09 GMT

Defending Against Neural Fake News by Zellers et al. presents a model for controllable text generation called Grover. This model can be used to create highly believable computer-generated news articles. The authors present this paper as a method of detecting and preventing the spread of fake news. They claim their model is 92% accurate at detecting fake news stories, partially due to artifacts that generators include in the generated text.

How to Evaluate Startup Offers - A Beginner's Guide

Avy Faingezicht — Mon, 20 Sep 2021 03:00:00 GMT

TL;DR, here is a sample spreadsheet you can duplicate to evaluate your offers. Este ensayo también está disponible en español. I recently decided to leave my big tech job at Apple, where I had been for 6 years, to join a startup called Vouch (we're hiring!). As I was considering the switch, multiple people told me I was making a big mistake, losing out on both job stability and compensation. Those people had no idea what they were talking about. If you're joining a startup for money, you're doing it for the wrong reasons, but that doesn't mean that money shouldn't factor into your decision. You can build financial models around salaries and equity, and I'll show you how to do that below, but remember your job experience will give you _intangibles_ that you can't represent in a spreadsheet. A FAANG offer will win on sheer numbers every time. If you have flexibility, pick your next job based on how it can help you get to where you want to end up. Only you can tell whether you value the stability of a brand name or learning how to navigate a bureaucracy more or less than seeing a new business come to life or experiencing hypergrowth. The people you'll work with and the network you'll build matter a lot, too. Make sure you have a clear north star, or you'll have a much harder time making a choice. For example, when I chose to join Vouch, I tried to maximize my learning about the venture ecosystem and how startups operate. If you are considering taking a role at a startup, the framework below should give you basic tools to compare startup offers, but it doesn't pretend to give you exact answers. ### 1. How venture capital funding works When you hear that a startup has raised a round with an eye-popping valuation, all that means is that there was a venture capital (VC) firm willing to buy a portion of their business for cash, and the startup was willing to take their deal. The figures they agree to imply the company's new valuation. Startups use other financial instruments to raise capital, but the standard venture fundraise is simply that: an inflow of dollars matched by the issue of new shares of **preferred stock**, creating value almost out of thin air. To illustrate the mechanics, let's imagine[^1] a company raising a seed round from a single firm: * Before the deal, the startup has 10M shares outstanding. * A VC firm values the company at $5M _pre-money_ and agrees to invest $1M on that price. This effectively gives the company a _post-money_ valuation of $6M. * The valuation implies a price for the company's outstanding shares. Since _pre-money_ the 10M shares are worth $5M, the price is $5M / 10M shares = $0.50/share * In exchange for the $1M, the company will issue new shares and hand them to the VC. The number of shares issued is mostly a matter of accounting: ($1M) / ($0.50/share) = 2M new shares. Now the company has 10M + 2M = 12M shares outstanding. * At this point, an additional set of **common shares** could be issued to create/restock the employee option pool. These shares would also factor into pricing but we'll ignore them to keep the analysis simple. For shareholders, the fundraise is usually good news. However, they now own a smaller piece of a larger pie. When your percentage of ownership is decreased via the issuance of new shares you're experiencing **dilution**. If you were the lucky holder of 100,000 shares, you used to own 1% of the company. After this round, you only own 0.83%, and future rounds of funding will further dilute your shares. There are many nuances to startup funding[^2], but the basics described above on how equity is priced and how those prices evolve throughout a company's life are a big part of the puzzle. Most startups fail, but if you're lucky, you'll join one of the few that is not worth $0 in the long run. ### 2. Compensation package structure and options On top of your salary and maybe a signing bonus, startups offer equity as part of their compensation. Employee ownership is ingrained in the startup mentality for a reason. Stock compensation aligns incentives, translating employees' hard work to potential shared gains. At the same time, it allows companies to offer the prospect of a big payout in the future in exchange for a smaller paycheck today. While bigger companies might offer stock in the form of RSUs, which are functionally similar to the public company shares you could buy on a brokerage account, smaller startups give their employees stock options instead. These securities come in many different flavors, mostly differing on tax rules, but to keep things simple we'll assume we're talking about incentive stock options (ISOs). An ISO gives the holder the right to buy a share of common stock in the future, at a prespecified price. This is called the **strike price**, and it comes from an independent appraisal of the fair market value (FMV) of the stock. It is interchangeably called the **409a valuation**, after the section in the IRS code that regulates it. The strike price you're offered will always be a fraction of the most recent preferred share price, set so that the current value of common stock makes your options nominally worth $0, ensuring no tax events are triggered when the employee receives them. Using the example from before, the $0.50 preferred stock price paid by the investors might translate to a strike price of $0.15[^3]. At this price, when a prospective employee is offered $50k in equity she'd get a grant for 100,000 options. If the company does poorly, those options might end up being underwater and worthless, but let's assume the company does well, and that a few years down the line their shares are priced at $50 (a 100x increase!). This lucky employee has the right to **exercise** her options, redeeming shares by paying $15,000 ($0.15 * 100,000 options) to get $5M worth of stock ($50 * 100,000 shares). But we're getting ahead of ourselves. Even before the equity gets to appreciate, it needs to **vest**. Big equity numbers sound exciting to a hypothetical new-joiner, but getting a grant upon signing her offer doesn't mean that equity will immediately be transferred to her. Once again this is about incentive alignment. If she immediately got her equity, she could quit the next day. To ensure people are committed for the long run, the industry-standard vesting schedule spreads the distribution over four years, with a one year **cliff**. During the first year of employment, the employee doesn't get any equity. After hitting the cliff at the one year mark, the first 25% of the grant vests, and every month after that another 2.0833% (75% over 36 months) hits her account. The terms discussed above vary a bit from company to company, and some might be cagier than others about giving up this information during negotiation. Two valuable terms to ask about are early exercise and exercise windows[^4]. In essence, at the time you leave the company, you'll have a limited amount of time to shell out cash to turn your vested options into shares. The standard window used to be 90 days after leaving, although some companies have started to change to favor employees with multi-year windows. These terms will have a big impact on net compensation. Transparency in valuation and prices is a signal of a well-run operation, an open culture, and a healthy business. If they don't share those numbers it might mean things aren't going that well. A dollar number on an equity package means nothing without the broader context. You should always ask for specific numbers for: * Total shares outstanding * Last preferred price * 409a/strike price * Number of options in your grant Armed with these values, and setting aside a lot of complexity about taxation, we can move on to the calculations that will allow us to compare various offers. ### 3. Expected value If you have taken a basic probability class, feel free to skip this section. Expected value is what it sounds like: a way to measure the potential outcome of something that has an unknown value today — such as the valuation of a startup in 5, 10, or 20 years. Let’s use a coin flip as an example. If it comes up heads, you get $100. If it comes up tails, you get $0. The expected value (denoted as E[x]) is defined as the sum of the outcomes weighted by the probability that each one might occur: E[x] = (½ * $0) + (½ * $100) = $50 True, under this payout scheme no one bet over a coin flip will ever reward you with $50, but the anticipated value is still clearly given by this weighted sum. If you repeat the coin flip a thousand times on average you'd end up at $50, as the law of large numbers tells us. More complex events can be modeled similarly. If you toss a standard 6-sided die, the expected value to be rolled is 3.5, even if no roll can ever have its outcome suspended between 3 and 4: (⅙ * 1) + (⅙ * 2) + (⅙ * 3) + (⅙ * 4) + (⅙ * 5) + (⅙ * 6) = 3.5 Applying this idea, you can estimate the future value of anything: the expected rewards from pulling the arm on a slot machine, the return on buying a new house, the price of Apple stock tomorrow, or the potential payoff of your startup IPOing. The nuance here is that unlike a coin flip where the odds are well understood and discrete (half the time you'll see heads, and half the time tails) and the payoffs are well defined ($0 or $100) most interesting problems involve more uncertainty. With a slot machine, the player doesn't really know the payoff structure, nor the probability distribution, even though they're both baked into the game by its designer. In the housing, stock, and venture markets, not only are the payoffs and their probability distributions unknown to us, but they are not defined by anyone else. They are emergent phenomena. Uncertainty about uncertainty is the reason why there's a whole industry devoted to analyzing patterns in the noise. Just like there is expected value, there's also a way to measure expected risk, which I'll leave for a different blog post. ### 4. Discounted cash flow analysis If you have taken a basic finance class, feel free to skip this section. Evaluating any financial project requires accounting for the time value of money. On top of the uncertainty aspects discussed earlier, the payment schedule for most startups is asymmetrical: You’re accepting less compensation today for the promise of a potential payout later. Setting aside any liquidity constraints (e.g. wanting to save $X/year for a down payment, or needing $Y/month to pay your student loans and rent), we need some way to account for that time difference to calculate the real value of any offer on the table. In short, the problem we're solving is that $1 today is worth more than $1 tomorrow. With $1 today, you can pay off $1 of debt, and avoid accruing interest on it; or you can put it in the bank and earn that interest yourself. The process of laying out projects side by side to compare their returns is called discounted cash flow analysis, or DCF. For example, assuming there's a bank willing to pay you a rate of r = 1% on your cash, you know that depositing $100 today means having $101 next year. The math also works the other way around, though, and we can see that if ($100 today) * (1 + 1%) = $101 next year then $100 today = ($101 next year) / (1 + 1%) Abstractly, $X today = $X next year/(1+r) Dividing by that factor is called discounting. You can use the same logic to discount further into the future, dividing each cash flow by (1+r) again each period. $X next year/(1+r) = ($X in two years/(1+r))/(1+r) = $X in two years/(1+r)^2 The further out you go the more the discounting will eat away at the value of those future cash flows. Picking the right value for r is a science of its own, but in some ways it represents the alternative uses you'd have for that cash if you had it today. For example, let's imagine two startups. One offers you a higher starting salary and steady pay increases, while the other lets you know they won't give you a raise after your first year, promising bigger payouts down the line.

	Year 1	Year 2	Year 3	Year 4
Offer A	$110	$115	$120	$125
Offer B	$100	$100	$120	$150

If we choose r = 5%, for example, we can discount those cash flows to get to each offers' net present value (NPV): NPV(A) = $110 + $115/(1+r) + $120/(1+r)^2 + $125/(1+r)^3 = $415.57 NPV(B) = $100 + $100/(1+r) + $120/(1+r)^2 + $150/(1+r)^3 = $413.01 If we naively add up each stream of payments they both add up to $470, but the pay structure is very different and (merely financially) you'd be better off taking A's offer than B's. The NPV you end up with depends entirely on the value you pick for r. ### Evaluating your offer Now let's put it all together, getting into scenario and sensitivity analysis. Let's go back to hypothetical companies, but now instead of just assuming sure exits, let's model out these payments based on expected values, including dilution and exercise costs. This is where scenario analysis comes in. You can guesstimate the odds of potential outcomes based on what you know about the market - just make sure to always err on the side of being too negative, not too optimistic. You can find a spreadsheet you can copy and edit to plug in your own values here. ![Equity Spreadsheet](https://cdn.faingezicht.com/evaluating_startup_offers/equity_tool.png "Equity Spreadsheet") The values in the light blue represent offer terms that will be given to you by the person you're negotiating with (this is why transparent terms matter). The values in red can be filled-in based on what you learned in the interviews, and your own research about the market. How big can this startup really get? Lastly, the values in yellow are rough rules of thumb around dilution which will depend on what round the company is currently on, and overall market sentiment. You can build a table like this one to model scenarios for the offers you have at hand. For simplicity, the sheet linked above makes assumptions across offers that don't need to hold. For example, all these fictional startups exit after 4 years, but you could model uncertainty around that as well. Take the sheet as a template and update it to fit your needs. You could further extend the calculations to include dilution from additional funding rounds, secondary sales, refresh/evergreen grants, etc. Ultimately, what we're looking for is to boil down each offer to a set of future expected cash flows, which we can then discount at different rates. In the sample linked above, for example, Foo Inc gives a very nice salary and bonus up front but meager exit potential, while Bar.ly and Baz.io's compensation weighs a lot more on the potential exit. On paper, it looks like Bar.ly isn't offering very compelling compensation. ![NPV Sensitivity Analysis](https://cdn.faingezicht.com/evaluating_startup_offers/NPV_sensitivity_analysis.png "NPV Sensitivity Analysis") Only your gut can tell you which one feels right from here on. VCs get to make many small bets with their fund. That gives them a lot more leeway to be wrong on multiple counts while still turning a profit. When picking your new job you are putting all your eggs in one basket, so you better pick one where the bottom is unlikely to drop out. This is why choosing a startup just for the financial possibilities is a bad idea. If you have a couple of options that check the boxes on what you're looking for, and you're equally excited about them, only then it comes down to the financials. If the equity is worth $0, will you have learned the things that you want to learn? Will you meet the right people? Will you work closely enough with the leadership team? These won't show up in any financial model, but probably matter a lot more in the long arc of your career. ### A few thoughts on negotiating Most startups are willing to trade equity for cash. If you need to pay off student loans, or are otherwise cash-strapped, you can ask for more cash and less equity. If you can live on a smaller paycheck, and truly believe in the vision, you can ask for more equity and take on that risk. If you feel like you have leverage, asking for two offers, one with high equity and low salary and vice versa, can give you a hint at how they value their own stock. Asking for more equity is often easier than more cash, depending on their funding and option pool situation, but having both offers can give you an idea of what an all-cash offer would look like for this startup, which is a valuable data point in itself. I'll leave the rest of my negotiating advice for a different post, but if you're looking for help on that front I'd recommend you read Patrick McKenzie's guide. Do you have your own modeling suggestions, negotiation tips, startup advice, general feedback, or any questions? I'd love to hear it. _This is the third in a series I’m writing about my recent career transition out of Apple. See part one, on things I learned in my time at Apple, and part two, on how I picked my role at Vouch. Make sure to check back in a few weeks for upcoming pieces on **things to ask in interviews with startups**, and **tips on how to make your job transition as smooth as possible**, etc._ Thanks to Hannah Doherty, Max Faingezicht, Moi Stern, and Sandeep Paruchuri for their feedback on drafts of this post.

Photo: Big Wheel, Las Vegas, by me. Previously posted on Vax'd Travel.

[^1]: Lifted from Y Combinator's fundraising guide [^2]: If you want to scratch the surface, Brad Feld's Venture Deals is the book to get started. [^3]: This implies a FMV of 30% of the latest round's valuation. While the appraisal is a much more complex process, depending on the round, ⅕ to ⅓ of the preferred price is a standard rule of thumb to follow. [^4]: Both of these are out of scope for this blog post, but the former can have a big impact on your taxes, while the latter might make your options worth even less than you thought if your financial situation doesn't allow you to acquire your stock.

Writing things from scratch and containers - 123dev #15

Justin Garrison's Homepage — Mon, 20 Sep 2021 03:00:00 GMT

Comments

Planning work

I’m sure many of you saw this gif in various places recently. There’s lots of similarities that are obvious from a software engineering perspective. To me it shows how difficult a re-write really is.

Estimating

“Most people overestimate what they can achieve in a year and underestimate what they can achieve in ten years.”

The same is true for what I can achieve in an afternoon and what I can achieve in a week.

Art Sunday: Untitled artwork I plan to print out and hang above the fireplace

Michael Honey: Writing - michaelhoney.com — Mon, 20 Sep 2021 01:55:16 GMT

Speedrunning Computer Games History with a 6yr Old - Part 4

duncanlock.net — Mon, 20 Sep 2021 00:04:34 GMT

Figure 1. Our hand-drawn map of the Kingdom of Hyrule. I drew a 2x1 grid in Inkscape and printed it out, then we filled it in as we went along.

It’s been about six months since the last update. We’ve mostly been camping, playing outside and generally enjoying the summer - although we have also squeezed in some gaming, here and there.

Favourite Games So Far

We spent almost all of our game time playing Legend of Zelda. It’s a towering achievement – a truly great game. I’m amazed that this thing is from 1986 – it doesn’t feel like it.

It’s also a great example of a different kind of co-op game. One where you play together by collaborating: discussing what to do & where to go next, how to tackle things, etc… We also took turns with the controller, with me often doing the “hard bits”.

Figure 2. Oh crap, Darknuts.

We got pretty stuck for a few weeks trying to get past the blue Darknuts in the 5th level dungeon. I almost caved and was considering editing the save game file, rather than having the kid lose interest – but it didn’t come to that.

We eventually completed it in the traditional, time-honoured way – sitting on the sofa in our pyjamas on Saturday morning, banging our heads against the level, until we eventually cracked it.

There was less reading in Zelda than I was hoping - but what little there was, the kid did, which was good practice. I did most of the cartography, but the kid did do some.

It was great fun and also, I think, the kid learnt something about being persistent and working towards longer term objectives.

Updates & Changes to the System

2021-09-04

I thought it was about time we started on some adventure games – partly to help with reading practice, partly because they’re great.

ScummVM

Maniac Mansion (1987)

2021-09-18

We tentatively started having play-dates again - two so far. In preparation for this, I added some more co-op games. All of these games are 2 to 4 player co-op games, except for Gods & Populous, which I just wanted to play.

These are a little but of a leap in time from where we were, but I think we need to speed up anyway, so…

NES

Amiga [1]

I’ll report back later…

We’ve started on Maniac Mansion and that went OK, considering that it’s a new type of game that the kid hasn’t tried before - and I’m making him do all the reading 😉.

We’ve also played a few games of Bomberman, which is as ever, loads of fun. I’ll report back later on how we get on.

Footnotes & References

The motherload of amiga games, in WHDL format. ↩

Note-Making on Physical Books in Drafts

trv.rs — Sun, 19 Sep 2021 20:21:00 GMT

Drafts 28 has been released and with it the new Scan Document feature. Even before I got my hands on it, I knew that this feature is huge… this is the missing link in getting notes and quotes from physical books into my digital Creativity System.

The simplest version is already done: simply use the Scan Documents feature. From a new draft, hold the + Button in the top left to select Scan Documents. Scan your document, and hey presto! The image is OCR'd and brought into a new draft.

Of course this is Drafts, and so more automation is possible. You can use the "Booknotes" family of Actions I wrote to build a "shelf" of books you are currently reading, then link scanned Drafts to your main booknote with an MLA styled link… or just use the "Insert MLA Citation" action I created to create a reference to your scanned note. It's up to you!

I'm sure over the coming months the community will figure out even more clever ways to manage this, but I'm thrilled to have OCR in the Drafts toolbox. I'll probably start writing actions here to auto-fix some of the quirks of the scanning process… combining newlines into paragraphs, prefixing with > for quotes… and honestly it'll probably cause me to buy some more physical books.

I'm eyeing a couple of volumes I haven't started reading yet… excited to be able to capture notes and quotes without having to retype the whole passage.

Rejections, Resilience, and Rejoicing: Getting a Tenure-Track Position

Ecce Homology on Bastian Grossenbacher Rieck's personal homepage — Sun, 19 Sep 2021 17:00:00 GMT

Soon, a new chapter in my career as a researcher beckons: I will join the Institute of AI for Health, an institute of the Helmholtz Zentrum München, to lead my own research group, the AIDOS Lab. I intend to use this position to reify my vision of topology-based methods for addressing challenges in biomedicine and healthcare. In this post, I want to briefly detail the long, rocky, and certainly winding road that led me to this position.

This is what it felt like traversing academia at times. In the end, I hope we all arrive where we want to arrive, even if it might take quite long. (Photo by Jack Anstey on Unsplash)

But why?

The social media streams are chock full of people winning awards, prestigious grants, and getting coveted professorships. Most of us tend to craft an image of perpetual success, soaring ever higher. Few of us tend to discuss the path that led to these successes. Like the proverbial iceberg, we only observe a fraction of the things that were required to get to a certain point. I want to give you the ‘full scoop’ here, in the hope that people find inspiration in it. I will also use this post to express certain misgivings about the application process. As for myself, I certainly learned a lot while applying to tenure-track positions, and I intend to improve the system as much as I can.

Applications

Let’s start with the raw stats. Like Derek Sivers, I assume that I am below average, so your numbers will hopefully look better. Over the course of 2020, I applied to 20 different institutions¹ with 23 individual applications.² Here’s a summary of all the feedback I received, shown as a Sankey diagram that was built using the awesome SankeyMATIC service:

In the following, I will briefly discuss a few aspects of this diagram.

Getting an interview (or not)

About 30% of my applications led to an interview. I count such an invitation as a roaring success because it means that my application materials made the cut! If you are at this point, I would suggest that you briefly celebrate this achievement because it means that you are being noticed by potential employers.

At the same time, 30% of my applications led to a direct rejection. There are many potential reasons for a direct rejection:

I might have been too inexperienced for some positions. If the administrative bar for applying to a job is sufficiently low, I tend to send in an application anyway even though it might be a long shot. This is a delicate balance, though: I am strongly opposed to wasting time by applying to positions that are probably not attainable for me. For instance, if a position is looking for an established professor with more than 5 years experience leading their group, I am not (yet) the target demographic. This attitude sometimes surprises people, but I can only recommend to ‘know thyself and thy market value’ before investing too much time.
My research output might be considered insufficient for certain places. Some places required me to enter my bibliometric information, i.e. the number of papers, the number of citations, and my h-index, using a service called Web of Knowledge. This service is utterly useless and should be avoided at all costs. It does not even index most machine learning conferences, leading to an extremely skewed profile of anyone’s research (at least in the field of machine learning; I can only hope that it is better for other disciplines).

If you as a department feel you have to rely on such statistics, consider using Google Scholar. It is infinitely better at keeping track of publications.
Competition might have been too fierce (which is somehow related to my first point) with many excellent people applying to the same job, forcing the hiring committee to weed out applications rather strictly. I did my homework for all positions and only applied when there was a non-zero chance of success without divine intervention but I do not know who my competitors were.
My research agenda might not have been aligned with what a department was looking for. With many departments implementing a strategy for cutting-edge machine learning methods nowadays, in a diverse field such as ours, it is never clear whether your research goals are a perfect fit for a certain place. As someone with a rather unorthodox research vision, some departments might be reluctant to invest time in reviewing my application further. That’s okay!

Almost 40% of my applications, however, resulted in no reaction at all. Not even a rejection (but at least an acknowledgement of the receipt of my application materials, so I guess, in some weird quantum fashion, I might still be in the race for certain positions) letter. Of course, I believe that I am sufficiently smart to know that after six months, it is unlikely that I will hear back, but I must say that it is disappointing to jump through a lot of administrative hoops in preparing your application materials just to be ghosted in the end.

If you are responsible for the hiring in your department, please heed these words: every candidate, in my opinion, deserves at least a rejection letter (even a canned response is fine).

Lessons learned

For most of us, multiple applications are needed in order to get a positive response. Try to treat every invitation to an interview as a ‘win,’ because fundamentally, this means that your application materials were sufficiently good. Plan for rejection letters to be largely devoid of content. Due to legal reasons, no place will be willing or able to tell you why they rejected you. You might inquire about this through the grapevine, but you will never (well, hardly ever…) get an official justification.

Nevertheless, I always send a polite message thanking the organisers for considering my application at least. For some places, I asked about the number of candidates they invited and how many applications they received. This gave me a way to update my perception of my own chances at these jobs. In fact, I treated any such rejection as a ’near miss.’ For instance, one position received over 60 applicants, but only 6 were invited for an interview. I therefore—probably incorrectly, but full of vim and vigour—concluded that I was ‘obviously’ placed at rank 7, rather than rank 60.

I would suggest that everyone includes such statistics in their rejection letters. That way, we few, we happy few, at least get a rough feeling about our chances.

During the interview

The interview process itself was relatively standardised: a few slides are prepared in advance and later on, you have one-on-one interviews with faculty members. I enjoyed this part of the process the most because it forced me to sharpen my skills when it comes to discussing the merits of my own work. Since faculty members might not all be familiar with the particulars of what you are doing, this is a great opportunity to improve your pedagogical skills!

All in all, interviews were really enjoyable. I particularly appreciated the fact that all the places I interviewed at accommodated my time zone requirements when having virtual interviews. This really made the process less stressful.

When it comes to one-on-one interviews, I enjoy tough questions the most—this demonstrates that a person is genuinely interested in your research and wants you to convince them!

Don’t forget that these interviews are two-sided: you are applying at the department, but the department is also trying to convince you to join them, so make use of this time to ask all the questions you had in mind. My most favourite question to ask faculty members is ‘Given that you also started out on a tenure-track position at some point, what advice do you wish you would have been given?’ This question was universally appreciated by all, and I like it because it is rather open to interpretation, making it easy to segue into other subjects.

Lessons learned

Prepare your materials well in advance, of course, and be sure to know how the respective presentation app works. I would also suggest to invest in good lighting and a good microphone, but this is more for the convenience of your listeners. I doubt that it makes a large difference in the outcome but a good microphone at least makes it a joy to listen to someone.

If you are not the interviewee but part of the department, please make sure to follow common virtual conference etiquette and keep your microphone muted when not talking. I had few instances in which I could hear faculty members doing something else in the background while I was talking (such as listening to a soccer game at full volume). I do not expect people to listen to my talk or engage with my ideas, but at least be a decent ’listener’ and mute yourself, please!

When participating in such a virtual meeting, I always appreciated the places that had dedicated persons for making sure that the candidate is in the right meeting, their audio works, etc. This can make all the difference between an average interviewing experience and a stellar one!

After the interview

As you can see from my own stats above, I received three offers in total, but also two rejections. A rejection at this stage usually means that they found someone whose research agenda is an even better fit than yours. For instance, one place was looking for someone with a better track record of industry collaborations; my profile was slightly leaning towards ‘blue skies research,’ so I was not the perfect fit. It is part of the interviewing experience that you learn about your own fit only during the interview, so rejections are also par for the course.

I always make sure to write honest ’thank you’ notes because everyone invests a considerable amount of time into interviewing you. In some cases, I received additional feedback along the lines of ‘Your profile is great, but it is not what we are looking for right now.’ I appreciate this type of feedback a lot because it proves that I did a good and convincing job.

To reiterate what I wrote above: given the diversity of research directions in our field, it is often unclear what a particular candidate brings to the table. The interview stage can be very helpful in that regard, and I certainly do not want to be a part of a department where I would not be a good fit. The idea is to build something splendid here, and if my goals and those of the other department members are fundamentally misaligned, life is bound to become rather tedious.

Concerning the offers that I received, I was of course over the moon for each and every one! It is so great to see that my work paid off. Notice that I did not get all the offers at the same time—I therefore had to think and deliberate quite a bit before sending in my final answer.

If you are part of a departmental search committee, it might seem like a foolish or ungrateful move for a candidate to reject an offer. However, I can honestly stress that I had valid reasons for each rejection. I can only speak for myself here, of course, but I wanted to ensure that the position would be an excellent fit for me, enabling me to pursue my research agenda and build a group that does marvellous science. If I felt that it would be impossible for me to fully commit—for whatever reason—I turned an offer down (in the spirit of open and honest communication). I rather want to commit to one thing that makes me say ‘Hell yeah!’ rather than being lukewarm about something.³

Parting thoughts

As you can see from my own case, it takes a certain amount of resilience—to stomach all the rejections—before you may finally rejoice. Make sure that you apply to positions that entice you even if you do not think you are the ‘perfect’ fit.⁴ However, also be realistic in your aspirations; every application requires you to invest some time, which could also be spent doing other things.

Be prepared for rejections, but also be prepared for some weird requirements (looking at you, Web of Knowledge…) or some weird conversations about your CV (at one of the places, I got criticised quite harshly for having ‘only’ top-tier ML conference publications, but not a lot of ML journal publications; I still don’t know what that was all about).

Finally, be prepared to have amazing in-depth discussion about your research. Even for the places that ultimately rejected my application, it was fantastic to be able to discuss my research agenda with such eminent scholars! A big ’thank you’ to everyone that had to deal with my applications. I am grateful for having come this far, and I look forward to an exciting future.

Until next time, dear reader, I wish you all the best for your own job search. ‘Screw your courage to the sticking place and you shall not fail!’

I shall refrain from naming all of them because this post should remain generic. The job market is in constant flux, and your profile for potential positions is probably a lot different from mine. ↩︎
Certain institutions had multiple positions at multiple departments, necessitating, you guessed it, multiple applications. ↩︎
I will maybe discuss the reasons for my own decision in another post. For now, let me just say that I am very excited about the new opportunities my upcoming position will entail. ↩︎
Remember that ’the perfect is the enemy of the good.’ ↩︎

Can Apple Watch Measure Your Body Weight? [Updated for 2022]

Misleading Metaphors — Sun, 19 Sep 2021 16:12:32 GMT

How does my apple watch know my weight?

Longboard Setup

Home on mfashby.net — Sun, 19 Sep 2021 12:20:49 GMT

Fuzzing Zig Code Using AFL++

ryanliptak.com — Sun, 19 Sep 2021 03:00:00 GMT

A stop-gap solution for fuzzing Zig code

Plain Org for iOS (a month later)

xenodium.com @alvaro — Sun, 19 Sep 2021 03:00:00 GMT

A month ago, I posted about my desire to bring org tasks/TODOs to iOS and make them quickly available from my iPhone.

Since then, I've received some great feedback, which I've been slowly chipping away at. My intent isn't so much to move my org workflow over to iOS, but to supplement Emacs while away from my laptop.

As of now, this is what the inline edit experience looks like:

If, like me, you prefer dark mode. The app's got ya covered:

Plain Org is not yet available on the App Store, but you can get a TestFlight invite if you send me an email address. Ping me on reddit, twitter, or email me at "plainorg" + "@" + "xenodium.com".

You can also check out progress over at the r/plainorg subreddit.

Against the Flood

Weblog on marginalia.nu — Sun, 19 Sep 2021 03:00:00 GMT

So hacker news apparently discovered my search engine, and really took a liking to the idea. Actually that’s a bit of an understatement, the thread has gotten 3.3k points and lingered on the front page for half a week. And I wasn’t planning for it to go quite that public yet. It has quietly been online for a while, but it was only very recently it started to feel like it was really coming together. It wasn’t perfect, there was still a lot of jankiness and limitations that could have been fixed with more time. The index was half the size it should have been. Someone discovered it and shared it. It took off like a rocket, and I’m still at a loss for words at the reception it’s gotten. I have received so many encouraging comments, emails, offers of collaboration, a few have even joined the patreon. I’ve been working through all the messages and I aim to reply to them all, but it takes time. I’m very grateful for all of this, since I half thought I was alone in this.

Catching the blur event on an element and its children

Muffin Man — Sun, 19 Sep 2021 03:00:00 GMT

Recently I implemented a fly out menu in React, and stumbled on the following problem - I had to catch a blur event on the menu, but it had multiple focusable children. When user is tabbing between these menu items, blur event is triggered every time on the parent, followed by the focus event on the next item. As I wanted to close the menu on blur, this would close it before user was able to get to the next menu item.

Solution is fairly simple and it is not React exclusive - it can be used with any other framework or vanilla JavaScript. The flow goes something like this:

Listen to a blur event on the parent element (in my case it was the menu). This will catch blur event on all of its children too.
In the handler, give browser time to focus the next element (by using requestAnimationFrame).
Check if the newly focused element is in our parent element.
If it is, do nothing, as focus hasn’t exited the parent yet.
If it is not, we left the parent element completely and it is safe to do our blur logic (in my case, it was closing the menu).

Code looks like this:

const handleBlur = (e) => {
  const currentTarget = e.currentTarget;

  // Give browser time to focus the next element
  requestAnimationFrame(() => {
    // Check if the new focused element is a child of the original container
    if (!currentTarget.contains(document.activeElement)) {
      // Do blur logic here!
    }
  });
};

React component #

I pulled out the logic and created a small React component:

const ChildrenBlur = ({ children, onBlur, ...props }) => {
  const handleBlur = useCallback(
    (e) => {
      const currentTarget = e.currentTarget;

      // Give browser time to focus the next element
      requestAnimationFrame(() => {
        // Check if the new focused element is a child of the original container
        if (!currentTarget.contains(document.activeElement)) {
          onBlur();
        }
      });
    },
    [onBlur]
  );

  return (
    <div {...props} onBlur={handleBlur}>
      {children}
    </div>
  );
};

and usage is pretty straight forward:

<ChildrenBlur
  onBlur={() => {
    doSomethingCoolOnBlur()
  }}
>
  <button>Button 1</button>
  <button>Button 2</button>
  <button>Button 3</button>
</ChildrenBlur>

Demo #

To see it live check the demo on Codepen:

I really like this technique, and I found it useful in multiple places like tooltips and dropdowns, where it can replace “outside click” listeners.

Rolling upgrades and three links I disagree with - 123dev #14

Justin Garrison's Homepage — Sun, 19 Sep 2021 03:00:00 GMT

Comments

Rolling upgrades

The gif this week is the first of many I’ve made with a focus on teaching engineering concepts. It is part of an article I wrote for The New Stack explaining different patterns for Kubernetes cluster upgrades. The full article be available later this month.

With a little bit of shared knowledge you know exactly what is happening in that gif with almost no words. I’ll still be sharing gifs I find funny or intriguing in the newsletter, but creating and sharing my own gifs was a desire from day 0.

Reluctance to Bear a Symbol

Lambda Land — Sun, 19 Sep 2021 03:00:00 GMT

I feel uncomfortable with many symbols. I might have opinions about a subject, but there’s rarely a camp that has some symbol, flag, slogan, etc. that I’m comfortable with adopting because that camp does not accurately reflect my opinion. All too often, a slogan takes on more than its surface meaning, and that can make using that slogan tricky.

Art Sunday: Omnispheric

Michael Honey: Writing - michaelhoney.com — Sun, 19 Sep 2021 02:05:37 GMT

Five ways to reduce variance in A/B testing

Bytepawn - Marton Trencseni — Sun, 19 Sep 2021 01:00:00 GMT

I use toy Monte Carlo simulations to demonstrate 5 ways to reduce variance in A/B testing: increase sample size, move towards a more even split, reduce variance in the metric definition, stratification and CUPED.

Maddy

Home on mfashby.net — Sun, 19 Sep 2021 00:05:32 GMT

Negative impacts of land digitization

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 18 Sep 2021 21:00:00 GMT

“Land is political, not technical”. Like all life, that is.

The Power of xdotool in X11

Greg Herlein — Sat, 18 Sep 2021 11:00:01 GMT

X11’s xdotool cli is snazzy, and solved an annoying window placement problem.

Just say no ⚽😑 - 123dev #13

Justin Garrison's Homepage — Sat, 18 Sep 2021 03:00:00 GMT

Comments

Do you know how to say no?

People often feel guilty for saying no to opportunities, but in reality you should say no when you really care about something but know you are not the best person. It doesn’t matter if you’re not interested, if you don’t have enough time, or you don’t have the right skills. If you understand saying no will make the project/event/collaboration/etc. more successful then you can better position yourself to say no and explain why.

Restart

Home on mfashby.net — Sat, 18 Sep 2021 01:41:39 GMT

Wiki-Loader: Loading Wikipedia into Neo4J

Joshua Rogers — Fri, 17 Sep 2021 15:00:00 GMT

In April, I put together a Python script that would load an XML export from Wikipedia into a local SQLite database because, once again, you never know when you'll inadvertently become a time traveller who needs knowledge of future events to return home. (Look, it's more interesting than just saying “because I find it amusing”, so just roll with me here.) This script satisfied the typical time traveler's use-case of “finding a bit of topical information quickly”, yes, but that is only one of the many use-cases that chronological explorers must concern themselves with before traipsing around through spacetime.

Of supply chains, or why global is personal these days

Welcome to Marco Fioretti's website! on Stop at Zona-M — Fri, 17 Sep 2021 07:50:00 GMT

Global is personal. It really is.

Your code is on fire 🔥 - 123dev #12

Justin Garrison's Homepage — Fri, 17 Sep 2021 03:00:00 GMT

Comments

Shipping code that doesn’t catch fire

Sometimes I push code into production that I’m certain will work well. Not because of test coverage or PR reviews, but usually just because the change was so simple and obvious there couldn’t possibly be another way to do it.

Many times those pushes work fine and my false confidence is reinforced to do it again another day. Often times I end up paying for an undocumented quick fix later. Sometimes those guaranteed-not-to-fail code pushes catch fire and roll production down a hill. Those are good learning experiences.

Create your first NFT Marketplace for selling cars in PHP Laravel and web3.js

Adnan's Random bytes — Thu, 16 Sep 2021 21:45:41 GMT

If you are into Crypto then chances are you have heard about NFT, especially after the incidents like Beeple sold his art for $69 million dollars or this friendship meme. NFT is usually considered is suitable for arts like pictures only but this is not true. NFT is all about uniqueness, authority, and the transfer of ownership. We are going to use NFT for selling cars. The video demo of this project can be seen below: In this post, I will be discussing how to mint ERC721 NFT tokens and then integrate them with new or existing websites. The user who buys a car will pay the amount of the car being bought and the minting fee. I came across many tutorials regarding NFT or ERC721 tokens but none of them actually provided end-end solutions. It was quite frustrating because NFT itself is nothing unless it is integrated with an app. In this post, I am coming up with a fictitious car-selling portal site developed in PHP Laravel and web3.js. As I said in a previous post, that the backend framework or language does not matter. I will be writing this tutorial in a way that will highlight the NFT functionality and its integration. In case you wonder why I am using PHP not Python, well, it is just an attempt to break the taboo that blockchain programming is all about Javascript-based technologies. What is NFT? According to Wikipedia: A non-fungible token (NFT) is a unit of data stored on a digital ledger, called a blockchain, that certifies a digital asset to be unique and therefore not interchangeable.[1] NFTs can be used to represent items such as photos, videos, audio, and other types of digital files. Access to any copy of the original file, however, is not restricted to the buyer of the NFT. While copies of these digital items are available for anyone to obtain, NFTs are tracked on blockchains to provide the owner with a proof of ownership that is separate from copyright. NFT is quite a in demand these days. Though the majority of NFTs are built on Ethereum based blockchains but not limited to it. Blockchains like Solana and many others also offer NFT on their respective platforms. I am not going to cover the Laravel code as such since this is not the scope of this post neither I am going to cover Truffle/Ganache and smart contract integration via web3.js. You can read about it here. Website Development The backend of the website is developed in Laravel. MySQL has been used for storing all data and IPFS is a decentralized CDN for images. Pinata Cloud which provides the pinning service was actually used to upload all images on IPFS. The flow of the website is pretty simple. Currently, dApps are considered those who are hybrid in nature that is, a few features are on blockchain, especially the payment one, and the rest of the info is in centralized systems. A user signs up and uploads the details of the car he wants to sell. The image is uploaded via Pinata API for pinning. The API returns a hash which along with other information is stored in the MySQL database. The cars table structure looks like the below: The function that uploads the inventory looks like the below: Don’t be happy as these keys do not exist anymore 😉 The car once uploaded appears like the below image: As you see I have used VIN here. The reason to use it for to bring uniqueness to the product since NFT is all about uniqueness. NFT Smart Contract Currently, Ethereum offers two types of smart contracts standards: ERC721 and ERC1155. I am using the very basic ERC721 one since it serves the purpose well. I am using the one provided by OpenZeppelin. Using existing templates spare you to come up with your own faulty implementations while on the other hand smart contracts by OZ guys have been thoroughly audited by security companies. dCars NFT smart contract looks like the below: Notice that dCars is inheriting ERC721 contract provided by OZ guys. I am also inheriting Ownable for ownership and authorization purposes thus make it secure. You can learn more about it here. I am only implementing the minting feature because I am changing the default implementation by introducing Counters, rest of the functions that are already available to us. Thanks to our friend, Inheritence. The mintNFT function accepts a tokenURI parameter which is actually directing to the Metadata of the individual NFT. The Metadata looks like the below. This is actually the metadata that helps to discover your NFT on NFT marketplaces like Openseas. Web3js Integration Here web3.js is being used for two reasons: First, asking the user to pay the price of the car itself, and second the gas fee for minting purposes. I am not going into detail about charging ETH from your user’s Metamask since I discussed it here and minting the NFT which is actually calling mintNFT function which I have discussed here. payEther function looks like the below. And createNFT JS function which is actually calling smart contract function looks like the below: Both of these functions are actually invoked via the click event of the BUY button which after successful transaction stores info in relevant DB tables. After this the detail page will look like the below: Visitors can see who is the current owner of this car and its history. It is quite possible that the owner makes it to resell to others, for that transferFrom smart contract function will be called which is not implemented here. Conclusion So in this post, you learn what is an NFT and how as a developer you can implement it in existing or new web apps. This is not a complete project, features like reselling have not been implemented in it. Like always, the code is available on Github. If you like this post then you should subscribe to my blog for future updates. * indicates required Email Address *

The post Create your first NFT Marketplace for selling cars in PHP Laravel and web3.js first appeared on Adnan's Random bytes.

The American Style of quotation mark punctuation makes no sense

Nicola Iarocci — Thu, 16 Sep 2021 09:05:25 GMT

Years ago, I translated an essay by Terry Windling, On Tolkien and Fairie-Stories, from American English to Italian. I remember arguing with the author about her use of periods in quotations. Each quotation would end with a period before the closing mark. I was puzzled. We don’t do that in Italy. More importantly, I read many English texts where the period was left outside the quotation itself. She insisted that her style was correct¹.

Why and how Amazon Counter could be good for the world

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 16 Sep 2021 08:50:00 GMT

If it re-done right, that is. Back in early 2020 (*), a friend of mine asked online what people thought about Amazon Counter. My first reply greatly surprised him, because he knows my opinions about Amazon, and that reply started with: “This really sounds like a consumer’s dream, and a green one too." The conversation continued as follows (edited for clarity). Friend: Did you call it green because it would enable ‘cosmo-local’ production, that is small-scale manufacturing of products designed once and then reused globally?

Practice, handling errors, and some python 🐍 - 123dev #11

Justin Garrison's Homepage — Thu, 16 Sep 2021 03:00:00 GMT

Comments

Practice

I’ve been on a diet and I’m pretty sure this cat has better sit-up form than I have. Just like anything we do it takes practice and if we don’t do it for a while we fall out of form. As much as I exercised in high school I have to relearn a lot after not doing much for the past 20 years.

Top 5 areas where senior architects can make the biggest difference

Stories by Billy Lo on Medium — Wed, 15 Sep 2021 22:46:30 GMT

Photo by Lex Photography from Pexels

Think of your technology systems as a car: a tool that your company’s leaders use to get to their destination.

When designing and building this car, critical elements must receive sufficient focus. Here is my Top 5 list.

The engine and the drivetrain: The operating platform must be reliable and adaptive so we can drive on highways, city roads and suburb trails. Cloud native is definitely the prominent trend here.
Speedometer and brakes: To be able to drive fast without going out of control, we need to have good sensors and control systems. In the context of software systems, these are your application performance & management (APM) components.
Drivers’ manuals and training: Drivers come from different backgrounds and we cannot assume they understand how the car can be used optimally. Systems are the same. Education, communication and coaching are high value areas. Architects can make a big difference because of their strong understanding on how systems support business processes.
GPS and on-dash displays: Road conditions are dynamic and we cannot stick to the same route every time. Analytics and reporting components is always worth your senior architects’ time. We need to get to the right place after all.
Driver’s comfort: The journey is measured in years not minutes. If you want your system to be successful, it has to feel comfortable. But how do you architect systems that feel right? There is no magic formula here. Ensure you have a good feedback gathering mechanism: something that you can use to make adjustments. The ability to observe how applications are used (e.g. tools such as HotJar, Google Analytics) is a good start.

You may notice that security is not on this list because it really is a given. If you have a Ferrari with no locks, you are not going to find it again for your 2nd trip. :-)

Focus means saying no to good, but less important things. I hope this can help you prioritize projects that need to be worked on.

Safe travels! 🚗 🚗 🚗

This is not the toilet I was looking for

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 15 Sep 2021 21:00:00 GMT

Never mind FACE recognition. Ass recognition is where it’s at.

Shang-Chi and the Legend of the Ten Rings

🦉 olshansky 🦁 — Wed, 15 Sep 2021 09:47:52 GMT

Olshansky's review of Shang-Chi and the Legend of the Ten Rings

advanced shell packaging: resholve YADM's nixpkg

travis a. everett — Wed, 15 Sep 2021 03:40:00 GMT

I've been working on resholve to improve Nix packaging for shell projects since early 2020, but the new v0.6.0 release is a decent leap forward in the fraction of Shell scripts and programs resholve can handle.

Adding Desktop Icons to sdkman managed packages

Random Notes by agilob — Wed, 15 Sep 2021 03:00:00 GMT

When you install package like jmc or VisualVM from sdkman, the installation is for your current user only, so it doesn’t create desktop shortcuts allowing you to start program from Gnome or KDE app launchers. Unfortunately this by design forces users to start application from command line, even when it’s a desktop application. To fix it, and be able to run jmc from Alt+F2 launcher you need to create your own desktop file like this:

Hidden work, systems design, compensation, and faster learning - 123dev #10

Justin Garrison's Homepage — Wed, 15 Sep 2021 03:00:00 GMT

Comments

Hidden work

I never thought about how donuts were filled. I should have guessed it was someone’s job, but I assumed it was automated. It makes me realize there’s a lot of work I never think about.

There is lots of work I do that no one else thinks about. This isn’t a bad thing. It’s important to view my work as a delightful filling to someone’s day. Hopefully, like this email is for you.

Facebook does opposite of what I said

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 15 Sep 2021 02:50:00 GMT

But hey, why should you bother? It works, does it not?

Max Size of Django message Data in Cookie Storage

Rob — Wed, 15 Sep 2021 00:46:14 GMT

I responded to a little challenge from Matt Layman of the Django Riffs podcast Episode 15, User Session Data. Toward the end of the episode, Matt asked what might happen if over 4k worth of text was passed into messages if the backend storage method was cookies? What I found was unexpected. First, you can… Continue reading

Test models thoroughly, test everything else less

Writing on Adam Craven — Tue, 14 Sep 2021 19:21:06 GMT

Model all state in an application

Writing on Adam Craven — Tue, 14 Sep 2021 19:04:58 GMT

Ensure UI components can be loaded separately from an application

Writing on Adam Craven — Tue, 14 Sep 2021 16:49:53 GMT

MediaWiki hotkeys (the useful ones)

River Writes - A MediaWiki Blog — Tue, 14 Sep 2021 10:15:21 GMT

A list of just the useful hotkeys/access keys in MediaWiki

New Solutions Creating Old Problems

Weblog on marginalia.nu — Tue, 14 Sep 2021 03:00:00 GMT

I’ve spent some time the last week optimizing how the search engine identifies appropriate search results, putting far more consideration into where and how the search terms appear in the page when determining the order they are presented.

Search-result relevance is a pretty difficult problem, but I do think the changes has brought the search engine in a very good direction.

A bit simplified, I’m building tiered indices, ranging from

Going back to the gym

Imran's Blog — Tue, 14 Sep 2021 03:00:00 GMT

The province that I am in lifted covid restrictions for the gym. I used to work out a lot before covid, during the pandemic I have co-opted more of a sedentary lifestyle.

I signed up for a new membership as soon as possible to the closest gym. The first workout (per body part) always sucks.

The weights that I am able to lift is far below what I used to be able to do
I am going to be sore for days after the workout and the worked out body parts will be quite useless

Despite the pain, it's a pleasant experience. It has a huge improvement on my mood, though going outside might also contribute to that. I am super hoping that the government does not lock things down now that vaccine administration is nearing 80ish percent.

My old gym goal used to be a 600 lb deadlift before I was 30. I think it might be doable to achieve by 35 (sitting pretty at 135 lbs at the moment).

Oh and in other happier news, the vertigo has gone (more like I just got used to it)! At the least it has not affected any of my workouts and I hope it remains that way.

Balance, free resources, lock-in, and whitepapers - 123dev #9

Justin Garrison's Homepage — Tue, 14 Sep 2021 03:00:00 GMT

Comments

Balance

Learning how to ride a bike requires you to balance in new ways. Riding a bike on a tight rope requires similar balance, and I can only image the perceived risks of failure makes it exponentially harder to focus.

Trying anything new will require some sort of balance. Physical, mental, time management, relationships all create a balance in our lives that can easily topple due to unexpected illnesses, changes in priority, or adding new plates to juggle. Balance is only achievable when we understand how we fall and reacting to the signals that indicate we are falling.

Abusing AWS Lambda to make an Aussie Search Engine

Ben E. C. Boyter — Tue, 14 Sep 2021 03:00:00 GMT

UPDATE 2025-05-19

Somewhat un-suprisingly AWS has closed the loophole that I was exploiting for this https://aws.amazon.com/service-terms/#30._AWS_Lambda and as such compiling data into your lambda for “free” storage is no longer allowed. As such bonzamate.com.au in its current iteration has been shut down.

Watch this space however! I have some ideas on how to bring it back, albeit in a different form.

TL/DR I wrote an Australian search engine. You can view it at bonzamate.com.au. It’s interesting because it runs its own index, only indexes Australian websites, is written by an Australian for Australians and hosted in Australia. It’s interesting technically because it runs almost entirely serverless using AWS Lambda, and uses bit slice signatures or bloom filters for the index similar to Bing. I also found out the most successful code I have ever written is PHP, despite never being a professional PHP developer.

Something meaningful about driverless, for a change

Welcome to Marco Fioretti's website! on Stop at Zona-M — Mon, 13 Sep 2021 21:00:00 GMT

This is refreshing, really.

Pico Ada Libraries Release 0.6.0

Jeremy Grosser — Mon, 13 Sep 2021 10:00:00 GMT

I’ve tagged release 0.6.0 of the rp2040_hal, pico_bsp, and pico_examples libraries for the Raspberry Pi Pico. Release notes are at the end of this page.

Things that are real

Tilting at Windmills — Mon, 13 Sep 2021 03:00:00 GMT

I just watched the documentary "Behind the Curve" on Netflix. It explores the Flat Earth movement and their efforts to prove that there is a…

Vertical auto scaling Daft Punk elliptic curves - 123dev #8

Justin Garrison's Homepage — Mon, 13 Sep 2021 03:00:00 GMT

Comments

One more feature

It’s difficult with software to know when to stop and release what you have. There’s always one more feature to create and one more bug to squash. Putting out software you know has bugs or doesn’t do everything you want is anxiety inducing but also freeing because it can only get better from here.

The Dig

🦉 olshansky 🦁 — Sun, 12 Sep 2021 12:04:36 GMT

Olshansky's review of The Dig

Demystifying AWS IAM Policies: Unraveling De Morgan’s Laws and S3 Buckets Policy

Brent's Blog — Sun, 12 Sep 2021 03:00:00 GMT

Before we get into it we need to review how Condition evaluation works:

How to learn and three projects to try - 123dev #7

Justin Garrison's Homepage — Sun, 12 Sep 2021 03:00:00 GMT

Comments

Company presentations and movie magic

I often watch conference talks from companies that appear to be complete internal products, but behind the scenes are more like the blackbird car in the gif above. Companies have high performance, purpose built solutions they create and then present them as the answer to everyone’s problem. This isn’t always a negative thing. You should be aware that their Ferrari may not have a windshield or AC.

No, THESE 9/11 losses are not Adobe's fault

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sun, 12 Sep 2021 02:50:00 GMT

No way they are. They are YOUR fault.

September 11th from Singapore

swyx's site RSS Feed — Sun, 12 Sep 2021 01:26:53 GMT

just some memories 20 years after 9/11.

Fun with speckle patterns

Anfractuosity — Sun, 12 Sep 2021 01:04:43 GMT

I recently came across a really fascinating video – https://www.youtube.com/watch?v=tYFLze9VwB0 where they make use of a micro laser projector to generate a speckle pattern on a surface and then photograph it with a DSLR. They then simply touch a surface with their hand/finger and then take a photo of the speckle pattern again after the […]

Not Just Bikes

voussoir.net/writing — Sat, 11 Sep 2021 07:56:04 GMT

Back to writing

[life]

Not Just Bikes

In May of 2021 I discovered the youtube channel Not Just Bikes, and immediately watched the whole thing. The experience was revelatory and painful.

NJB has shown me that American urban and road design took a nosedive during the postwar period, where the American Dream was to own a house with a yard, drive a car everywhere you go, and, for some reason, have nothing of utility around you. It's very easy to hop around a map of the US and find houses where the nearest grocery store is literally an hour's walk away.

So you drive, and you park in the parking lots. The parking lots are enormous because everybody's driving instead of walking. They're so enormous, they push all the actual destinations farther and farther apart from each other, so that walking becomes even more useless and you have to do even more driving. Then the parking lots have to get bigger.

It would be no big deal to visit several small stores for a few items each on foot, but nobody likes starting their car multiple times to visit multiple stores across an asphalt ocean, and finding a new parking space each time. So, shopping trips are usually single-destination only, preferring the Walmarts and Targets over the small, specialized, personalized, and/or more informed grocers and boutiques.

Our stroads are wide, straight, and multilaned, so that it's easy for us to go really really fast and crash into buildings, because those are our priorities. All other forms of transit infrastructure are designed first and foremost to avoid disrupting the fragile egos of the motorists:

Because heaven forbid we impose upon the helpless motorists inside their cushy, air conditioned, noise-proof boxes, with radio and navigation and heated seats, that require no physical effort to launch at speeds higher than humanity could have dreamed of just two hundred years ago. Don't make them wait at a crossing. Life is already so hard for them.

Here's my favorite NJB quote:

For centuries, cities were designed such that everybody had what they needed within walking distance. Once you got where you were going, you could access everything you needed to on foot. These are the kinds of places Americans go to on vacation.

How Suburban Development Makes American Cities Poorer

He says "centuries", but I think he means "since the dawn of civilization". Automobiles are less than 150 years old. You might recognize a number of beautiful, comfortable, enviable cities which are older than that. And you'll never find yourself thinking "commuting to work in ancient Rome must have been so difficult without a car!", because it wasn't. The car didn't become a necessity until we intentionally stamped out all of the alternatives.

In America, young people love to say that the DMV needs to be more aggressive with re-testing elderly people and revoking their drivers license because they cause more danger in traffic. And while that's true and probably a good idea, it doesn't get implemented because license revocation is a prison sentence and absolutely everyone knows it. In the cities of respectable nations, children and the elderly don't have any problems going to the shop or cafe on their own because it's only five minutes away from their front door. In the States, a car is compulsory for all travel unless you live in the very densest of areas. Without it, you're stuck. Taking away your grandmother's license is taking away her independence. A whole lifetime of stars-and-stripes 🦅 American 🦅 freedom and independence comes toppling down with the removal of that one stone. Then Grandma spends the rest of her life trapped in the house because there's nowhere to walk and not enough bus service and everything's too far for even an electric mobility scooter to be useful, and she whithers away into mental and physical atrophy and dies.

One of the first excuses that springs to peoples' minds as to why America must be car dependent is that it is so big. Yes, America is geographically big, and if you go looking you can find very isolated cities that would take a long time to travel between. But the width of the continent as a whole has absolutely nothing to do with the placement of essentials within a single city. Even people who own very large houses still keep the toilet paper within arm's reach of the commode.

The people who built the oldest parts of our country must not have known how big the continent was, because they put living quarters right on top of shops back then! If today's cities would suck in their gut and bring more mixed use back into the center of downtown, people could get all of their daily business done by walking instead of driving, regardless of the center-to-center distance between them and the next city over. It's a non-sequitur.

The phrase that's been in my mind for a while is "too stupid to live". It is something of a wonder that we Americans haven't already extinguished ourselves by forgetting how to breathe or eat. You may have heard that sloths can starve to death on a full stomach because their only source of food is practically indigestible. You'd think they'd figure it out and try something else.

When my family gathers around the dinner table, one topic that always comes up for conversation is the vehicular idiocy that was seen throughout the week. Red light runs, excessive speeding, drivers in the shoulder lane, crashes, and near misses. A majority of people are not to be trusted with the task of driving, but we force them to do it anyway because we, collectively, are too stupid to put together better systems that don't necessitate it. It is easier to concentrate our moments of frustration towards the individuals who make short-term mistakes like running red lights than it is to realize the frustration that should be directed at the makers of long-term, diffuse, and pervasive mistakes like building suburban sprawl and bulldozing transit options.

Another part of the problem is that it feels good to punish people. We are a sadistic species and we like to coerce people into crossing our boundaries so that we can feel justified in retaliating. It is not as fun to design boundaries that don't get crossed in the first place. We design roadways that encourage unsafe driving and then we put up signs justifying punishment. It's dissonance. Here's a big fat 50mph five lane road that runs straight for miles, but you'd better slow down to 25mph now because there's an elementary school here. Nevermind that the 25mph stretch of road is cut from the same cloth as the 50mph road — it's the posted sign that makes the difference. Here's a series of traffic lights that don't coordinate with each other, and will stop you over and over again just as soon as you get going, but you'd better not run any reds and you sure as hell better not get stuck in the intersection during a light change because boy I'm gonna, um, honk at you, yeah that's what I'm gonna do, and you're gonna deserve it. It is difficult to convince people that car crashes are a symptom of deep, systemic design flaws affecting average people of average driving skill, rather than the fault of uniquely stupid individuals. Which is not to say the average skill level is very high, but even the people who'd like to opt out do not have a choice.

There is a freeway near me that is currently undergoing expansion, which is proven to induce more demand. Intelligent species have known this for decades. The mbillions of dollars being used to put more cars on the road are not being used to improve walkability, bikeability, or reduce the need for driving in the first place by fixing our wack zoning and just allowing a grocery to open on more corners.

Our zoning is the first thing that must change before anything else can happen. American cities are designed with huge contiguous swaths of land zoned as residential, especially R1 which denotes single-family housing. This is the zoning that prevents you from walking to the shop because there are no shops. This is the zoning that requires you to exodus by car to do absolutely anything except admire your neighbors' yards. This is the zoning for which every American deserves forty lashes.

With mixed-use zoning we can allow destinations to exist among the living space instead of being wholly separate. We should start with small shops, cafes, and restaurants. The kinds of things that are needed often by the residents and won't draw excessive traffic from far away because the people who live far away have their own. This will give the NIMBYs less room to complain, though they will try anyway.

In the study of occupational safety, there is something called the hierarchy of controls. Commonly represented as an inverted pyramid, it suggests you should do everything you can to eliminate a hazard using the most effective methods available to you, and only proceed to less effective methods when absolutely necessary. In between each of these levels, you could insert the phrase "but if you can't, then, ...".

For example, an electrician wearing rubber gloves (PPE) could touch a live wire without too much risk, but it makes a lot more sense to turn the circuit off in the first place (Elimination) unless there are substantial justifications for working hot.

Applying this to suburban motorsports, we might find these levels correspond to:

Elimination: Do not ride any dangerous vehicle, just walk. But if you can't, then,
Substitution: Get ordinary motorists off the road because they have proven themselves untrustworthy; substitute them with buses operated by drivers with higher training standards, and trains which run on rails and are backed by substantial safety measures. But if you can't, then,
Engineering controls: Increase separation of motorways from other modes of transit. Add comprehensive networks of bicycle paths. But if you can't, then,
Administrative controls: Design roadways to be less conducive to speeding and reckless driving by making them less convenient to motorists. Raise pedestrian sidewalks so motorists must slow down, lest they wreck their suspension by speeding over it. Move traffic lights to the near side of the intersection. Bring destinations closer together with better zoning. But if you can't, then,
PPE: Mandate airbags and seatbelts.

Fail, fail, fail, fail, pass. Great job America, you got seatbelts in your cars. The only measure you've managed to accomplish so far is the least effective.

We are the sloths. We are too stupid to live. We'd rather challenge ourselves with the invention of self-driving cars and the manufacture of unspeakable quantities of batteries for EVs [1] than consider the possibility that maybe spending thousands of hours in traffic isn't an inherent part of life around which all else must bow down. You'd think they'd figure it out and try something else.

NJB has made me ashamed to be American. I am ashamed that a country with this much flagrant disregard for sense touts itself as the best nation in the world. I am ashamed that the closest we can get to eco-friendliness is to make our cars run on massive batteries and steer themselves, because simply getting out of the car was never considered an option. I am ashamed that these are my surroundings and that I did not wake up from this Matrix sooner.

One of the other big problems with cars, which I don't think NJB has mentioned yet, is that they turn everyone into anonymous metal bubbles. Even without tinted windows, cars provide very little person-to-person visibility, communication, and most importantly, empathy. It is easy to become hateful towards a driver on the road in a way you'd never be towards a person you bumped into on foot, because they are nothing more than a big hotwheel that's sitting in your way. Also because bumping into someone on foot doesn't normally incur thousands of dollars of property damage.

Tom Scott made a video about Peachtree City, Georgia [2]. They have a network of asphalt paths designed specifically for travel by golf cart, which I think is really cool and sets a good example for the rest of the country to follow. Small vehicles provide an intermediate option between bicycles and 3,000 pound automobiles, ideal for many of the around-town trips we have to make where a bike might not be enough:

Travelling with a child or multiple children, or a person with limited mobility.
Travelling moderate distances in unpleasant weather.
Shopping for things that are too big or awkward to carry on a bike.
More things that I currently don't know about because I don't ride bikes.

NJB has talked about microcars, and while the video is still excellent, these don't strike my fancy the same way the golf carts did. Most of the microcars shown in his video follow too much in the design language of regular cars, with their high pillars and rectangulish porthole windows.

If we were to adopt small vehicles in more areas, I'd like them to remain mostly open-air, or wrapped in clear glass only. Slower modes of travel allow everyone to be closer to each other, and this coupled with high person-to-person visibility will ease traffic interactions and lessen the visceral savage hate that leads to so many of our stupid, pointless crashes. A switch to small vehicles is an opportunity to solve multiple traffic problems at once — let's not squander it by following the same old metal-bubble design. I don't expect actual golf carts to be what sweeps the nation, I just want small vehicles about that size, plus some comforts like weatherproofing and some speakers.

(See also Micah Toll's glowing review of a 25 mph electric mini-truck from China: part 1, part 2)

Finally, we need more public transit and it needs to provide benefits that cars can't. Sometimes I'll pick a destination on Google maps and ask it to route by transit. For destinations that are within "day trip" distance, the results are usually three to five times as long as travel by car, blowing all possibility of making a day trip out of it. I tried sampling a few destinations from here to the other coast and transit never seems to break even, always 1.5× to 2.5× the travel time of the car.

I have very little experience riding transit, so I'm not sure how much timing overhead is to be expected, or if these numbers are Americanly bad. Naturally, some time overhead is worth it in exchange for not having to do the driving, and transit routes can never be as direct as driving there yourself. But I really was expecting the trains to go just a little bit faster to make up for it. Notwithstanding that at these distances you're better off enjoying stops along the way rather than gunning for the finish.

As it stands, it is undeniable that transit is always significantly slower and less practical for day-to-day travel than driving except in the very densest of cities, and only exists for those who have no other options. Improved transit could be a great benefit to our society and atmosphere [3], but only if it provides enough advantages to be desirable to a person who otherwise could drive a car.

I can also recommend City Beautiful. In the meantime, try breaking traffic waves.

Here's an exchange from How to Get Ahead in Advertising, a movie I'm surprised I hadn't heard of earlier:

- [Dennis, to his boil] I'd like to see you suffer.

- [Boil] A typically communist statement.

- I'm not a communist!

- Yes, you are. You want to take everyone's car away.

- I do not want to take anything from anyone. I want to give them the choice of something better.

- Oh, yes? What?

- Trains.

- Trains? Trains are no good. They're old-fashioned. I hate trains. They're rotten.

- Only because they don't consume. Only because they're already there and don't eat up more and more and more. That's why you hate them. That's why government hates them. That's why they're old-fashioned and rotten.

- You commies don't half talk a lot of shit.

- Shut up! I'm not a communist!

And here's a fun, angry comment from Hacker News user brailsafe:

The suburbs are ironically, but not so surprisingly filled with crazy assholes that feel like the world belongs to them, or otherwise have some absurd levels of entitlement. They feel like they own the sidewalks, or if there's no sidewalks, they own the street and everything on it or not on it in front of their house, they feel like they should control whether or not a marginally higher density building should be built down the street or whether a neighbor can paint their house a different colour. When there's no land left, they want more cul-de-sacs built for them because it's the suburban dream and fuck wetlands anyway, and when there's no social services left, they want new ones built for them at everyone else's expense and they don't really want to be part of a community; they want to be by themselves, on their fat asses, in front of their fucking TV, the only quality of which they understand is how big it is.

https://news.ycombinator.com/item?id=35106661

[1] I look forward to EVs replacing ICEs, but they are still not as good as just using fewer cars.

[2] not to be confused with...

[3] That's the place we keep our breathable air.

View this document's history

Contact me: writing@voussoir.net

If you would like to subscribe for more, add this to your RSS reader: https://voussoir.net/writing/writing.atom

Snow Lake

Sam Hooke — Sat, 11 Sep 2021 04:30:00 GMT

View of Snow Lake as you descend the ridge.

Summary

Snow Lake is incredibly popular, and for good reason. The hike is beautiful from start to finish, especially with the low lying blankets of mist that roll through the trees and across the lake. It’s barely an hour’s drive from Seattle, yet feels like remote, unspoilt wilderness.

Code Coverage for Zig with Callgrind

ryanliptak.com — Sat, 11 Sep 2021 03:00:00 GMT

Generating code coverage information for Zig tests/binaries using Callgrind

🔒 Creating IAM and handling exceptions - 123dev #6

Justin Garrison's Homepage — Sat, 11 Sep 2021 03:00:00 GMT

Comments

The weight of nothing

Sometimes we make decisions in software and design that appear to have no impact on what we’re building. After many years of unchecked decisions we may find our systems breaking from a million — seemingly weightless — snowflakes.

Future of open source

Open source is changing and so are the people and communities. What started as free software stored in a drawer back when people used to share computers has evolved into big business and companies trying to protect the type of open source that benefits them. A new generation of users and contributors are getting involved and have their own motives.

Graphene is the new plastic

Welcome to Marco Fioretti's website! on Stop at Zona-M — Sat, 11 Sep 2021 02:50:00 GMT

Not necessarily in the good ways.

Implicit Hiring Criteria

Dan Palmer — Sat, 11 Sep 2021 02:00:00 GMT

At Thread I’m involved in hiring engineers for frontend, backend and iOS roles. One of the things I have become more aware of as I have gained experience in hiring and interviewing is how my biases affect the outcomes of interviews. This is something I’m always trying to improve – to understand what biases I have, to mitigate their effects – and in the process I have found a mental model that has helped me.

How I Modified a Thule Bike Trailer Hitch for an E-Bike with a 12mm Axle

Mike Kasberg's Blog — Fri, 10 Sep 2021 19:00:00 GMT

Thule makes several popular bike trailers, including the Cadence and Chariot. The trailers come with the Thule ezHitch to attach easily to most quick-release or solid axle wheels, and Thule also sells an adapter to make the ezHitch fit on thru-axle wheels. This covers most common bicycle setups, but it leaves out many e-bikes, which often use 12mm solid rear axles. And that’s really unfortunate because towing a trailer is a great way to use an e-bike! I figured out how to modify a Thule ezHitch to fit on my e-bike with a 12mm solid axle, and I’ll show you how I did it below.

The Thule ezHitch is compatible with most solid axle bicycles, but most solid axle bicycles use a 3/8” (~10mm) axle. Larger rear axles are common on e-bikes because many e-bike designs apply torque against the rear axle and therefore rely on a sturdy solid axle, often with additional measures (such as anti-spin washers) to prevent the axle from rotating against the frame. For example, my Juiced CrossCurrent e-bike has an M12x1.25 rear axle and some Rad Power Bikes also have a 12mm rear axle. Unfortunately, Thule doesn’t offer any adapter that will fit a 12mm axle, so Thule trailers are incompatible with many e-bikes – unless you modify the hitch to make it work!

Do this at your own risk!

The following is not officially supported by Thule. It will almost certainly void your warranty, and could result in serious injury or death. I'm not recommending that you do this or that it's safe to do; I'm just showing you what I did.

Well, if the hole on the Thule ezHitch is too small, the obvious solution is to make the hole a little bigger – and that’s actually pretty easy to do. You can drill through stainless steel using a cobalt drill bit and an ordinary drill. I picked up a 1/2” cobalt drill bit for about $25 at Home Depot. 1/2” is 12.7mm, so this is slightly larger than the axle – but 1/2” is a common drill bit size and the extra 0.7mm gives just a little extra room for error. When researching this idea, I found a YouTube video on the RadCity eBike Vlog that shows a similar approach for a different trailer brand – you could watch that if you want to see what to expect.

When drilling steel, you should go slower than you would if you were drilling wood. Running the drill slowly will reduce the amount of heat that builds up, which will reduce damage to the drill bit. It took several minutes of drilling for me to get through the layer of steel. While you might be able to use a drill bit that isn’t cobalt, it will probably damage the drill bit. My cobalt drill bit seems to have held up great!

When you’re done drilling, the project is finished. The whole thing only takes about 5 minutes and you’re ready to install the hitch on your e-bike! The axle on my Juiced CrossCurrent was long enough that the extra space for the hitch wasn’t a problem. My hitch now fits perfectly on my e-bike, and pulling my son around in the trailer is a blast! It’s really nice to have the extra power from the e-bike when pulling the trailer up a hill or travelling longer distances – the e-bike with a trailer is a great combo and I’m glad I found a solution to make my e-bike work with a Thule trailer!

2024 Update

Some people are curious to know how this has held up for me over time. Well, it's been more than two years (probably 40+ rides), and the trailer hitch on the e-bike continues to work great for me!

Error Handling Links

Blog on blog.gnoack.org — Fri, 10 Sep 2021 17:54:54 GMT

This is a collection of interesting literature on the subject of error handling which I had collected to research my own blog post on the subject a while ago.

“A Philosophy of Software Design” by John Ousterhout has a significant section on error handling that I found worth a read. It takes a slightly different angle and gives some good examples on how to design programs in a way so that errors cannot happen, or only happen in the right places.
The Smalltalk-80 Blue book had the weirdest error handling mechanism: You’d call the error: method on your own object, and the interactive system pops up a dialog. Some Smalltalk errors are recoverable in unusual ways through the interactivity of the system - you can define methods on the go if your program calls a method that doesn’t exist… :)
Common Lisp has a “Conditions” system which is similar to exceptions in that it propagates up the stack, but it inserts a layer in the middle between handling (“catching”) and signaling (“raising”) exceptions. The middle layer can define recovery strategies which the handling layer can then choose from.

I found the Common Lisp stuff in the “Practical Common Lisp” book. I cannot claim that I managed to wrap my head around that one. The chapter is online at https://gigamonkeys.com/book/beyond-exception-handling-conditions-and-restarts.html . Apparently someone wrote a full book on the Common Lisp condition system recently, but I haven’t read that. https://amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X

Apart from that, the books I looked through were not very helpful. I feel that the Ousterhout approach of explaining this is a useful one, talking about ways to design programs so that the complications of error handling are reduced.

Other notable “error handling philosophies” I have only some links to…

https://python.org/dev/peps/pep-0020/ The Zen of Python (“errors should never pass silently”)
The Midori error model http://joeduffyblog.com/2016/02/07/the-error-model/ (Midori was an experimental OS developed at Microsoft)
My own research culminated in this article: https://blog.gnoack.org/post/error_handling/. In hindsight, I took a much too academic and prescriptive approach there, and so it doesn’t get read much. The article might be biased towards the “deployed and monitored” kind of software.

Another thing I heard people discuss is the rule to “act on an error in only one place”, but I failed to find a canonical source for it. (It’s the rule that is most commonly violated if someone is logging the same error at multiple layers in the stack, leading to log spam.)

Abstract Space

Writing on Adam Craven — Fri, 10 Sep 2021 13:53:24 GMT

The quest for a humidifier

Liza Shulyayeva — Fri, 10 Sep 2021 12:20:00 GMT

Humidifiers and I have a several-year-long history, in which I’ve considered countless humidifiers and tried…a couple. My problem with humidifiers is that I’m lazy. I do not want to scrub them clean. I don’t want to even wipe them after each refill. I don’t want to find yellow goo under the tank that I have to clean by hand.

Django Project Initialisation Process

nutcroft — Fri, 10 Sep 2021 03:00:00 GMT

When I start a new Django project I can never remember what are the steps to arrive at a functional boilerplate with one Django app, the simplest view to get started, and all dependencies set.

This is a documentation of this process.

Preparation

We assume our project’s name is earth.

We assume we have these installed:

bash
git
python3

Setup project

mkdir earth && cd earth
git init
python3 -m venv venv
source venv/bin/activate

Create `requirements.in`

cat << EOF > requirements.in
django
EOF

Create `requirements_dev.txt`

cat << EOF > requirements_dev.txt
pip-tools
ipdb
ipython
isort
flake8
black
coverage
EOF

Create `Makefile`

cat << EOF > Makefile
.PHONY: format lint cov

lint:
   flake8 --exclude=.git/,venv/ --ignore=E203,E501,W503
   isort --check-only --profile black .
   black --check --exclude '/(\.git|venv)/' .

format:
   black --exclude '/(\.git|venv)/' .
   isort --profile black .

cov:
   coverage run --source='.' --omit 'venv/*' manage.py test
   coverage report -m
EOF

Setup dependencies

pip install -r requirements_dev.txt
pip install --upgrade pip
pip-compile
pip install -r requirements.txt

Start django project

django-admin startproject earth .
django-admin startapp main

Reformat based on black

make format

Create `.gitignore`

cat << EOF > .gitignore
*.pyc
venv/
EOF

Initial commit

git add .
git commit -m "init django project"

Create `main/views.py`

cat << EOF > main/views.py
from django.http import HttpResponse


def index(request):
    return HttpResponse("ok")
EOF

Create `main/urls.py`

cat << EOF > main/urls.py
from django.urls import path

from main import views

urlpatterns = [
    path("", views.index, name="index"),
]
EOF

Edit `earth/urls.py`

Add the include import:

from django.urls import path, include

And the index path to include all main app urls:

path("", include("main.urls")),

Edit `earth/settings.py`

import os

SECRET_KEY = os.environ.get("SECRET_KEY", "nonrandom_secret")

DEBUG = True if os.environ.get("NODEBUG") is None else False

ALLOWED_HOSTS = [
    "localhost",
    "127.0.0.1",
]

INSTALLED_APPS = [
    "main.apps.MainConfig",
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]

STATIC_URL = "/static/"
STATIC_ROOT = os.path.join(BASE_DIR, "static")

Migrate and run

python manage.py migrate
python manage.py runserver

HTML, queues, and whiteboarding - 123dev #5

Justin Garrison's Homepage — Fri, 10 Sep 2021 03:00:00 GMT

Comments

HTML

There were some conversations on Twitter this past weekend that HTML is not a programming language. I don’t care about semantics, but I DO care to recognize people that write HTML as developers.

My first full-time job I wrote code to solve problems even though I didn’t have “developer” in my title. I automated a job that used to take 3 weeks down to 3 hours with the code I wrote. It just so happened that my IDE was Excel.

Prove them right! Not wrong.

unsungNovelty — Thu, 09 Sep 2021 14:36:37 GMT

License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 _International Public License (CC BY-NC-ND _4.0)

Conversations are important to learn and understand things better. It is one of the fundamental ways we learn and grow. It helps us to reach conclusions and validates our thoughts/ideas on what is “right” and what is “wrong”. But if you have observed, we end up having conversations which are pretty much one sided most times. Or it ends up being a debate where the last man standing wins. A productive conversation is getting hard to come by these days. Cos we have our own knowledge that defines what is right and what is wrong. That is not the problem though. The problem is that we are not ready to open up to the idea that we could be wrong about what is “right” and what is “wrong”.

So how could we understand our biases better? How can we open up to the idea that we could be wrong? How can we have more productive conversations? An easy way to test your biases is to try to talk to the person you are having conversations and prove them right! Thereby trying to prove yourself “wrong”. This puts your thoughts and ideas into testing and helps you listen. So instead of noding along until the person stops talking so that you can prove them wrong, you tolerate and hopefullly listen better about what the other person has to say.

This also makes it easy for you to callout your own biases easily in a conversation. To be very clear, this doesn’t mean you should defend everything the other person says though. It is just a simple test to check how biased you are when going into a conversation. Where the byproduct could be better listening and tolerance to opposite views.

This only works when you are open to the possibility of thinking that you could be wrong. So be open next time you are having a conversation. Try to prove them “right” when you are trying to have a conversation with someone. And maybe you would be able to uncover your biases. Or validate your thoughts. So, prove them right. Not wrong!

References / links

Have thoughts or feedback regarding my post/writing? DM me on Twitter @unsungNovelty.

One more proof that cyberspace does not exist? Great!

Welcome to Marco Fioretti's website! on Stop at Zona-M — Thu, 09 Sep 2021 11:00:00 GMT

So we can solve REAL problems, instead.

Learn Crystal with Practice Problems - Part 1: Variables Math and Strings

Bartlomiej Mika — Thu, 09 Sep 2021 06:38:40 GMT

Are you interested in learning #Crystal-Lang but having difficulty because you are a tactile learner? The purpose of this tutorial is to provide you practice problems you can try out at your leisure.

Error Handling in Go REST APIs

Alex Richey — Thu, 09 Sep 2021 03:00:00 GMT

A pattern for distinguishing and handling client and internal errors in Go that leverages the power of Go’s interfaces.

Docker Gripes

janissary — Thu, 09 Sep 2021 03:00:00 GMT

The Curious Case of the Dot-Com Link Farms

Weblog on marginalia.nu — Thu, 09 Sep 2021 03:00:00 GMT

I spent some time today weeding out yet more link-farms from my search engine’s index.

Typically what I would do is just block the subnet assigned to the VPS provider they’re on, and that does seem to work rather well. The cloud providers that don’t police what they host is almost always home to quite a lot of this stuff, so I don’t particularly mind scorching some earth in the name of a clean index.

📃 Make your resume stand out and write your docs with semantic line breaks - 123dev #4

Justin Garrison's Homepage — Thu, 09 Sep 2021 03:00:00 GMT

Comments

Open Source

There’s been a lot of open source news this week between my current employer and Elastic. I have nothing to do with it directly, but it reminds me that open source is people which means there will always be drama.

Even when it’s billion dollar companies it often comes down to a small number of people making decisions. I think this weeks drama hurt the image of open source more than it helped.

Promising news for right to repair from Germany

Welcome to Marco Fioretti's website! on Stop at Zona-M — Wed, 08 Sep 2021 21:00:00 GMT

New rules say something obvious, but never said enough.

Daft Punk's legendary Alive 2007 concert with remastered audio and 4K video

Nicola Iarocci — Wed, 08 Sep 2021 09:05:25 GMT

Someone did it –the perfect Daft Punk tribute after their break up. Also ideal for those hard-coding sessions.

For the first time in 14 years, experience Daft Punk like never before! The first professionally recorded live show; revived meticulously by remastering and restoring the audio and video. The Alive 2007 show is considered a staple in live performances featuring the signature ‘Pyramid’ and groundbreaking visuals, which is now complemented by a fully remastered audio track which aims to maximize the experience. Presenting the ultimate authentic Daft Punk experience.

The Small Website Discoverability Crisis

Weblog on marginalia.nu — Wed, 08 Sep 2021 03:00:00 GMT

There are a lot of small websites on the Internet: Interesting websites, beautiful websites, unique websites.

Unfortunately they are incredibly hard to find. You cannot find them on Google or Reddit, and while you can stumble onto them with my search engine, it is not in a very directed fashion.

It is an unfortunate state of affairs. Even if you do not particularly care for becoming the next big thing, it’s still discouraging to put work into a website and get next to no traffic beyond the usual bots.

Gradle run two tasks of the same type

Random Notes by agilob — Wed, 08 Sep 2021 03:00:00 GMT

I have this setup in a single project which handles backend and frontend generation of server and client code. This requires to run openapi-generator twice, once for backend with spring generator and once for frontend with typescript-angular generator. I need backend code to be generated to build directory - so it is not committed to version control. TypeScript code needs to be reformatted and committed to git. TypeScript code also requires additional mapping due to non-standard structure of my specification.

AWS as a framework, pair terminals, and bash script template - 123dev #3

Justin Garrison's Homepage — Wed, 08 Sep 2021 03:00:00 GMT

Comments

Learning to code

The more programming languages I learn the more it resembles human language for me. It takes a lot of practice to write content that humans want to read. It takes similar practice to write code that computers want to execute.

If I start writing in another language the meaning of the words might be the same, but how they will be interpreted by the readers could be completely different.

Book review & notes - Looking for Information

River Writes - A MediaWiki Blog — Tue, 07 Sep 2021 23:33:43 GMT

A review of, including chapter-by-chapter notes on, “Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior”

If reskilling is not quick enough, it's not COVID's fault

Welcome to Marco Fioretti's website! on Stop at Zona-M — Tue, 07 Sep 2021 21:00:00 GMT

It cannot be. It just can’t.

Learning loops

Minireference blog — Tue, 07 Sep 2021 17:34:11 GMT

I was talking with friends recently about an interesting phenomenon that all self-taught programmers have observed, which we ended up calling “learning loops.” A learning loop is a process in which learners are motivated to advance their knowledge thanks to the positive feedback on their performance. In this blog post, I want to look at […]

Connect Jupyter to Remote

Luke Salamone's Blog — Tue, 07 Sep 2021 17:10:56 GMT

Here’s how to connect to a remote Jupyter notebook.

Create an ssh tunnel to your remote machine:

ssh -L 8080:localhost:8080 user@12.34.56.78

# or use a .pem file to connect to ec2
ssh -L 8080:localhost:8080 -i "aws.pem" ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com

Start Jupyter on that machine in headless mode:

jupyter notebook --no-browser --port=8080

Use a browser to open one of the urls that Jupyter presents:
http://localhost:8080/?token=xyz

DIY cloud gaming setup with VFIO, Parsec and AMD

./techtipsy — Tue, 07 Sep 2021 06:00:00 GMT

A build log of how I set up my own 'cloud gaming' setup, including all the issues I faced and attempts to fix those.

The Toyota Way

june.kim — Tue, 07 Sep 2021 03:00:00 GMT

Comparing SIMD on x86-64 and arm64

Code & Visuals — Tue, 07 Sep 2021 03:00:00 GMT

1. Introduction
2. 4-Wide Ray Bounding Box Intersection
3. Test Program Setup
4. Shared SSE/Neon Structs

5. Scalar Implementations
6. SSE Implementation
7. Neon Implementation
8. Auto-Vectorized Implementation

9. ISPC Implementation
10. Final Results and Conclusions
11. Addenda
12. References

Introduction

I recently wrote a big two-part series about a ton of things that I learned throughout the process of porting my hobby renderer, Takua Renderer, to 64-bit ARM. In the second part, one of the topics I covered was how the Embree ray tracing kernels library gained arm64 support by utilizing the sse2neon project to emulate x86-64 SSE2 SIMD instructions using arm64’s Neon SIMD instructions. In the second part of the series, I had originally planned on diving much deeper into comparing writing vectorized code using SSE intrinsics versus using Neon intrinsics versus other approaches, but the comparison write-up became so large that I wound up leaving it out of the original post with the intention of making the comparison into its own standalone post. This post is that standalone comparison!

As discussed in my porting to arm64 series, a huge proportion of the raw compute power in modern CPUs is located in vector SIMD instruction set extensions, and lots of things in computer graphics happen to be be workload types that fit vectorization very well. Long-time readers of this blog probably already know what SIMD instructions do, but for the unfamiliar, here’s a very brief summary. SIMD stands for single instruction, multiple data, and is a type of parallel programming that exploits data level parallelism instead of concurrency. What the above means is that, unlike multithreading, in which multiple different streams of instructions simultaneously execute on different cores over different pieces of data, in a SIMD program, a single instruction stream executes simultaneously over different pieces of data. For example, a 4-wide SIMD multiplication instruction would simultaneously execute a single multiply instruction over four pairs of numbers; each pair is multiplied together at the same time as the other pairs. SIMD processing makes processors more powerful by allowing the processor to process more data within the same clock cycle; many modern CPUs implement SIMD extensions to their base scalar instruction sets, and modern GPUs are at a very high level broadly similar to huge ultra-wide SIMD processors.

Multiple approaches exist today for writing vectorized code. The four main ways available today are: directly write code using SIMD assembly instructions, write code using compiler-provided vector intrinsics, write normal scalar code and rely on compiler auto-vectorization to emit vectorized assembly, or write code using ISPC: the Intel SPMD Program Compiler. Choosing which approach to use for a given project requires considering many different tradeoffs and factors, such as ease of programming, performance, and portability. Since this post is looking at comparing SSE2 and Neon, portability is especially interesting here. Auto-vectorization and ISPC are the most easily portable approaches, while vector intrinsics can be made portable using sse2neon, but each of these approaches requires different trade-offs in other areas.

In this post, I’ll compare vectorizing the same snippet of code using several different approaches. On x86-64, I’ll compare implementations using SSE intrinsics, using auto-vectorization, and using ISPC emitting SSE assembly. On arm64, I’ll compare implementations using Neon intrinsics, using SSE intrinsics emulated on arm64 using sse2neon, using auto-vectorization, and using ISPC emitting Neon assembly. I’ll also evaluate how each approach does in balancing portability, ease-of-use, and performance.

4-wide Ray Bounding Box Intersection

For my comparisons, I wanted to use a small but practical real-world example. I wanted something small since I wanted to be able to look at the assembly output directly, and keeping things smaller makes the assembly output easier to read all at once. However, I also wanted something real-world to make sure that whatever I learned wasn’t just the result of a contrived artificial example. The comparison that I picked is a common operation inside of ray tracing: 4-wide ray bounding box intersection. By 4-wide, I mean intersecting the same ray against four bounding boxes at the same time. Ray bounding box intersection tests are a fundamental operation in BVH traversal, and typically account for a large proportion (often a majority) of the computational cost in ray intersection against the scene. Before we dive into code, here’s some background on BVH traversal and the role that 4-wide ray bounding box intersection plays in modern ray tracing implementations.

Acceleration structures are a critical component of ray tracing; tree-based acceleration structures convert tracing a ray against a scene from being a O(N) problem into a O(log(N)) problem, where N is the number of objects that are in the scene. For scenes with lots of objects and for objects made up of lots of primitives, lowering the worst-case complexity of ray intersection from linear to logarithmic is what makes the difference between ray tracing being impractical and practical. From roughly the late 90s through to the early 2010s, a number of different groups across the graphics field put an enormous amount of research and effort into establishing the best possible acceleration structures. Early on, the broad general consensus was that KD-trees were the most efficient acceleration structure for ray intersection performance, while BVHs were known to be faster to build than KD-trees but less performant at actual ray intersection. However, advancements over time improved BVH ray intersection performance [Stich et al. 2009] to the point where today, BVHs are now the dominant acceleration structure used in pretty much every production ray tracing solution. For a history and detailed survey of BVH research over the past twenty-odd years, please refer to Meister et al. [2021]. One interesting thing to note when looking through the past twenty years of ray tracing acceleration research are the author names; many of these authors are the same people that went on to create the modern underpinnings of Embree, Optix, and the ray acceleration hardware found in NVIDIA’s RTX GPUs.

A BVH is a tree structure where bounding boxes are placed over all of the objects that need to be intersected, and then these bounding boxes are grouped into (hopefully) spatially local groups. Each group is then enclosed in another bounding box, and these boxes are grouped again, and so on and so forth until a top-level bounding box is reached that contains everything below. In university courses, BVHs are traditionally taught as being binary trees, meaning that each node within the tree structure bounds two children nodes. Binary BVHs are the simplest possible BVH to build and implement, hence why they’re usually the standard version taught in schools. However, the actual branching factor at each BVH node doesn’t have to be binary; the branching factor can be any integer number greater than 2. BVHs with 4 and even 8 wide branching factors have largely come to dominate production usage today.

The reason production BVHs today tend to have wide branching factors originates in the need to vectorize BVH traversal in order to utilize the maximum possible performance of SIMD-enabled CPUs. Early attempts at vectorizing BVH traversal centered around tracing groups, or packets, of multiple rays through a BVH together; packet tracing allows for simultaneously intersecting N rays against a single bounding box at each node in the hierarchy [Wald et al. 2001], where N is the vector width. However, packet tracing only really works well for groups of rays that are all going in largely the same direction from largely the same origin; for incoherent rays, divergence in the traversal path each incoherent ray needs to take through the BVH destroys the efficacy of vectorized packet traversal. To solve this problem, several papers concurrently proposed a different solution to vectorizing BVH traversal [Wald et al. 2008, Ernst and Greiner 2008, Dammertz et al. 2008]: instead of simultaneously intersecting N rays against a single bounding box, this new solution simultaneously intersects a single ray against N bounding boxes. Since the most common SIMD implementations are at least 4 lanes wide, BVH implementations that want to take maximum advantage of SIMD hardware also need to be able to present four bounding boxes at a time for vectorized ray intersection, hence the move from a splitting factor of 2 to a splitting factor of 4 or even wider. In addition to being more performant when vectorized, a 4-wide splitting factor also tends to reduce the depth and therefore memory footprint of BVHs, and 4-wide BVHs have also been demonstrated to be able to outperform 2-wide BVHs even without vectorization [Vegdahl 2017]. Vectorized 4-wide BVH traversal can also be combined with the previous packet approach to yield even better performance for coherent rays [Tsakok 2009].

All of the above factors combined are why BVHs with wider branching factors are more popularly used today on the CPU; for example, the widely used Embree library [Wald et al. 2014] offers 4-wide as the minimum supported split factor, and supports even wider split factors when vectorizing using wider AVX instructions. On the GPU, the story is similar, although a little bit more complex since the GPU’s SIMT (as opposed to SIMD) parallelism model changes the relative importance of being able to simultaneously intersect one ray against multiple boxes. GPU ray tracing systems today use a variety of different split factors; AMD’s RDNA2-based GPUs implement hardware acceleration for a 4-wide BVH [AMD 2020]. NVIDIA does not publicly disclose what split factor their RTX GPUs assume in hardware, since their various APIs for accessing the ray tracing hardware are designed to allow for changing out for different, better future techniques under the hood without modification to client applications. However, we can guess that support for multiple different splitting factors seems likely given that Optix 7 uses different splitting factors depending on whether an application wants to prioritize BVH construction speed or BVH traversal speed [NVIDIA 2021]. While not explicitly disclosed, as of writing, we can reasonable guess based off of what Optix 6.x implemented that Optix 7’s fast construction mode implements a TRBVH [Karras and Aila 2013], which is a binary BVH, and that Optix 7’s performance-optimized mode implements a 8-wide BVH with compression [Ylitie et al. 2017].

Since the most common splitting factor in production CPU cases in a 4-wide split, and since SSE and Neon are both 4-wide vector instruction sets, I think the core simultaneous single-ray-4-box intersection test is a perfect example case to look at! To start off, we need an efficient intersection test between a single ray and a single axis-aligned bounding box. I’ll be using the commonly utilized solution by Williams et al. [2005]; improved techniques with better precision [Ize 2013] and more generalized flexibility [Majercik 2018] do exist, but I’ll stick with the original Williams approach in this post to keep things simple.

Test Program Setup

Everything in this post is implemented in a small test program that I have put in an open Github repository, licensed under the Apache-2.0 License. Feel free to clone the repository for yourself to follow along using or to play with! To build and run the test program yourself, you will need a version of CMake that has ISPC support (so, CMake 3.19 or newer), a modern C++ compiler with support for C++17, and a version of ISPC that supports Neon output for arm64 (so, ISPC v1.16.1 or newer); further instructions for building and running the test program is included in the repository’s README.md file. The test program compiles and runs on both x86-64 and arm64; on each processor architecture, the appropriate implementations for each processor architecture are automatically chosen for compilation.

The test program runs each single-ray-4-box intersection test implementation N times, where N is an integer that can be set by the user as the first input argument to the program. By default, and for all results in this post, N is set to 100000 runs. The four bounding boxes that the intersection tests run against are hardcoded into the test program’s main function and are reused for all N runs. Since the bounding boxes are hardcoded, I had to take some care to make sure that the compiler wasn’t going to pull any optimization shenanigans and not actually run all N runs. To make sure of the above, the test program is compiled in two separate pieces: all of the actual ray-bounding-box intersection functions are compiled into a static library using -O3 optimization, and then the test program’s main function is compiled separately with all optimizations disabled, and then the intersection functions static library is linked in.

Ideally I would have liked to set up the project to compile directly to a Universal Binary on macOS, but unfortunately CMake’s built-in infrastructure for compiling multi-architecture binaries doesn’t really work with ISPC at the moment, and I was too lazy to manually set up custom CMake scripts to invoke ISPC multiple times (once for each target architecture) and call the macOS lipo tool; I just compiled and ran the test program separately on an x86-64 Mac and on an arm64 Mac. However, on both the x86-64 and arm64 systems, I used the same operating system and compilers. For all of the results in this post, I’m running on macOS 11.5.2 and I’m compiling using Apple Clang v12.0.5 (which comes with Xcode 12.5.1) for C++ code and ISPC v1.16.1 for ISPC code.

For the rest of the post, I’ll include results for each implementation in the section discussing that implementation, and then include all results together in a results section at the end. All results were generated by running on a 2019 16 inch MacBook Pro with a Intel Core i7-9750H CPU for x86-64, and on a 2020 M1 Mac Mini for arm64 and Rosetta 2. All results were generated by running the test program with 100000 runs per implementation, and I averaged results across 5 runs of the test program after throwing out the highest and lowest result for each implementation to discard outliers. The timings reported for each implementation are the average across 100000 runs.

Defining Structs Usable with Both SSE and Neon

Before we dive into the ray-box intersection implementations, I need to introduce and describe the handful of simple structs that the test program uses. The most widely used struct in the test program is FVec4, which defines a 4-dimensional float vector by simply wrapping around four floats. FVec4 has one important trick: FVec4 uses a union to accomplish type punning, which allows us to access the four floats in FVec4 either as separate individual floats, or as a single __m128 when using SSE or a single float32x4_t when using Neon. __m128 on SSE and float32x4_t on Neon serve the same purpose; since SSE and Neon use 128-bit wide registers with four 32-bit “lanes” per register, intrinsics implementations for SSE and Neon need a 128-bit data type that maps directly to the vector register when compiled. The SSE intrinsics implementation defined in <xmmintrin.h> uses __m128 as its single generic 128-bit data type, whereas the Neon intrinsics implementation defined in <arm_neon.h> defines separate 128-bit types depending on what is being stored. For example, Neon intrinsics use float32x4 as its 128-bit data type for four 32-bit floats, and uses uint32x4 as its 128-bit data type for four 32-bit unsigned integers, and so on. Each 32-bit component in a 128-bit vector register is often known as a lane. The process of populating each of the lanes in a 128-bit vector type is sometimes referred to as a gather operation, and the process of pulling 32-bit values out of the 128-bit vector type is sometimes referred to as a scatter operation; the FVec4 struct’s type punning makes gather and scatter operations nice and easy to do.

One of the comparisons that the test program does on arm64 machines is between an implementation using native Neon intrinsics, and an implementation written using SSE intrinsics that are emulated with Neon intrinsics under the hood on arm64 via the sse2neon project. Since for this test program, SSE intrinsics are available on both x86-64 (natively) and on arm64 (through sse2neon), we don’t need to wrap the __m128 member of the union in any #ifdefs. We do need to #ifdef out the Neon implementation on x86-64 though, hence the check for #if defined(__aarch64__). Putting everything above all together, we can get a nice, convenient 4-dimensional float vector in which we can access each component individually and access the entire contents of the vector as a single intrinsics-friendly 128-bit data type on both SSE and Neon:

struct FVec4 {
    union {  // Use union for type punning __m128 and float32x4_t
        __m128 m128;
#if defined(__aarch64__)
        float32x4_t f32x4;
#endif
        struct {
            float x;
            float y;
            float z;
            float w;
        };
        float data[4];
    };

    FVec4() : x(0.0f), y(0.0f), z(0.0f), w(0.0f) {}
#if defined(__x86_64__)
    FVec4(__m128 f4) : m128(f4) {}
#elif defined(__aarch64__)
    FVec4(float32x4_t f4) : f32x4(f4) {}
#endif

    FVec4(float x_, float y_, float z_, float w_) : x(x_), y(y_), z(z_), w(w_) {}
    FVec4(float x_, float y_, float z_) : x(x_), y(y_), z(z_), w(0.0f) {}

    float operator[](int i) const { return data[i]; }
    float& operator[](int i) { return data[i]; }
};

Listing 1: FVec4 definition, which defines a 4-dimensional float vector that can be accessed as either a single 128-bit vector value or as individual 32-bit floats.

The actual implementation in the test project has a few more functions defined as part of FVec4 to provide basic arithmetic operators. In the test project, I also define IVec4, which is a simple 4-dimensional integer vector type that is useful for storing multiple indices together. Rays are represented as a simple struct containing just two FVec4s and two floats; the two FVec4s store the ray’s direction and origin, and the two floats store the ray’s tMin and tMax values.

For representing bounding boxes, the test project has two different structs. The first is BBox, which defines a single axis-aligned bounding box for purely scalar use. Since BBox is only used for scalar code, it just contains normal floats and doesn’t have any vector data types at all inside:

struct BBox {
    union {
        float corners[6];        // indexed as [minX minY minZ maxX maxY maxZ]
        float cornersAlt[2][3];  // indexed as corner[minOrMax][XYZ]
    };

    BBox(const FVec4& minCorner, const FVec4& maxCorner) {
        cornersAlt[0][0] = fmin(minCorner.x, maxCorner.x);
        cornersAlt[0][1] = fmin(minCorner.y, maxCorner.y);
        cornersAlt[0][2] = fmin(minCorner.z, maxCorner.z);
        cornersAlt[1][0] = fmax(minCorner.x, maxCorner.x);
        cornersAlt[1][1] = fmax(minCorner.y, maxCorner.y);
        cornersAlt[1][2] = fmax(minCorner.x, maxCorner.x);
    }

    FVec4 minCorner() const { return FVec4(corners[0], corners[1], corners[2]); }

    FVec4 maxCorner() const { return FVec4(corners[3], corners[4], corners[5]); }
};

Listing 2: Struct holding a single bounding-box.

The second bounding box struct is BBox4, which stores four axis-aligned bounding boxes together. BBox4 internally uses FVec4s in a union with two different arrays of regular floats to allow for vectorized operation and individual access to each component of each corner of each box. The internal layout of BBox4 is not as simple as just storing four BBox structs; I’ll discuss how the internal layout of BBox4 works a little bit later in this post.

Williams et al. 2005 Ray-Box Intersection Test: Scalar Implementations

Now that we have all of the data structures that we’ll need, we can dive into the actual implementations. The first implementation is the reference scalar version of ray-box intersection. The implementation below is pretty close to being copy-pasted straight out of the Williams et al. 2005 paper, albeit with some minor changes to use our previously defined data structures:

bool rayBBoxIntersectScalar(const Ray& ray, const BBox& bbox, float& tMin, float& tMax) {
    FVec4 rdir = 1.0f / ray.direction;
    int sign[3];
    sign[0] = (rdir.x < 0);
    sign[1] = (rdir.y < 0);
    sign[2] = (rdir.z < 0);

    float tyMin, tyMax, tzMin, tzMax;
    tMin = (bbox.cornersAlt[sign[0]][0] - ray.origin.x) * rdir.x;
    tMax = (bbox.cornersAlt[1 - sign[0]][0] - ray.origin.x) * rdir.x;
    tyMin = (bbox.cornersAlt[sign[1]][1] - ray.origin.y) * rdir.y;
    tyMax = (bbox.cornersAlt[1 - sign[1]][1] - ray.origin.y) * rdir.y;
    if ((tMin > tyMax) || (tyMin > tMax)) {
        return false;
    }
    if (tyMin > tMin) {
        tMin = tyMin;
    }
    if (tyMax < tMax) {
        tMax = tyMax;
    }
    tzMin = (bbox.cornersAlt[sign[2]][2] - ray.origin.z) * rdir.z;
    tzMax = (bbox.cornersAlt[1 - sign[2]][2] - ray.origin.z) * rdir.z;
    if ((tMin > tzMax) || (tzMin > tMax)) {
        return false;
    }
    if (tzMin > tMin) {
        tMin = tzMin;
    }
    if (tzMax < tMax) {
        tMax = tzMax;
    }
    return ((tMin < ray.tMax) && (tMax > ray.tMin));
}

Listing 3: A direct implementation of "An Efficient and Robust Ray-Box Intersection Algorithm" by Amy Williams et al. 2005.

For our test, we want to intersect a ray against four boxes, so we just write a wrapper function that calls rayBBoxIntersectScalar() four times in sequence. In the wrapper function, hits is a reference to a IVec4 where each component of the IVec4 is used to store either 0 to indicate no intersection, or 1 to indicate an intersection:

void rayBBoxIntersect4Scalar(const Ray& ray,
                            const BBox& bbox0,
                            const BBox& bbox1,
                            const BBox& bbox2,
                            const BBox& bbox3,
                            IVec4& hits,
                            FVec4& tMins,
                            FVec4& tMaxs) {
    hits[0] = (int)rayBBoxIntersectScalar(ray, bbox0, tMins[0], tMaxs[0]);
    hits[1] = (int)rayBBoxIntersectScalar(ray, bbox1, tMins[1], tMaxs[1]);
    hits[2] = (int)rayBBoxIntersectScalar(ray, bbox2, tMins[2], tMaxs[2]);
    hits[3] = (int)rayBBoxIntersectScalar(ray, bbox3, tMins[3], tMaxs[3]);
}

Listing 4: Wrap and call rayBBoxIntersectScalar() four times sequentially to implement scalar 4-way ray-box intersection.

The implementation provided in the original paper is easy to understand, but unfortunately is not in a form that we can easily vectorize. Note the six branching if statements; branching statements do not bode well for good vectorized code. The reason branching doesn’t go well with SIMD code is because with SIMD code, the same instruction has to be executed in lockstep across all four SIMD lanes; the only way for different lanes to execute different branches is to run all branches across all lanes sequentially, and for each branch mask out the lanes that the branch shouldn’t apply to. Contrast with normal scalar sequential execution where we process one ray-box intersection at a time; each ray-box test can independently choose what codepath to execute at each branch and completely bypass executing branches that never get taken. Scalar code can also do fancy things like advanced branch prediction to further speed things up.

In order to get to a point where we can more easily write vectorized SSE and Neon implementations of the ray-box test, we first need to refactor the original implementation into an intermediate scalar form that is more amenable to vectorization. In other words, we need to rewrite the code in Listing 3 to be as branchless as possible. We can see that each of the if statements in Listing 3 is comparing two values and, depending on which value is greater, assigning one value to be the same as the other value. Fortunately, this type of compare-and-assign with floats can easily be replicated in a branchless fashion by just using a min or max operation. For example, the branching statement if (tyMin > tMin) { tMin = tyMin; } can be easily replaced with the branchless statement tMin = fmax(tMin, tyMin);. I chose to use fmax() and fmin() instead of std::max() and std::min() because I found fmax() and fmin() to be slightly faster in this example. The good thing about replacing our branches with min/max operations is that SSE and Neon both have intrinsics to do vectorized min and max operations in the form of _mm_min_ps and _mm_max_ps for SSE and vminq_f32 and vmaxq_f32 for Neon.

Also note how in Listing 3, the index of each corner is calculated while looking up the corner; for example: bbox.cornersAlt[1 - sign[0]]. To make the code easier to vectorize, we don’t want to be computing indices in the lookup; instead, we want to precompute all of the indices that we will want to look up. In Listing 5, the IVec4 values named near and far are used to store precomputed lookup indices. Finally, one more shortcut we can make with an eye towards easier vectorization is that we don’t actually care what the values of tMin and tMax are in the event that the ray misses the box; if the values that come out of a missed hit in our vectorized implementation don’t exactly match the values that come out of a missed hit in the scalar implementation, that’s okay! We just need to check for the missed hit case and instead return whether or not a hit has occurred as a bool.

Putting all of the above together, we can rewrite Listing 3 into the following much more compact, more more SIMD friendly scalar implementation:

bool rayBBoxIntersectScalarCompact(const Ray& ray, const BBox& bbox, float& tMin, float& tMax) {
    FVec4 rdir = 1.0f / ray.direction;
    IVec4 near(int(rdir.x >= 0.0f ? 0 : 3), int(rdir.y >= 0.0f ? 1 : 4),
            int(rdir.z >= 0.0f ? 2 : 5));
    IVec4 far(int(rdir.x >= 0.0f ? 3 : 0), int(rdir.y >= 0.0f ? 4 : 1),
            int(rdir.z >= 0.0f ? 5 : 2));

    tMin = fmax(fmax(ray.tMin, (bbox.corners[near.x] - ray.origin.x) * rdir.x),
                fmax((bbox.corners[near.y] - ray.origin.y) * rdir.y,
                    (bbox.corners[near.z] - ray.origin.z) * rdir.z));
    tMax = fmin(fmin(ray.tMax, (bbox.corners[far.x] - ray.origin.x) * rdir.x),
                fmin((bbox.corners[far.y] - ray.origin.y) * rdir.y,
                    (bbox.corners[far.z] - ray.origin.z) * rdir.z));
                    
    return tMin <= tMax;
}

Listing 5: A much more compact implementation of Williams et al. 2005; this implementation does not calculate a negative tMin if the ray origin is inside of the box.

The wrapper around rayBBoxIntersectScalarCompact() to make a function that intersects one ray against four boxes is exactly the same as in Listing 4, just with a call to the new function, so I won’t bother going into it.

Here is how the scalar compact implementation (Listing 5) compares to the original scalar implementation (Listing 3). The “speedup” columns use the scalar compact implementation as the baseline:

	x86-64:	x86-64 Speedup:	arm64:	arm64 Speedup:	Rosetta2:	Rosetta2 Speedup:
Scalar Compact:	44.5159 ns	1.0x.	41.8187 ns	1.0x.	81.0942 ns	1.0x.
Scalar Original:	44.1004 ns	1.0117x	78.4001 ns	0.5334x	90.7649 ns	0.8935x
Scalar No Early-Out:	55.6770 ns	0.8014x	85.3562 ns	0.4899x	102.763 ns	0.7891x

The original scalar implementation is actually ever-so-slightly faster than our scalar compact implementation on x86-64! This result actually doesn’t surprise me; note that the original scalar implementation has early-outs when checking the values of tyMin and tzMin, whereas the early-outs have to be removed in order to restructure the original scalar implementation into the vectorization-friendly compact scalar implementation. To confirm that the original scalar implementation is faster because of the early-outs, in the test program I also include a version of the original scalar implementation that has the early-outs removed. Instead of returning when the checks on tyMin or tzMin fail, I modified the original scalar implementation to store the result of the checks in a bool that is stored until the end of the function and then checked at the end of the function. In the results, this modified version of the original scalar implementation is labeled as “Scalar No Early-Out”; this modified version is considerably slower than the compact scalar implementation on both x86-64 and arm64.

The more surprising result is that the original scalar implementation is slower than the compact scalar implementation on arm64, and by a considerable amount! Even more interesting is that the original scalar implementation and the modified “no early-out” version perform relatively similarly on arm64; this result strongly hints to me that for whatever reason, the version of Clang I used just wasn’t able to optimize for arm64 as well as it was able to for x86-64. Looking at the compiled x86-64 assembly and the compiled arm64 assembly on Godbolt Compiler Explorer for the original scalar implementation shows that the structure of the output assembly is very similar across both architectures though, so the cause of the slower performance on arm64 is not completely clear to me.

For all of the results in the rest of the post, the compact scalar implementation’s timings are used as the baseline that everything else is compared against, since all of the following implementations are derived from the compact scalar implementation.

SSE Implementation

The first vectorized implementation we’ll look at is using SSE on x86-64 processors. The full SSE through SSE4 instruction set today including contains 281 instructions, introduced over the past two decades-ish in a series of supplementary extensions to the original SSE instruction set. All modern Intel and AMD x86-64 processors from at least the past decade support SSE4, and all x86-64 processors ever made support at least SSE2 since SSE2 is written into the base x86-64 specification. As mentioned earlier, SSE uses 128-bit registers that can be split into two, four, eight, or even sixteen lanes; the most common (and original) use case is four 32-bit floats. AVX and AVX2 later expanded the register width from 128-bit to 256-bit, and the latest AVX-512 extensions introduced 512-bit registers. For this post though, we’ll just stick with 128-bit SSE.

In order to program directly using SSE instructions, we can either write SSE assembly directly, or we can use SSE intrinsics. Writing SSE assembly directly is not particularly ideal for all of the same reasons that writing programs in regular assembly is not particularly ideal for most cases, so we’ll want to use intrinsics instead. Intrinsics are functions whose implementations are specially handled by the compiler; in the case of vector intrinsics, each function maps directly to a known single or small number of vector assembly instructions. Intrinsics kind of bridge between writing directly in assembly and using full-blown standard library functions; intrinsics are higher level than assembly, but lower level than what you typically find in standard library functions. The headers for vector intrinsics are defined by the compiler; almost every C++ compiler that supports SSE and AVX intrinsics follows a convention where SSE/AVX intrinsics headers are named using the pattern *mmintrin.h, where * is a letter of the alphabet corresponding to a specific subset or version of either SSE of AVX (for example, x for SSE, e for SSE2, n for SSE4.2, i for AVX, etc.). For example, xmmintrin.h is where the __m128 type we used earlier in defining all of our structs comes from. Intel’s searchable online Intrinsics Guide is an invaluable resource for looking up what SSE intrinsics there are and what each of them does.

The first thing we need to do for our SSE implementation is to define a new BBox4 struct that holds four bounding boxes together. How we store these four bounding boxes together is extremely important. The easiest way to store four bounding boxes in a single struct is to just have BBox4 store four separate BBox structs internally, but this approach is actually really bad for vectorization. To understand why, consider something like the following, where we perform an min operation between the ray tMin and a distance to a corner of a bounding box:

fmax(ray.tMin, (bbox.corners[near.x] - ray.origin.x) * rdir.x);

Now consider if we want to do this operation for four bounding boxes in serial:

fmax(ray.tMin, (bbox0.corners[near.x] - ray.origin.x) * rdir.x);
fmax(ray.tMin, (bbox1.corners[near.x] - ray.origin.x) * rdir.x);
fmax(ray.tMin, (bbox2.corners[near.x] - ray.origin.x) * rdir.x);
fmax(ray.tMin, (bbox3.corners[near.x] - ray.origin.x) * rdir.x);

The above serial sequence is a perfect example of what we want to fold into a single vectorized line of code. The inputs to a vectorized version of the above should be a 128-bit four-lane value with ray.tMin in all four lanes, another 128-bit four-lane value with ray.origin.x in all four lanes, another 128-bit four-lane value with rdir.x in all four lanes, and finally a 128-bit four-lane value where the first lane is a single index of a single corner from the first bounding box, the second lane is a single index of a single corner from the second bounding box, and so on and so forth. Instead of an array of structs, we need the bounding box values to be provided as a struct of corner value arrays where each 128-bit value stores one 32-bit value from each corner of each of the four boxes. Alternatively, the BBox4 memory layout that we want can be thought of as an array of 24 floats, which is indexed as a 3D array where the first dimension is indexed by min or max corner, the second dimension is indexed by x, y, and z within each corner, and the third dimension is indexed by which bounding box the value belongs to. Putting the above together with some accessors and setter functions yields the following definition for BBox4:

struct BBox4 {
    union {
        FVec4 corners[6];             // order: minX, minY, minZ, maxX, maxY, maxZ
        float cornersFloat[2][3][4];  // indexed as corner[minOrMax][XYZ][bboxNumber]
        float cornersFloatAlt[6][4];
    };

    inline __m128* minCornerSSE() { return &corners[0].m128; }
    inline __m128* maxCornerSSE() { return &corners[3].m128; }

#if defined(__aarch64__)
    inline float32x4_t* minCornerNeon() { return &corners[0].f32x4; }
    inline float32x4_t* maxCornerNeon() { return &corners[3].f32x4; }
#endif

    inline void setBBox(int boxNum, const FVec4& minCorner, const FVec4& maxCorner) {
        cornersFloat[0][0][boxNum] = fmin(minCorner.x, maxCorner.x);
        cornersFloat[0][1][boxNum] = fmin(minCorner.y, maxCorner.y);
        cornersFloat[0][2][boxNum] = fmin(minCorner.z, maxCorner.z);
        cornersFloat[1][0][boxNum] = fmax(minCorner.x, maxCorner.x);
        cornersFloat[1][1][boxNum] = fmax(minCorner.y, maxCorner.y);
        cornersFloat[1][2][boxNum] = fmax(minCorner.x, maxCorner.x);
    }

    BBox4(const BBox& a, const BBox& b, const BBox& c, const BBox& d) {
        setBBox(0, a.minCorner(), a.maxCorner());
        setBBox(1, b.minCorner(), b.maxCorner());
        setBBox(2, c.minCorner(), c.maxCorner());
        setBBox(3, d.minCorner(), d.maxCorner());
    }
};

Listing 6: Struct holding four bounding boxes together with values interleaved for optimal vectorized access.

Note how the setBBox function (which the constructor calls) has a memory access pattern where a single value is written into every 128-bit FVec4. Generally scattered access like this is extremely expensive in vectorized code, and should be avoided as much as possible; setting an entire 128-bit value at once is much faster than setting four separate 32-bit segments across four different values. However, something like the above is often inevitably necessary just to get data loaded into a layout optimal for vectorized code; in the test program, BBox4 structs are initialized and set up once, and then reused across all tests. The time required to set up BBox and BBox4 is not counted as part of any of the test runs; in a full BVH traversal implementation, the BVH’s bounds at each node should be pre-arranged into a vector-friendly layout before any ray traversal takes place. In general, figuring out how to restructure an algorithm to be easily expressed using vector intrinsics is really only half of the challenge in writing good vectorized programs; the other half of the challenge is just getting the input data into a form that is amenable to vectorization. Actually, depending on the problem domain, the data marshaling can account for far more than half of the total effort spent!

Now that we have four bounding boxes structured in a way that is amenable to vectorized usage, we also need to structure our ray inputs for vectorized usage. This step is relatively easy; we just need to expand each component of each element of the ray into a 128-bit value where the same value is replicated across every 32-bit lane. SSE has a specific intrinsic to do exactly this: _mm_set1_ps() takes in a single 32-bit float and replicates it to all four lanes in a 128-bit __m128. SSE also has a bunch of more specialized instructions, which can be used in specific scenarios to do complex operations in a single instruction. Knowing when to use these more specialized instructions can be tricky and requires extensive knowledge of the SSE instruction set; I don’t know these very well yet! One good trick I did figure out was that in the case of taking a FVec4 and creating a new __m128 from each of the FVec4’s components, I could use _mm_shuffle_ps instead of _mm_set1_ps(). The problem with using _mm_set1_ps() in this case is that with a FVec4, which internally uses __m128 on x86-64, taking a element out to store using _mm_set1_ps() compiles down to a MOVSS instruction in addition to a shuffle. _mm_shuffle_ps(), on the other hand, compiles down to a single SHUFPS instruction. _mm_shuffle_ps() takes in two __m128s as input and takes two components from the first __m128 for the first two components of the output, and takes two components from the second __m128 for the second two components of the output. Which components from the inputs are taken is assignable using an input mask, which can conveniently be generated using the _MM_SHUFFLE() macro that comes with the SSE intrinsics headers. Since our ray struct’s origin and direction elements are already backed by __m128 under the hood, we can just use _mm_shuffle_ps() with the same element from the ray as both the first and second inputs to generate a __m128 containing only a single component of each element. For example, to create a __m128 containing only the x component of the ray direction, we can write: _mm_shuffle_ps(rdir.m128, rdir.m128, _MM_SHUFFLE(0, 0, 0, 0)).

Translating the fmin() and fmax() functions is very straightforward with SSE; we can use SSE’s _mm_min_ps() and _mm_max_ps() as direct analogues. Putting all of the above together allows us to write a fully SSE-ized version of the compact scalar ray-box intersection test that intersects a single ray against four boxes simultaneously:

void rayBBoxIntersect4SSE(const Ray& ray,
                        const BBox4& bbox4,
                        IVec4& hits,
                        FVec4& tMins,
                        FVec4& tMaxs) {
    FVec4 rdir(_mm_set1_ps(1.0f) / ray.direction.m128);
    /* use _mm_shuffle_ps, which translates to a single instruction while _mm_set1_ps involves a
    MOVSS + a shuffle */
    FVec4 rdirX(_mm_shuffle_ps(rdir.m128, rdir.m128, _MM_SHUFFLE(0, 0, 0, 0)));
    FVec4 rdirY(_mm_shuffle_ps(rdir.m128, rdir.m128, _MM_SHUFFLE(1, 1, 1, 1)));
    FVec4 rdirZ(_mm_shuffle_ps(rdir.m128, rdir.m128, _MM_SHUFFLE(2, 2, 2, 2)));
    FVec4 originX(_mm_shuffle_ps(ray.origin.m128, ray.origin.m128, _MM_SHUFFLE(0, 0, 0, 0)));
    FVec4 originY(_mm_shuffle_ps(ray.origin.m128, ray.origin.m128, _MM_SHUFFLE(1, 1, 1, 1)));
    FVec4 originZ(_mm_shuffle_ps(ray.origin.m128, ray.origin.m128, _MM_SHUFFLE(2, 2, 2, 2)));

    IVec4 near(int(rdir.x >= 0.0f ? 0 : 3), int(rdir.y >= 0.0f ? 1 : 4),
            int(rdir.z >= 0.0f ? 2 : 5));
    IVec4 far(int(rdir.x >= 0.0f ? 3 : 0), int(rdir.y >= 0.0f ? 4 : 1),
            int(rdir.z >= 0.0f ? 5 : 2));

    tMins = FVec4(_mm_max_ps(
        _mm_max_ps(_mm_set1_ps(ray.tMin), 
                   (bbox4.corners[near.x].m128 - originX.m128) * rdirX.m128),
        _mm_max_ps((bbox4.corners[near.y].m128 - originY.m128) * rdirY.m128,
                   (bbox4.corners[near.z].m128 - originZ.m128) * rdirZ.m128)));
    tMaxs = FVec4(_mm_min_ps(
        _mm_min_ps(_mm_set1_ps(ray.tMax),
                   (bbox4.corners[far.x].m128 - originX.m128) * rdirX.m128),
        _mm_min_ps((bbox4.corners[far.y].m128 - originY.m128) * rdirY.m128,
                   (bbox4.corners[far.z].m128 - originZ.m128) * rdirZ.m128)));

    int hit = ((1 << 4) - 1) & _mm_movemask_ps(_mm_cmple_ps(tMins.m128, tMaxs.m128));
    hits[0] = bool(hit & (1 << (0)));
    hits[1] = bool(hit & (1 << (1)));
    hits[2] = bool(hit & (1 << (2)));
    hits[3] = bool(hit & (1 << (3)));
}

Listing 7: SSE version of the compact Williams et al. 2005 implementation.

The last part of rayBBoxIntersect4SSE() where hits is populated might require a bit of explaining. This last part implements the check for whether or not a ray actually hit the box based on the results stored in tMin and tMax. This implementation takes advantage of the fact that misses in this implementation produce inf or -inf values; to figure out if a hit has occurred, we just have to check that in each lane, the tMin value is less than the tMax value, and inf values play nicely with this check. So, to conduct the check across all lanes at the same time, we use _mm_cmple_ps(), which compares if the 32-bit float in each lane of the first input is less-than-or-equal than the corresponding 32-bit float in each lane of the second input. If the comparison succeeds, _mm_cmple_ps() writes 0xFFF into the corresponding lane in the output __m128, and if the comparison fails, 0 is written instead. The remaining _mm_movemask_ps() instruction and bit shifts are just used to copy the results in each lane out into each component of hits.

I think variants of this 4-wide SSE ray-box intersection function are fairly common in production renderers; I’ve seen something similar developed independently at multiple studios and in multiple renderers, which shouldn’t be surprising since the translation from the original Williams et al. 2005 paper to a SSE-ized version is relatively straightforward. Also, the performance results further hint at why variants of this implementation are popular! Here is how the SSE implementation (Listing 7) performs compared to the scalar compact representation (Listing 5):

	x86-64:	x86-64 Speedup:	Rosetta2:	Rosetta2 Speedup:
Scalar Compact:	44.5159 ns	1.0x.	81.0942 ns	1.0x.
SSE:	10.9660 ns	4.0686x	13.6353 ns	5.9474x

The SSE implementation is almost exactly four times faster than the reference scalar compact implementation, which is exactly what we would expect as a best case for a properly written SSE implementation. Actually, in the results listed above, the SSE implementation is listed as being slightly more than four times faster, but that’s just an artifact of averaging together results from multiple runs; the amount over 4x is basically just an artifact of the statistical margin of error. A 4x speedup is the maximum speedup we can possible expect given that SSE is 4-wide for 32-bit floats. In our SSE implementation, the BBox4 struct is already set up before the function is called, but the function still needs to translate each incoming ray into a form suitable for vector operations, which is additional work that the scalar implementation doesn’t need to do. In order to make this additional setup work not drag down performance, the _mm_shuffle_ps() trick becomes very important.

Running the x86-64 version of the test program on arm64 using Rosetta 2 produces a more surprising result: close to a 6x speedup! Running through Rosetta 2 means that the x86-64 and SSE instructions have to be translated to arm64 and Neon instructions, and the 8x speedup here hints that for this test, Rosetta 2’s SSE to Neon translation ran much more efficiently than Rosetta 2’s x86-64 to arm64 translation. Otherwise, a greater-than-4x speedup should not be possible if both implementations are being translated with equal levels of efficiency. I did not expect that to be the case! Unfortunately, while we can speculate, only Apple’s developers can say for sure what Rosetta 2 is doing internally that produces this result.

Neon Implementation

The second vectorized implementation we’ll look at is using Neon on arm64 processors. Much like how all modern x86-64 processors support at least SSE2 because the 64-bit extension to x86 incorporated SSE2 into the base instruction set, all modern arm64 processors support Neon because the 64-bit extension to ARM incorporates Neon in the base instruction set. Compared with SSE, Neon is a much more compact instruction set, which makes sense since SSE belongs to a CISC ISA while Neon belongs to a RISC ISA. Neon includes a little over a hundred instructions, which is less than half the number of instructions that the full SSE to SSE4 instruction set contains. Neon has all of the basics that one would expect, such as arithmetic operations and various comparison operations, but Neon doesn’t have more complex high-level instructions like the fancy shuffle instructions we used in our SSE implementation.

Much like how Intel has a searchable SSE intrinsics guide, ARM provides a helpful searchable intrinsics guide. Howard Oakley’s recent blog series on writing arm64 assembly also includes a great introduction to using Neon. Note that even though there are fewer Neon instructions in total than there are SSE instructions, the ARM intrinsics guide lists several thousand functions; this is because of one of the chief differences between SSE and Neon. SSE’s __m128 is just a generic 128-bit container that doesn’t actually specify what type or how many lanes it contains; what type a __m128 value is or how many lanes a __m128 value contains interpreted as is entirely up to each SSE instruction. Contrast with Neon, which has explicit separate types for floats and integers, and also defines separate types based on width. Since Neon has many different 128-bit types, each Neon instruction has multiple corresponding intrinsics that differ simply by the input types and widths accepted in the function signature. As a result of all of the above differences from SSE, writing a Neon implementation is not quite as simple as just doing a one-to-one replacement of each SSE intrinsic with a Neon intrinsic.

…or is it? Writing C/C++ code utilizing Neon instructions can be done by using the native Neon intrinsics found in <arm_neon.h>, but another option exists through the sse2neon project. When compiling for arm64, the x86-64 SSE <xmmintrin.h> header is not available for use because every function in the <xmmintrin.h> header maps to a specific SSE instruction or group of SSE instructions, and there’s no sense in the compiler trying to generate SSE instructions for a processor architecture that SSE instructions don’t even work on. However, the function definitions for each intrinsic are just function definitions, and sse2neon project reimplements every SSE intrinsic function with a Neon implementation under the hood. So, using sse2neon, code originally written for x86-64 using SSE intrinsics can be compiled without modification on arm64, with Neon instructions generated from the SSE intrinsics. A number of large projects originally written on x86-64 now have arm64 ports that utilize sse2neon to support vectorized code without having to completely rewrite using Neon intrinsics; as discussed in my previous Takua on ARM post, this approach is the exact approach that was taken to port Embree to arm64.

The sse2neon project was originally started by John W. Ratcliff and a few others at NVIDIA to port a handful of games from x86-64 to arm64; the original version of sse2neon only implemented the small subset of SSE that was needed for their project. However, after the project was posted to Github with a MIT license, other projects found sse2neon useful and contributed additional extensions that eventually fleshed out full coverage for MMX and all versions of SSE from SSE1 all the way through SSE4.2. For example, Syoyo Fujita’s embree-aarch64 project, which was the basis of Intel’s official Embree arm64 port, resulted in a number of improvements to sse2neon’s precision and faithfulness to the original SSE behavior. Over the years sse2neon has seen contributions and improvements from NVIDIA, Amazon, Google, the Embree-aarch64 project, the Blender project, and recently Apple as part of Apple’s larger slew of contributions to various projects to improve arm64 support for Apple Silicon. Similar open-source projects also exist to further generalize SIMD intrinsics headers (simde), to reimplement the AVX intrinsics headers using Neon (AvxToNeon), and Intel even has a project to do the reverse of sse2neon: reimplement Neon using SSE (ARM_NEON_2_x86_SSE).

While learning about Neon and while looking at how Embree was ported to arm64 using sse2neon, I started to wonder how efficient using sse2neon versus writing code directly using Neon intrinsics would be. The SSE and Neon instruction sets don’t have a one-to-one mapping to each other for many of the more complex higher-level instructions that exist in SSE, and as a result, some SSE intrinsics that compiled down to a single SSE instruction on x86-64 have to be implemented on arm64 using many Neon instructions. As a result, at least in principle, my expectation was that on arm64, code written directly using Neon intrinsics typically should likely have at least a small performance edge over SSE code ported using sse2neon. So, I decided to do a direct comparison in my test program, which required implementing the 4-wide ray-box intersection test using Neon:

inline uint32_t neonCompareAndMask(const float32x4_t& a, const float32x4_t& b) {
    uint32x4_t compResUint = vcleq_f32(a, b);
    static const int32x4_t shift = { 0, 1, 2, 3 };
    uint32x4_t tmp = vshrq_n_u32(compResUint, 31);
    return vaddvq_u32(vshlq_u32(tmp, shift));
}

void rayBBoxIntersect4Neon(const Ray& ray,
                        const BBox4& bbox4,
                        IVec4& hits,
                        FVec4& tMins,
                        FVec4& tMaxs) {
    FVec4 rdir(vdupq_n_f32(1.0f) / ray.direction.f32x4);
    /* since Neon doesn't have a single-instruction equivalent to _mm_shuffle_ps, we just take
    the slow route here and load into each float32x4_t */
    FVec4 rdirX(vdupq_n_f32(rdir.x));
    FVec4 rdirY(vdupq_n_f32(rdir.y));
    FVec4 rdirZ(vdupq_n_f32(rdir.z));
    FVec4 originX(vdupq_n_f32(ray.origin.x));
    FVec4 originY(vdupq_n_f32(ray.origin.y));
    FVec4 originZ(vdupq_n_f32(ray.origin.z));

    IVec4 near(int(rdir.x >= 0.0f ? 0 : 3), int(rdir.y >= 0.0f ? 1 : 4),
            int(rdir.z >= 0.0f ? 2 : 5));
    IVec4 far(int(rdir.x >= 0.0f ? 3 : 0), int(rdir.y >= 0.0f ? 4 : 1),
            int(rdir.z >= 0.0f ? 5 : 2));

    tMins =
        FVec4(vmaxq_f32(vmaxq_f32(vdupq_n_f32(ray.tMin),
                                (bbox4.corners[near.x].f32x4 - originX.f32x4) * rdirX.f32x4),
                        vmaxq_f32((bbox4.corners[near.y].f32x4 - originY.f32x4) * rdirY.f32x4,
                                (bbox4.corners[near.z].f32x4 - originZ.f32x4) * rdirZ.f32x4)));
    tMaxs = FVec4(vminq_f32(vminq_f32(vdupq_n_f32(ray.tMax),
                                    (bbox4.corners[far.x].f32x4 - originX.f32x4) * rdirX.f32x4),
                            vminq_f32((bbox4.corners[far.y].f32x4 - originY.f32x4) * rdirY.f32x4,
                                    (bbox4.corners[far.z].f32x4 - originZ.f32x4) * rdirZ.f32x4)));

    uint32_t hit = neonCompareAndMask(tMins.f32x4, tMaxs.f32x4);
    hits[0] = bool(hit & (1 << (0)));
    hits[1] = bool(hit & (1 << (1)));
    hits[2] = bool(hit & (1 << (2)));
    hits[3] = bool(hit & (1 << (3)));
}

Listing 8: Neon version of the compact Williams et al. 2005 implementation.

Even if you only know SSE and have never worked with Neon, you should already be able to tell broadly how the Neon implementation in Listing 8 works! Just from the name alone, vmaxq_f32() and vminq_f32() obviously correspond directly to _mm_max_ps() and _mm_min_ps() in the SSE implementation, and understanding how the ray data is being loaded into Neon’s 128-bit registers using vdupq_n_f32() instead of _mm_set1_ps() should be relatively easy too. However, because there is no fancy single-instruction shuffle intrinsic available in Neon, the way the ray data is loaded is potentially slightly less efficient.

The largest area of difference between the Neon and SSE implementations is in the processing of the tMin and tMax results to produce the output hits vector. The SSE version uses just two intrinsic functions because SSE includes the fancy high-level _mm_cmple_ps() intrinsic, which compiles down to a single CMPPS SSE instruction, but implementing this functionality using Neon takes some more work. The neonCompareAndMask() helper function implements the hits vector processing using four Neon intrinsics; a better solution may exist, but for now this is the best I can do given my relatively basic level of Neon experience. If you have a better solution, feel free to let me know!

Here’s how the native Neon intrinsics implementation performs compared with using sse2neon to translate the SSE implementation. For an additional point of comparison, I’ve also included the Rosetta 2 SSE result from the previous section. Note that the speedup column for Rosetta 2 here isn’t comparing how much faster the SSE implementation running over Rosetta 2 is with the compact scalar implementation running over Rosetta 2; instead, the Rosetta 2 speedup columns here compare how much faster (or slower) the Rosetta 2 runs are compared with the native arm64 compact scalar implementation:

	arm64:	arm64 Speedup:	Rosetta2:	Rosetta2 Speedup over Native:
Scalar Compact:	41.8187 ns	1.0x.	81.0942 ns	0.5157x
SSE:	-	-	13.6353 ns	3.0669x
SSE2NEON:	12.3090 ns	3.3974x	-	-
Neon:	12.2161 ns	3.4232x	-	-

I originally also wanted to include a test that would have been the reverse of sse2neon: use Intel’s ARM_NEON_2_x86_SSE project to get the Neon implementation working on x86-64. However, when I tried using ARM_NEON_2_x86_SSE, I discovered that the ARM_NEON_2_x86_SSE isn’t quite complete enough yet (as of time of writing) to actually compile the Neon implementation in Figure 8.

I was very pleased to see that both of the native arm64 implementations ran faster than the SSE implementation running over Rosetta 2; which means that my native Neon implementation is at least halfway decent, and which also means that sse2neon works as advertised. The native Neon implementation is also just a hair faster than the sse2neon implementation, which indicates that at least here, rewriting using native Neon intrinsics instead of mapping from SSE to Neon does indeed produce slightly more efficient code. However, the sse2neon implementation is very very close in terms of performance, to the point where it may well be within an acceptable margin of error. Overall, both of the native arm64 implementations get a respectable speedup over the compact scalar reference, even though the speedup amounts are a bit less than the ideal 4x. I think that the slight performance loss compared to the ideal 4x is probably attributable to the more complex solution required for filling the output hits vector.

To better understand why the sse2neon implementation performs so close to the native Neon implementation, I tried just copy-pasting every single function implementation out of sse2neon into the SSE 4-wide ray-box intersection test. Interestingly, the result was extremely similar to my native Neon implementation; structurally they were more or less identical, but the sse2neon version had some additional extraneous calls. For example, instead of replacing _mm_max_ps(a, b) one-to-one with vmaxq_f32(a, b), sse2neon’s version of _mm_max_ps(a, b) is vreinterpretq_m128_f32(vmaxq_f32(vreinterpretq_f32_m128(a), vreinterpretq_f32_m128(b))). vreinterpretq_m128_f32 is a helper function defined by sse2neon to translate an input __m128 into a float32x4_t. There’s a lot of reinterpreting of inputs to specific float or integer types in sse2neon; all of the reinterpreting in sse2neon is to convert from SSE’s generic __m128 to Neon’s more specific types. In the specific case of vreinterpretq_m128_f32, the reinterpretation should actually compile down to a no-op since sse2neon typedefs __m128 directly to float32x4_t, but many of sse2neon’s other reinterpretation functions do require additional extra Neon instructions to implement.

Even though the Rosetta 2 result is definitively slower than the native arm64 results, the Rosetta 2 result is far closer to the native arm64 results than I normally would have expected. Rosetta 2 usually can be expected to perform somewhere in the neighborhood of 50% to 80% of native performance for compute-heavy code, and the Rosetta 2 performance for the compact scalar implementation lines up with this expectation. However, the Rosetta 2 performance for the vectorized version lends further credence to the theory from the previous section that Rosetta 2 somehow is better able to translate vectorized code than scalar code.

Auto-vectorized Implementation

The unfortunate thing about writing vectorized programs using vector intrinsics is that… vector intrinsics can be hard to use! Vector intrinsics are intentionally fairly low-level, which means that when compared to writing normal C or C++ code, using vector intrinsics is only a half-step above writing code directly in assembly. The vector intrinsics APIs provided for SSE and Neon have very large surface areas, since a large number of intrinsic functions exist to cover the large number of vector instructions that there are. Furthermore, unless compatibility layers like sse2neon are used, vector intrinsics are not portable between different processor architectures in the same way that normal higher-level C and C++ code is. Even though I have some experience working with vector intrinsics, I still don’t consider myself even remotely close to comfortable or proficient in using them; I have to rely heavily on looking up everything using various reference guides.

One potential solution to the difficulty of using vector intrinsics is compiler auto-vectorization. Auto-vectorization is a compiler technique that aims to allow programmers to better utilize vector instructions without requiring programmers to write everything using vector intrinsics. Instead of writing vectorized programs, programmers write standard scalar programs which the compiler’s auto-vectorizer then converts into a vectorized program at compile-time. One common auto-vectorization technique that many compilers implement is loop vectorization, which takes a serial innermost loop and restructures the loop such that each iteration of the loop maps to one vector lane. Implementing loop vectorization can be extremely tricky, since like with any other type of compiler optimization, the cardinal rule is that the originally written program behavior must be unmodified and the original data dependencies and access orders must be preserved. Add in the need to consider all of the various concerns that are specific to vector instructions, and the result is that loop vectorization is easy to get wrong if not implemented very carefully by the compiler. However, when loop vectorization is available and working correctly, the performance increase to otherwise completely standard scalar code can be significant.

The 4-wide ray-box intersection test should be a perfect candidate for auto-vectorization! The scalar implementations are implemented as just a single for loop that calls the single ray-box test once per iteration of the loop, for four iterations. Inside of the loop, the ray-box test is fundamentally just a bunch of simple min/max operations and a little bit of arithmetic, which as seen in the SSE and Neon implementations, is the easiest part of the whole problem to vectorize. I originally expected that I would have to compile the entire test program with all optimizations disabled, because I thought that with optimizations enabled, the compiler would auto-vectorize the compact scalar implementation and make comparisons with the hand-vectorized implementations difficult. However, after some initial testing, I realized that the scalar implementations weren’t really getting auto-vectorized at all even with optimization level -O3 enabled. Or, more precisely, the compiler was emitting long stretches of code using vector instructions and vector registers… but the compiler was just utilizing one lane in all of those long stretches of vector code, and was still looping over each bounding box separately. As a point of reference, here is the x86-64 compiled output and the arm64 compiled output for the compact scalar implementation.

Finding that the auto-vectorizer wasn’t really working on the scalar implementations led me to try to write a new scalar implementation that would auto-vectorize well. To try to give the auto-vectorizer as good of a chance at possible at working well, I started with the compact scalar implementation and embedded the single-ray-box intersection test into the 4-wide function as an inner loop. I also pulled apart the implementation into a more expanded form where every line in the inner loop carries out a single arithmetic operation that can be mapped to exactly to one SSE or Neon instruction. I also restructured the data input to the inner loop to be in a readily vector-friendly layout; the restructuring is essentially a scalar implementation of the vectorized setup code found in the SSE and Neon hand-vectorized implementations. Finally, I put a #pragma clang loop vectorize(enable) in front of the inner loop to make sure that the compiler knows that it can use the loop vectorizer here. Putting all of the above together produces the following, which is as auto-vectorization-friendly as I could figure out how to rewrite things:

void rayBBoxIntersect4AutoVectorize(const Ray& ray,
                                    const BBox4& bbox4,
                                    IVec4& hits,
                                    FVec4& tMins,
                                    FVec4& tMaxs) {
    float rdir[3] = { 1.0f / ray.direction.x, 1.0f / ray.direction.y, 1.0f / ray.direction.z };
    float rdirX[4] = { rdir[0], rdir[0], rdir[0], rdir[0] };
    float rdirY[4] = { rdir[1], rdir[1], rdir[1], rdir[1] };
    float rdirZ[4] = { rdir[2], rdir[2], rdir[2], rdir[2] };
    float originX[4] = { ray.origin.x, ray.origin.x, ray.origin.x, ray.origin.x };
    float originY[4] = { ray.origin.y, ray.origin.y, ray.origin.y, ray.origin.y };
    float originZ[4] = { ray.origin.z, ray.origin.z, ray.origin.z, ray.origin.z };
    float rtMin[4] = { ray.tMin, ray.tMin, ray.tMin, ray.tMin };
    float rtMax[4] = { ray.tMax, ray.tMax, ray.tMax, ray.tMax };

    IVec4 near(int(rdir[0] >= 0.0f ? 0 : 3), int(rdir[1] >= 0.0f ? 1 : 4),
            int(rdir[2] >= 0.0f ? 2 : 5));
    IVec4 far(int(rdir[0] >= 0.0f ? 3 : 0), int(rdir[1] >= 0.0f ? 4 : 1),
            int(rdir[2] >= 0.0f ? 5 : 2));

    float product0[4];

#pragma clang loop vectorize(enable)
    for (int i = 0; i < 4; i++) {
        product0[i] = bbox4.corners[near.y][i] - originY[i];
        tMins[i] = bbox4.corners[near.z][i] - originZ[i];
        product0[i] = product0[i] * rdirY[i];
        tMins[i] = tMins[i] * rdirZ[i];
        product0[i] = fmax(product0[i], tMins[i]);
        tMins[i] = bbox4.corners[near.x][i] - originX[i];
        tMins[i] = tMins[i] * rdirX[i];
        tMins[i] = fmax(rtMin[i], tMins[i]);
        tMins[i] = fmax(product0[i], tMins[i]);

        product0[i] = bbox4.corners[far.y][i] - originY[i];
        tMaxs[i] = bbox4.corners[far.z][i] - originZ[i];
        product0[i] = product0[i] * rdirY[i];
        tMaxs[i] = tMaxs[i] * rdirZ[i];
        product0[i] = fmin(product0[i], tMaxs[i]);
        tMaxs[i] = bbox4.corners[far.x][i] - originX[i];
        tMaxs[i] = tMaxs[i] * rdirX[i];
        tMaxs[i] = fmin(rtMax[i], tMaxs[i]);
        tMaxs[i] = fmin(product0[i], tMaxs[i]);

        hits[i] = tMins[i] <= tMaxs[i];
    }
}

Listing 9: Compact scalar version written to be easily auto-vectorized.

How well is Apple Clang v12.0.5 able to auto-vectorize the implementation in Listing 9? Well, looking at the output assembly on x86-64 and on arm64… the result is disappointing. Much like with the compact scalar implementation, the compiler is in fact emitting nice long sequences of vector intrinsics and vector registers… but the loop is still getting unrolled into four repeated blocks of code where only one lane is utilized per unrolled block, as opposed to produce a single block of code where all four lanes are utilized together. The difference is especially apparent when compared with the hand-vectorized SSE compiled output and the hand-vectorized Neon compiled output.

Here are the results of running the auto-vectorized implementation above, compared with the reference compact scalar implementation:

	x86-64:	x86-64 Speedup:	arm64:	arm64 Speedup:	Rosetta2:	Rosetta2 Speedup:
Scalar Compact:	44.5159 ns	1.0x.	41.8187 ns	1.0x.	81.0942 ns	1.0x.
Autovectorize:	34.1398 ns	1.3069x	38.1917 ns	1.0950x	59.9757 ns	1.3521x

While the auto-vectorized version certainly is faster than the reference compact scalar implementation, the speedup is far from the 3x to 4x that we’d expect from well vectorized code that was properly utilizing each processor’s vector hardware. On arm64, the speed boost from auto-vectorization is almost nothing.

So what is going on here? Why is compiler failing so badly at auto-vectorizing code that has been explicitly written to be easily vectorizable? The answer is that the compiler is in fact producing vectorized code, but since the compiler doesn’t have a more complete understanding of what the code is actually trying to do, the compiler can’t set up the data appropriately to really be able to take advantage of vectorization. Therein lies what is, in my opinion, one of the biggest current drawbacks of relying on auto-vectorization: there is only so much the compiler can do without a higher, more complex understanding of what the program is trying to do overall. Without that higher level understanding, the compiler can only do so much, and understanding how to work around the compiler’s limitations requires a deep understanding of how the auto-vectorizer is implemented internally. Structuring code to auto-vectorize well also requires thinking ahead to what the vectorized output assembly should be, which is not too far from just writing the code using vector intrinsics to begin with. At least to me, if achieving maximum possible performance is a goal, then all of the above actually amounts to more complexity than just directly writing using vector intrinsics. However, that isn’t to say that auto-vectorization is completely useless- we still did get a bit of a performance boost! I think that auto-vectorization is definitely better than nothing, and when it does work, it works well. But, I also think that auto-vectorization is not a magic bullet perfect solution to writing vectorized code, and when hand-vectorizing is an option, a well-written hand-vectorized implementation has a strong chance of outperforming auto-vectorization.

ISPC Implementation

Another option exists for writing portable vectorized code without having to directly use vector intrinsics: ISPC, which stands for “Intel SPMD Program Compiler”. The ISPC project was started and initially developed by Matt Pharr after he realized that the reason auto-vectorization tends to work so poorly in practice is because auto-vectorization is not a programming model [Pharr 2018]. A programming model both allows programmers to better understand what guarantees the underlying hardware execution model can provide, and also provides better affordances for compilers to rely on for generating assembly code. ISPC utilizes a programming model known as SPMD, or single-program-multiple-data. The SPMD programming model is generally very similar to the SIMT programming model used on GPUs (in many ways, SPMD can be viewed as a generalization of SIMT): programs are written as a serial program operating over a single data element, and then the serial program is run in a massively parallel fashion over many different data elements. In other words, the parallelism in a SPMD program is implicit, but unlike in auto-vectorization, the implicit parallelism is also a fundamental component of the programming model.

Mapping to SIMD hardware, writing a program using a SPMD model means that the serial program is written for a single SIMD lane, and the compiler is responsible for multiplexing the serial program across multiple lanes [Pharr and Mark 2012]. The difference between SPMD-on-SIMD and auto-vectorization is that with SPMD-on-SIMD, the compiler can know much more and rely on much harder guarantees about how the program wants to be run, as enforced by the programming model itself. ISPC compiles a special variant of the C programming language that has been extended with some vectorization-specific native types and control flow capabilities. Compared to writing code using vector intrinsics, ISPC programs look a lot more like normal scalar C code, and often can even be compiled as normal scalar C code with little to no modification. Since the actual transformation to vector assembly is up to the compiler, and since ISPC utilizes LLVM under the hood, programs written for ISPC can be written just once and then compiled to many different LLVM-supported backend targets such as SSE, AVX, Neon, and even CUDA.

Actually writing an ISPC program is, in my opinion, very straightforward; since the language is just C with some additional builtin types and keywords, if you already know how to program in C, you already know most of ISPC. ISPC provides vector versions of all of the basic types like float and int; for example, ISPC’s float<4> in memory corresponds exactly to the FVec4 struct we defined earlier for our test program. ISPC also adds qualifier keywords like uniform and varying that act as optimization hints for the compiler by providing the compiler with guarantees about how memory is used; if you’ve programmed in GLSL or a similar GPU shading language before, you already know how these qualifiers work. There are a variety of other small extensions and differences, all of which are well covered by the ISPC User’s Guide.

The most important extension that ISPC adds to C is the foreach control flow construct. Normal loops are still written using for and while, but the foreach loop is really how parallel computation is specified in ISPC. The inside of a foreach loop describes what happens on one SIMD lane, and the iterations of the foreach loop are what get multiplexed onto different SIMD lanes by the compiler. In other words, the contents of the foreach loop is roughly analogous to the contents of a GPU shader, and the foreach loop statement itself is roughly analogous to a kernel launch in the GPU world.

Knowing all of the above, here’s how I implemented the 4-wide ray-box intersection test as an ISPC program. Note how the actual intersection testing happens in the foreach loop; everything before that is setup:

typedef float<3> float3;

export void rayBBoxIntersect4ISPC(const uniform float rayDirection[3],
                                const uniform float rayOrigin[3],
                                const uniform float rayTMin,
                                const uniform float rayTMax,
                                const uniform float bbox4corners[6][4],
                                uniform float tMins[4],
                                uniform float tMaxs[4],
                                uniform int hits[4]) {
    uniform float3 rdir = { 1.0f / rayDirection[0], 1.0f / rayDirection[1],
                            1.0f / rayDirection[2] };

    uniform int near[3] = { 3, 4, 5 };
    if (rdir.x >= 0.0f) {
        near[0] = 0;
    }
    if (rdir.y >= 0.0f) {
        near[1] = 1;
    }
    if (rdir.z >= 0.0f) {
        near[2] = 2;
    }

    uniform int far[3] = { 0, 1, 2 };
    if (rdir.x >= 0.0f) {
        far[0] = 3;
    }
    if (rdir.y >= 0.0f) {
        far[1] = 4;
    }
    if (rdir.z >= 0.0f) {
        far[2] = 5;
    }

    foreach (i = 0...4) {
        tMins[i] = max(max(rayTMin, (bbox4corners[near[0]][i] - rayOrigin[0]) * rdir.x),
                    max((bbox4corners[near[1]][i] - rayOrigin[1]) * rdir.y,
                        (bbox4corners[near[2]][i] - rayOrigin[2]) * rdir.z));
        tMaxs[i] = min(min(rayTMax, (bbox4corners[far[0]][i] - rayOrigin[0]) * rdir.x),
                    min((bbox4corners[far[1]][i] - rayOrigin[1]) * rdir.y,
                        (bbox4corners[far[2]][i] - rayOrigin[2]) * rdir.z));
        hits[i] = tMins[i] <= tMaxs[i];
    }
}

Listing 10: ISPC implementation of the compact Williams et al. 2005 implementation.

In order to call the ISPC function from our main C++ test program, we need to define a wrapper function on the C++ side of things. When an ISPC program is compiled, ISPC automatically generates a corresponding header file named using the name of the ISPC program appended with “_ispc.h”. This automatically generated header can be included by the C++ test program. Using ISPC through CMake 3.19 or newer, ISPC programs can be added to any normal C/C++ project, and the automatically generated ISPC headers can be included like any other header and will be placed into the correct place by CMake.

Since ISPC is a separate language and since ISPC code has to be compiled as a separate object from our main C++ code, we can’t pass the various structs we’ve defined directly into the ISPC function. Instead, we need a simple wrapper function that extracts pointers to the underlying basic data types from our custom structs, and passes those pointers to the ISPC function:

void rayBBoxIntersect4ISPC(const Ray& ray,
                        const BBox4& bbox4,
                        IVec4& hits,
                        FVec4& tMins,
                        FVec4& tMaxs) {
    ispc::rayBBoxIntersect4ISPC(ray.direction.data, ray.origin.data, ray.tMin, ray.tMax,
                                bbox4.cornersFloatAlt, tMins.data, tMaxs.data, hits.data);
}

Listing 11: Wrapper function to call the ISPC implementation from C++.

Looking at the assembly output from ISPC for x86-64 SSE4 and for arm64 Neon, things look pretty good! The contents of the foreach loop have been compiled down to a single straight run of vectorized instructions, with all four lanes filled beforehand. Comparing ISPC’s output with the compiler output for the hand-vectorized implementations, the core of the ray-box test looks very similar between the two, while ISPC’s output for all of the precalculation logic actually seems slightly better than the output from the hand-vectorized implementation.

Here is how the ISPC implementation performs, compared to the baseline compact scalar implementation:

	x86-64:	x86-64 Speedup:	arm64:	arm64 Speedup:	Rosetta2:	Rosetta2 Speedup:
Scalar Compact:	44.5159 ns	1.0x.	41.8187 ns	1.0x.	81.0942 ns	1.0x.
ISPC:	8.2877 ns	5.3835x	11.2182 ns	3.7278x	11.3709 ns	7.1317x

The performance from the ISPC implementation looks really good! Actually, on x86-64, the ISPC implementation’s performance looks too good to be true: at first glance, a 5.3835x speedup over the compact scalar baseline implementation shouldn’t be possible since the maximum expected possible speedup is only 4x. I had to think about this result a while; I think the explanation for this apparently better-than-possible speedup is because the setup versus the actual intersection test parts of the 4-wide ray-box test need to be considered separately. The actual intersection part is the part that is an apples-to-apples comparison across all of the different implementations, while the setup code can vary significantly both in how it is written and in how well it can be optimized across different implementations. The reason for the above is that the setup code is more inherently scalar. I think that the reason the ISPC implementation has an overall more-than-4x speedup over the baseline is because in the baseline implementation, the scalar setup code is not much out of the -O3 optimization level, whereas the ISPC implementation’s setup code is both getting more out of ISPC’s -O3 optimization level and is additionally just better vectorized on account of being ISPC code. A data point that lends credence to this theory is that when Clang and ISPC are both forced to disabled all optimizations using the -O0 flag, the performance difference between the baseline and ISPC implementations falls back into a much more expected multiplier below 4x.

Generally, I really like ISPC! ISPC delivers on the promise of write-once compiler-and-run-anywhere vectorized code, and unlike auto-vectorization, ISPC’s output compiler assembly performs as we expect for well-written vectorized code. Of course, ISPC isn’t 100% fool-proof magic; care still needs to be taken in writing good ISPC programs that don’t contain excessive amounts of execution path divergence between SIMD lanes, and care still needs to be taken in not doing too many expensive gather/scatter operations. However, these types of considerations are just part of writing vectorized code in general and are not specific to ISPC, and furthermore, these types of considerations should be familiar territory for anyone with experience writing GPU code as well. I think that’s a general strength of ISPC: writing vector CPU code using ISPC feels a lot like writing GPU code, and that’s by design!

Final Results and Conclusions

Now that we’ve walked through every implementation in the test program, below are the complete results for every implementation across x86-64, arm64, and Rosetta 2. As mentioned earlier, all results were generated by running on a 2019 16 inch MacBook Pro with a Intel Core i7-9750H CPU for x86-64, and on a 2020 M1 Mac Mini for arm64 and Rosetta 2. All results were generated by running the test program with 100000 runs per implementation; the timings reported are the average time for one run. I ran the test program 5 times with 100000 runs each time; after throwing out the highest and lowest result for each implementation to discard outliers, I then averaged the remaining three results for each implementation for each architecture. In the results, the “speedup” columns use the scalar compact implementation as the baseline for comparison:

			Results
	x86-64:	x86-64 Speedup:	arm64:	arm64 Speedup:	Rosetta2:	Rosetta2 Speedup:
Scalar Compact:	44.5159 ns	1.0x.	41.8187 ns	1.0x.	81.0942 ns	1.0x.
Scalar Original:	44.1004 ns	1.0117x	78.4001 ns	0.5334x	90.7649 ns	0.8935x
Scalar No Early-Out:	55.6770 ns	0.8014x	85.3562 ns	0.4899x	102.763 ns	0.7891x
SSE:	10.9660 ns	4.0686x	-	-	13.6353 ns	5.9474x
SSE2NEON:	-	-	12.3090 ns	3.3974x	-	-
Neon:	-	-	12.2161 ns	3.4232x	-	-
Autovectorize:	34.1398 ns	1.3069x	38.1917 ns	1.0950x	59.9757 ns	1.3521x
ISPC:	8.2877 ns	5.3835x	11.2182 ns	3.7278x	11.3709 ns	7.1317x

In each of the sections above, we’ve already looked at how the performance of each individual implementation compares against the baseline compact scalar implementation. Ranking all of the approaches (at least for the specific example used in this post), ISPC produces the best performance, hand-vectorization using each processor’s native vector intrinsics comes in second, hand-vectorization using a translation layer such as sse2neon follows very closely behind using native vector intrinsics, and finally auto-vectorization comes in a distant last place. Broadly, I think a good rule of thumb is that auto-vectorization is better than nothing, and that for large complex programs where vectorization is important and where cross-platform is required, ISPC is the way to go. For smaller-scale things where the additional development complexity of bringing in an additional compiler isn’t justified, writing directly using vector intrinsics is a good solution, and using translation layers like sse2neon to port code written using one architecture’s vector intrinsics to another architecture without a total rewrite can work just as well as rewriting from scratch (assuming the translation layer is as well-written as sse2neon is). Finally, as mentioned earlier, I was very surprised to learn that Rosetta 2 seems to be much better at translating vector instructions than it is at translating normal scalar x86-64 instructions.

Looking back over the final test program, around a third of the total lines of code in the test program aren’t ray-box intersection code at all. Around a third of the code is made up of just defining data structures and doing data marshaling to make sure that the actual ray-box intersection code can be efficiently vectorized at all. I think that in most applications of vectorization, figuring out the data marshaling to enable good vectorization is just as important of a problem as actually writing the core vectorized code, and I think the data marshaling can often be even harder than the actual vectorization part. Even the ISPC implementation in this post only works because the specific memory layout of the BBox4 data structure is designed for optimal vectorized access.

For much larger vectorized applications, such as full production renderers, planning ahead for vectorization doesn’t just mean figuring out how to lay out data structures in memory, but can mean having to incorporate vectorization considerations into the fundamental architecture of the entire system. A great example of the above is DreamWorks Animation’s Moonray renderer, which has an entire architecture designed around coalescing enough coherent work in an incoherent path tracer to facilitate ISPC-based vectorized shading [Lee et al. 2017]. Weta Digital’s Manuka renderer goes even further by fundamentally restructuring the typical order of operations in a standard path tracer into a shade-before-hit architecture, also in part to facilitate vectorized shading [Fascione et al. 2018]. Pixar and Intel have also worked together recently to extend OSL with better vectorization for use in RenderMan XPU, which has necessitated the addition of a new batched interface to OSL [Liani and Wells 2020]. Some other interesting large non-rendering applications where vectorization has been applied through the use of clever rearchitecting include JPEG encoding [Krasnov 2018] and even JSON parsing [Langdale and Lemire 2019]. More generally, the entire domain of data-oriented design [Acton 2014] revolves around understanding how to structure data layout according to how computation needs to access said data; although data-oriented design was originally motivated by a need to efficiently utilize the CPU cache hierarchy, data-oriented design is also highly applicable to structuring vectorized programs.

In this post, we only looked at 4-wide 128-bit SIMD extensions. Vectorization is not limited to 128-bits or 4-wide instructions, of course; x86-64’s newer AVX instructions use 256-bit registers and, when used with 32-bit floats, AVX is 8-wide. The newest version of AVX, AVX-512, extends things even wider to 512-bit registers and can support a whopping 16 32-bit lanes. Similarly, ARM’s new SVE vector extensions serve as a wider successor to Neon (ARM also recently introduced a new lower-energy lighter weight companion vector extension to Neon, named Helium). Comparing AVX and SVE is interesting, because their design philosophies are much further apart than the relatively similar design philosophies behind SSE and Neon. AVX serves as a direct extension to SSE, to the point where even AVX’s YMM registers are really just an expanded version of SSE’s XMM registers (on processors supporting AVX, the XMM registers physically are actually just the lower 128 bits of the full YMM registers). Similar to AVX, the lower bits of SVE’s registers also overlap Neon’s registers, but SVE uses a new set of vector instructions separate from Neon. The big difference between AVX and SVE is that while AVX and AVX-512 specify fixed 256-bit and 512-bit widths respectively, SVE allows for different implementations to define different widths from a minimum of 128-bit all the way up to a maximum of 2048-bit, in 128-bit increments. At some point in the future, I think a comparison of AVX and SVE could be fun and interesting, but I didn’t touch on them in this post because of a number of current problems. In many Intel processors today, AVX (and especially AVX-512) is so power-hungry that using AVX means that the processor has to throttle its clock speeds down [Krasnov 2017], which can in some cases completely negate any kind of performance improvement. The challenge with testing SVE code right now is… there just aren’t many arm64 processors out that actually implement SVE yet! As of the time of writing, the only publicly released arm64 processor in the world that I know of that implements SVE is Fujitsu’s A64FX supercomputer processor, which is not exactly an off-the-shelf consumer part. NVIDIA’s upcoming Grace arm64 server CPU is also supposed to implement SVE, but as of 2021, the Grace CPU is still a few years away from release.

At the end of the day, for any application where vectorization is a good fit, not using vectorization means leaving a large amount of performance on the table. Of course, the example used in this post is just a single data point, and is a relatively small example; your mileage may and likely will vary for different and larger examples! As with any programming task, understanding your problem domain is crucial for understanding how useful any given technique will be, and as seen in this post, great care must be taken to structure code and data to even be able to take advantage of vectorization. Hopefully this post has served as a useful examination of several different approaches to vectorization! Again, I have put all of the code in this post in an open Github repository; feel free to play around with it yourself (or if you are feeling especially ambitious, feel free to use it as a starting point for a full vectorized BVH implementation)!

Addendum 2022-09-07

After I published this post, Romain Guy wrote in with a suggestion to use -ffast-math to improve the auto-vectorization results. I gave the suggestion a try, and the result was indeed markedly improved! Across the board, using -ffast-math cut the auto-vectorization timings by about half, corresponding to around a doubling of performance. Using ffast-math, the auto-vectorized implementation still trails behind the hand-vectorized and ISPC implementations, but by a much narrower margin than before, and overall is much much better than the compact scalar baseline. Romain previously presented a talk in 2019 about Google’s Filament real-time rendering engine, which includes many additional tips for making auto-vectorization work better.

References

Mike Acton. 2014. Data-Oriented Design and C++. In CppCon 2014.

AMD. 2020. “RDNA 2” Instruction Set Architecture Reference Guide. Retrieved August 30, 2021.

ARM Holdings. 2021. ARM Intrinsics. Retrieved August 30, 2021.

ARM Holdings. 2021. Helium Programmer’s Guide. Retrieved September 5, 2021.

ARM Holdings. 2021. SVE and SVE2 Programmer’s Guide. Retrieved September 5, 2021.

Holger Dammertz, Johannes Hanika, and Alexander Keller. 2008. Shallow Bounding Volume Hierarchies for Fast SIMD Ray Tracing of Incoherent Rays. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering 27, 4 (Jun. 2008), 1225-1234.

Manfred Ernst and Günther Greiner. 2008. Multi Bounding Volume Hierarchies. In Proc. of IEEE Symposium on Interactive Ray Tracing (RT 2008). 35-40.

Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 31.

Romain Guy and Mathias Agopian. 2019. High Performance (Graphics) Programming. In Android Dev Summit ‘19. Retrieved September 7, 2021.

Intel Corporation. 2021. Intel Intrinsics Guide. Retrieved August 30, 2021.

Intel Corporation. 2021. Intel ISPC User’s Guide. Retrieved August 30, 2021.

Thiago Ize. 2013. Robust BVH Ray Traversal. Journal of Computer Graphics Techniques 2, 2 (Jul. 2013), 12-27.

Tero Karras and Timo Aila. 2013. Fast Parallel Construction of High-Quality Bounding Volume Hierarchies. In Proc. of High Performance Graphics (HPG 2013). 89-88.

Vlad Krasnov. 2017. On the dangers of Intel’s frequency scaling. In Cloudflare Blog. Retrieved August 30, 2021.

Vlad Krasnov. 2018. NEON is the new black: fast JPEG optimization on ARM server. In Cloudflare Blog. Retrieved August 30, 2021.

Geoff Langdale and Daniel Lemire. 2019. Parsing Gigabytes of JSON per Second. The VLDB Journal. 28 (2019), 941-960.

Mark Lee, Brian Green, Feng Xie, and Eric Tabellion. 2017. Vectorized Production Path Tracing. In Proc. of High Performance Graphics (HPG 2017). Article 10.

Max Liani and Alex M. Wells. 2020. Supercharging Pixar’s RenderMan XPU with Intel AVX-512. In ACM SIGGRAPH 2020 Exhibitor Sessions.

Alexander Majercik, Cyril Crassin, Peter Shirley, and Morgan McGuire. 2018. A Ray-Box Intersection Algorithm and Efficient Dynamic Voxel Rendering. Journal of Computer Graphics Techniques 7, 3 (Sep. 2018), 66-81.

Daniel Meister, Shinji Ogaki, Carsten Benthin, Michael J. Doyle, Michael Guthe, and Jiri Bittner. 2021. A Survey on Bounding Volume Hierarchies for Ray Tracing. Computer Graphics Forum (Proc. of Eurographics) 40, 2 (May 2021), 683-712.

NVIDIA. 2021. NVIDIA OptiX 7.3 Programming Guide. Retrieved August 30, 2021.

Howard Oakley. 2021. Code in ARM Assembly: Lanes and loads in NEON. In The Eclectic Light Company. Retrieved September 7, 2021.

Matt Pharr. 2018. The Story of ISPC. In Matt Pharr’s Blog. Retrieved July 18, 2021.

Matt Pharr and William R. Mark. 2012. ispc: A SPMD compiler for high-performance CPU programming. In Proc. of nnovative Parallel Computing (InPar 2012). 184-196.

Martin Stich, Heiko Friedrich, and Andreas Dietrich. 2009. Spatial Splits in Bounding Volume Hierarchies. In Proc. of High Performance Graphics (HPG 2009). 7-13.

John A. Tsakok. 2009. Faster Incoherent Rays: Multi-BVH Ray Stream Tracing. In Proc. of High Performance Graphics (HPG 2009). 151-158.

Nathan Vegdahl. 2017. BVH4 Without SIMD. In Psychopath Renderer. Retrieved August 20, 2021.

Ingo Wald, Carsten Benthin, and Solomon Boulos. 2008. Getting Rid of Packets - Efficient SIMD Single-Ray Traversal using Multi-Branching BVHs. In Proc. of IEEE Symposium on Interactive Ray Tracing (RT 2008). 49-57.

Ingo Wald, Philipp Slusallek, Carsten Benthin, and Markus Wagner. 2001. Interactive Rendering with Coherent Ray Tracing. Computer Graphics Forum (Proc. of Eurographics 20, 3 (Sep. 2001), 153-165.

Ingo Wald, Sven Woop, Carsten Benthin, Gregory S. Johnson, and Manfred Ernst. 2014. Embree: A Kernel Framework for Efficient CPU Ray Tracing. ACM Transactions on Graphics (Proc. of SIGGRAPH) 33, 4 (Jul. 2014), Article 143.

Amy Williams, Steve Barrus, Keith Morley, and Peter Shirley. 2005. An Efficient and Robust Ray-Box Intersection Algorithm. _Journal of Graphics Tools) 10, 1 (Jan. 2005), 49-54.

Henri Ylitie, Tero Karras, and Samuli Laine. 2017. Efficient Incoherent Ray Traversal on GPUs Through Compressed Wide BVHs. In Proc. of High Performance Graphics (HPG 2017). Article 4.

Wikipedia. 2021. Advanced Vector Extensions. Retrieved September 5, 2021.

Wikipedia. 2021. Automatic Vectorization. Retrieved September 4, 2021.

Wikipedia. 2021. AVX-512. Retrieved September 5, 2021.

Wikipedia. 2021. Single Instruction, Multiple Threads. Retrieved July 18, 2021.

Wikipedia. 2021. SPMD. Retrieved July 18, 2021.

Closest Vertex	Translation
A (Origin)	\( (x,y) \rightarrow (2x, 2y) \)
B (1,0)	\( (x,y) \rightarrow (1 - 2(1-x), 2y) \rightarrow (2x - 1, 2y) \)
C (0,1)	\( (x,y) \rightarrow (2x, 1 - 2(1-y)) \rightarrow (2x, 2y - 1) \)

WIP	TH	CT	Notes
0	0	2	\(CT_0\)
1	1/2	2
2	1	2
3	2/3	2
4	2	2
6	3	2
8	4	2	\(TH_\Omega\) reached; end of flat \(CT_0\)
16	4	4	CT grows
32	4	8

WIP	TH	CT	Notes
0	0	2
1	1/2	2
2	2/3	3	CT grows, \(TH_\Omega\) not yet reached
3	1	3
4	1	4
6	6/5	5
8	4/5	10
16	4/3	12
32	2	16