Hacker News Personal Blogs 2020 | Top 100 Blogs

Underusing the NHS

Krishna's blog — Thu, 31 Dec 2020 02:00:00 GMT

A personal reflection on the author's experiences with the UK's National Health Service (NHS) and the challenges of overusing and underusing healthcare.

My Path to Developer Relations and Thoughts on the Future of DevRel

swyx's site RSS Feed — Thu, 31 Dec 2020 00:06:02 GMT

Alex Lakatos interviewed me for the Devrel Advent calendar he put together

My Journey from No CS Degree to AWS from Age 30 - 34

swyx's site RSS Feed — Wed, 30 Dec 2020 23:21:47 GMT

I was interviewed by Pete from No CS Degree on my journey, here are some extracts!

Swyx Interview on Daily Dev Tips

swyx's site RSS Feed — Wed, 30 Dec 2020 23:04:35 GMT

I was interviewed by Chris Bongers on Daily Dev Tips, here are some extracts.

The Road Chose Me Volume 2: Three years and 54,000 miles around Africa

The Road Chose Me — Wed, 30 Dec 2020 05:04:54 GMT

I’m extremely proud to announce I’ve published my second major book in print! ‘The Road Chose Me Volume 2’ documents my entire three year African expedition from start to finish. It covers the adventures,...

2020 Year in Review

Ben Overmyer's Site — Mon, 28 Dec 2020 02:00:00 GMT

Last year I didn't write a retrospective. This year I felt it was necessary, given all that's happened.

2020 has been a rough year. Covid-19, George Floyd's murder and the ensuing riots, and Donald Trump's increasingly bizarre behavior are all pretty well known and covered elsewhere. My sister lost three cats this year; two to old age and one small kitten to disease. My wife's grandfather passed away due to Covid.

Going fully remote was relatively easy for my wife and me, as we both have professions that mesh well with it. In March just after the coronavirus lockdown began in the USA I changed clients. I have never seen anyone at my current client in person; we have only ever interacted in Teams and Zoom. Video chat is definitely preferable to voice-only, but we still need spatially-aware video chat before it can get close to replicating in-person interaction. Side conversations and awareness of eye contact are sorely lacking.

My wife and I went on our first cruise together in late February and early March. This was right after the first cruise ship off East Asia got quarantined, but right before lockdown began. The ship was filled with hand sanitizer stations and we were required to wash our hands before entering the buffet restaurants, but everything was otherwise no different from normal operation. The daily email updates about the increasing number of ports that were off-limits due to Covid-19 were disconcerting, though. Thankfully, the cruise was in the Caribbean, so it was one of the last regions to be affected by Covid-19 directly, and our ship was never infected.

Work has been good to me this year. At the end of last year, I won my employer's biggest award, the SDG Award. At the end of this year, I won it again. Throughout the year I got a few smaller awards too. That felt really good, and I've felt very appreciated at Solution Design Group. I joined the Employee Experience Committee and gave a presentation on roguelike development at last year's internal SDG Presents event. The Tabletop Games company community that I lead went quiet for the most part over 2020. I regret that a bit; the events we held for it last year were a lot of fun for everyone. In hindsight, running them virtually would have been a nice change of pace from all the negative events of the year.

From a learning perspective, I dove deep into Chef this year. It went so well that not only do I no longer hate Chef, I've become an advocate for it in some scenarios. I also began studying Estonian history and the Estonian language. Estonia is a major focus in my personal life right now as we gear up to move there. Their history is fascinating and so much of it is unknown outside the Baltic region itself. I was blown away when I found out that Estonia is effectively the Silicon Valley of Europe, with services like Skype being born there. More on all of this will come in a future blog post.

With dramatically reduced in-person social events this year, I needed to find new ways to interact with people. The Dungeons & Dragons campaign that I played in went virtual, and I stayed with it for several months, but ultimately it was too big for me to play in virtually. For technical reasons we ended up dropping video chat in favor of voice chat, and that meant it was difficult to get attention without interrupting someone else. I ended up leaving that campaign and joining a smaller one, and that is working out much better. I also have attended a handful of virtual happy hours over the year, and those have been fun, but sporadic. An unexpected social outlet came about in the form of the tilde community that I belong to - Tilde Town. I interacted with them much more this year than in any of the previous five years I've been a part of it, and started to make some friends. Discord communities have also provided a social environment to a lesser degree, and I've become more involved in the Royal Manticoran Navy, a fan organization dedicated to David Weber's Honorverse novels.

My personal project Iron Arachne has seen a wealth of development in 2020. Near the beginning of the year I started running into limitations of the Golang ecosystem and rewrote the entire project into a Laravel monolith. This worked much better, even if certain operations became much slower. The site expanded to the point where users had accounts and could save their generated results, and the heraldry generator grew and became more useful and complex. Unfortunately, the infrastructure costs to support the site also grew, and in the last couple months I could no longer justify their expense. So I rewrote the site again, this time as a single-page app in Vue.js that I could host for free on Netlify. The site lost a lot of functionality in the process, but it now costs nothing to maintain. In a happy turn of events, the star system generator actually became much faster in the conversion to JavaScript, going from a 3-5 second generation time to under 60ms. I will talk more about my plans for Iron Arachne in my first blog post of 2021.

The last big change of 2020 that I'll mention here is my return to the Apple ecosystem. I replaced my Pixel 2 with an iPhone 11 Pro last year, and this year I replaced my custom Windows/Linux PC with a 2019 16" MacBook Pro, and my Fitbit Charge 3 with an Apple Watch. I don't regret this decision at all. While there have been some periods of adjustment, it has been a positive experience overall. Gaming on the MacBook Pro is not as good as it was on my custom rig, but Bootcamp lets me play most of my PC games without issue. I wish the GPU was better but otherwise it works fine.

All in all, 2020 has been a lot better than it could have been for me personally. I am deeply grateful for being in a position to weather the storm this way. Hopefully 2021 will be better for the world in general. The vaccines and new American leadership give me hope.

The year in books: 20 to recommend in 2020

Notes on software development — Sun, 27 Dec 2020 02:00:00 GMT

This year I finished 47 books, up from last year but not a personal best. The breakdown was 17 non-fiction and 30 fiction. Another 20-30 remain started but unfinished this year.

Non-fiction

The 8 non-fiction books I most recommend are:

Fashionapolis: The Price of Fast Fashion and the Future of Clothes (Must read)
Effective Python: 90 Specific Ways to Write Better Python (Must read; truly excellent for Python programmers, I recommend this to anyone I work with)
The Machine that Changed the World (Must read)
Europe: The Struggle for Supremacy from 1453 to the Present
Wind, Sand and Stars
American Colussus: The Triumph of Capitalism, 1865-1900
Making Common Sense of Japan
The German Genius

The 3 books I recommend you not to waste time on are: "The Two Koreas", "The Price of Inequality", and "Ninety Percent of Everything: Inside Shipping".

The whole list

The Two Koreas by Don Oberdorfer
- Interesting but not a huge fan, seemed pretty biased against South Korea somehow
Forbidden Nation: A History of Taiwan by Jonathan Manthorpe
Effective Python: 90 Specific Ways to Write Better Python by Brett Slatkin
A Philosophy of Software Design by John Ousterhout
- Came as a recommendation from someone on Twitter, ultimately not a huge fan. Still looking for high quality books on software design
The Price of Inequality by Joseph E. Stiglitz
- Agreed with the premise but the book was incoherent and too self-assuring
Paris Reborn: Napoléon III, Baron Haussmann, and the Quest to Build a Modern City by Stephane Kirkland
Fashionapolis: The Price of Fast Fashion and the Future of Clothes by Dana Thomas
A Moveable Feast by Ernest Hemingway
- I normally love Hemingway's writing but this particular book was not very coherent
Europe: The Struggle for Supremacy from 1453 to the Present by Brendan Simms
- Such an excellent introduction to the continent for Americans who otherwise don't have great background
Wind, Sand and Stars by Antoine de Saint-Exupéry
- A beautiful memoir of flights by the author of The Little Prince, very similar in style to Hemingway
American Colussus: The Triumph of Capitalism, 1865-1900 by H.W. Brands
- Baby's first primer on unions, (I need more recommendations on the history of unions)
Making Common Sense of Japan by Steven R. Reed
- It can be difficult to find English translations of Korean, Japanese history by Korean and Japanese authors; this is a good one by an American professor
The Machine that Changed the World by James P. Womack
- An excellent, well-researched history of automobile manufacturing in the US, Europe and Japan from the 1900s to 1990; how Japan ate everyone's lunch
The United States of Europe by T.R. Reid
- Very light introduction to the European Union
The Soul of a New Machine by Tracy Kidder
- Overhyped by the internets, but not bad
The German Genius by Peter Watson
- Dense but excellent introduction to many famous Germans in many fields throughout time
Ninety Percent of Everything: Inside Shipping by Rose George

Fiction

I'm trying to read more from non-English authors. If you see non-English authors in the vein of these here that you can recommend, I'd love to hear from you.

The 12 fiction books I most recommend are:

Planet of the Apes (Must read, yes even if you've seen the film)
All Quiet on the Western Front (Must read)
The Mouse That Roared (Must read)
The Dead Mountaineer's Inn
The Golem and the Jinni
Neverwhere
Dubliners
Old Man's War
The Inspector Barlach Mysteries: The Judge and His Hangman and Suspicion
Fantômas
Foundation
Out of the Silent Planet

The only book I really didn't like was "Invisible Cities".

The whole list

March Violets by Philip Kerr (Scottish)
Liberty Bar by Georges Simenon (Belgian)
The Late Monsieur Gallet by Georges Simenon (Belgian)
Dubliners by James Joyce (Irish)
Tales of the City by Amistead Maupin (American)
The Third Policeman by Flann O'Brien (Irish)
44 Scotland Street by Alexander McCall Smith (British-African)
Knots and Crosses by Ian Rankin (Scottish)
I Hear Your Voice by Kim Young Ha (South Korean)
The Golem and the Jinni by Helene Wecker (American)
The Tokyo Zodiac Murders by Shimada Sōji (Japanese)
The Screwtape Letters by C.S. Lewis (English)
Neverwhere by Neil Gaiman (English)
Old Man's War by John Scalzi (American)
Tales from Earthsea by Ursula K. Le Guin (American)
Solaris by Stanisław Lem (Polish)
A Wizard of Earthsea by Ursula K. Le Guin (American)
Planet of the Apes by Pierre Boulle (French)
The Dead Mountaineer's Inn by Arkady Strugatsky (Russian)
Invisible Cities by Italo Calvino (Cuban-born Italian)
The Inspector Barlach Mysteries: The Judge and His Hangman and Suspicion by Friedrich Dürrenmatt (Swiss)
Fantômas by Marcel Allain (French)
All Quiet on the Western Front by Erich Maria Remarque (Germany)
A Crime in Holland by Georges Simenon (Belgian)
The Wonderful Adventure of Nils Holversson by Selma Lagerlöf (Swedish)
Foundation by Isaac Asimov (Russian-born American)
Out of the Silent Planet by C.S. Lewis (English)
The Spy Who Came in from the Cold by John le Carré (English)
The Bat by Jo Nesbø (Norwegian)
The Mouse That Roared by Leonard Wibberley (Irish-born American)

Out of 47 books read this year, here's the 20 I recommend to you (gave them 4/5 stars or better). I'm trying to read more non-English authors so I'd love to hear if there are authors with similar style on this list you'd recommend!https://t.co/FjHcvHpRSr
— Phil Eaton (@phil_eaton) December 27, 2020

You're Allowed To Make Your Own Tools

swyx's site RSS Feed — Sat, 26 Dec 2020 20:04:11 GMT

Making personal side projects for fun and profit.

An Annotated Guide to React Server Components

swyx's site RSS Feed — Wed, 23 Dec 2020 03:18:46 GMT

React Server Components are a nuanced, sweeping addition to React's existing capabilities. This is my guide.

Static analysis with semgrep: practical examples using Docker

Notes on software development — Sun, 20 Dec 2020 02:00:00 GMT

In this post we'll get a basic semgrep environment set up in Docker running some custom rules against our code.

Existing linters

Linters like pylint for Python or eslint for JavaScript are great for general, broad language standards. But what about common nits in code review like using print statements instead of a logger, or using a defer statement inside a for loop (Go specific), or the existence of multiple nested loops.

Most developers don't have experience working with language parsing. So it's fairly uncommon in small- and medium-sized teams to see custom linting rules. And while no single linter or language is that much more complex than the other (it's all just AST operations), there is a small penalty to learning the AST and framework for each language linter.

Semgrep

Semgrep is a generic tool for finding patterns in source code. Unlike traditional regex (and traditional grep) it can find recursive patterns. This makes it especially useful as a tool to learn for finding patterns in any language.

An advantage of semgrep rules is that you can learn the semgrep pattern matching syntax (which is surprisingly easy) and then you can write rules for any language you'd like to write rules for.

And while the online rule tester is awesome, I had a hard time going from that to a working sample on my own laptop with Docker. We'll do just that.

Catching print statements in Python

Let's say we want a script to fail on any use of print statements in Python:

$ cat test/python/simple-print.py
def main():
  print("DEBUG: here")
  print("DEBUG: ", "now here")

The current default example shown in the online editor happens to be for just this. Click the Advanced tab and you'll see the following:

rules:
- id: fail-on-print
  pattern: |
    print("...")
  message: |
    Semgrep found a match
  severity: WARNING

Copy this into config.yml. Let's modify the pattern to warn on all print calls, not just print calls with a single string argument:

rules:
- id: fail-on-print
  pattern: |
    print(...)
  message: |
    Semgrep found a match
  severity: WARNING

The editor doesn't mention it (nor do any docs I can find) but we also need to include two keys in the individual rule object: mode and languages.

rules:
- id: fail-on-print
  pattern: |
    print(...)
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

Semgrep fails really weirdly if you set mode to anything other than search, but it won't warn you that what you set is garbage. The languages setting is similarly fickle and doesn't give you much feedback if you set it incorrectly.

Also, I'm using the "generic" language here because I don't understand the difference between languages and as far as I'm concerned the syntax I'm using here is already pretty generic.

We run the semgrep Docker image:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=config.yml test/python
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/python/simple-print.py
severity:warning rule:fail-on-print: Semgrep found a match

2:print("DEBUG: here")
ran 1 rules on 1 files: 1 findings"")

And there we've got our warning!

Not completely clear to me why we're getting warned about a new version when we've pulled latest as the linked docs suggest. Maybe there's a newer version that hasn't made it into a Docker image yet.

Catching fmt.Print* statements in Go

Let's say we also want to fail on print statements in Go (because we should use a logger instead):

$ cat test/golang/simple-print.go
package main

import "fmt"

func main() {
  a := fmt.Sprintf("here")
  fmt.Println(a)
  fmt.Printf("%s\n", a)
  e := fmt.Errorf("My crazy error")
}

We could try to look for any import "fmt" code in a file but that would fail on uses of fmt.Sprintf or fmt.Errorf which are fine. Instead we'll just focus on uses of fmt.Printf or fmt.Println:

$ cat go-config.yml
rules:
- id: fail-on-print
  pattern-either:
    - pattern: fmt.Printf(...)
    - pattern: fmt.Println(...)
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

Run the Go config against the Go files:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config.yml test/golang
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/golang/simple-print.go
severity:warning rule:fail-on-print: Semgrep found a match

8:fmt.Printf("%s\n", a)
--------------------------------------------------------------------------------
7:fmt.Println(a)
ran 1 rules on 1 files: 2 findings

Cool! Making some sense. Now let's try a harder pattern.

Catching triple-nested for loops

Let's try to warn on the triple-nested loop in this code:

$ cat test/golang/loopy.go
package main

import "log"

func main() {
  doneFirst := false
  for i := 0; i < 10; i++ {
    log.Print(i)

    for j := 0; j < 100; j++ {
      c := i * j

      going := true
      k := 0
      for going {
        if k == c {
          break
        }

        k++
        log.Print(k)
      }
    }

    doneFirst = true
  }
}

If we want to catch the use of nested for loops here then we'll need to search for the loops surrounded by arbitrary syntax. Semgrep's ... syntax makes this easy.

$ cat go-config2.yml
rules:
- id: fail-on-3-loop
  pattern: |
    for ... {
      ...
      for ... {
        ...

        for ... {
          ...
        }
        ...
      }
      ...
    }
  message: |
    Semgrep found a match
  severity: WARNING
  mode: search
  languages: ["generic"]

And run semgrep:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config2.yml test/golang
A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
running 1 rules...
test/golang/loopy.go
severity:warning rule:fail-on-3-loop: Semgrep found a match

7:for i := 0; i < 10; i++ {
8:              log.Print(i)
9:
10:             for j := 0; j < 100; j++ {
11:                     c := i * j
12:
13:                     going := true
14:                     k := 0
15:                     for going {
16:                             if k == c {
-------- [hid 10 additional lines, adjust with --max-lines-per-finding] --------
ran 1 rules on 2 files: 1 findings

That's just swell.

Limits of static analysis

Now let's say we refactor one of the inner loops into its own function.

$ cat test/golang/loopy.go
package main

import "log"

func inner(i, j int) {
  c := i * j

  going := true
  k := 0
  for going {
    if k == c {
      break
    }

    k++
    log.Print(k)
  }
}

func main() {
  doneFirst := false
  for i := 0; i < 10; i++ {
    log.Print(i)

    for j := 0; j < 100; j++ {
      inner(i, j)
    }

    doneFirst = true
  }
}

And run semgrep again:

$ docker run -v "${PWD}:/src" returntocorp/semgrep --config=go-config2.yml test/golang
 A new version of Semgrep is available. Please see https://github.com/returntocorp/semgrep#upgrading for more information.
 running 1 rules...
 ran 1 rules on 2 files: 0 findings

Well great. The 3-nested loop still exists but we can't find it anymore because it's not syntactically obvious anymore.

At this point we'd need to start getting into linting based on runtime analysis. If you know of a tool that does this and lets you write rules like semgrep for it, please tell me!

In summary

In the end though, it's still very useful to be able to learn a single language for writing syntax rules at a high level to enforce behavior in code. Furthermore, a generic syntax matcher helps you write easily write rules for things that don't already have linters like YAML or JSON configuration or Vagrantfiles.

It can be annoying to work around some missing docs in semgrep but overall it's a great tool for the kit.

#semgrep is a really neat tool for syntactic analysis. Here are a few simple examples (catch print statements, triple nested loops, etc.) using Docker. Includes some necessary info the docs don't get intohttps://t.co/UDHEH5JmOa
— Phil Eaton (@phil_eaton) December 20, 2020

2020–12–19: Away from keyboard for the rest of the month

megi's PinePhone Development Log — Sat, 19 Dec 2020 01:00:00 GMT

How to Design Almost Any Backend and Deploy It to AWS with No Code

swyx's site RSS Feed — Fri, 18 Dec 2020 21:42:48 GMT

With the Amplify Sandbox, it is really easy to model and think through any app backend scenario

Sheffield Is The New Derby; Also Arsenal

infrequent oscillations — Fri, 18 Dec 2020 12:20:42 GMT

In which we make dire predictions.

Making statistics lie for the 2020 Presidential election

Bytepawn - Marton Trencseni — Thu, 17 Dec 2020 01:00:00 GMT

After the 2020 US presidential election, the Trump campaign filed over 50 lawsuits and attacked the integrity of the elections by claiming there was voter fraud. One of the last lawsuits was filed in the Supreme Court of the United States by the state of Texas. Here I look at the statistical claims made in this lawsuit that were supposed to show irregularities in the Georgia vote.

Why You Should Pre-Sell

swyx's site RSS Feed — Tue, 15 Dec 2020 20:00:25 GMT

My thoughts on preselling, from having done it successfully exactly one time.

2020–12–15: Battery thermistor measurements

megi's PinePhone Development Log — Tue, 15 Dec 2020 01:00:00 GMT

Benchmarking Rustpython With Criterion

Tom Forbes — Mon, 14 Dec 2020 15:27:30 GMT

I’ve been very excited about RustPython since I saw the project being presented at FOSDEM 2019. Last week I finally got to contributing something to the project ! For context, RustPython benchmarks specific Python files against CPython, the “reference implementation” for Python (read: the only real ...

Announcing the AutoEPM Patch Tracker

Subclassed — Mon, 14 Dec 2020 03:05:16 GMT

UPDATE: This service has been discontinued; this post is left as historical reference. If you need this sort of thing, contact me on LinkedIn.

With the recent release of EPM 11.2, I thought it would be a good time to make public a tool I've built and used "behind the scenes" for quite some time, the AutoEPM Patch Tracker.

It's a patch tracker focused on EPM products, automatically aggregating updates and pointing to readmes, hopefully a bit faster to peruse than My Oracle Support. It also recommends Critical Product Updates that can be applied to the EPM stack, and the most common "standard" updates (only for 11.1.2.4 at the moment). With a free account via LinkedIn sign-on, all lists can be downloaded in CSV, JSON, and XML (Atom).

It's only a small showcase of what we can do at AutoEPM by Targlet. If you'd like to find out how to integrate this level of automation and intelligence to your EPM enviroments, feel free to reach out! I'll be at OpenWorld London next week as well, always happy to have a chat.

Mimicry vs Reflexivity

swyx's site RSS Feed — Mon, 14 Dec 2020 00:35:02 GMT

Two opposing forces that describe how perception and reality do battle.

You Can Run Containers on AWS Amplify Now

swyx's site RSS Feed — Fri, 11 Dec 2020 16:24:37 GMT

AWS just added the easiest way to deploy long running serverful code alongside a serverless architecture.

Email whitelisting

infrequent oscillations — Thu, 10 Dec 2020 19:17:53 GMT

Settings -> Mail -> Accounts -> [appropriate account] -> turn on Notes Go to iOS Notes. Go to the Folders view and find the same account as step 1. Create a note with the first line “Whitelist”, then a blank line, then each domain you want to whitelist on its own line. Wait for the automatic processing to pick up the changes (a few times a day.) Note this isn’t a pure whitelist - instead it adds a base score of -12.

Bloom Filters - Much, much more than a space efficient hashmap!

Ben E. C. Boyter — Thu, 10 Dec 2020 02:00:00 GMT

A bloom filter is one of those data structures you are probably already aware of, or have at least heard about. For those looking for a simple recap, they are a probabilistic data structure which can be used to determine if something is in a set or not, giving a slight chance of returning a false positive result for some checks but while using less space than a full hashmap.

The Strange Case of the Danish Aarhus Mafia

swyx's site RSS Feed — Wed, 09 Dec 2020 17:46:36 GMT

Why are so many CS pioneers from Denmark?

Update on Old Posts

infrequent oscillations — Sun, 06 Dec 2020 10:52:43 GMT

In which we look back and update ourselves.

Language Servers are the New Frameworks

swyx's site RSS Feed — Sat, 05 Dec 2020 22:51:05 GMT

Developer Experience is shifting left, all the way to onKeyUp.

2020–12–05: Alternate EG25G userspace project

megi's PinePhone Development Log — Sat, 05 Dec 2020 01:00:00 GMT

The UX of Proving Our Humanity To Machines

swyx's site RSS Feed — Thu, 03 Dec 2020 18:13:27 GMT

What happens when we can't convince machines that we are human?

Comparing conversion at control and treatment sites

Bytepawn - Marton Trencseni — Thu, 03 Dec 2020 01:00:00 GMT

In real-life, non-digital situations, it's often not feasible to run true A/B tests. In such cases, we can compare before and after rollout conversions at a treatment site, while using a similar control site to measure and correct for seasonality. The post discusses how to compute increasingly correct p-values and bayesian probabilities in such scenarios.

2020–12–02: Fixing fallout from the new WiFi driver

megi's PinePhone Development Log — Wed, 02 Dec 2020 01:00:00 GMT

2020–11–30: Improved ANX7688 driver and increased I2C speed

megi's PinePhone Development Log — Mon, 30 Nov 2020 01:00:00 GMT

Writing on books - Fantastic Beasts and Where to Find Them

Nemo's Home — Sun, 29 Nov 2020 02:00:00 GMT

“Fantastic Beasts and Where to Find Them” is a curious book:

The in-universe book is written by Newt Scamander and was published in 1927.
The first edition of the companion book was published in 2001. This is apparently the 52nd in-universe edition with a foreword from Dumbledore and was released to the muggle world for charity.
The 2001 edition pretends to be Harry’s copy of the book as of the end of Harry’s 4th year at Hogwarts. As such it includes hand-written comments from the trio. (Yes, Hermione writes on books!)

However, with the release of the film of the same name in 2016 - a new edition was released with lots of changes:

6 new beasts that made an appearance in the film were added to the book¹:
- Hidebehind
- Hodag
- Horned Serpent
- Snallygaster
- Thunderbird
- Wampus cat
The hand-lettering was removed.
Dumbledore’s foreword is removed from the book, in favor of a in-universe foreword from Newt Scamander.
“About the Author” section changes Newt’s background. He no longer graduates from Hogwarts, just “leaves” it, as portrayed in the film.

All of these changes are meant to fix the inconsistencies in the book with the canon, however that also makes the book much less charming. I got myself a copy of the Hogwarts Library boxset a few years ago, which includes the newer edition of the book (Bloomsbury) - that means no witty comments from Ron.

Since it didn’t have the hand-lettering, I took it upon myself to fix that mistake. Thankfully, lists of all the comments in 2001 edition are available on the internet. The trickiest part was the “this book belongs to” page, which is missing from the newer edition. I ended up creating a faux-library card for that instead.

Here is what it looks like:

Ron plays hangman and loses.
Hermione writes on books!

Thanks to Bhavya for helping with the troll illustration.

I didn’t like the new additions, they sound less like a textbook and more like a transcript of what happened in the film. ↩

A weirdness with tinydns

infrequent oscillations — Fri, 27 Nov 2020 18:36:33 GMT

In which tinydns needs better logging.

My Favorite New Podcasts in 2020

swyx's site RSS Feed — Fri, 27 Nov 2020 18:18:14 GMT

An update to my favorite podcasts list with the ones I have found this year

John Oliver Thirst Bot

infrequent oscillations — Fri, 27 Nov 2020 11:31:08 GMT

In which John Oliver is Thirsty.

Emulating linux/AMD64 userland: interpreting an ELF binary

Notes on software development — Thu, 26 Nov 2020 02:00:00 GMT

In this post we'll stumble toward a working emulator for a barebones C program compiled for linux/AMD64. The approach will be slightly more so based on observation than by following a spec; a great way to quickly become familiar with a topic, and a bad way to guarantee correctness.

The goal:

$ cat tests/simple.c
int main() {
  return 4;
}
$ gcc tests/simple.c
$ go build -o main
$ ./main a.out && echo $?
4

This may look ridiculously simple but when you don't know how to deal with a binary or how instructions are encoded, it will take a few hours to write an emulator that can generally handle this program!

Code for this project is available on Github.

Background

AMD64, x86_64 or x64 are different names for AMD's widely adopted 64-bit extension to Intel's x86 instruction set (i.e. the encoding and semantics of x86 binaries). AMD64 is a superset of x86 (introducing 64-bit registers and operations) and thus backwards compatible with x86 programs.

A year and a half ago I first got into emulation with an AMD64 emulator in JavaScript. The JavaScript emulator interpreted the textual representation of AMD64 programs (e.g. MOV RBP, RSP, Intel's assembly syntax). A C program had to be compiled with -S to produce an assembly file that the JavaScript emulator could read (i.e. gcc -S tests/simple.c) This was a great way to get started with emulation by ignoring the complexity of encoded instructions and executable formats.

If we dig into the binary file produced by gcc on Linux we learn that it is an ELF file.

$ gcc test/simple.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=d0b5c742b9fbcbcca4dfa9438a8437a8478a51bb, for GNU/Linux 3.2.0, not stripped

ELF is responsible for surrounding the actual binary-encoded program instructions with metadata on exported and imported C identifiers and program entrypoint. But for simple programs like this initial emulator, we can ignore export/imports. We'll only use the ELF metadata to find out where the instructions for our main function start.

Where is main?

If we use an ELF reader+disassembler on the binary generated by gcc and search for main we can find its address.

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax
  40110f:       5d                      pop    %rbp
  401110:       c3                      retq
  401111:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  401118:       00 00 00
  40111b:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

0000000000401120 <__libc_csu_init>:

This means that the function, main, starts at address 0x401106 in memory. Furthermore, this implies that the binary must be loaded into CPU memory such that the CPU can jump here to execute our program.

In truth, main is not this program's entrypoint. If we run objdump -x a.out we can see that the ELF entrypoint is 0x401020.

$ objdump -x a.out
a.out:     file format elf64-x86-64
a.out
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000401020

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
             filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r--

This is because the actual entrypoint gcc sets up is a function called _start. The libc prelude beginning with _start is responsible for initializing the libc runtime, calling our main function and executing the exit syscall with the return value of main.

objdump -d a.out | grep -A10 '<_start>'
0000000000401020 <_start>:
  401020:       f3 0f 1e fa             endbr64
  401024:       31 ed                   xor    %ebp,%ebp
  401026:       49 89 d1                mov    %rdx,%r9
  401029:       5e                      pop    %rsi
  40102a:       48 89 e2                mov    %rsp,%rdx
  40102d:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
  401031:       50                      push   %rax
  401032:       54                      push   %rsp
  401033:       49 c7 c0 90 11 40 00    mov    $0x401190,%r8
  40103a:       48 c7 c1 20 11 40 00    mov    $0x401120,%rcx

But because all this libc initialization is relatively complicated we're just going to skip the actual ELF entrypoint for now. Our emulator will locate main, load the binary into memory, jump to the start of main, and set the exit code of the emulator to the result of main.

As you can see, this ELF binary has its own hard-coded view of where it will be in memory. What if our CPU were to run multiple process at once? We might give each process its own virtual memory space and map back to a real memory space so each process (and by extension, compilers) doesn't have to think about how they fit into memory relative to other processes.

The last question to figure out is where to load the ELF binary into emulator memory so that addresses in memory are where the program expects.

As it turns out, there is a piece of metadata called section headers that contain an address and a offset from the start of the ELF file. By subtracting this we can get the location the file expects to be in memory.

$ objdump -x a.out
a.out:     file format elf64-x86-64
a.out
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000401020

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
             filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r--

That is: 0x400040 (vaddr) - 0x40 (off) = 0x400000. Judging from a Google search this seems to be a pretty common address where ELF binaries are loaded into memory.

ELF and Go

Binary file formats tend to be a pain to work with because, to enable greater compression, everything ends up being a pointer to something else. So you end up jumping all around the file just to stitch information back together.

So the one third-party-ish library we'll use is Go's builtin debug/elf package. With this library we can load an ELF binary and iterate over symbols and sections to discover the location of main and the start address for the binary in memory.

Editing in main.go:

package main

import (
    "bytes"
    "debug/elf"
    "fmt"
    "io/ioutil"
    "log"
    "os"
)

type process struct {
    startAddress uint64
    entryPoint   uint64
    bin          []byte
}

func readELF(filename, entrySymbol string) (*process, error) {
    bin, err := ioutil.ReadFile(filename)
    if err != nil {
        return nil, err
    }

    elffile, err := elf.NewFile(bytes.NewReader(bin))
    if err != nil {
        return nil, err
    }

    symbols, err := elffile.Symbols()
    if err != nil {
        return nil, err
    }

    var entryPoint uint64
    for _, sym := range symbols {
        if sym.Name == entrySymbol && elf.STT_FUNC == elf.ST_TYPE(sym.Info) && elf.STB_GLOBAL == elf.ST_BIND(sym.Info) {
            entryPoint = sym.Value
        }
    }

    if entryPoint == 0 {
        return nil, fmt.Errorf("Could not find entrypoint symbol: %s", entrySymbol)
    }

    var startAddress uint64
    for _, sec := range elffile.Sections {
        if sec.Type != elf.SHT_NULL {
            startAddress = sec.Addr - sec.Offset
            break
        }
    }

    if startAddress == 0 {
        return nil, fmt.Errorf("Could not determine start address")
    }

    return &process{
        startAddress: startAddress,
        entryPoint:   entryPoint,
        bin:          bin,
    }, nil
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Binary not provided")
    }

    proc, err := readELF(os.Args[1], "main")
    if err != nil {
        panic(err)
    }

    fmt.Printf("Start: 0x%x\nEntry: 0x%x\n", proc.startAddress, proc.entryPoint)
}

We can test on a basic compiled C program:

$ cat tests/simple.c
int main() {
  return 4;
}
$ gcc tests/simple.c
$ go build -o main
$ ./main a.out
Start: 0x400000
Entry: 0x401106

And verify against objdump:

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp'>'

And that's it for dealing with ELF. Now we can sketch out a virtual CPU and how we deal with interpreting instructions starting at this address.

The CPU

AMD64 counts on being able to store values in registers and memory, sometimes through direct addressing and sometimes indirectly using stack operations (push and pop). And userland processes count on being loaded into CPU memory so the CPU can jump to the process entrypoint and process.

type cpu struct {
    proc    *process
    mem     []byte
    regfile *registerFile
    tick    chan bool
}

func newCPU(memory uint64) cpu {
    return cpu{
        mem:     make([]byte, memory),
        regfile: &registerFile{},
        tick:    make(chan bool, 1),
    }
}

The tick channel is so that later on we can wrap the emulator in a terminal debugger. But by default we'll just set up a goroutine to tick forever.

func repl(c *cpu) {
  // TODO
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Binary not provided")
    }

    proc, err := readELF(os.Args[1], "main")
    if err != nil {
        panic(err)
    }

    debug := false
    for _, arg := range os.Args[1:] {
        switch arg {
        case "--debug":
            fallthrough
        case "-d":
            debug = true
        }
    }

    // 10 MB
    cpu := newCPU(0x400000 * 10)

    go cpu.run(proc)
    if debug {
        repl(&cpu)
    } else {
        for {
            cpu.tick <- true
        }
    }
}

Registers

To emulate a simple program like our tests/simple.c, we'll only need to support a few common registers. The order is important so that we can use the Go identifiers when we want to refer to the encoded integer value of the register.

type register int

const (
    // These are in order of encoding value (i.e. rbp is 5)
    rax register = iota
    rcx
    rdx
    rbx
    rsp
    rbp
    rsi
    rdi
    r8
    r9
    r10
    r11
    r12
    r13
    r14
    r15
    rip
    rflags
)

var registerMap = map[register]string{
    rax:    "rax",
    rcx:    "rcx",
    rdx:    "rdx",
    rbx:    "rbx",
    rsp:    "rsp",
    rbp:    "rbp",
    rsi:    "rsi",
    rdi:    "rdi",
    r8:     "r8",
    r9:     "r9",
    r10:    "r10",
    r11:    "r11",
    r12:    "r12",
    r13:    "r13",
    r14:    "r14",
    r15:    "r15",
    rip:    "rip",
    rflags: "rflags",
}

type registerFile [18]uint64

func (regfile *registerFile) get(r register) uint64 {
    return regfile[r]
}

func (regfile *registerFile) set(r register, v uint64) {
    regfile[r] = v
}

Of immediate importance will be rip, rsp, and rax registers. rip is used to track the current instruction to process. It will generally be incremented except for when dealing with function calls and returns. rsp is used as a pointer to the top of a stack in memory. It is incremented and decremented as values are pushed and popped on this stack. Finally, rax is used to pass function return values.

Loading a program

Running a program is a matter of loading the program into memory, setting the stack pointer to the last address of memory (in x86 the stack grows down), pointing rip at the entrypoint, and looping until the entrypoint function returns.

func writeBytes(to []byte, start uint64, bytes int, val uint64) {
    for i := 0; i < bytes; i++ {
        to[start+uint64(i)] = byte(val >> (8 * i) & 0xFF)
    }
}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        // TODO: deal with instructions

        // move to next instruction
        c.regfile.set(rip, ip+1)
    }
}

func (c *cpu) run(proc *process) {
    copy(c.mem[proc.startAddress:proc.startAddress+uint64(len(proc.bin))], proc.bin)
    c.regfile.set(rip, proc.entryPoint)
    initialStackPointer := uint64(len(c.mem)-8)
    writeBytes(c.mem, initialStackPointer, 8, initialStackPointer)
    c.regfile.set(rsp, initialStackPointer)
    c.loop(initialStackPointer)
    os.Exit(int(c.regfile.get(rax)))
}

We write the initial stack pointer address into the stack so that when the program final returns, it will return to this address at which pointer we can exit the program.

And now we're ready to start interpreting instructions.

Instruction decoding

Using objdump we get a sense for what the program decodes to.

$ objdump -d a.out | grep -A10 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax
  40110f:       5d                      pop    %rbp
  401110:       c3                      retq
  401111:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
  401118:       00 00 00
  40111b:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

0000000000401120 <__libc_csu_init>:

We see that 0x55 means push %rbp. And we also see that instructions aren't a fixed number of bytes. Some are one byte, some are seven. Some (not shown) are even longer.

Thankfully instructions follow some fairly simple patterns. There are a set of prefix instructions and a set of real instructions. So far we should be able to tell on the first byte whether the instruction is a prefix instruction and, if not, how many bytes the instruction will take up on the whole.

push

To support a new instruction, we'll look up 0x55 in an opcode table like this. Clicking on 55 in the opcode index we see that this is indeed a push instruction. 50+r means that we have to subtract 0x50 from the opcode to determine the register we should push.

The register will be 0x55 - 0x50 = 5 which if we look up in a register table is rbp. Since we set up our register enum in code in this order, we'll be able to just use the constant rbp in Go code.

Finally, since the next instruction numerically is 0x58 we know that this instruction is identified by being between 0x50 and 0x57 inclusive. This is all the info we need to handle this instruction.

// helper for dumping byte arrays as hex
func hbdebug(msg string, bs []byte) {
    str := "%s:"
    args := []interface{}{msg}
    for _, b := range bs {
        str = str + " %x"
        args = append(args, b)
    }
    fmt.Printf(str+"\n", args...)
}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        if inb1 >= 0x50 && inb1 < 0x58 { // push
            regvalue := c.regfile.get(register(inb1 - 0x50))
            sp := c.regfile.get(rsp)
            writeBytes(c.mem, sp-8, 8, regvalue)
            c.regfile.set(rsp, uint64(sp-8))
        } else {
            hbdebug("prog", c.mem[ip:ip+10])
            panic("Unknown instruction")
        }

        c.regfile.set(rip, ip+1)
    }
}

If we try this out now we should expect it to panic on the second byte, 0x48.

$ go build -o main
$ ./main a.out
prog: 48 89 e5 b8 4 0 0 0 5d c3
panic: Unknown instruction

goroutine 19 [running]:
main.(*cpu).loop(0xc000086c30, 0x2800000)
        /home/phil/tmp/goamd/main.go:168 +0x16d
main.(*cpu).run(0xc000086c30, 0xc000086c00)
        /home/phil/tmp/goamd/main.go:180 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:211 +0x286

Looking good.

mov

Taking a look at the next two instructions with objdump we see mov encoded two different ways.

$ objdump -d a.out | grep -A4 '<main>'
0000000000401106 <main>:
  401106:       55                      push   %rbp
  401107:       48 89 e5                mov    %rsp,%rbp
  40110a:       b8 fe 00 00 00          mov    $0xfe,%eax

Looking up 0x48 we see that this is a prefix instruction that turns on 64-bit mode for the instruction. Some instructions like pop and push don't need this prefix to be in 64-bit mode. In any case, this just means we'll have to have a size flag that switches from 32-bit to 64-bit mode on seeing this instruction. This flag will be reset each time we start reading an instruction.

To deal with prefixes in general we'll loop through bytes when processing an instruction until we no longer see a prefix bytes. As we see prefix bytes we'll handle them accordingly.

var prefixBytes = []byte{0x48}

func (c *cpu) loop(entryReturnAddress uint64) {
    for {
        <-c.tick

        ip := c.regfile.get(rip)
        if ip == entryReturnAddress {
            break
        }

        inb1 := c.mem[ip]

        widthPrefix := 32
        for {
            isPrefixByte := false
            for _, prefixByte := range prefixBytes {
                if prefixByte == inb1 {
                    isPrefixByte = true
                    break
                }
            }

            if !isPrefixByte {
                break
            }

            // 64 bit prefix signifier
            if inb1 == 0x48 {
                widthPrefix = 64
            } else {
                hbdebug("prog", c.mem[ip:ip+10])
                panic("Unknown prefix instruction")
            }

            ip++
            inb1 = c.mem[ip]
        }

        if inb1 >= 0x50 && inb1 < 0x58 { // push

...

Moving past this prefix we get to 0x89. This instruction is for copying one register into another. The register operands are encoded in the second byte, 0xe5, called the ModR/M byte. Pulling out the two registers is just a matter of shifting and bitmasking the right 3 bits for each.

With this knowledge we can expand the instruction handling code.

        if inb1 >= 0x50 && inb1 < 0x58 { // push
            regvalue := c.regfile.get(register(inb1 - 0x50))
            sp := c.regfile.get(rsp)
            writeBytes(c.mem, sp-8, 8, regvalue)
            c.regfile.set(rsp, uint64(sp-8))
        }  else if inb1 == 0x89 { // mov r/m16/32/64, r/m16/32/64
            ip++
            inb2 := c.mem[ip]
            rhs := register((inb2 & 0b00111000) >> 3)
            lhs := register(inb2 & 0b111)
            c.regfile.set(lhs, c.regfile.get(rhs))
        } else {
            hbdebug("prog", c.mem[ip:ip+10])
            panic("Unknown instruction")
        }

Try emulating a.out again now. It will panic on the next unknown instruction, 0xb8. From objdump disassembly we see this is another mov instruction.

Hurray! There are apparently multiple ways the same instruction can be encoded. Looking it up in the opcode table, we see 0xB8 is for when the value to be copied is a literal number. The operand will be 32-bits, or four bytes, presumably because it doesn't have the 0x48 prefix.

// helper for converting up to 8 bytes into a single integer
func readBytes(from []byte, start uint64, bytes int) uint64 {
    val := uint64(0)
    for i := 0; i < bytes; i++ {
        val |= uint64(from[start+uint64(i)]) << (8 * i)
    }

    return val
}

func (c *cpu) loop(entryReturnAddress uint64) {

    ...

    for {

        ...

        } else if inb1 >= 0xB8 && inb1 < 0xC0 { // mov r16/32/64, imm16/32/64
            lreg := register(inb1 - 0xB8)
            val := readBytes(c.mem, ip+uint64(1), widthPrefix/8)
            ip += uint64(widthPrefix / 8)
            c.regfile.set(lreg, val)
        }

        ...

Two more instructions to go: pop and ret.

A terminal debugger

Taking a break for a moment, our system is already too complex to understand. It would be helpful to have a REPL so we can step through instructions and print register and memory values.

func (c *cpu) resolveDebuggerValue(dval string) (uint64, error) {
    for reg, val := range registerMap {
        if val == dval {
            return c.regfile.get(reg), nil
        }
    }

    if len(dval) > 2 && (dval[:2] == "0x" || dval[:2] == "0X") {
        return strconv.ParseUint(dval[2:], 16, 64)
    }

    return strconv.ParseUint(dval, 10, 64)
}

func repl(c *cpu) {
    fmt.Println("go-amd64-emulator REPL")
    help := `commands:
    s/step:             continue to next instruction
    r/registers [$reg]:     print all register values or just $reg
    d/decimal:          toggle hex/decimal printing
    m/memory $from $count:      print memory values starting at $from until $from+$count
    h/help:             print this`
    fmt.Println(help)
    scanner := bufio.NewScanner(os.Stdin)

    intFormat := "%d"
    for {
        fmt.Printf("> ")
        if !scanner.Scan() {
            break
        }
        input := scanner.Text()
        parts := strings.Split(input, " ")

        switch parts[0] {
        case "h":
            fallthrough
        case "help":
            fmt.Println(help)

        case "m":
            fallthrough
        case "memory":
            msg := "Invalid arguments: m/memory $from $to; use hex (0x10), decimal (10), or register name (rsp)"
            if len(parts) != 3 {
                fmt.Println(msg)
                continue
            }

            from, err := c.resolveDebuggerValue(parts[1])
            if err != nil {
                fmt.Println(msg)
                continue
            }

            to, err := c.resolveDebuggerValue(parts[2])
            if err != nil {
                fmt.Println(msg)
                continue
            }

            hbdebug(fmt.Sprintf("memory["+intFormat+":"+intFormat+"]", from, from+to), c.mem[from:from+to])

        case "d":
            fallthrough
        case "decimal":
            if intFormat == "%d" {
                intFormat = "0x%x"
                fmt.Println("Numbers displayed as hex")
            } else {
                intFormat = "%d"
                fmt.Println("Numbers displayed as decimal")
            }

        case "r":
            fallthrough
        case "registers":
            filter := ""
            if len(parts) > 1 {
                filter = parts[1]
            }

            for i := 0; i < len(registerMap); i++ {
                reg := register(i)
                name := registerMap[reg]
                if filter != "" && filter != name {
                    continue
                }

                fmt.Printf("%s:\t"+intFormat+"\n", name, c.regfile.get(reg))
            }

        case "s":
            fallthrough
        case "step":
            c.tick <- true
        }
    }
}

Let's try it out:

$ go build -o main
$ ./main a.out --debug
go-amd64-emulator REPL
commands:
        s/step:                         continue to next instruction
        r/registers [$reg]:             print all register values or just $reg
        d/decimal:                      toggle hex/decimal printing
        m/memory $from $count:          print memory values starting at $from until $from+$count
        h/help:                         print this
> r
rax:    0
rcx:    0
rdx:    0
rbx:    0
rsp:    41943040
rbp:    0
rsi:    0
rdi:    0
r8:     0
r9:     0
r10:    0
r11:    0
r12:    0
r13:    0
r14:    0
r15:    0
rip:    4198662
rflags: 0
> m rip 10
memory[4198662:4198672]: 55 48 89 e5 b8 4 0 0 0 5d
> s
> m rip 10
memory[4198663:4198673]: 48 89 e5 b8 4 0 0 0 5d c3
> r
rax:    0
rcx:    0
rdx:    0
rbx:    0
rsp:    41943032
rbp:    0
rsi:    0
rdi:    0
r8:     0
r9:     0
r10:    0
r11:    0
r12:    0
r13:    0
r14:    0
r15:    0
rip:    4198663
rflags: 0
> ^D

Now we can inspect the system interactively.

pop

Reemersing in the state of things, we now panic on 0x5D.

./main a.out
prog: 5d c3 66 2e f 1f 84 0 0 0
panic: Unknown instruction

goroutine 5 [running]:
main.(*cpu).loop(0xc000098ae0, 0x2800000)
        /home/phil/tmp/goamd/main.go:219 +0x2c5
main.(*cpu).run(0xc000098ae0, 0xc000098ab0)
        /home/phil/tmp/goamd/main.go:231 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:358 +0x286

Looking this up we see this is part of 58+r, pop. Similar to push we subtract 0x58 from the byte to get the register to pop onto. The stack operation is the reverse of push.

func (c *cpu) loop(entryReturnAddress uint64) {

    ...

    for {

        ...

        } else if inb1 >= 0x58 && inb1 < 0x60 { // pop
            lhs := register(inb1 - 0x58)
            sp := c.regfile.get(rsp)
            c.regfile.set(lhs, readBytes(c.mem, sp, 8))
            c.regfile.set(rsp, uint64(sp+8))
        }
        ...

Build and run for the final panic:

$ go build -o main
$ ./main a.out
prog: c3 66 2e f 1f 84 0 0 0 0
panic: Unknown instruction

goroutine 8 [running]:
main.(*cpu).loop(0xc000060c30, 0x2800000)
        /home/phil/tmp/goamd/main.go:224 +0x345
main.(*cpu).run(0xc000060c30, 0xc000060c00)
        /home/phil/tmp/goamd/main.go:236 +0xac
created by main.main
        /home/phil/tmp/goamd/main.go:363 +0x286

ret

Looking up 0xC3 we see that it is indeed ret. This function's responsibilty is to pop the stack onto rip, jumping back to caller.

        } else if inb1 == 0xC3 { // ret
            sp := c.regfile.get(rsp)
            retAddress := readBytes(c.mem, sp, 8)
            c.regfile.set(rsp, uint64(sp+8))
            c.regfile.set(rip, retAddress)
            continue
        }

Build and run:

$ go build -o main
$ ./main a.out
$ echo $?
4

If we modify tests/simple.c?

$ cat tests/simple.c
int main() {
  return 254;
}
$ gcc tests/simple.c
$ ./main a.out && echo $?
254

Not bad!

Process and next steps

Getting this far took a lot of trial and error, much of it hidden in this post. Setting up the REPL was critical to debugging mistakes. But aggressively unit testing would probably have been similarly fruitful. In the end, the most bug-prone aspects are basic arithmetic (off by one errors and converting bytes to/from integers). The part that's not terribly hard is actually interpreting instructions! But it's made easier by greatly simplifying the problem and ignoring legion cases.

Along the way it would have been helpful to also disassemble so that instead of just dumping memory at the instruction pointer we print the instructions we thought we were going to process. That may be a next goal.

Otherwise the typical goals are around getting syscall support, function call support, and porting these simple examples to Windows and macOS for the experience.

Here's take two on writing an emulator for linux/amd64 in Go. This time we're starting with ELF binaries, but still ignoring libc and jumping straight to main.https://t.co/A87r2RY21c
— Phil Eaton (@phil_eaton) November 26, 2020

2020–11–25: New multi-distro and p-boot released, WiFi power savings

megi's PinePhone Development Log — Wed, 25 Nov 2020 01:00:00 GMT

Write Errors That Don't Make Me Think

swyx's site RSS Feed — Thu, 19 Nov 2020 21:33:35 GMT

How we plan to fail with Thoughtful Error Message Design at Temporal.

DevOps Team Charter

smcleod.net — Wed, 18 Nov 2020 16:00:00 GMT

A template team charter for software and platform engineering teams.

Research¹ shows there is great value in people embracing a shared vision and ideas that are bigger than themselves.

When it comes to a team charter, it’s important to remember that it’s not a static document. It’s a living document that should be updated as the team evolves and grows.

Keep in mind that

One size doesn’t fit all.
Consider your (teams) mission, cultural background and values.
While not exhaustive - this is on the longer side, you may wish to distill as is practical, but I implore you not to only pick the easier items - that’s missing the point.
These can give you talking points when performing postmortems, retros and during times of conflict.

Exploring a new Postgres database

CRAIG KERSTIENS — Sat, 14 Nov 2020 21:30:56 GMT

At past jobs I’d estimate we had 100 different production apps that in some way were powering key production systems. Sure some were backend/internal apps while others key production apps such as the dashboard itself. At other companies we had a smaller handful of Heroku apps that powered our cloud service, about 5-10 in total. Even just working with those internal apps it’s a number of things to keep context on. But when it comes to interacting with something you don’t know getting a lay of the land quickly is key. In helping a customer optimize and tune, or even just understand what is going on in their app an understanding of the data model is key.

As I just started a few months back at Crunchy Data I found myself digging into a lot of new systems and quickly trying to ramp up and get a feel for them.

Over the past 10 years I’ve pretty well codified my steps to getting a feel for a new database. While I’m not a DBA and a small portion of my job is spent inside a database being able to quickly navigate one saves me hours each month and days out of the year. I’m sure my process isn’t perfect, but hopefully it helps other when navigating a new Postgres database for the first time.

First the tooling

For any new database my go to tool is psql. The built-in Postgres CLI is going to be the quickest thing for me to navigate around. If you use a CLI for anything else then this should be your preference here as well. I’m also going to have a psqlrc file setup that has some good defaults. My go to defaults in my psqlrc are:

-- Automatically format output based on result length and screen
\x auto


-- Prettier nulls
\pset null '#'

-- Save history based on database name
\set HISTFILE ~/.psql_history- :DBNAME

-- Turn on automatic query timing
\timing

Getting a feel for the tables

The first thing I’m going to do is just table a look at which objects exist within the database with \d. This will spit out a mix of tables, views, sequences all within your database. A cleaner version of this may be \dt which is only tables:


\d

 List of relations
 Schema | Name | Type | Owner
--------+-------------------------------+----------+----------------
 public | pg_stat_statements | view | postgres
 public | schema_migrations | table | postgres
 public | sessions | table | postgres
 public | sessions_id_seq | sequence | postgres
 public | tasks | table | postgres
 public | teams | table | postgres
 public | users | table | postgres
(7 rows)

\dt

 List of relations
 Schema | Name | Type | Owner
--------+-------------------------------+----------+----------------
 public | schema_migrations | table | postgres
 public | sessions | table | postgres
 public | tasks | table | postgres
 public | teams | table | postgres
 public | users | table | postgres
(5 rows)

We can also use the describe operation (\d) on specific relations as well such as tables to get a feel for how they look:


\d users

 Table "public.users"
 Column | Type | Collation | Nullable | Default
---------------------------+--------------------------+-----------+----------+--------------------
 id | uuid | | not null | uuid_generate_v4()
 email | text | | not null | ''::text
 encrypted_password | text | | not null | ''::text
 reset_password_token | text | | |
 reset_password_sent_at | timestamp with time zone | | |
 remember_created_at | timestamp with time zone | | |
 last_sign_in_at | timestamp with time zone | | |
 created_at | timestamp with time zone | | not null |
 updated_at | timestamp with time zone | | not null |
 name | text | | not null |
 team_id | uuid | | not null |
 deleted_at | timestamp with time zone | | |
Indexes:
 "users_pkey" PRIMARY KEY, btree (id)
 "index_users_on_email" UNIQUE, btree (email)
 "index_users_on_reset_password_token" UNIQUE, btree (reset_password_token)
Foreign-key constraints:
 "users_team_id_fkey" FOREIGN KEY (team_id) REFERENCES teams(id)

Human readable output

Of course you may want to go one step further and actually get a sense for the data. Here a basic SELECT tends to work with a limit 1. As you don’t quite know the shape of the data this is where having \x auto setup within your .psqlrc file is helpful to autoformat the output to your screen. You can also just manually run \x auto in your SQL session to get cleaner output.


SELECT *
FROM users
LIMIT 1;

-[ RECORD 1 ]-------------+--------------------------------------------------------------
id | 0a7a3cde-3613-4073-86a7-6a19b4e62bbe
email | craig.kerstiens@gmail.com
encrypted_password | $#IJ937Gmsdf00297sEmdfu12234
reset_password_token | #
reset_password_sent_at | #
remember_created_at | 2016-07-14 14:31:01.414795+00
last_sign_in_at | 2020-02-12 21:32:53.629246+00
created_at | 2016-02-18 03:03:26.403108+00
updated_at | 2020-02-14 23:16:16.080729+00
name | Craig
team_id | d46e864-1886-45e6-b538-8991562d2e99
deleted_at | #

Time: 91.592 ms

Most databases I work with now leverage JSONB. It’s a great tool for mixing semi-structured data with more structured data.

If you’re using JSON or JSONB then there is also a handy utility function to clean up that output - jsonb_pretty(yourcolumnhere). This will take care of making that huge JSON blob nice and readable.

Feel more at home when you encounter a new database

It doesn’t take hours of reading an ERD or schema files. In about 5-10 minutes of connecting to a new database I’m able to get a sense of how things are structured and able to actually start digging in. Don’t get me wrong, I’m by no means and expert in that time, but knowing some of these basic commands will really help the next time you encounter a database and are asked to help out or glean insights from it.

The Roaring 2020's

swyx's site RSS Feed — Sat, 14 Nov 2020 15:25:42 GMT

Five themes I am optimistic about for the next decade - and more open questions I have no idea about.

Blog update

Linux Engineer's random thoughts — Sat, 14 Nov 2020 01:00:00 GMT

I recently updated the server where this blog is hosted, so I tought I'd do an update on the original post explaining the tech stack used to run it. Once in seven years shouldn't be too meta.

The design

First of all, kudos to Pascal Navière, a very talented polymath …

M1 Memory and Performance

metablog — Fri, 13 Nov 2020 14:46:29 GMT

The M1 Macs are out now, and not only does Apple claim they're absolutely smokin', early benchmarks seem to confirm those claims. I don't find this surprising, Apple has been highly focused on performance ever since Tiger, and as far as I can tell hasn't let up since.

One maybe somewhat surprising aspect of the M1s is the limitation to "only" 16 Gigabytes of memory. As someone who bought a 16 Kilobyte language card to run the Merlin 6502 assembler on his Apple ][+ and expanded his NeXT cube, which isn't that different from a modern Mac, to a whopping 16 Megabytes, this doesn't actually seem that much of a limitation, but it did cause a bit of consternation.

I have a bit of a theory as to how this "limitation" might tie in to how Apple's outside-the-box approach to memory and performance has contributed to the remarkable achievement that is the M1.

The M1 is apparently a multi-die package that contains both the actual processor die and the DRAM. As such, it has a very high-speed interface between the DRAM and the processors. This high-speed interface, in addition to the absolutely humongous caches, is key to keeping the various functional units fed. Memory bandwidth and latency are probably the determining factors for many of today's workloads, with a single access to main memory taking easily hundreds of clock cycles and the CPU capable of doing a good number of operations in each of these clock cycles. As Andrew Black wrote: "[..] computation is essentially free, because it happens 'in the cracks' between data fetch and data store; ..".

The tradeoff is that you can only fit so much DRAM in that package for now, but if it fits, it's going to be super fast.

So how do we make sure it all fits? Well, where Apple might have been "focused" on performance for the last 15 years or so, they have been completely anal about memory consumption. When I was there, we were fixing 32 byte memory leaks. Leaks that happened once. So not an ongoing consumption of 32 bytes again and again, but a one-time leak of 32 bytes.

That dedication verging on the obsessive is one of the reasons iPhones have been besting top-of-the-line Android phone that have twice the memory. And not by a little, either.

Another reason is the iOS team's steadfast refusal to adopt tracing garbage collection as most of the rest of the industry did, and macOS's later abandonment of that technology in favor of the reference counting (RC) they've been using since NeXTStep 4.0. With increased automation of those reference counting operations and the addition of weak references, the convenience level for developers is essentially indistinguishable from a tracing GC now.

The benefit of sticking to RC is much-reduced memory consumption. It turns out that for a tracing GC to achieve performance comparable with manual allocation, it needs several times the memory (different studies find different overheads, but at least 4x is a conservative lower bound). While I haven't seen a study comparing RC, my personal experience is that the overhead is much lower, much more predictable, and can usually be driven down with little additional effort if needed.

So Apple can afford to live with more "limited" total memory because they need much less memory for the system to be fast. And so they can do a system design that imposes this limitation, but allows them to make that memory wicked fast. Nice.

Another "well-known" limitation of RC that has made it the second choice compared to tracing GC is the fact that updating those reference counts all the time is expensive, particularly in a multi-threaded environment where those updates need to be atomic. Well...

fun fact: retaining and releasing an NSObject takes ~30 nanoseconds on current gen Intel, and ~6.5 nanoseconds on an M1
— David Smith (@Catfish_Man) November 10, 2020

How?

We got that working on x86-64 too :) this further improvement is because uncontended acquire-release atomics are about the same speed as regular load/store on A14
— David Smith (@Catfish_Man) November 10, 2020

Problem solved. I guess it helps if you can make your own Silicon ;-)

So Apple's focus on keeping memory consumption under control, which includes but is not limited to going all-in on reference counting where pretty much the rest of the industry has adopted tracing garbage collection, is now paying off in a majory way ("bigly"? Too soon?). They can get away with putting less memory in the system, which makes it possible to make that memory really fast. And that locks in an advantage that'll be hard to duplicate.

It also means that native development will have a bigger advantage compared to web technologies, because native apps benefit from the speed and don't have a problem with the memory limitations, whereas web-/electron apps will fill up that memory much more quickly.

My First PC

Ben E. C. Boyter — Thu, 12 Nov 2020 02:00:00 GMT

At my current workplace we have been doing themed Friday stand-ups. The most recent was about your first PC love, or whatever technology you look back on fondly. What follows is my one.

My first PC was I think a Commodore PC40-III. I’m a little fuzzy on the details and its hard to find it based on pictures. Was not my first taste of technology, but the one I look on back most fondly.

The Light and Dark Side of the API Economy

swyx's site RSS Feed — Wed, 11 Nov 2020 18:07:36 GMT

The 'API Economy' is a popular term for VC's and tech media, however Developers seem ironically out of the loop despite their central importance to the whole story. Here's my explanation, together with a dash of economics and social implications.

The impact of management teams as a decision-making group, in startups and enterprise

Notes on software development — Wed, 11 Nov 2020 02:00:00 GMT

Ambitious companies form management teams at every level above you, sometimes including you. Management teams meet periodically and have private chat rooms. They discuss customers, product and organizational direction. Sometimes discussions are well documented and periodically public. Sometimes decisions are poorly telegraphed out.

Management teams do no inherent harm in a company with customers; employees outside of the management team can unearth customer usage data to discover meaningful places to contribute. For example, graphing historic server logs to discover slowest requests, figure out why and how to fix. Or even just paying attention to the most frequent questions sales asks and finding ways to clarify. (All of this under the assumption that even when there is solid product direction, good employees tend to have extra time at work and want to make good use of it.)

For the first few years even in a well-funded startup with solid founders, there are few customers. Even under a solid product team, the product direction is not yet completely clear. The management team includes founders and non-engineering executives. As a decision making group they are opaque. Employees outside the management team face a barrier in finding ways to meaningful contribute. Ambitious, dedicated folks outside the team leave.

So what?

It is not clear to me how the natural (and not inherently bad) concept of management teams attracts and retains ambitious, dedicated non-founders at small companies. Maybe disenfranchisement is not important, or even necessary.

Or maybe management teams as a decision-making group are too easily a substitute for developing a grassroots culture of collaboration and trust between marketing, sales, product and development.

New post! "management teams as a decision-making group are too easily a substitute for developing a grassroots culture of collaboration and trust between marketing, sales, product and development."https://t.co/7RukBMI59h
— Phil Eaton (@phil_eaton) November 16, 2020

Ferrari F12 Berlinetta

smcleod.net — Sun, 08 Nov 2020 18:00:00 GMT

Last weekend I had a chance encounter with a Ferrari F12 Berlinetta and the owner kindly let me jump in and take a few photos.

Specs

6.3L v12
544 KW (740 Horsepower)
690 Nm at 6000RPM
1525 kg
0-100 in 3.1s
0-200 in 8.5s
Maximum speed: ‘over 340 km/h’
Was ‘The fastest road car Ferrari ever built’

Photos

Temporal Transparency Update

swyx's site RSS Feed — Sat, 07 Nov 2020 05:04:37 GMT

Our 9th transparency update describing how we are shipping the upcoming Authentication feature

Naked Emperors in Tech

swyx's site RSS Feed — Fri, 06 Nov 2020 15:01:55 GMT

Some things we often repeat as truth just aren't. We should call bullshit more often.

Uvinji

Ben Overmyer's Site — Tue, 03 Nov 2020 02:00:00 GMT

As soon as the pre-expansion patch launched, I fired up World of Warcraft with the intent to create my new main character and begin my adventure anew.

Well, that first night there was a bit of a hiccup, as everyone else had the same idea. It took a lot longer to unlock the Vulpera allied race than I thought it would, as there was a quest chain for me to complete and it was very buggy. On top of that, I kept getting logged out. Eventually, I gave up for the night and logged back in the next day. Everything went smoothly after that.

I created Uvinji, my Vulpera Shaman, and logged in. I expected to start at the new new-player island, but instead it started me off at level 10 and skipped all of that. I was a little disappointed, but oh well.

After a little bit of running around Orgrimmar, I went on my way to Northrend to do the Wrath of the Lich King content. It was fun, at first, especially because I started in an area that I'd never actually done before. I made my way through all the quests in the Borean Tundra and ended up in Dragonblight.

At this point, I'd leveled up to 31. With the level cap being 50 now, that was quite a bit of progress. However, at this point, I got bored of the grind. The content was familiar, and honestly, I hate the Scourge and all of the undead-filled areas. They're very monotone in color and boring to level through. So, I used my one-time Shadowlands level-up-to-50 token on Uvinji to power him all the way up, and went through the "you're a brand new 50" tutorial.

I'm looking forward to the pre-expansion event that starts next Tuesday. Here's hoping it's more exciting than the last time the Scourge event happened...

2020–10–31: New p-boot released, new pre-built/updated kernels

megi's PinePhone Development Log — Sat, 31 Oct 2020 01:00:00 GMT

Just More Color

Ben Overmyer's Site — Fri, 30 Oct 2020 02:00:00 GMT

Today I reworked my site's CSS a bit to add more color.

This is a work in progress. Over time I plan to do more to liven the site up, while still trying to keep the overall size small.

Next, I'm considering whether to add more images to some parts of the site.

Unevenness at the edges

Bytepawn - Marton Trencseni — Fri, 30 Oct 2020 01:00:00 GMT

Sometimes we look at the top performers in a field and see obviously uneven representations of groups (gender, ethnicity, etc). There a multitude of factors that can lead to it, such as unfair bias in access to opportunities. Here I will show one unintuitive mathematical effect that can contribute to such unevenness in the case of normal distributions.

Notes on Serverless GraphQL with AWS AppSync

swyx's site RSS Feed — Thu, 29 Oct 2020 01:28:50 GMT

My Notes on Slobodan Stojanovic's Serverless GraphQL with AppSync talk

2020–10–29: New p-boot features – reboot to FEL/eMMC, 60FPS fix

megi's PinePhone Development Log — Thu, 29 Oct 2020 01:00:00 GMT

Let's Fix the News

Tilting at Windmills — Wed, 28 Oct 2020 02:00:00 GMT

As another US presidential election draws near, the brokenness of the media is becoming ever more glaring. It's not exactly a new phenomenon…

On Forever Projects

Ben Overmyer's Site — Wed, 28 Oct 2020 02:00:00 GMT

I just finished reading a blog post entitled The Forever Project by Scott Turner of Here Dragons Abound. It made me reflect on Iron Arachne and what I'm trying to do with it.

Lately I've been taking a break from Iron Arachne, and while on that break, I've found that I lack motivation or interest in continuing it. But this blog post made me reconsider that. It's not that I lack motivation or interest in Iron Arachne, but rather that I've been too focused on developing the site as a tool for others to use and not near enough on what I originally started it for.

Iron Arachne is my "forever project," as the article calls it. It's an endless creative exploration meant to delve into many, many different topics at the drop of a hat. It's not meant to be complete, or bug-free, or even useful. I had forgotten that.

In the context of this revelation, I'm taking a look at what Iron Arachne is currently and where I'd like to take it. I definitely don't want to do another rewrite. However, I think I would like to start exploring more random concepts. To that end, I'm going to follow some of the advice from Scott and write a bunch of new issues in GitLab that are just ideas of things to try - small ones, medium ones, big ones, and pie-in-the-sky ones.

I also will need to reinstate my weekly ritual of coding on Sunday mornings, but I think I'll broaden the scope of that a bit to be "work on Iron Arachne" instead of coding. That means anything that falls within that context is fair game - so, I could code, or I could do artwork, or read about geology, or write issues, or write a blog post, or anything else that fits the mold.

Now that I'm getting this down in writing, I'm finally getting excited about Iron Arachne again. That's a tremendous sign that I'm on the right track.

Found-Money Startups

swyx's site RSS Feed — Mon, 26 Oct 2020 17:02:59 GMT

'I will find free money for you' is one of my favorite business models. Here's a quick definition of Found-Money Startups and a short list I've been keeping.

Standard ML in 2020

Notes on software development — Sun, 25 Oct 2020 03:00:00 GMT

Incredibly, Standard ML implementations are still actively developed. MLton, Poly/ML, MLKit, SML# and SML/NJ are the most prominent. Discussion on the future direction of Standard ML remains healthy as well.

And somehow OCaml's lesser known cousin still beats out OCaml for multicore threading support (in Poly/ML).

While MLton hasn't merged with MultiMLton or RTMLton to support multicore, a new fork of MLton with parallelism is pretty far along and in active development at CMU.

A commentor shared Manticore, another implementation with parallelism support in active development at UChicago.

Furthermore, the last few years have welcomed some entirely new implementations. WebML, by a prominent open source hacker, is written in Rust and compiles Standard ML to WebAssembly. SOSML is an interpreter written in TypeScript by former students of Saarland University. It features a nifty online IDE.

A commenter shared SomewhatML, an actively developing compiler for Standard ML written in Rust.

There have also been some new experimental spins on Standard ML in the last few years. Morel is an interpreter with some nice syntax extensions written in Java by the author of Apache Calcite. And Bright ML is a spin on Standard ML and OCaml written in Standard ML (and using the abandoned Moscow ML compiler of all implementations).

So if you're looking for an easy intro to the ML family of languages, I still recommend the simplicity and performance of Standard ML and its small but definitely, surprisingly, not dead community. :)

Additional resources:

Are you using Standard ML? Let me know how/why!

Standard ML implementations are still in active development! There have even been some interesting new implementations pop up in the last few years.https://t.co/6kOcMKVfQa
— Phil Eaton (@phil_eaton) October 25, 2020

Moderating a 220k Developer Community

swyx's site RSS Feed — Tue, 20 Oct 2020 20:19:41 GMT

Reflections from two-and-a bit years helping to run /r/Reactjs

Svelte for Sites, React for Apps

swyx's site RSS Feed — Sun, 18 Oct 2020 04:08:02 GMT

Why we should stop trying to use the same tool for different jobs.

AmpliBox - a Self Hosted File Storage App with AWS Amplify

swyx's site RSS Feed — Mon, 12 Oct 2020 18:35:04 GMT

A demo using AWS S3 (Simple Storage Service) using Amplify Storage, Svelte, and Tailwind CSS

Notes on Time Management from a Dying Professor

swyx's site RSS Feed — Mon, 12 Oct 2020 03:10:32 GMT

Notes on Randy Pausch's lecture on Time Management

2020–10–12: Backlight changes when swithcing between USB-C power modes

megi's PinePhone Development Log — Mon, 12 Oct 2020 01:00:00 GMT

Choice Architecture

swyx's site RSS Feed — Sat, 10 Oct 2020 19:29:19 GMT

Reflection on how both gyms and startups do choice architecture and how we can be more conscious of this

The Best Of - 2020 Edition

smcleod.net — Sat, 10 Oct 2020 19:00:00 GMT

Near the end of every year I note down a summary of the best apps, hardware & podcasts I’ve enjoyed throughout the year (and often for some time before).

This post has been superseded. You can find the latest version of this post here.

Software and Services

Software - Mobile

Browser Addons

Hardware

Podcasts

YouTube Channels

The Absolute Worst Software I’ve Experienced

💩 Microsoft Office365 💩
💩 Microsoft Teams 💩
💩 Atlassian Jira 💩
💩 Atlassian Bamboo 💩
💩 Atlassian Confluence 💩
💩 McAffee Endpoint Security 💩
💩 Facebook 💩

Email Shenanigans

infrequent oscillations — Thu, 08 Oct 2020 18:33:06 GMT

In which we proxy emails.

Rest in Peace Jimothy

smcleod.net — Thu, 08 Oct 2020 18:20:00 GMT

You magnificent girl, you brought me 14 years of love and happiness

Photos

Service as a Service

swyx's site RSS Feed — Wed, 07 Oct 2020 20:16:35 GMT

Before you start offering Software as a Service, perhaps just offer a Service.

Follow Up

swyx's site RSS Feed — Tue, 06 Oct 2020 13:29:38 GMT

Following up and following through is a well known formula for success. Yet people don't do it. Why?

2020–10–05: Reboot to FEL, skipping SD card boot, external display fixes

megi's PinePhone Development Log — Mon, 05 Oct 2020 01:00:00 GMT

Why Tailwind CSS

swyx's site RSS Feed — Sun, 04 Oct 2020 05:48:22 GMT

Why I changed my mind on Tailwind CSS, and why I now consider it the Goldilocks Styling Solution

Beware The Fallacy of Composition

swyx's site RSS Feed — Thu, 01 Oct 2020 03:00:00 GMT

Many things that make sense individually don't make sense as a group.

Leading Up to Shadowlands

Ben Overmyer's Site — Wed, 30 Sep 2020 03:00:00 GMT

The Shadowlands expansion for World of Warcraft comes out next month. A lot of things seem to be happening in October this year. I'm opting to focus my gaming time on WoW, given the new expansion.

A month ago, I was playing a lot of WoW. The new expansion offers a lot of opportunities for starting new characters, and I wanted to unlock the Vulpera allied race to try something new. Unlocking that race previously required completing a particular story arc and reaching Exalted status with the Vulpera. I'd finished the story arc, but the reputation grind was quite a slog, and I hadn't come close to finishing it.

Then, I discovered that the pre-patch for Shadowlands was going to remove the reputation grind from unlocking allied races. This completely killed my interest in playing WoW before the pre-patch hits, and I haven't played since, despite keeping my subscription active.

As soon as the pre-patch is live, I'm diving back into the game with a vengeance. My new character will be a Vulpera shaman. I want to play a healer, and the Vulpera shaman's totems are fox-shaped which I love.

The new leveling experience takes you from 1 to 50 (the new pre-expansion cap) in a single expansion's content. I'm going to go through Wrath of the Lich King as my choice, since that remains my favorite expansion so far. It remains to be seen if I will blog about this process or not, but it might be fun to journal my experience. I don't think I'll be able to reach level 50 with my Vulpera before the expansion lands, but that's going to be my goal anyway.

I don't know if pet battle levels will be rewired too, but if they are, I definitely want to try and bring at least one group of pets up to level cap during my leveling. We'll see how that works out.

So yeah. I'll be retiring Exsecratus as my main, which is only the second time I've ever changed mains since I first started playing WoW back in 2004. Here's to new beginnings!

Faster Literal String Matching in Go

Ben E. C. Boyter — Wed, 30 Sep 2020 03:00:00 GMT

TL/DR: I wrote a fast literal string matching library in Go get it here https://github.com/boyter/go-string/

Recently I rebuilt searchcode in Go, as mentioned.

While there was a variety of reasons for this one was that performance in the previous Python version was not great. Please note this was mostly due to my own shortcomings and not the language itself. However I have always used searchcode as a test bed for interesting problems, since it gets enough traffic to verify things at scale, and I wanted to get better at Go.

2020–09–29: New codec driver in my 5.9 kernel

megi's PinePhone Development Log — Tue, 29 Sep 2020 01:00:00 GMT

5 TILs about Node.js Fundamentals from the Node.js Design Patterns Book

swyx's site RSS Feed — Sun, 27 Sep 2020 11:48:33 GMT

5 Things I Learned about Node.js Fundamentals from the Node.js Design Patterns Book

REST Endpoint + CRUD with AWS Lambda and DynamoDB in 2 minutes

swyx's site RSS Feed — Wed, 23 Sep 2020 17:31:01 GMT

A lightning fast overview of everything you need to know to set up a REST endpoint with full CRUD capability with AWS Lambda, DynamoDB and AWS Amplify in 2 minutes.

On Blogging Versus Social Media

Ben Overmyer's Site — Tue, 22 Sep 2020 03:00:00 GMT

I read Rubenerd's post about seriousness, and it got me to thinking. Most people these days are content to post on social media. Very few have their own blogs, let alone a dedicated personal website.

But that's not the web I grew up with.

I've had many blogs over the years. They've been powered by different services or software, and had different focuses. But taken as a whole, they've painted a picture remarkably similar to modern social media posts - snapshots of my life at the time. Things that I've been thinking about, things I'm excited about, things I'm angry about.

There are two key differences, though. Blog posts tend to run a lot longer than social media posts, and I fully control the platform on which they appear.

Also, people who read blogs tend to be more willing to read longer content. On social media, the vast majority of users seem to only be interested in reacting to short sentences and images. There's nothing wrong with that, exactly, but it's not the kind of interaction I'm looking for when I write longer content.

The other thing that Rubenerd made me remember is back when I drew cartoon faces for my earliest websites to reflect my mood for any given update. Maybe I should resurrect that practice and give my site a little more playfulness.

2020–09–20: Downsizing the multi-boot image

megi's PinePhone Development Log — Sun, 20 Sep 2020 01:00:00 GMT

2020–09–20: Some ways to improve Pinephone safety

megi's PinePhone Development Log — Sun, 20 Sep 2020 01:00:00 GMT

2020–09–18: Let's talk about safety of Pinephone

megi's PinePhone Development Log — Fri, 18 Sep 2020 01:00:00 GMT

Rust 2021: Lower the barriers

Krishna's blog — Thu, 17 Sep 2020 03:00:00 GMT

Rust's success is guaranteed on a long enough timescale, but lowering the barrier to entry by making it easier to learn is crucial for its adoption rate.

2020–09–17: Video acceleration experiments with PinePhone

megi's PinePhone Development Log — Thu, 17 Sep 2020 01:00:00 GMT

Notes from Amir Shevat on Measuring & Managing Developer Relations

swyx's site RSS Feed — Wed, 16 Sep 2020 21:40:08 GMT

Notes from an a16z podcast about developer relations that I thought was valuable

Analysing the Indian government cyberspace

Nemo's Home — Wed, 16 Sep 2020 03:00:00 GMT

I recently did some work on analysing the Indian government cyberspace, thought I should document them somewhere outside of my Twitter¹.

List of GoI websites

I’d made a list of Indian government websites in Jan 2019:

I ran @18F /pulse on Indian Government websites to see how many of them support HTTPS. A quick summary:

Total Websites: 14183
Total Live Websites: 11710 (82%)
Websites with Valid HTTPS: 4753 (40% of all live websites)

Raw Dataset for now: docs.google.com/spreadsheets
— Nemo (@captn3m0) January 15, 2019

The dataset was from 2 sources:

GoI Directory
crt.sh (All certificates ending in .gov.in were used)

I re-ran the scripts to get an updated list (12842 domains), then tabulated them against the public-suffix² for each. There is a long-tail, and I’ve published results here. Here are the top public suffixes for Indian government sites:

Public Suffix	Domains
nic.in	2454
gov.in	7259
in	528
ac.in	490
com	568
co.in	171
org.in	168
edu.in	117
org	844
res.in	134
net.in	12
net	38

Sanskari Proxy

This was a long standing idea on my ideas repo:

A lot of Indian Government websites are inaccessible on the public internet, because they geo-fence it to within Indian Boundaries. The idea is to make a Indian Proxy service that specifically works only for the Geo-fenced Indian government websites.

For eg, if uidai.gov.in is inaccessible, hitting uidai.gov.sanskariproxy.in will get you the same result, proxied via our servers.

Since I’d made an updated list of GoI websites, this seemed easy enough. I realized that setting up uidai.gov.sanskariproxy.in would likely count as impersonation under the Indian law, so I did the next best thing: run an actual proxy. Here’s the announcement tweet:

Are you a security researcher outside India? Do you hate getting geoblocked to Indian government websites?

Well, I made a proxy for security researchers outside India to access Indian government websites without resorting to shady VPNs.
captn3m0/sanskari-proxy
— Nemo (@captn3m0) September 5, 2020

Project page is https://github.com/captn3m0/sanskari-proxy, and if you’d like to get access - please reach out.

Cyberspace Ownership

I’d planned to get a complete list of geoblocked websites next. While I’m progressing on this front, the results have been inconsistent/inaccurate so far. As an intermediate step, I’d made a list of IPs against every domain³, which looked like this:

Domain	IP Address
aavin.tn.gov.in	164.100.134.148
abnhpm.gov.in	14.143.233.34
agnii.gov.in	13.232.216.65
ap.gov.in	117.254.92.53
aponline.gov.in	125.16.9.130
appolice.gov.in	118.185.110.147
attendance.gov.in	164.100.166.114
cgg.gov.in	112.133.222.115

While running numerous nmap scans (and failing), I start checking the ASN⁴ for some of these IPs to see who was hosting each website - especially the ones I was finding were blocked.

I stumbled upon a bulk IP to ASN service by Cymru, ran all the IPs against it and published the results. Here’s the important graph:

As you can expect, NIC⁵ has the highest share, with NKN⁶, BSNL, and CtrlS following at roughly 5% each. There are a few other chart on the twitter thread, and the raw data is available here with interactive versions of each visualization.

What next?

I’m working on running and comparing connectivity scans to these IPs to get a better understanding of the geoblocking situation. There’s also some issues with the domain list, as it seems to be missing lots of domains - so more corrections are needed.

Twitter decided to suspend 12 different accounts I had access to recently - I’m starting to get wary of using Twitter for archival now. ↩
A “public suffix” is one under which Internet users can (or historically could) directly register names. For eg - nic.in or github.io. Mozilla manages the list at https://publicsuffix.org/. ↩
There are issues with this approach, since domains do resolve to multiple IPs. But this is okay for the rudimentary analysis I’ve been doing so far. ↩
Autonomous Systems (AS) is how the internet is sliced up and managed by different entities. Each AS (usually an ISP) is responsible for routing within its network, while announcing network routes on how it can be reached. ↩
The primary government office (under MeitY) that provides infrastructure and support for government IT services. ↩
National Knowledge Network is a multi-gigabit research and education network that provides a high speed network backbone for educational institutions in India. ↩

2020–09–16: PineBook Pro and Levinboot again

megi's PinePhone Development Log — Wed, 16 Sep 2020 01:00:00 GMT

Typesafe, Fullstack React & GraphQL with AWS Amplify

swyx's site RSS Feed — Wed, 16 Sep 2020 00:23:42 GMT

This is the blog version of a talk I am giving at React Summit 2020

Pointers are Easy, Optimization is Complicated

metablog — Tue, 15 Sep 2020 12:01:17 GMT

Just recently came across Ralf Jung's 2018 post titled Pointers are Complicated. The central thesis is that the model that most C (and assembly language) programmers have that a pointer is just an integer that happens to be a machine address is wrong, in fact the author flat out states: "Pointers are definitely not integers."

That's a strong statement. I like strong statements, because they make a discussion possible. So let's respond in kind: the claim that pointers are definitely not integers is wrong.

The example

The example the author uses to show that pointers are definitely not integers is the following:

int test() {
    auto x = new int[8];
    auto y = new int[8];
    y[0] = 42;
    int i = /* some side-effect-free computation */;
    auto x_ptr = &x[i];
    *x_ptr = 23;
    return y[0];
}

And this is the crux of the reasoning:

It would be beneficial to be able to optimize the final read of y[0] to just return 42. The justification for this optimization is that writing to x_ptr, which points into x, cannot change y.

So pointers are "hard" and "not integers" because they conflict with this optimization that "would be beneficial".

I find this fascinating: a "nice to have" optimzation is so obviously more important than a simple and obvious pointer model that it doesn't even need to be explained as a possible tradeoff, never mind justified as to why the tradeoff is resolved in favor of the nice-to-have optimization.

I prefer the simple and obvious pointer model. Vastly.

This way of placing the optimizer's concerns far ahead of the programmer's is not unique, if you check out Chris Lattner's What Every C Programmer Should Know About Undefined Behavior, you will note the frequent occurrence of the phrase "enables ... optimizations". It's pretty much the only justification ever given.

I call this now industry-dominating style of programming Compiler Optimizer Creator Oriented Programming (COCOP). It was thoroughly critiqued in What every compiler writer should know about programmers or “Optimization” based on undefined behaviour hurts performance (pdf).

Pointers as Integers

There are certainly machines where pointers are not integers, the most prominent being 8086/80286 16 bit segmented mode, where a (far) pointer consists of a segment and an offset. On 8086, the segment is simply shifted left 4 bits and added to the offset, on 80286 the segment can be located anywhere in memory or not be resident, implementing a segmented virtual memory. AFAIK, these modes are simplified variants of the iAPX 432 object memory model.

What's important to note in this context is that the iAPX 432 and its memory model failed horribly, and industry actively and happily moved away from the x86 segmented model to what is called a "flat address space", common on other architectures and finally also adopted by Intel with the 386.

The salient feature of a "flat address space" is that a pointer is an integer, and in fact this eqivalence is also rather influential on CPU architecture, with address-space almost universally tied to the CPU's integer size. So although the 68K was billed as a 16 bit CPU (or 16/32), its registers were actually 32 bits, and IIRC its address ALUs were fully 32 bit, so if you wanted to do some kinds of 32 bit aritmetic, the LEA (Load Effective Address) instruction was your friend. The reason for the segmented architecturee on the 8086 was that it was a true 16 bit machine, with 16 bit registers, but Intel wanted to have a 20 bit address space.

So not only was and is there an equivalence of pointers and integers, this state of affairs was one that was actively sought and joyously received once we achieved it again. Giving it up for nice-to-have optimizations seems at best debatable, but at the very least it is something that should be discussed/debated, rather than simply assumed away.

Book Review - Shoe Dog by Phil Knight

swyx's site RSS Feed — Mon, 14 Sep 2020 06:30:30 GMT

One of the greatest autobiographies by a business icon and great writer

Notes on Adversarial Interoperability

swyx's site RSS Feed — Mon, 14 Sep 2020 02:37:49 GMT

Summarizing thoughts from Seth Godin and Cory Doctorow on Interoperability

2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card

megi's PinePhone Development Log — Mon, 14 Sep 2020 01:00:00 GMT

Modelling a Wealth Tax Correctly

Krishna's blog — Sun, 13 Sep 2020 03:00:00 GMT

This article examines the accuracy of modelling a wealth tax and critiques Paul Graham's argument against it.

I Want to Fix Goodreads

PREPEND — Sat, 12 Sep 2020 03:00:00 GMT

I use goodreads a lot because I like to read books and want to find more books to read. This morning I was reading a thread on hackernews on Sarah Manavis’s NewStatesman article and it reminds me of how much I want, I yearn, for useful book recommendations. I think about this pretty much every time I open the goodreads page or app and hope that maybe this is the day they finally fix recommendations, so I figure I would capture some ideas I have for fixing their recommendations.

I Love Goodreads

First, I love Goodreads. My profile says I joined May 2012. But I think that’s just whenever I started this profile and I have multiple, old profiles that I can’t find. My memory says I started using them soon after launch, but maybe I’m misremembering. But I’ve used them for a long time because I like to read books. I love browsing bookstores and libraries looking for new books. I experience joy finding a 50 or 100 year old book that I didn’t know about. One of my favorite things is reading a book so good that I can’t go to bed, or want to walk an extra mile. But that’s pretty rare for me, I’m lucky if I get one of those a year.

I have goodreads on my link stack with the tag “purdah” that I think means it is part of the stuff I do that uniquely identifies me.

Any time I get a recommendation from a friend or article or wherever, I add it to my “to read” shelf on goodreads. That helps me when I’m trying to buy a new book. So at least I can read through the 245 books that at least a past me wanted to read. This works well and is better than a vanilla list.

I also like seeing what my friends read and especially what they like or dislike. My friends have different tastes so it’s not as simple as just reading everything they read, but it helps a little. It’s one of the few places that I think facebook helps because at least I get to find out facebook friends who also are on goodreads. Every once in a while I find a new book through this feed, maybe once every 2 years. This might be because I don’t have many friends on goodreads (only 42), but I think it’s also because different people have different tastes in books.

I also like that goodreads will tell me when an author I’ve read releases a new book, or is having a book chat or something. Although I wish they would tell me when they come to a local book signing.

The Problem As I See It

This gets me to what I think is wrong and want to fix. I have read and rated 802 books with my average rating of 3.97. This makes sense as I’m probably only going to read books I think I’ll like, but I think this is the average goodreads rating, not the average of my ratings. Looking at what I rated, I’ve only given out 3 one stars, 42 two stars and tons of five stars. I keep scrolling on and on.

The user interface for goodreads is pretty bad- information is hard to find, layout is wasted, search doesn’t work well, hard to tell ads from user content- but that’s not what I’m talking about here. The main problem is in finding books that I will like. Maybe a UX overhaul from a great designer would help, but not that much. A craigslist interface would be great if they could just solve this one problem…

What is the single book that I will love most in all the world?

I want to find the next amazing experience. Books are great because they are cheap and portable. It’s not like wine where the best is more than I’ll ever pay. Or travel where a first class flight can cost $20k. If I can find a book, I can read it. And maybe someone I know can read it too.

If I can mix in more wonderful books, my life will be better. My family’s life will be better. Etc etc. Maybe this is the answer to world enlightenment. Probably not, but it will be great, I think.

When I click on recommendations, goodreads shows me five books that I don’t want to read. And it’s weirdly bad. Some of it makes since why it’s bad. They recommend different versions of books I’ve read. They recommend two different versions of Lord of the Rings (one of my favorite books), but I guess they don’t know these are the same book. That’s just a waste of time and it’s weird that amazon isn’t smart enough to avoid duplicative books.

But then they recommend stuff like Stephen King’s The Long Walk. This is perplexing because I only have one book that goodreads knows I’ve read from King and I rated it two stars. I don’t have any of his books in my “to read” list. Why would they recommend this book? Two of my friends rated it four stars and one even wrote a recommendation. So maybe that’s why. I’m aware of this book and don’t want to read it. But other books like Shoe Dog by Phil Knight has no relationship to me. Why would they recommend this? I also know about this book and don’t want to read it. I wish there was some way to note books that I don’t want to read. Then at least they would stop showing me the same book over and over.

So it seems like goodreads is making pretty basic, rules-based recommendations. And they don’t help me pick new books.

My Perfect Wish

My idea for an 11-star experience ¹ in finding new books is that Goodreads knows me even better than I know myself and constantly recommends the perfect book. They recommend books that teach me, that spark enlightenment, that I hate but teach me to appreciate other books, books for when I want to be happy, and books for when I need to be sad. I want something like Jane from Ender’s Game who experiences everything I know but is fixated on finding me a good book. So the output of a book sommelier whose passion is seeing me sit there and love a book.

That’s what I would like.

A Little Story About Jeff Bezos

Once I was lucky enough to go to TED. It was great on so many levels, I highly recommend it and 2/3 of the cost was tax deductible. It was full of great stuff. I got to wait in the registration line in front of Neil Gaiman and Amanda Palmer, that was neat.

Anyway, one night Neil Gaiman was doing in a midnight ghost story reading in some neat four story townhouse type building ² that was totally empty and spooky and a great place for the event. The reading was on the top floor and there was a huge crowd waiting for the single elevator in this place. The lights were off and there weren’t any staff around so it was maybe 100 people waiting to go up three at a time. So I looked around the corner and took the stairs. About 2 or 3 other people had the same idea so we started racing up the stairs to get to the top and avoid the crowds.

photo by Joe Fox

This building was awesome. It seemed 100 years old, rare in Vancouver, and as we went up the stairs we saw each floor. The stairwell opened up to a big room that could probably seat a 50 person wedding dinner. The lights were off, and someone had opened a few windows and the white, wispy curtains were blowing with a little breeze. Even though it didn’t feel like a breeze. I said “cool, haunted house” and stopped climbing the stairs to look around. The person next to me said “yeah, cool” and we started quickly running around checking out the empty room. We did this on the third floor too and me and this random person did quick 60 second explorations of this spooky house.

It all happened quickly. Maybe 3 minutes to go up the stairs, run around each floor, and get to the top. About halfway through I realized this random dude was Jeff Bezos. I thought that was neat. I’ve bought books from Amazon since 1995 and I’m pretty sure I telnetted into it to order some books (although maybe that was cdnow before they bought them), so it was cool to think that Jeff Bezos likes haunted houses and still likes exploring empty buildings.

We got to the top and went into the venue and were maybe the first 10 people. Comically got there maybe after two or three elevators full of people. We went to the front where Gaiman was setting up at the podium. I got a comfy chair right in the front about five feet in front of Gaiman. Jeff Bezos sat in the chair to my left. The chair on my right had Neil’s wife, Amanda Palmer, who was carrying a ukelele case. I didn’t know it was a ukulele case until she told me. She also said she might play, but she didn’t.

Gaiman finished setting up and came up to talk to Bezos. Bezos asked if Gaiman got the books he sent. Gaiman said he did and they were really great and he enjoyed reading them. It seemed like this was the most recent in a series of book shipments.

Aside from nerd joy from getting to witness this small talk of people who interest me, I thought, at the time and many times since,

How cool is it that Neil-freaking Gaiman is still finding new books that are great, and how cool would it be to get book recommendations from Jeff Bezos. I bet he has access to the best book recommendations in the world.

Also, TED has something called the “TED book club” where they mail you this box of books a month before the conference. This was the closest I’ve come to my dream. They maybe sent 10 and 5 are still on my bookshelf. But not the shelf behind me, so I don’t remember them specifically. I’ll look them up one day.

That’s probably better than Jane making recommendations. Amazon had just bought goodreads a year earlier and I was hopeful that any day, goodreads would roll out a wonderful recommendation engine any day now.

What I Want to Do to Fix

What I would like to do is to just get a data dump of mine and everyone on goodreads recommendations. I don’t need to know the identities, just to know what recommendations go with what individual. It would be neat to protect privacy and find individuals to potentially follow them on goodreads, but I think for the analysis I want to do, I don’t need to know the actual identities.

I think if I have this data, I could try to find people who rate a few hundred books like I have, then look at what books they’ve read the most, or books that they’ve rated highest. I think that would be a good signal that I might like it.

It would also be cool to see books they hated. Just to see uniqueness amongst the millions of readers.

It would also be cool to find my exact opposite. People who rate the absolute opposite from me, my antipode. ³

If I just had this data, I would do what goodreads isn’t doing for me. Maybe they’ve already tried and it’s impossible to achieve. Maybe it’s just one of those things where we have to increase opportunities and leverage serendipity.

They do have an api but it seems meant for integrators to do goodreads functions from an app or something. I can get my own recommendations, but there’s no way to get other people’s ratings, and unless I know the name of the book I can’t look up ratings. And that’s sort of the problem since I don’t know the name of books I don’t know. They do allow me to view a specific user’s info and even lets us compare shelves.

I wish they had a data dump somewhere. It’s strange to me that lots of users allow their ratings to be public (eg, this random friend user), so I can theoretically scrape all the reviews from just public users, but that’s a lot of work. I share my ratings because I want to help others. I assume that other users do so for the same reason so I wish Goodreads/Amazon would help facilitate this. Since userids are just 8 digits, I supposed I can use their API to read one user a second (their requested rate limit) and it would just take me and it would just take me 3.17 years, or maybe less since they may not have 99,999,999 users, but Statistica claims they had 90 million registered users on July 2019. ⁴

Housekeeping

Unanswered Questions

Low quality of reading random people’s reviews and comments

How to identify yourself, when yourself changes? Neal Stephenson’s PURDAH.

Future Blog Ideas

Describe the link stack idea.
Emergent awesomeness. When you can’t explicitly do something, you just have to increase the rate of random awesome things and appreciate them when they happen.

Stuff I Read While Working on This Post (That You Might Want to Read Too)

Footnotes

https://mastersofscale.com/brian-chesky-handcrafted/ Masters of Scale podcast with Brian Chesky where he describes AirBNB process of trying to design experiences where users rate something 11 stars (out of 5). I find it funny that AirBNB has never given me an 11 star experience, but lots of 5 stars. And I think that’s the message here. ↩
https://vancouverclub.ca/ I think it was The Vancouver Club, so imagine this place but at midnight, empty and abandoned. Wikipedia says it was founded in 1889. ↩
https://en.wikipedia.org/wiki/Antipodes I would also like to see this antipode functionality for twitter. Who is the person with the exact opposite of me on Twitter. Would be neat to read their ideas. ↩
https://www.statista.com/statistics/252986/number-of-registered-members-on-goodreadscom ↩

KotSF is now Tilting at Windmills

Tilting at Windmills — Fri, 11 Sep 2020 03:00:00 GMT

According to WHOIS, I first registered the domain kotsf.com, a little over 12 years ago on August 18, 2008. Since then it's been my catch…

2020–09–11: Adding postmarket OS to multi-distro image

megi's PinePhone Development Log — Fri, 11 Sep 2020 01:00:00 GMT

2020–09–11: Ways to help improve Pinephone kernel

megi's PinePhone Development Log — Fri, 11 Sep 2020 01:00:00 GMT

2020–09–11: PinePhone multi-boot image deduplication tool complete

megi's PinePhone Development Log — Fri, 11 Sep 2020 01:00:00 GMT

Go Modules have a v2+ Problem

Donat Studios — Thu, 10 Sep 2020 09:11:12 GMT

Go has a problem. Go modules place a strange naming requirement on modules version 2 or greater. Module names on modules v2+ must end in the major version ala …/v2, and communication of this rule has been weak. It's non-obvious, and the community at large does not understand it.

I have seen many very large projects including Google owned projects get it wrong.

I brought the issue up at my local Go meetup, and no one had ever heard about the rule. They were very skeptical the rule existed at all.

A little history

For a long time Go did not include a built in method for versioning dependencies. It did include the sometimes controversial go get method of fetching packages. The go get tool until recently simply cloned the HEAD of a packages primary branch based on URL and placed it in your $GOPATH.

In what has been oft linked to Go's "extremist position" on backwards compatibility, the expectation was largely that if you published a library, you were expected to maintain compatibility when at all possible.

A plethora of tools popped up over the years to fill this void, gopkg.in, glide, and even the at one time "official experiment" dep.

dep was the heir apparent to dependency management with backing from Google.

Seemingly suddenly however Russ "rsc" Cox sprung vgo (aka Go Modules) onto the world as the official Go solution. There was seemingly no input from the community in its creation, to the chagrin of many.

Whereas dep was a largely standard dependency manager in the vein of npm and the like, Go modules is a decidedly opinionated "Go" solution. It's handled as part of the language as a build step. When you go build if you are missing the versioned dependency, the build tool will fetch it.

The dependencies requirements are listed in go.mod and the exact versions expected are in go.sum. Both files are in their own sort of domain specific language. The prior has a syntax reminiscent of Go itself. The latter is a space separated list of dependency versions largely not intended for human consumption.

The build step will add to these files as necessary, and running go mod tidy will clean them up.

The v2+ problem in detail

Go modules place a very strange requirement on package developers. When a module hits major version 2 or higher, the module name must end in the major version. The advantage to this is it creates a separate package. The result is that a project can now depend on multiple major versions of the same library.

There are two tactics to achieving /v2 modules:

The first method is to change the module name in your go.mod

An example may seen in the go.mod of github.com/google/go-github seems simple enough, but requires you to also change any cross references within your package. While not the end of the world, it is error prone and even Google missed some.

The biggest problem with this method is that it breaks subpackages in non-module-aware versions of Go. As time goes by this becomes less important, but in older versions of Go the package name had to exactly match it's location. https://github.com/pkg/foo could not be github.com/pkg/foo/v2 when there is no v2 directory. The Go Team acknowledging this trouble and patched rudimentary module support into point releases 1.9.7 and 1.10.3 of the Go language to help ease the transition.

The second method and the method the Go Team themselves promote is leaving your v1 project as is in the root of your repo, and actually creating a v2/ subfolder containing your v2 library.

This feels weird, but the advantage is it works cleanly with older versions of Go that do not understand modules. Off the top of my head I only knew of a single project that went this route, and they appear to have gone back on it.

For what it's worth, the official Go Blog tried to clear the situation up some, and posted a write-up about it.

blog.golang.org/v2-go-modules

Even as a seasoned developer, I found this write-up somewhat impenetrable. For how important and unusual of a requirement this is, the write-up is not accessible enough.

Many projects including Buildkite's terminal-to-html simply didn't know about the requirement until they forget to handle it, and end up publishing broken releases. A v2+ Go library with a go.mod file, but not respecting the module v2+ rules is a compile time error for dependants. This is worse for users than not being a Go module at all.

Some projects like GORM sidestep the issue entirely by tagging their 2.0 releases as far flung 1.x releases. That something of a solution, but smells terrible.

What's to be done?

First, I think the Go Team needs to do a better job of shining a light on this rule. They need to scream it from the rooftops. They need at the very least a section of the documentation that lays it out clearly and in layman's terms.

Beyond this, there's room for warnings from the Go tooling. Subpackages importing older versions of the root package should throw a warning. It's way too easy an issue to miss.

There's room for similar notes on godoc.org / pkg.go.dev, although the usefulness is questionable.

The best and least likely option would be for Go to drop the requirement and make it optional. I believe this is a workable solution. Packages who want to allow users to include multiple versions can follow the v2 guidelines, and others don't need to.

An option I would personally promote, but is likely never going to happen, is to require /v0 and /v1 as well, such that you'd hit the issue right away rather than years into a project. Teach the convention early.

Discussion at: news.ycombinator.com/item?id=24429045

Against Namespacing Personal Sites

swyx's site RSS Feed — Thu, 10 Sep 2020 01:15:08 GMT

I care a lot about creating Cool URIs so I have been paralyzed more than I should be about what I'm calling "URL Architecture". In true fashion, I'm blogging about it.

2020–09–10: PinePhone multi-boot image deduplication

megi's PinePhone Development Log — Thu, 10 Sep 2020 01:00:00 GMT

How to Set Up SvelteKit with Tailwind CSS

swyx's site RSS Feed — Tue, 08 Sep 2020 19:30:58 GMT

A quick 3 step guide for myself on how to set up Svelte with Tailwind CSS

2020–09–08: PinePhone multi-boot finishing touches / modem improvements

megi's PinePhone Development Log — Tue, 08 Sep 2020 01:00:00 GMT

2020–09–08: PinePhone multi-boot image optimizations

megi's PinePhone Development Log — Tue, 08 Sep 2020 01:00:00 GMT

If You Sell Anything Online, This Book Will Make You Money

swyx's site RSS Feed — Mon, 07 Sep 2020 22:16:56 GMT

My glowing review of Rob Hope's Landing Page Hot Tips Book

The case for comments in code

Notes on software development — Mon, 07 Sep 2020 03:00:00 GMT

When I first started programming, especially when asked for code samples, my comments lacked purpose and would often duplicate in English what the code clearly indicated. I knew that "commenting is good" but as a beginner I had no further insight.

Over time with the help of books like Clean Code, I grew disdainful of comments. Good code should be self-documenting. Whenever I needed to write a comment to explain something, I'd realize I could easily rename some key variable or function. I grew more comfortable with variables and functions with a few words in the title. Better to spend time on good code structure and naming.

I have always left TODOs though, since TODOs can't so easily be expressed in variable names. But even these TODOs concerned me because they existed in my issue tracker, or maybe should have.

As I watched mature open source projects and mature engineers, I came to value well-documented pull requests. Solid pull requests include or link to all necessary background, opportunities failed or ignored, how to use, links to external bugs requiring workarounds and the results of performance evaluation.

Beyond pull request descriptions, when I really wanted to grease a pull request I'd use the pull request UI to add comments calling reviewer attention to key changes in lines of the diff.

Both kinds of guidance are a massive aid to reviewers, saving a lot of time.

But when I'd find a bug in code -- and I knew there was good pull request documentation, even for pull requests as recent as six months ago -- I've been repeatedly failed by the pull request and pull request comment search exposed by Github and Gitlab.

I knew there were links to documented oddities or bug reports in pull request threads. But practically speaking, for historic pull requests, pull request comments are useless.

This is the single biggest reason I've started to push for more comments in code. More so than all other tools (issue tracker, code management system, etc.) comments in code have the greatest chance of still being around and easily searchable if they haven't been deleted.

Don't get me started on pull request documentation in an external medium like Slack. It's so rewarding to get or give instant feedback on changes on instant messengers, but good luck finding that discussion 3 months later.

Every time I have to call out a line of code in a pull request, that's immediate cause for that code to be modified with comments.

Maybe I wouldn't do this if Github/Gitlab exposed a Google Docs-like interface for browsing code line by line with links to all pull request comment threads.

The biggest reason to add comments in code (often linking to documented oddities or bug reports) is because it's impossible to search pull request threads historically in every source control management UI I've used.https://t.co/JlHWfbUH5z
— Phil Eaton (@phil_eaton) September 8, 2020

2020–09–07: PinePhone multi-boot image boot testing

megi's PinePhone Development Log — Mon, 07 Sep 2020 01:00:00 GMT

2020–09–05: PinePhone multi-boot image

megi's PinePhone Development Log — Sat, 05 Sep 2020 01:00:00 GMT

2020–09–04: Pinebook Pro and Levinboot

megi's PinePhone Development Log — Fri, 04 Sep 2020 01:00:00 GMT

2020–09–02: Progress on the multi-boot image

megi's PinePhone Development Log — Wed, 02 Sep 2020 01:00:00 GMT

2020–09–01: More p-boot cleanups and an example configuration

megi's PinePhone Development Log — Tue, 01 Sep 2020 01:00:00 GMT

Effective Data Visualization Part 3: Line charts and stacked area charts

Bytepawn - Marton Trencseni — Tue, 01 Sep 2020 01:00:00 GMT

Most charts should be line charts or stacked area chart, because they communicate valuable trend information and are easy to parse for the human eyes and brain.

2020–08–31: Releasing p-boot GUI bootloader

megi's PinePhone Development Log — Mon, 31 Aug 2020 01:00:00 GMT

2020–08–31: Getting started

megi's PinePhone Development Log — Mon, 31 Aug 2020 01:00:00 GMT

How to Create Luck

swyx's site RSS Feed — Sun, 30 Aug 2020 00:23:06 GMT

Your entire worldview changes when you realize you can create luck.

Putting your Keyboard on Steroids with Karabiner Elements

swyx's site RSS Feed — Fri, 28 Aug 2020 05:10:46 GMT

I did a livestream with John Lindquist from Egghead.io today, and he blew my mind on how much mileage you can get out of your keyboard!

Mass delete of Gmail emails

Linux Engineer's random thoughts — Fri, 28 Aug 2020 00:00:00 GMT

Gmail used to be the reference for "infinite email storage". Not anymore. The space growth stopped, and then Google started selling storage space.

It wouldn't be an issue if it were easy to mass delete emails; unfortunately batch delete operations can be quite long, and even lock you out of …

First Look at AWS Amplify Flutter (Developer Preview)

swyx's site RSS Feed — Tue, 25 Aug 2020 21:25:14 GMT

AWS Amplify launched a Flutter Integration, so I thought I would record a quick video and blogpost

Effective Data Visualization Part 2: Formatting numbers

Bytepawn - Marton Trencseni — Sun, 23 Aug 2020 01:00:00 GMT

Format numbers for human consumption. What is more readable, 1.539e+5 or 153,859? Showing numbers effectively on spreadsheets, charts, dashboards, reports is a basic ingredient for readability, like formatting code in programming.

Effective Data Visualization Part 1: Categorical data

Bytepawn - Marton Trencseni — Sat, 22 Aug 2020 01:00:00 GMT

Making clear, readable charts is part of the craftmanship minimum for any data related role. In part one, I look at how to present categorical data.

Cheatsheet for moving from Master to Main

swyx's site RSS Feed — Fri, 21 Aug 2020 06:04:51 GMT

Notes I have collected on how to renaming the Git default/primary branch

Multi-armed bandits and false positives

Bytepawn - Marton Trencseni — Fri, 21 Aug 2020 01:00:00 GMT

I use Monte Carlo simulations to explore the false positive rate of Multi-armed bandits.

Prerecording Talks for Online Conferences

swyx's site RSS Feed — Tue, 18 Aug 2020 02:26:34 GMT

Tips for Speakers Prerecording Talks for Online Conferences

Thank You TailwindCSS

Brightball Articles — Mon, 17 Aug 2020 14:50:18 GMT

In the last 2-3 weeks, after seeing a lot of recommendations, I decided to read the Refactoring UI book and then dove into TailwindCSS. Design has always been a huge gap in my skillset. As frontend complexity increased over the last decade, it seemed less and less worth it to invest time to fill it. Then came TailwindCSS.

Factorio and Software Engineering

Krishna's blog — Sun, 16 Aug 2020 03:00:00 GMT

How playing the game Factorio reminds me of software engineering, from managing complexity to teamwork and automation

Writing a simple Python compiler: 1. hello, fibonacci

Notes on software development — Sun, 16 Aug 2020 03:00:00 GMT

In this post we'll write a Python to C compiler in Python. This is especially easy to do since Python has a builtin parser library and because a number of CPython internals are exposed for extension writers.

By the end of this post, in a few hundred lines of Python, we'll be able to compile and run the following program:

$ cat tests/recursive_fib.py
def fib(n):
    if n == 0 or n == 1:
        return n

    return fib(n - 1) + fib(n - 2)


def main():
    print(fib(40))
$ python3 pyc tests/recursive_fib.py
$ ./bin/a.out
102334155

This post implements an extremely small subset of Python and completely gives up on even trying to manage memory because I cannot fathom manual reference counting. Maybe some day I'll find a way to swap in an easy GC like Boehm.

Source code for this project is available on Github.

Dependencies

We'll need Python3, GCC, libpython3, and clang-format.

On Fedora-based systems:

$ sudo dnf install gcc python3-devel clang-format python3

And on Debian-based systems:

$ sudo apt install gcc python3-dev clang-format python3

This program will likely work as well on Windows, Mac, FreeBSD, etc. but I haven't gone through the trouble of testing this (or providing custom compiler directives). Pull requests welcome!

A hand-written first-pass

Before we get into the compiler, let's write the fibonacci program by hand in C using libpython.

As described in the Python embedding guide we'll need to include libpython and initialize it in our main.c:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int main(int argc, char *argv[]) {
  Py_Initialize();

  return 0;
}

To compile against libpython, we'll use python3-config installed as part of python3-devel to tell us what should be linked at each step during compilation.

$ gcc -c -o main.o $(python3-config --cflags) main.c
$ gcc $(python3-config --ldflags) main.o
$ ./a.out; echo $?
0

Cool! Now as we think about translating the fibonacci implementation, we want to keep everything as Python objects for as long as possible. This means passing and receiving PyObject* to and from all functions, and converting all C integers to PyLong*, a "subtype" of PyObject*. You can imagine that everything in Python is an object until you operate on it.

For more information on objects in Python, check out the Data model page in Python docs.

To map a C integer to a PyLong* we use PyLong_FromLong. To map in reverse, we use PyLong_AsLong.

To compare two PyObject*s we can use PyObject_RichCompareBool which will handle the comparison regardless of the type of the two parameters. Without this helper we'd have to write complex checks to make sure that the two sides are the same and if they are, unwrap them into their underlying C value and compare the C value.

We can use PyNumber_Add and PyNumber_Subtract for basic arithmetic, and there are many similar helpers available to us for operations down the line.

Now we can write a translation:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

PyObject* fib(PyObject* n) {
  PyObject* zero = PyLong_FromLong(0);
  PyObject* one = PyLong_FromLong(1);
  if (PyObject_RichCompareBool(n, zero, Py_EQ) || PyObject_RichCompareBool(n, one, Py_EQ)) {
    return n;
  }

  PyObject* left = fib(PyNumber_Subtract(n, one));

  PyObject* two = PyLong_FromLong(2);
  PyObject* right = fib(PyNumber_Subtract(n, two));

  return PyNumber_Add(left, right);
}

int main(int argc, char *argv[]) {
  Py_Initialize();

  PyObject* res = fib(PyLong_FromLong(7)); // Should be 13

  return PyLong_AsLong(res);
}

Compile and run it:

$ gcc -c -o main.o $(python3-config --cflags) main.c
$ gcc $(python3-config --ldflags) main.o
$ ./a.out; echo $?
13

That's great! But we cheated in one place. We assumed that the input to the fib function was an integer, and we propagated that assumption everywhere we wrote PyNumber_* operations. When we write the compiler, we'll need to check that both arguments are an integer before we call a numeric helper, otherwise we may need to call a string concatenation helper or something else entirely.

Compiler Architecture

We'll break the code into four major parts:

libpyc.c: helper functions for generated code
pyc/context.py: utilities for scope and writing code in memory
pyc/codegen.py: for generating C code from a Python AST
pyc/__main__.py: the entrypoint

When I'm writing a new compiler using an existing parser I almost always start with the entrypoint and code generator so I can explore the AST. However, it's easiest to explain the code if we start with the utilities first.

We'll also want an empty pyc/__init__.py.

libpyc.c

This C file will contain three helper functions for safely adding, subtracting, and printing. It will be concatenated to the top of the generated C file. We'll only support integers for now but this structure sets us up for supporting more types later on.

We'll use PyLong_Check before calling number-specific methods.

#define PY_SSIZE_T_CLEAN
#include <Python.h>

inline PyObject* PYC_Add(PyObject* l, PyObject* r) {
  // TODO: allow __add__ override

  // Includes ints and bools
  if (PyLong_Check(l) && PyLong_Check(r)) {
    return PyNumber_Add(l, r);
  }

  // TODO: handle str, etc.

  // TODO: throw exception
  return NULL;
}

inline PyObject* PYC_Sub(PyObject* l, PyObject* r) {
  // TODO: allow __add__ override

  // Includes ints and bools
  if (PyLong_Check(l) && PyLong_Check(r)) {
    return PyNumber_Subtract(l, r);
  }

  // TODO: handle str, etc.

  // TODO: throw exception
  return NULL;
}

inline PyObject* PYC_Print(PyObject* o) {
  PyObject_Print(o, stdout, Py_PRINT_RAW);
  printf("\n");
  return Py_None;
}

That's it! We could generate these as strings in Python but it gets hairy to do so. By using a dedicated C file, we can take advantage of syntax highlighting since this file is only C code. And since we've marked all functions as inline, there's no runtime cost to using not embedding these as strings in Python.

pyc/context.py

This file will contain a Context class for managing identifiers in scope and for proxying to a Writer class that contains helpers for writing lines of C code.

We'll have two instances of the Writer class in Context so that we can write to a body (or current/primary) region and an initialization region.

The initialization region is necessary in case there are any variables declared at the top-level. We can't initialize these variables in C outside of a function since every PyObject* must be created after calling Py_Initialize. This section will be written into our C main function before we enter a compiled Python main function.

import copy


class Writer():
    content = ""

    def write(self, exp: str, indent: int = 0):
        self.content += ("  " * indent) + exp

    def writeln(self, stmt: str, indent: int = 0):
        self.write(stmt + "\n", indent)

    def write_statement(self, stmt: str, indent: int = 0):
        self.writeln(stmt + ";", indent)


class Context():
    initializations = Writer()
    body = Writer()
    indentation = 0

    scope = 0
    ret = None
    namings = {}
    counter = -1

    def __getattr__(self, name: str) -> object:
        # Helpers to avoid passing in self.indentation every time
        outputs = [initializations", "body"]
        for output in outputs:
            if name.startswith(output):
                return lambda s, i=None: getattr(getattr(self, output), name[len(output)+1:])(s, i if i is not None else self.indentation)

        return object.__getattr__(self, name)

    def get_local(self, source_name: str) -> dict:
        return self.namings[source_name]

    def register_global(self, name: str, loc: str):
        self.namings[name] = {
            "name": loc,
            "scope": 0,
        }

    def register_local(self, local: str = "tmp") -> str:
        self.counter += 1
        self.namings[local] = {
            "name": f"{local}_{self.counter}",
            # naming dictionary is copied, so we need to capture scope
            # at declaration
            "scope": self.scope,
        }
        return self.namings[local]["name"]

    def copy(self):
        new = copy.copy(self)
        # For some reason copy.deepcopy doesn't do this
        new.namings = dict(new.namings)
        return new

    def at_toplevel(self):
        return self.scope == 0

This is all pretty boring boilerplate. Let's move on.

pyc/main.py

The entrypoint is responsible for reading source code, parsing it, calling the code generator, writing the source code to a C file, and compiling it.

First, we read and parse the source code:

import ast
import os
import subprocess
import shutil
import sys

from context import Context
from codegen import generate

BUILTINS = {
    "print": "PYC_Print",
}


def main():
    target = sys.argv[1]
    with open(target) as f:
        source = f.read()
    tree = ast.parse(source, target)

Then we write libpyc.c into the body, register builtins, and run code generation:

...

def main()
    ...

    ctx = Context()
    with open("libpyc.c") as f:
        ctx.body_write(f.read() + "\n")

    for builtin, fn in BUILTINS.items():
        ctx.register_global(builtin, fn)

    generate(ctx, tree)

Next, we create a clean output directory and write main.c with the generated code and a main function to initialization Python and any global variables:

...

def main():
   ...

    # Create and move to working directory
    outdir = "bin"
    shutil.rmtree(outdir, ignore_errors=True)
    os.mkdir(outdir)
    os.chdir(outdir)

    with open("main.c", "w") as f:
        f.write(ctx.body.content)

        main = ctx.namings.get("main")["name"]
        f.write(f"""int main(int argc, char *argv[]) {{
  Py_Initialize();

  // Initialize globals, if any.
{ctx.initializations.content}
  PyObject* r = {main}();
  return PyLong_AsLong(r);
}}""")

Finally, we run clang-format and gcc against the generated C code:

...

def main():
    ...

    subprocess.run(["clang-format", "-i", "main.c"])

    cflags_raw = subprocess.check_output(["python3-config", "--cflags"])
    cflags = [f.strip() for f in cflags_raw.decode().split(" ") if f.strip()]
    cmd = ["gcc", "-c", "-o", "main.o"] + cflags + ["main.c"]
    subprocess.run(cmd)

    ldflags_raw = subprocess.check_output(["python3-config", "--ldflags"])
    ldflags = [f.strip() for f in ldflags_raw.decode().split(" ") if f.strip()]
    cmd = ["gcc"] + ldflags + ["main.o"]
    subprocess.run(cmd)

All together:

import ast
import os
import subprocess
import shutil
import sys

from context import Context
from codegen import generate

BUILTINS = {
    "print": "PYC_Print",
}


def main():
    target = sys.argv[1]
    with open(target) as f:
        source = f.read()
    tree = ast.parse(source, target)

    ctx = Context()
    with open("libpyc.c") as f:
        ctx.body_write(f.read() + "\n")

    for builtin, fn in BUILTINS.items():
        ctx.register_global(builtin, fn)

    generate(ctx, tree)

    # Create and move to working directory
    outdir = "bin"
    shutil.rmtree(outdir, ignore_errors=True)
    os.mkdir(outdir)
    os.chdir(outdir)

    with open("main.c", "w") as f:
        f.write(ctx.body.content)

        main = ctx.namings.get("main")["name"]
        f.write(f"""int main(int argc, char *argv[]) {{
  Py_Initialize();

  // Initialize globals, if any.
{ctx.initializations.content}
  PyObject* r = {main}();
  return PyLong_AsLong(r);
}}""")

    subprocess.run(["clang-format", "-i", "main.c"])

    cflags_raw = subprocess.check_output(["python3-config", "--cflags"])
    cflags = [f.strip() for f in cflags_raw.decode().split(" ") if f.strip()]
    cmd = ["gcc", "-c", "-o", "main.o"] + cflags + ["main.c"]
    subprocess.run(cmd)

    ldflags_raw = subprocess.check_output(["python3-config", "--ldflags"])
    ldflags = [f.strip() for f in ldflags_raw.decode().split(" ") if f.strip()]
    cmd = ["gcc"] + ldflags + ["main.o"]
    subprocess.run(cmd)


main()

Done!

pyc/codegen.py

Lastly we write the translation layer from Python AST to C. We'll break this out into 10 helper functions. It is helpful to have the AST spec for reference.

1/10: generate

The entrypoint of the code generator is generate(ctx: Context, exp). It generates code for any object with a body attribute storing a list of statements. This function will generate code for objects like modules, function bodies, if bodies, etc.

The statements we'll support to begin are:

ast.Assign
ast.FunctionDef
ast.Return
ast.If
and ast.Expr

For each statement, we'll simply pass on generation to an associated helper function. In the case of expression generation though, we'll also add a noop operation on the result of the expression otherwise the compiler will complain about an unused variable.

def generate(ctx: Context, module):
    for stmt in module.body:
        if isinstance(stmt, ast.Assign):
            generate_assign(ctx, stmt)
        elif isinstance(stmt, ast.FunctionDef):
            generate_function_def(ctx, stmt)
        elif isinstance(stmt, ast.Return):
            generate_return(ctx, stmt)
        elif isinstance(stmt, ast.If):
            generate_if(ctx, stmt)
        elif isinstance(stmt, ast.Expr):
            r = generate_expression(ctx, stmt.value)
            ctx.body_writeln("// noop to hide unused warning")
            ctx.body_write_statement(f"{r} += 0")
        else:
            raise Exception(f"Unsupported statement type: {type(stmt)}")

Remember to throw exceptions aggressively otherwise you'll have a bad time debugging programs using new syntax.

Let's dig into these helpers.

2/10: generate_assign

To generate assignment code, we need to check if we're at the top-level or not. If we're at the top-level we can declare the variable but we can't initialize it yet. So we add the initialization code to the initialization section of the program.

If we're not at the top-level, we can declare and assign in one statement.

Before doing either though, we register the variable name so we can get a safe local name to use in generated code. Then we compile the right-hand side so we can assign it to the left-hand side.

import ast

from context import Context


def initialize_variable(ctx: Context, name: str, val: str):
    if ctx.at_toplevel():
        decl = f"PyObject* {name}"
        ctx.body_write_statement(decl, 0)

        init = f"{name} = {val}"
        ctx.initializations_write_statement(init)
    else:
        ctx.body_write_statement(f"PyObject* {name} = {val}")


def generate_assign(ctx: Context, stmt: ast.Assign):
    # TODO: support assigning to a tuple
    local = ctx.register_local(stmt.targets[0].id)
    val = generate_expression(ctx, stmt.value)
    initialize_variable(ctx, local, val)

We're going to need to implement generate_expression to make this work.

3/10: generate_expression

Just like for statements in generate, there are a few kinds of expressions we need to implement:

ast.Num
ast.BinOp
ast.BoolOp
ast.Name
ast.Compare
and ast.Call

For ast.Num, we just need to wrap the literal number as a PyLong*. And for ast.Name we just need to look up the local name in context. Otherwise we delegate to more helper functions.

def generate_expression(ctx: Context, exp) -> str:
    if isinstance(exp, ast.Num):
        # TODO: deal with non-integers
        tmp = ctx.register_local("num")
        initialize_variable(ctx, tmp, f"PyLong_FromLong({exp.n})")
        return tmp
    elif isinstance(exp, ast.BinOp):
        return generate_bin_op(ctx, exp)
    elif isinstance(exp, ast.BoolOp):
        return generate_bool_op(ctx, exp)
    elif isinstance(exp, ast.Name):
        return ctx.get_local(exp.id)["name"]
    elif isinstance(exp, ast.Compare):
        return generate_compare(ctx, exp)
    elif isinstance(exp, ast.Call):
        return generate_call(ctx, exp)

    raise Exception(f"Unsupported expression: {type(exp)}")

For every code generation helper that is an expression, we store the expression in a local variable and return the variable's name so that parent nodes in the AST can refer to the child. This can result in inefficient code generation (useless assignment) but that's not really a big deal for a project like this and will likely be optimized away by GCC anyway. The more annoying aspect is that useless assignment just makes the generated code harder to read.

4/10: generate_bin_op

For binary operators we need to support addition and subtraction. Other binary operators like equality or and/or are represented in ast.Compare and ast.BoolOp.

This is easy to write because we already prepared helpers in libpyc.c: PYC_Sub and PYC_Add.

def generate_bin_op(ctx: Context, binop: ast.BinOp) -> str:
    result = ctx.register_local("binop")

    l = generate_expression(ctx, binop.left)
    r = generate_expression(ctx, binop.right)

    if isinstance(binop.op, ast.Add):
        ctx.body_write_statement(f"PyObject* {result} = PYC_Add({l}, {r})")
    elif isinstance(binop.op, ast.Sub):
        ctx.body_write_statement(f"PyObject* {result} = PYC_Sub({l}, {r})")
    else:
        raise Exception(f"Unsupported binary operator: {type(binop.op)}")

    return result

Easy enough.

5/10: generate_bool_op

We only need to support or for the fibonacci program, but or in Python is more complicated than in C. In Python, the first value to be truthy short-circuits the expression and the result is its value, not True.

We'll use goto to short-circuit and we'll use PyObject_IsTrue to do the truthy check:

def generate_bool_op(ctx: Context, boolop: ast.BoolOp) -> str:
    result = ctx.register_local("boolop")
    ctx.body_write_statement(f"PyObject* {result}")

    if isinstance(boolop.op, ast.Or):
        done_or = ctx.register_local("done_or")

        for exp in boolop.values:
            v = generate_expression(ctx, exp)
            ctx.body_write_statement(f"{result} = {v}")
            ctx.body_writeln(f"if (PyObject_IsTrue({v})) {{")
            ctx.body_write_statement(f"goto {done_or}", ctx.indentation+1)
            ctx.body_writeln("}")

        ctx.body_writeln(f"{done_or}:\n", 0)

    return result

Now that I write this down I see we could probably move this function into libpyc.c if we used a loop. Maybe in the next iteration.

We move on.

6/10: generate_compare

This function handles equality and inequality checks. We'll adapt the PyObject_RichCompareBool helper we used in the hand-written translation.

The only additional thing to keep in mind is that the right-hand side is passed as an array. So we have to iterate through it and apply the equality/inequality check on all objects in the list.

def generate_compare(ctx: Context, exp: ast.Compare) -> str:
    result = ctx.register_local("compare")
    left = generate_expression(ctx, exp.left)
    ctx.body_write_statement(f"PyObject* {result} = {left}")

    for i, op in enumerate(exp.ops):
        v = generate_expression(ctx, exp.comparators[i])

        if isinstance(op, ast.Eq):
            ctx.body_write_statement(f"{result} = PyObject_RichCompare({result}, {v}, Py_EQ)")
        elif isinstance(op, ast.NotEq):
            ctx.body_write_statement(f"{result} = PyObject_RichCompare({result}, {v}, Py_NE)")
        else:
            raise Exception(f"Unsupported comparison: {type(op)}")

    return result

7/10: generate_call

The last expression is simple enough. We compile the call's arguments first, then the function itself, then we call the function with the arguments like any C function. Calling the C function directly will have ramifications for interacting with Python libraries (basically, we won't be able to interact with any) but it's the easiest way to get started.

def generate_call(ctx: Context, exp: ast.Call) -> str:
    args = ', '.join([generate_expression(ctx, a) for a in exp.args])
    fun = generate_expression(ctx, exp.func)
    res = ctx.register_local("call_result")

    # TODO: lambdas and closures need additional work
    ctx.body_write_statement(
        f"PyObject* {res} = {fun}({args})")
    return res

And that's it for expressions! Just a few more statement helpers to support.

8/10: generate_function_def

This is a fun one. First we register the function name in scope. Then we copy the context so variables within the function body are contained within the function body. We increment scope so we know we've left the top-level. Finally, we compile the body.

def generate_function_def(ctx: Context, fd: ast.FunctionDef):
    name = ctx.register_local(fd.name)

    childCtx = ctx.copy()
    args = ", ".join([f"PyObject* {childCtx.register_local(a.arg)}" for a in fd.args.args])
    ctx.body_writeln(f"PyObject* {name}({args}) {{", 0)

    childCtx.scope += 1
    childCtx.indentation += 1
    generate(childCtx, fd)

    if not childCtx.ret:
        childCtx.body_write_statement("return Py_None")

    ctx.body_writeln("}\n", 0)

The check for childCtx.ret isn't strictly necessary because we could just emit a return even if there already was one. Asking generate_return to set this attribute and having generate_function_def check it just makes the generate code a little prettier.

9/10: generate_return

Very straightforward, we just compile the value to be returned and then we emit a return statement.

We store the return value so that the function definition can know whether to add a return PyNone statement.

def generate_return(ctx: Context, r: ast.Return):
    ctx.ret = generate_expression(ctx, r.value)
    ctx.body_writeln("")
    ctx.body_write_statement(f"return {ctx.ret}")

And we've got one last statement to support!

10/10: generate_if

You know the deal: compile the test and if the test is truthy, enter the compiled body. We'll deal with the else body another time.

def generate_if(ctx: Context, exp: ast.If):
    test = generate_expression(ctx, exp.test)
    ctx.body_writeln(f"if (PyObject_IsTrue({test})) {{")
    ctx.indentation += 1
    generate(ctx, exp)
    # TODO: handle exp.orelse
    ctx.indentation -= 1
    ctx.body_writeln("}\n")

And we're done the compiler!

Trying it out

As promised:

$ cat tests/recursive_fib.py
def fib(n):
    if n == 0 or n == 1:
        return n

    return fib(n - 1) + fib(n - 2)


def main():
    print(fib(40))
$ python3 pyc tests/recursive_fib.py
$ ./bin/a.out
102334155

Microbenchmarking, or making compiler Twitter unhappy

Keep in mind this implementation does a small fraction of what CPython is doing.

If you time the generated code:

$ python3 pyc tests/recursive_fib.py
$ time ./bin/a.out
102334155
./bin/a.out  18.69s user 0.03s system 99% cpu 18.854 total

And CPython (with main() append to the source):

time python3 tests/recursive_fib.py
102334155
python3 tests/recursive_fib.py  76.24s user 0.11s system 99% cpu 1:16.81 total

The only reason I mention this is because when I did a similar compiler project for JavaScript targeting C++/libV8, the generated code was about the same or a little slower in speed.

I haven't gotten that much better at writing these compilers.

Latest post up, on writing a simple Python to C compiler (in Python).https://t.co/4kkji0XXbp
— Phil Eaton (@phil_eaton) August 16, 2020

Developer Exception Engineering

swyx's site RSS Feed — Sat, 15 Aug 2020 20:20:15 GMT

It's time we look beyond the easy questions in developer experience, and start addressing the uncomfortable ones.

Scary Terrorism Math

PREPEND — Sat, 15 Aug 2020 03:00:00 GMT

The August 4th explosions in Beirut were very jarring to me due to the loss of human life, and the numerous, direct videos that looked like something out my 80s youth nightmares. Hearing that it was caused by 2,750 tonnes of ammonium nitrate, I started working on my phone to quantify my fear by figuring out how easy it would be to ship that much, and how much it would cost.

According to wikipedia, there were at least 171 deaths; 6,000 injuries; 300,000 people homeless; and US$10-15 billion in destruction with a blast radius causing damage up to two miles away. ¹

After my initial shock, I heard 2,750 tonnes of ammonium nitrate mentioned in the news but didn’t know much about that and wanted to know how big that was. It surprised me that it seems to cost a little over half a million to buy and ship this much It ends up being really cheap, really common, and really small and that scared me by how simple of a risk this is for misuse. I’m hoping that anti-terrorism agencies are working to prevent this but since I didn’t see much coverage on this, I’m posting my notes here in case others were doing similar research.

If you can, please consider donating to help those impacted. Choosing a charity is hard, but I noticed that Nassim Nicholas Taleb recommended Beirut Emergency Fund 2020, a co-worker recommended Lebanese Red Cross, and Google has a nice Donate with Google page set up with the Center for Disaster Philanthropy.

This is unrelated to technology, but I think random researching outside of comfort areas is really important for design and debugging. Also, I think user empathy is really important for figuring out the solution so I frequently have to learn about whatever is important to my users. That being said, if this isn’t interesting, please skip.

My question

My initial wondering was “what is ammonium nitrite and how big is 2,750 tonnes of it?” And that led to a chain of reasoning that made me worry more and more…

How much space does 2,750 tonnes take up?
How much space does that take up?
How many shipping containers is that?
How much would it cost to buy that much?
How much would it cost to ship to that to a harbor near New York?

Nothing here was too complicated but it ate up part of Sunday morning while my wife and I were watching the news. I work on trying to collect references for lines of reasoning and so I started bookmarking some sites as I answered each of these questions.

Breaking down the arithmetic

There are many assumptions I made and I’m not experienced in chemistry or mining or export or shipping so I think that I’m likely quite a bit off in my calculations, so I figure I may be off by a power of 10. But that’s a big caveat before I start making naive calculations.

How big?

Ammonium nitrate is commonly used as a fertilizer and as an industrial explosive. To figure out how much space this much would take up I want to multiply out by its density. According to wikipedia the density of ammonium nitrate is 1.725 grams / cm3, ² so to figure out how much space this takes up I need to convert this mass to a dimension.

1 metric ton = 1,000,000 grams = 1,000 kilograms = 1 megagram
1,000,000 / 1.725 = 579710.144927536231884 cm3 = .5797 m3 = 20.47 cubic feet
2,750 * .5797 = 1594.175 m3

So a single tonne of ammonium nitrate takes up .5797 m3 or 20.47 cubic feet.

Why did I convert that to cubic feet? I wanted to try to think of that in car cargo size and those are displayed in cubic feet.

So a BMW 330 Gran Turismo hatchback has a 18.4 cubic foot capacity with the seats up and 56.5 with the seats down. ⁴ So it won’t fit a whole tonne with the seats up, but it can hold almost three with the seats down. Of course, the axles can’t bear even a half tonne so it’s moot for reality, but I’m now thinking of 2,750 tonnes in terms of hatchbacks.

So, the entire batch of ammonium nitrate in Beirut takes up 1594 m3. That’s too big to think of in hatchbacks, how many shipping containers is that?

A shipping container, or an intermodal container, are those standard sized steel boxes that we see on trucks and stacked up in ports and is used for shipping pretty much everything for freight transport. They come in 20 foot and 40 foot versions and shipping is described in twenty-foot equivalent units or TEUs. This is one of those funny situations where the standard is in imperial units (kind of like tire sizes) but the dimensions are 6.1m long, 2.6m high, and 2.9m wide and have an internal volume of 33.1 m3. ⁵

1594.175 / 33.1 = 48.16238670694864

So 2,750 tonnes takes up 48.16 TEU shipping containers. Still kind of hard to visualize. How many does one of those cargo ships carry? Is 50 containers a lot?

Modern container ships can carry over 23,000 TEUs. ⁶ So 50 containers is just a fraction of the overall cargo.

So I can kind of visualize 50 TEUs and the neat web site The Measure of Things tells me that’s 3/4 the size of a hot air balloon.

So big, but not humongous. Certainly something that the Joker or Simon Gruber could arrange to ship with 25-50 semi trucks.

How expensive?

How much would this much cost? The price seems to vary depending on country, this chart from 2007 shows the cheapest price from Russia at $156/tonne. ⁷ And the current retail price on Alibaba.com is $192-217/tonne. ⁸

The world’s total output was 21.6M tonnes in 2017 and the top five producing countries were Russia (9.9M), Egypt (4.3M), Uzbekistan (1.4M), the US (1.4M), and Poland (1.3M).

So 2,750 seems like a purchasable amount. Assuming that the 2007 price from Russia is reasonable for a bulk price gets us to $429,000 as the raw cost of the material.

156 x 2,750 = $429,000

How much to ship?

So if a bad guy has 50 containers full of ammonium nitrate, how much does that cost to ship?

An explosion in any US port would be terrible, but just to estimate prices I calculated shipping it to the Red Hook Terminal in Brooklyn, NY as a two mile radius from there would reach much of New York City.

Freightos.com shows that four quotes of 50 containers shipped from St. Petersburg, Russia (RULED) to New York (USNYC) range from $106,200-138,000 and will take 13-19 days.

Conclusion

So it seems like the total cost to purchase and ship 2,750 tonnes of ammonium nitrate is $567,000. That seems way too cheap to cause so much damage.

According to CNN, the fertilizer was on a Russian ship called the MVRhosus and was bound for Madagascar when it stopped in Beirut.

This is one of those risks that I don’t know any way to practically cope with, so I have to trust that multiple government agencies are working every day making sure stuff like this doesn’t happen.

But if someone is working on a Bond script, or an Austin Powers script, this might eventually show up in a story.

All the calculations together

$156 x 2,750 tonnes = $429,000
1,000,000 grams / 1.725 g/cm3 = 579710.144927536231884 cm3 = .5797 m3
2,750 tonnes * .5797 m3 = 1594.175 m3
1594.175 m3 / 33.1 m3/TEU = 48.16238670694864 TEU
$429,000 chemicals + $138,000 shipping = $567,000

Unanswered questions

Questions I didn’t find that definitely impact overall cost:

I’m not sure what mixture and ingredients are needed beyond raw ammonium nitrate.
I’m not sure if stacked containers will result in a similar explosion.
Not sure how much it costs to rent all those containers.
St. Petersburg may not be the best shipping port from Russia.
How hard it is to arrange customs forms.

The History of Containers

Stuff I read while working on this post

References

shipping container image from Business Insider
bmw image from Autocar

Footnotes

The Bigger the Tos

infrequent oscillations — Thu, 13 Aug 2020 14:32:21 GMT

In which we recreate an infographic.

Realtime Offline-First Chat App in 100 Seconds

swyx's site RSS Feed — Thu, 13 Aug 2020 01:27:32 GMT

A quick demo of how easy it is to add realtime, offline persistence to an app with Amplify DataStore

A Developer's Guide to Startup Fundraising

swyx's site RSS Feed — Wed, 12 Aug 2020 03:21:55 GMT

A Brief Guide to Startup Fundraising Terminology for Beginners

5 Q&A's on Writing and Selling My First Book

swyx's site RSS Feed — Tue, 11 Aug 2020 02:24:57 GMT

I answered some questions from someone planning to write their first book

Text to "Go String" Tool

Donat Studios — Sat, 08 Aug 2020 09:14:19 GMT

Go currently lacks any sort of heredoc or similar syntax, and it's backtick syntax cannot contain backticks. This can make correctly encoded creating go strings, particularly for large strings that contain many backticks, a bit of a challenge.

Here's a tool to safely encode any given input to Go string. It's written in Go using Go's own string encoder and compiled with WASM.

The code as well as a CLI version of it are available on GitHub.

Fork my source on GitHub!

Daddy, did you really need to buy an electric car?

Svedic.org — Fri, 07 Aug 2020 16:36:13 GMT

I made a mistake: I bought an electric car. EV articles I have read on Hacker News and Reddit didn’t prepare me for a dozen EV infrastructure problems in my part of the EU. Anecdotes below explain lessons I learned … Continue reading →

A/B testing and Multi-armed bandits

Bytepawn - Marton Trencseni — Fri, 07 Aug 2020 01:00:00 GMT

Multi-armed bandits minimize regret when performing A/B tests, trading off between exploration and exploitation. Monte Carlo simulations shows that less exploration yields less statistical significance.

Random Stats About URLs

infrequent oscillations — Wed, 05 Aug 2020 16:36:46 GMT

In which we count some URLs and domains.

Wednesday roundup for 2020-08-05

infrequent oscillations — Wed, 05 Aug 2020 12:36:55 GMT

Various things, links, and excitement.

Stay Safe on the Road

Ben E. C. Boyter — Sun, 02 Aug 2020 03:00:00 GMT

SI live in an area with a popular driving road. As such you see a lot of people flying around in fancy cars, motorbikes and bike riders. Between where I am and the start there is an awful bit of road that funnels everyone together. As such a lot of people jump the line to get past the bike riders because there isn’t enough space for them to have their own lane (I keep complaining to my local member about this).

Football summary 2019

infrequent oscillations — Wed, 29 Jul 2020 21:43:31 GMT

In which we lament the introduction of VAR.

Anomalous Parity

infrequent oscillations — Wed, 29 Jul 2020 11:11:42 GMT

In which we fiddle with some code.

Notes on "Growing a Language"

swyx's site RSS Feed — Tue, 28 Jul 2020 23:50:13 GMT

Revisiting one of my favorite talks ever, on designing programming languages for user extensibility and collaboration

4 Things I Learned from Mastering Mongoose.js

swyx's site RSS Feed — Sun, 26 Jul 2020 12:53:19 GMT

A quick book review of the new Mongoose.js book from Val Karpov

A single-node Kubernetes cluster without virtualization or a container registry

Notes on software development — Sat, 25 Jul 2020 03:00:00 GMT

This post is a recipe for setting up a minimal Kubernetes cluster on Fedora without requiring virtualization or a container registry. These two features make the system cloud-agnostic and the cluster entirely self-contained. The post will end with us running a simple Flask app from a local container.

This setup is primarily useful for simple CI environments or application development on Linux. (Docker Desktop has better tooling for development on Mac or Windows.)

Getting Kubernetes

The core of this effort is K3s, a Kubernetes distribution that allows us to run on a single node without virtualization.

But first off, install Docker.

Then install K3s:

$ curl -sfL https://get.k3s.io | sh -

It may prompt you to adjust some SELinux policies like so:

$ sudo dnf install -y container-selinux selinux-policy-base
$ sudo rpm -i https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm

Swap these out with whatever it prompts and retry the K3s install.

Finally, install kubectl:

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

Now copy the global K3s kubeconfig into ~/.kube/config:

$ sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
$ sudo chown $USER:$GROUP ~/.kube/config

And enable K3s:

$ sudo systemctl enable k3s

If you're on Fedora 31+ you'll need to disable cgroups v2 and reboot:

$ sudo grubby --args="systemd.unified_cgroup_hierarchy=0" --update-kernel=ALL
$ sudo reboot

Finally, you can run kubectl:

$ kubectl get pods
No resources found in default namespace.

A simple application

We'll create a small Flask app, containerize it, and write a Kubernetes deployment and service config for it.

We begin with app.py:

from flask import Flask

app = Flask(__name__)


@app.route('/')
def index():
    return 'Hello World, Flask!'


if __name__ == '__main__':
    app.run(debug=True)

Then a Dockerfile:

FROM python:3-slim

RUN pip install flask
COPY . /app

CMD python3 /app/app.py

Then the deployment in manifest.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
spec:
  selector:
    matchLabels:
      name: helloworld
  template:
    metadata:
      labels:
        name: helloworld
    spec:
      containers:
        - image: helloworld
          name: helloworld

Running in Kubernetes

First we build, save, and import the image into k3s:

$ docker build . -t helloworld
$ docker save helloworld > helloworld.tar
$ sudo k3s ctr image import helloworld.tar
$ kubectl apply -f ./manifest.yaml
$ kubectl port-forward $(kubectl get pods | grep helloworld | cut -d ' ' -f 1) 5000 > log 2>&1 &
$ curl localhost:5000
Hello World, Flask

And that's it!

Latest post is a recipe for creating a self-contained, single-node Kubernetes cluster for CI environments using a basic Flask app.https://t.co/fegAZFEQzO
— Phil Eaton (@phil_eaton) July 25, 2020

8 Landing Page Tips from Rob Hope

swyx's site RSS Feed — Fri, 24 Jul 2020 01:19:19 GMT

I submitted my book's landing page for Rob Hope's review and he obliged! Brutal feedback incoming.

8 Q&A's for Bootcamp Students in 2020

swyx's site RSS Feed — Thu, 23 Jul 2020 20:37:24 GMT

I did a Q&A for Fullstack Academy Bootcamp Prep students - copying out my answers here!

Unofficial VS Code Snippets for AWS Amplify

swyx's site RSS Feed — Thu, 23 Jul 2020 19:08:16 GMT

making my own vs code snippets helpers for working with AWS Amplify

Tuesday roundup for 2020-07-21

infrequent oscillations — Tue, 21 Jul 2020 13:31:28 GMT

Various things, links, and excitement.

Columnarised Highlights

infrequent oscillations — Mon, 20 Jul 2020 20:38:46 GMT

In which we format some text, with colours.

On right action and American politics

Ben Overmyer's Site — Mon, 20 Jul 2020 03:00:00 GMT

We try to do what's right in the best way we can. This is a statement that generally guides human behavior. You can summarize this as "right action."

What defines right action differs from person to person, however. In looking at the approaches of Americans to the problem of politics and national leadership, we can observe two primary examples.

The first defines the quality of being right as the degree of perceived adherence to one's world view. The second defines the quality of being right as the degree to which a measurable result adheres to the desired outcome.

So, when judging the policy of a political figure, someone with the first approach will judge that policy as correct if it conforms to one's value set. Someone with the second approach will judge that policy as correct if it has been measurably proven to be so.

These two approaches overlap to varying degrees in a single individual. The policy in question will be judged more dispassionately and in greater alignment with the second approach if the policy has less alignment with an existing identity value of the individual. That is, a person is more likely to be rational about a topic if that topic doesn't matter to them much. However, if the topic impedes or aligns with a core value of the person in question, they are far more likely to be irrational in their reaction to policy on that topic.

It should seem reasonable to suggest that an empirical approach to every decision is ideal. The problem is that humans are incapable of being rational about every decision in their lives. However, in striving to be rational about as much as possible, one can approach the ideal even if it remains elusive.

Judging a policy based on its measured results or on historical measured data is far preferable to the alternative. When evaluating those measurements, it's important to trust the opinion of experts over the opinion of unqualified individuals. This lends itself to more accurate prediction of results, which in turn allows for more effective policy choice.

Further, once a policy has been in place for an amount of time that allows for statistically significant results, that policy can and should be reevaluated to determine its correctness. This reinforces a structure of continual improvement.

Too often in American politics, policymakers make value judgments based on personal identity and not on measured results. Voters decide on their representatives the same way. This conflicts with the original philosophical climate that gave birth to the United States of America in the first place. Benjamin Franklin, Thomas Jefferson, and James Madison were all proponents of Enlightenment philosophy. The phrasing of the Declaration of Independence includes language to this effect; in describing the right of people to establish new government, the Declaration states that they lay "... its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness."

It's not enough to content oneself with baser instincts. Instead of choosing the path of blind acceptance, we must choose to temper our molten feelings with the cool water of empirical logic. Only then can we forge a strong future.

Semi-Automatic npm and GitHub Releases with `gh-release` and `auto-changelog`

swyx's site RSS Feed — Fri, 17 Jul 2020 21:57:45 GMT

A snippet I use all the time

Screwed by Lufthansa and the German Government, Saved by PayPal

Svedic.org — Wed, 15 Jul 2020 20:33:22 GMT

The travel and tourism sectors are suffering because of the COVID-19 pandemic, and governments are scrambling to offer subsidies to affected businesses. But, there is little talk on how companies are transferring their costs to citizens, with government approval. In … Continue reading →

The National Broadband Network (NBN) A Massive Dissapointment

Ben E. C. Boyter — Tue, 14 Jul 2020 03:00:00 GMT

I write this mostly so I can move on. The NBN has been a source of constant frustration and anger for me. While I will never forgive those who caused the current shambles I can perhaps redirect my anger elsewhere will keeping it in the back of my mind.

So here is my brief time-line of NBN.

I moved to area that was due to get NBN FTTP in ~2014. The fact that it was due to get such a connection influenced the decision. With the change in government that ended up going away. I was then promised by Senator Fifield (who replaced Malcolm Turnbull as communications Minister) while on the campaign trail for the 2016 election that NBN would start construction in the area in 2016. This however was pushed back to 2017. What followed was it being pushed out roughty every 3 months. Finally told it will be done in June 2018.

Infrequent Roundup, part N

infrequent oscillations — Fri, 10 Jul 2020 15:44:49 GMT

Various things, links, and excitement.

Voronants

infrequent oscillations — Mon, 06 Jul 2020 11:19:43 GMT

In which we explore a fractal landscape with ants.

Lessons and Regrets from My $25000 Book Launch

swyx's site RSS Feed — Fri, 03 Jul 2020 07:06:47 GMT

Reflections on the Coding Career book launch

Launching the Coding Career Handbook!

swyx's site RSS Feed — Wed, 01 Jul 2020 16:08:37 GMT

I'm launching my career advice book today!

AWS Amplify - Access Denied

Tilting at Windmills — Mon, 29 Jun 2020 03:00:00 GMT

In my continuing journey to learn more about Gatsby, React, and all things serverless and single page app I hosted the prototype of my…

Versioned Docs in 30 Seconds with Amplify Console's Branch Subdomains

swyx's site RSS Feed — Fri, 26 Jun 2020 19:34:09 GMT

Amplify Console just got the ability to create a custom subdomain for every new branch. This makes creating versioned docs a cinch!

Minimal CSS

Ben Overmyer's Site — Thu, 25 Jun 2020 03:00:00 GMT

After years of using a CSS framework for my site, I've now dropped it entirely.

Inspired by the movement to reduce the filesize and cruft of websites that have no need for graphics or heavy JavaScript, I decided to cut out as much as possible. This started a few months ago with removing my web fonts and replacing them with system fonts.

Today, I removed Bulma from my site. I opted instead for the most basic CSS I could. I still write it as SCSS, and I rely on several CSS3 features, but the file size has dropped dramatically. So has the load time.

This may be just a phase, but for now, I'm really enjoying the new minimal aesthetic.

The game breaking Deal Breaker card

Nemo's Home — Wed, 24 Jun 2020 03:00:00 GMT

I have a presentation I sometimes give about Monopoly being a terrible game¹. I usually end it by pointing the audience to Monopoly Deal, which I introduce it as “the only Monopoly edition you can enjoy”.

The advantages are obvious:

Much shorter game length.
No dice rolls.

Even if you lose badly on luck, you’ve only lost 15 minutes. While it is better than Monopoly, that isn’t to say it is a well designed game (6.3 on BoardGameGeek).

The one major dent in the otherwise decent game is the “Deal Breaker” card, which breaks the game. I’ve house-ruled the game since forever to keep the card out of the game, since it breaks the most important rule of game design:

Games should be fun²

Deal Breaker stops people from doing that. How? Read on.

A lot of metagaming discussion with friends resulted in the following observations:

The Deal Breaker is a very powerful card (It takes you 1/3rd of the way to a victory).
You can assume the single Deal Breaker card to be worth a complete set.
The best use of such a card is to win the game. Using it earlier means giving other players a chance to drag you down from 2->1 set. But if you use it to win the game, the game ends immediately.
Hence, Deal Breaker will always end up being the last card of the game.

If you are playing a game with the Deal Breaker card, you’d want to save it till the very end, and win the game with it. The only possible case for not winning is the other player having a “Just Say No” card, and playing it on the Deal Breaker to negate your move.

Ergo, the metagame converges to the following:

Any game with Deal Breaker will end up having the Deal Breaker as the last turn.
The only way to prevent someone else from winning with the Deal Breaker is to play a Just Say No on the Deal Breaker.
If you have a Just Say No card, you must save it till the end of the game for the Deal Breaker.

There are 2 Deal Breakers, and 3 Just Say Nos in the game. However, considering a single Deal Breaker is enough to win the game, and the chances of you getting a second of either card are fairly small - both the Deal Breaker and Just Say No cards will end up getting hoarded for the endgame.

What this results in is something that breaks the fundamental rule of game design:

Players are disincentivized from playing the Just Say No card.

As any Exploding Kittens player can confirm, playing a Just Say No card is one of the coolest moves in the game. It lets you stick it to the player who dares ask you for 8M³ rent. It lets you pretend you’re counting your money, and then pull out a trump card and feel awesome! By disincentivizing players from playing the coolest card in the game, the Deal Breaker card makes things less fun. And that breaks our “rule of fun”.

In fact, the mere existence of a Deal Breaker card changes the equation. Note that there may be cases where you lose with a Just Say No card because you were hoarding it for the eventual Deal Breaker (which might never come). Someone asks you for 4M rent, and you have to pay up despite having a Just Say No card, because you must save the damn card for when someone steals your set. There are a few rare exceptions, but the Deal Breaker creates too many plays where not playing the Just Say No is indeed the correct move.

So here is the more interesting corollary observation:

The mere existence of the Deal Breaker card breaks the game by making the Just Say No card unplayable and worthless.⁴

Hence, if you’re playing Monopoly Deal, please house-rule the Deal Breaker card and make it easier for everyone. Two easy ways are:

Remove the card from the game entirely.
Reduce the power of the game to be same as a Forced Deal, except let it break a set.

I have the slides here, but they don’t stand well on their own. ↩
Any game that eliminates players from the game breaks this rule. Popular examples are Monopoly and Mafia/Werewolf. Also see this amazing post on the biggest mistake that Guillotine makes (The “Callous Guard” card). ↩
Turns out, that there is no symbol in the unicode for Monopoly Money ↩
We decided Deal Breaker might make sense in 6+ player games with many more cards, where it might help even the playing grounds a bit for a losing player (much more likely) instead of helping the almost-winning-player score a victory. ↩

A survey of OneCardToRuleThemAll companies

Nemo's Home — Mon, 22 Jun 2020 03:00:00 GMT

A lot of companies have come up with the idea for reducing all your cards into a single piece of plastic. Here’s a summary of all the ones I could find, and their fate. Beware: this field is very much a startup graveyard. The only remaining survivor seems to be Curve¹ but it’s also the first one that’s attempting this outside of US.

There seem to be a lot of challenges (regulatory, financial, and technical) before such a thing becomes reality. And there’s Apple/Samsung Pay as well. Here’s a summary of all the companies I could find in this space. If I’ve missed any, please let me know, and I’ll add them here.

Curve (2015-)

Curve allows you to spend from any of your accounts using just one card, clearing the clutter from your wallet and simplifying your finances.

Curve has raised a total of $74M at a valuation north of $250M, out of which £6M were from a crowdfunding campaign in 2019.
Their waitlist is at 800,000+ users.
Curve faced flak for failing to disclose its usage numbers (<100,000 monthly active users out of total 500,000) to crowdfund investors.
Only works with Visa/MasterCard. Used to work with Amex, but Curve has a history of working and being blocked by Amex repeatedly.
See this page for some details on how it works.

Coin (YC W13) (2012-2017)

Coin raised a total of $15.6M.
Shipped in 2015 April.
Acquired by Fitbit in May 2016.
Shutdown by Fitbit in Feb 2017.
Related: Google acquired FitBit in 2019 Nov.

Plastc (2014-2017)

Single dynamic card with e-Ink touchscreen. See obligatory launch video with fancy AR.
Crowdfunded $9M on pre-orders in October 2014.
Delayed launch to April 2016, and then again to September 2016.
Shut down and declared bankruptcy in April 2017.
Complete story via digg.com.

Stratos (2015-2015)

Was supposed to cost $95/year.
Announced May 2015.
Raised $6.63 million over three rounds of financing.
Ran out of money by December 2015.
Sold to Ciright One to avoid collapse.
Seems to be dead now.

Swyp (2014-2017)

Had a pre-order campaign in 2014.
Raised $5M from Khosla Ventures in 2017.
Tried to pivot in 2017, failed.
Offered customers a debit card with an app called Hoot in 2017 as an alternative to refunds. I don’t think the Hoot card ever materialized.
Tilt (Swyp’s payment processor) also ceased operations in 2017, due to unrelated reasons.
Swyp finally shut down in December 2017.

Final (YC W15) (2014-2017)

Final was with a credit card with a different number for every website. As a independent card, Final doesn’t exactly fit in this list, but it’s a very relevant and loved product.
Announced itself with a snazzy video in mid 2014
Raised $4M and launched in August 2016, but remained invite-only till the very end.
Final’s blog has some interesting content: A Request for Credit Cards program to build card-issuance backed businesses and the Payment card landscape for 2017.
Shut down in december 2017 and acquired by Goldman Sachs in an acqui-hire.

Indian Landscape

India hasn’t seen a true single-card app yet, but there have been lots of related attempts:

Infino promises a single-card, but it hasn’t launched yet. It had a public roadmap, but never launched and ultimately pivoted to a Coupon Aggregator called Meet Donut, which shut down.
IndusInd bank has tried a dual-chip debit+credit card. Union Bank has a similar card.
IndusInd has also tried an interactive credit card with 3 buttons.
FamPay, Fold, Niyo, OneCard, Donut🪦, vCard🪦, Slice ² and lots of other startups are doing co-branded issuance, but that’s not the same thing.

With the emergence of UPI, and low penetration rate of credit cards, I don’t see a market in India - but I’d love to be proven wrong.

Did I miss anything? Reach out and let me know.

Thanks to Harman for reviewing drafts of this, and PRL for getting me interested enough to document this.

Curve used to live on imaginecurve.com, then switched to curve.app and now to curve.com, which must have cost them millions. ↩
Unlike others on the list, vCard is entirely a virtual card, and supports UPI transfers from your credit limit. ↩

Notes on RSI for Developers

swyx's site RSS Feed — Fri, 19 Jun 2020 21:45:25 GMT

I'm starting to feel some RSI in my left hand. It's a matter of time. I decided to collect some information about it to make improvements now rather than later.

#HeyForGood

swyx's site RSS Feed — Fri, 19 Jun 2020 00:37:56 GMT

Recounting my process of how I started with 2 HEY email invites and ended up raising $2575 in Donations to Support Diversity in Tech!

My Life as a Con Man

swyx's site RSS Feed — Sun, 14 Jun 2020 22:46:24 GMT

Confidence is a dual edged sword. I trafficked in confidence when I was in finance, and now I see it everywhere I look.

The Curious Case of Swift's Adoption of Smalltalk Keyword Syntax

metablog — Sun, 14 Jun 2020 18:13:02 GMT

I was really surprised to learn that Swift recently adopted Smalltalk keyword syntax: [Accepted] SE-0279: Multiple Trailing Closures. That is: a keyword terminated by a colon, followed by an argument and without any surrounding braces.

The mind boggles.

A little.

Of course, Swift wouldn't be Swift if this weren't a special case of a special case, specifically the case of multiple trailing closures, which is a special case of trailing closures, which are weird and special-casey enough by themselves. Below is an example:


UIView.animate(withDuration: 0.3) {
  self.view.alpha = 0
} completion: { _ in
  self.view.removeFromSuperview()
}

Note how the arguments to animate() would seem to terminate at the closing parenthesis, but that's actually not the case. The curly braces after the closing paren start a closure that is actually also an argument to the method, a so-called trailing closure. I have a little bit of sympathy for this construct, because closures inside of the parentheses look really, really awkward. (Of course, all params apart from a sole x inside f(x) look awkward, but let's not quibble. For now.).

Another thing this enables is methods that reasonably resemble control structures, which I heard is a really great idea.

The problem is that sometimes you have more than one closure argument, and then just stacking them up behind what appears to be end of the function/method call gets really, really awkward, and you can't tell which block is which argument, because the trailing closure doesn't get a keyword.

Well, now it does. And we now have 4 different method syntaxes in one!

Traditional C/Pascal/C++/Java function call syntax x.f()
The already weird-ish addition of Smalltalk/Objective-C keywords inside the f(x) syntax: f(arg:x)
Original trailing-closure syntax, which is just its own thing, for the first closure
Smalltalk non-brackted keyword syntax for the 2nd and subsequent closures.

That is impressive, in a scary kind of way.

Swift is a crescendo of special cases stopping just short of the general; the result is complexity in the semantics, complexity in the behaviour (i.e. bugs), and complexity in use (i.e. workarounds).
— Which features overcomplicate Swift, Rob Rix

In understand that this proposal was quite controversial, with heated discussion between opponents and proponents. I understand and sympathize with both sides. On the one hand, this is markedly better than alternatives. On the other hand it is a special case of a special case that is difficult to justify as an addition of all that is already there.

Special cases beget special cases beget special cases.

Of course the answer was always there: Smalltalk keyword syntax is not just the only reasonable solution in this case, it also solves all the other cases. It is the general solution. Here's how this could look in Objective-Smalltalk (which uses curly braces instead for closures instead of Smalltalk-80's square brackets):


UIView animate:{ self.view.alpha ← 0. } withDuration:0.3 completion:{ self view removeFromSuperview. }.

No special cases, every argument is labeled, no syntax mush of brackets inside parentheses etc. And yes, this also handles user-defined control structures, to:do: is just a method on NSNumber:


1 to:10 do:{:i | stdout println:"I will not introduce {i} special cases willy nilly.".}.

And since keywords naturally go between their arguments, there is no need for "operators", as a very different and special syntax form. You just allow some "binary" keywords to look a little different, so instead of 2 multiply:3 you can write 2 * 3. And when you have 2 raisedTo:3 instead of pow(2,3) (with the signature: func pow(_ x: Decimal, _ y: Int) -> Decimal), do you really neeed to go to the trouble of defining an "operator"?

Or Swift's a as b, another special kind of syntax. How about a as:b? (Yes I know there are details, but those are ... details.). And so on and so forth.

But of course, it's too late now. When I chose Smalltalk as the base syntax for the language that has turned into Objective-Smalltalk, it wasn't just because I just like it or have gotten used to it via Objective-C. Smalltalk's syntax is surprisingly flexible and general, Smalltalk APIs look a lot like DSLs, without any of the tooling or other overheads.

And that's the frustrating part: this stuff was and is available and well-known. At least if you bother to look and/or ask. But instead, we just choose these things willy-nilly and everybody has to suffer the consequences.

UPDATE:

I guess what I am trying to get at is that if you'd thought things through just a little bit, you could have had almost the entire syntax of your language for the cost (complexity, implementation size and brittleness, cognitive load, etc.) of this one special case of a special case. And it would have been overall better to boot.

Unicode support. What does that actually mean? 😃

Ben E. C. Boyter — Sun, 14 Jun 2020 03:00:00 GMT

Unicode support in my experience seems to be one of those hand wavy things where most people respond to the question of “Do you support unicode?” with

Yeah we support emojis, so yes we support unicode! 😃😭😈

Which is partially correct… It’s certainly a good start. However unicode is more than emojis.

So lets take a step back. What is unicode? It’s actually easy to explain, here it is taken from the wiki page “A standard for representing the worlds text”. That means it includes all the languages in the world, such as chinese characters, japanese writing systems, arabic and even nordic runes. As such you get all sorts of interesting things in the standard such as characters that display as white space but actually are not because they represent a nordic rune carved on the side of the stone not visible from the front.

Generating a full-stack application from a database

Notes on software development — Sun, 14 Jun 2020 03:00:00 GMT

DBCore can now generate a TypeScript/React CRUD UI that is automatically hooked up to the generated REST API (in Go).

The UI has full support for login, viewing (and filtering), editing, and creating database entities.

PostgreSQL, SQLite and MySQL are supported.

How to use?

The goal of this project is primarily to provide as much useful boilerplate as possible for full-stack applications. The system is probably not sufficient to be an entire application development platform. It's currently missing hooks, overrides, and per-row/per-table authorization.

The UI code generation may be even less useful in the long-term than the API because UIs are by necessity very diverse. But it is good not to need to build the same browser-side API, authentication, and routing logic again now that it's taken care of in code generation.

Screenshots

Here are a few screenshots of the examples/todo application. Every page here is auto-generated after reading the database schema. The browser application is hooked up to the similarly auto-generated API.

Creating a table entity

Viewing all table entities

Filtering table entities

Viewing an individual table entity

Editing a table entity

What's new in DBCore: a TypeScript/React UI generated from your database schema and hooked up to the similarly generated Go REST API.

So you can now generate an entire full stack application from your database schema. Screenshots in the post.https://t.co/BTGRVBsfUR
— Phil Eaton (@phil_eaton) June 14, 2020

Random roundup for 2020-06-12

infrequent oscillations — Fri, 12 Jun 2020 11:38:09 GMT

Various things, links, and excitement.

4 Q&A's on Blogging for Developers

swyx's site RSS Feed — Fri, 12 Jun 2020 02:17:37 GMT

Answering Q&A's

His Dark Materials Season 1 Readthrough

Nemo's Home — Sun, 07 Jun 2020 03:00:00 GMT

A long time ago, I tried to do a readthrough for Game of Thrones (Book 1) alongside the first season. I managed to reach Episode 5, before I sped through the rest of the book, but I tried.

Trying something similar for His Dark Materials, which is a great series if you’re looking to watch something new. Instead of noting down Chapter/Book equivalence (like I tried last time), going to write down my thoughts here as I’m reading along. Spoiler Warning for the entire first season obviously.

Overall

The show is very tightly knit with the book with a few over-arching changes from the story-telling perspective:

You get to see other points-of-view, other than just Lyra. Helps establish what else is happening, especially in the other worlds.
A lot of infodumps are prevented, or better, broken down into multiple sessions.
The major change from the first book is ofcourse showing Will’s PoV and our earth.

Chapter 1

The opening scene with the great flood sets some context, but isn’t in the books.
The Master/Butler chat on the wine poisoining happens much later in the books.
There is a lot of foreshadowing around Lyra’s parentage that happens in the first 2 chapters that is entirely missed in the show.

Chapter 2

The entire Grumman’s skull and hunt sub-plot hasn’t shown up in the book so far (presumably because we only see Lyra’s PoV).
The party scene is very-well handled (with all the subtle changes for the better. Superb acting as well :)
The hiding-lyra-in-the-boat scene is merely given a passing mention in the book, but so well done in the show.

Chapter 3

Splitting Lyra’s parentage reveal (Coulter reveals her father) is a smart move in the show.
The show changes Lyra’s kidnappers from Turkish slavers to Gobblers.
The Alethiometer reveal happens with both Father Coram and John Fa in the book. The section also has a huge infodump, especially since it involves the parentage reveal. The show breaks it into 3: alethiometer reveal with Father Coram, a previous interrogation of Lyra with John Fa, and Lyra’s parentage reveal (mother) with Ma Costa.

Chapter 4

The one notable “not-in-the-book” scene is the Coulter’s meeting with Iofur.

Chapter 5

Interesting to note that the characters of Billy and Roger are fused in both the film and the TV adaptations.

Chapter 6

Lyra starts a fire in the books, but the show makes it more dramatic by destroying the machine.
The balloon ride covers a lot more in the books.

Learnings

Overall, the show has been nicely adapted so far, and I think there’s a few reasons:

The show barely messes with Lyra’s timeline. Important to ensure this to avoid creating cascading issues down the path.
Majority of the changes are either made on kill-able subplots, or side-plots that show us what’s going on elsewhere.
And finally, the show spends time on where the medium works best. The scaring scene in Chapter 2, for eg.

I’m still sad that the ghosts in the crypt don’t get to be seen, though.

CVE-2020-13254

Dan Palmer — Sun, 07 Jun 2020 02:00:00 GMT

Information Exposure Vulnerability with Django and Memcached

On Wednesday April 29th, Thread started experiencing a partial outage of our main backend service. We traced the issue down to the existence of malformed Memcached keys and corrected the issue on thread.com. Along the way we suspected that this could be exploited on some Django sites using Memcached to cause private data exposure – either internal service data or data about other users. The only issue on Thread was HTTP 500 server errors seen by a small number of users, no private data was leaked.

Generating a REST API from a database

Notes on software development — Sat, 06 Jun 2020 03:00:00 GMT

I recently published an alpha version of a code generation tool, DBCore, that reads a database schema from PostgreSQL or MySQL and generates an entire Go API with CRUD operations, pagination, filtering, and authentication.

But more than just generating code like xo/xo or gnorm, DBCore defines a standard REST API that can be implemented in any language -- and includes a reference implementation in Go. I'm eager to add Java and Ruby implementations as well. And I'd be more than happy to accept community contributions.

Boilerplate & code generation

Web application boilerplate is boring. You should do it once from scratch (preferably down to the socket layer) and never do it again. I struggled for the last few years to find the right system to reduce boilerplate. If I were building a new line-of-business application as an employee I'd pick one of Rails, ASP.NET, Spring, Django, or similar.

I've never worked on one of those frameworks professionally and I've never been able to force myself to learn any of them in my free time. But even if I could use one of these, none of them get close to giving you an entire functioning application with authentication, pagination, filtering all based on your existing database.

Over the last few years though I've relied heavily on code generation for Go projects. Code generation is basically the only way to conserve type-safe code in Go. But it's similarly popular in more powerful languages like Java.

However none of the existing projects give you much flexibility or provide you with enough templates to be useful.

DBCore

DBCore is written in F# and can be distributed as a static binary on all systems .NET now supports (read: not just Windows!).

Reading from MySQL or PostgreSQL is supported but I'd like to see that extended to include SQLite, Oracle, and MS SQL at least.

As mentioned, currently DBCore only provides a Go REST API template. That only solves half the problem of building an application though. And while there are some projects that can generate an admin CRUD interface for you, I want to see that more tightly integrated into DBCore. So I'll be introducing a new template for a browser application as well. For each table in the database it will generate a page showing paginated entries and allow you to create, update, and delete.

Finally, while the tool only currently has a concept of "browser" and "api" templates, the project should be able to accept any kind of template and generate any text based on any database schema.

New blog post, background and goals for dbcorehttps://t.co/XW9gUCtvr0
— Phil Eaton (@phil_eaton) June 7, 2020

crickybo

infrequent oscillations — Wed, 03 Jun 2020 03:00:00 GMT

Alas, due to the current world situation, crickybo is on lockdown. Fediverse: @crickybo@social.rjp.is Twitter Telegram

Making AWS Amplify work with Rollup

swyx's site RSS Feed — Wed, 03 Jun 2020 01:22:28 GMT

AWS Amplify assumes CommonJS, which Rollup is allergic to. I recently discovered that you can make it work with Rollup with a few tweaks.

MPWTest Only Tests Frameworks

metablog — Mon, 01 Jun 2020 14:10:28 GMT

It should be noted, if it wasn't obvious, that MPWTest is opinionated software, meaning it achieves some of its smoothness by gleefully embracing constraints that some might view as potentially crippling limitations.

Maybe the biggest of these constraints, mentioned in the previous post, is that MPWTest only tests frameworks. This means that the following workflow is not supported out of the box:

The original OCUnit (that I wrote after @KentBeck's paper and was out at about the same time as JUnit) friction was not "high" IMHO: add the test framework, write a subclass of TestCase, launch your app with a -Test argument, results are logged in ProjectBuilder' console.
— Marco Scheurer (@phink0) May 30, 2020

The point being that this is a workflow I not just somewhat indifferently do not want, but rather emphatically and actively want to avoid. Tests that are run (only?) when launching the app are application tests. My perspective is that unit tests are an integral part of the class. This may seem a subtle distinction, but subtle differences in something you do constantly can have huge impacts. "Steter Tropfen höhlt den Stein."

Another aspect is that launching the app for testing as a permanent and fixed part of your build process seems highly annoying at best. Linker finishes, app pops up, runs for a couple of seconds, shuts down again. I don't see that as viable. For testing to be integral and pervasive, it has to be invisible when the tests succeed.

The testing pyramid is helpful here: my contention is that you want to be at the bottom of that pyramid, ideally all of the time. Realistically, you're probably not going to get there, but you should push really, really hard, even making sacrifices that appear to be unreasonable to achieve that goal.

Framework-oriented programming

Only testing frameworks begs the question as to how to test those parts of the application not in frameworks. For me the answer is simple: there isn't any production code outside of frameworks.

None. Not the UI, not the application delegate. Only the auto-generated main().

The benefits of this approach are plentiful, the effort minimal. And if you think this is an, er, eccentric position to take, the program you almost certainly use to create apps for iOS/macOS etc. takes the same eccentric position: Xcode's main executable is 45K in size and only contains a main() function and some Swift boilerplate.

If all your code is in frameworks, only testing frameworks is not a problem. That may seem like a somewhat extreme case of sour grapes, with the arbitrary limitations of a one-off unit testing framework driving major architectural decisions, but the causality is the other way around: I embraced framework-oriented programming before and independently of MPWTest.

iOS

Another issue is iOS. Running a command-line tool that dynamically loads and tests frameworks is at least tricky and may be impossible, so that approach currently does not work. My current approach is that I view on-device and on-simulator tests as higher-up in the testing hierarchy: they are more costly, less numerous and run less frequently.

The vast majority of code lives in cross-platform frameworks (see: Ports and Adapters) and is developed and tested primarily on macOS. I have found this to be much faster than using the simulator or a device in day-to-day programming, and have used this "mac-first" technique even on projects where we were using XCTest.

Although not testing on the target platform may be seen as a problem, I have found discrepancies to be between exceedingly rare and non-existent, with "normal" code trending towards the latter. One of the few exceptions in the not-quite-so-normal code that I sometimes create was the change of calling conventions on arm64, which meant that plain method pointers (IMPs) no longer worked, but had to be cast to the "correct" pointer type, only on device. Neither macOS nor the simulator would show the problem.

For that purpose, I hacked together a small iOS app that runs the tests specified in a plist in the app bundle. There is almost certainly a better way to handle this, but I haven't had the cycles or motivation to look into it.

How to approximate

So you can't or don't want to adopt MPWTest. That doesn't mean you can't get at least some of the benefits of the approach. As a start, instead of using Cmd-B in Xcode to build, just use Cmd-U instead. That's what I did when working on Wunderlist, where we used XCTest.

Second, adopt framework-oriented programming and the Ports and Adapters style as much as possible. Put all your code in frameworks, and as much as possible in cross-platform frameworks that you can test/run on macOS, and even if you are developing exclusively for iOS, create a macOS target for that framework. This makes using Cmd-U to build much less painful.

Third, adhere to a strict 1:1 mapping between production classes and test classes, and place your test classes in the same file as the class they are testing.

Thanks for sharing! I do this all the time in Rust, where tests are at the bottom of the source. I love it. Some people complain that now we're mingling testing code and prod code, but I think that doesn't hold: we're also adding logs and assertions to our prod code.
— Benedikt Terhechte @ 🏠 (@terhechte) June 1, 2020

With OCUnit you didn't have to but your tests in a separate bundle, target etc. You could have them at the bottom of your class file in a subclass of TestCase instead of a "(testing) category". The difference is not that big.
— Marco Scheurer (@phink0) May 31, 2020

My practical experience with both JUnit and XCTest on medium-sized projects does not square with the assertion that the difference is not that big: you still have to create these additional classes, they have to communicate with the class under tests (self in MPWTest), you have to track changes etc. And of course, you have to know to configure und use the framework differently from the way it was built, intended and documented. And what I've seen of OCUnit use was that the tests were not co-located with the class, but in a separate part of the project.

A final note is that the trick of interchangeably using the class as the test fixture is only really possible in a language like Objective-C where classes are first class objects. It simply wouldn't be possible in Java. This is how the class can test itself, and the tests become an integral part of the class, rather than something that's added somewhere else.

MPWTest: Reducing Test Friction by Going Beyond the xUnit Model

metablog — Sun, 31 May 2020 09:59:55 GMT

By popular demand, a quick rundown of MPWTest (“The Simplest Testing Framework That Could Possibly Work”), my own personal unit testing framework, and how it makes TDD fast, fun, and frictionless.

I created MPWTest because once I had been bitten by the TDD bug, I definitely did not want to write software without TDD ever again, if I could help it. This was long before XCTest, and even its precursor SenTestKit was in at best in parallel development, I certainly wasn't aware of it.

It is a bit different, and the differences make it sufficiently better that I much prefer it to the xUnit variants that I've worked with (JUnit, some SUnit, XCTest). All of these are vastly better than not doing TDD, but they introduce significant amounts of overhead, friction, that make the testing experience much more cumbersome than it needs to be, and to me at least partly explains some of the antipathy I see towards unit testing from developers.

The attitude I see is that testing is like eating your vegetables, you know it's supposed to be good for you and you do it, grudgingly, but it really is rather annoying and the benefits are more something you know intellectually.

For me with MPWTest, TDD is also still intellectually a Good Thing™, but also viscerally fun, less like vegetables and more like tasty snacks, except that those snacks are not just yummy, but also healthy. It helps me stay in the flow and get things done.

I would second MPWTest.

Mind you, I don't know how it's normally done—but the fact that everything is in one file && part of the build process makes it WAY WAY WAY faster.

Given the advent of SwiftUI—it's like Xcode previews—but for tests.

1/
— 𝔾𝕦𝕤𝕥𝕒𝕧𝕠 𝕄𝕦𝕔𝕙𝕠 𝕃𝕠𝕧𝕖 👌🏻 (@LongMuchoLove) May 28, 2020

What it does is let me change code quickly and safely, the key to agile:

Contrary to what you may have read in the Agile literature, the key to agility is the ability to change code quickly and safely. And the key to that is the ability to re-test code quickly and effectively. Fast-running automated tests ("unit tests") are the key to agility.
— Jason Gorman (only, more indoors than usual) (@jasongorman) April 18, 2020

Here is how it works.

Setup

First you need to build the testlogger binary of the MPWTest project. I put mine in /usr/local/bin and forget about it. You can put it anywhere you like, but will have to adjust the paths in what follows.

Next, add a "Script" build phase to your (framework) project. MPWTest currently only tests frameworks.


tester=/usr/local/bin/testlogger
framework=${TARGET_BUILD_DIR}/${FULL_PRODUCT_NAME}

if  [ -f ${tester}  ]  ; then
    $tester ${framework}
else
    echo "projectfile:0:1: warning: $tester or  $framework  not found, tests not run"
fi

The bottom of the Build Phases pane of your project should then look something roughly like the following:

There is no separate test bundle, no extra targets, nada. This may not seem such a big deal when you have just a single target, but once you start getting having a few frameworks, having an additional test target for each really starts to add up. And adds a decision-point: should I really create an additional test bundle for this project? Maybe I can just repurpose this existing one?

Code

In the class to be tested, add the +(NSArray*)testSelectors method, returning the list of tests to run/test methods to execute. Here is an example from the JSON parser I've been writing about:


+testSelectors
{
  return @[
            @"testParseJSONString",
            @"testParseSimpleJSONDict",
            @"testParseSimpleJSONArray",
            @"testParseLiterals",
            @"testParseNumbers",
            @"testParseGlossaryToDict",
            @"testDictAfterNumber",
            @"testEmptyElements",
            @"testStringEscapes",
            @"testUnicodeEscapes",
            @"testCommonStrings",
            @"testSpaceBeforeColon",
          ];
}

You could also determine these names automagically, but I prefer the explicit list as part of the specification: these are the tests that should be run. Otherwise it is too easy to just lose a test to editing mistakes and be none the wiser for it.

Then just implement a test, for example testUnicodeEscapes:


+(void)testUnicodeEscapes
{
	MPWMASONParser *parser=[MPWMASONParser parser];
	NSData *json=[self frameworkResource:@"unicodeescapes" category:@"json"];
	NSArray *array=[parser parsedData:json];
	NSString *first = [array objectAtIndex:0];
	INTEXPECT([first length],1,@"length of parsed unicode escaped string");
	INTEXPECT([first characterAtIndex:0], 0x1234, @"expected value");
	IDEXPECT([array objectAtIndex:1], @"\n", @"second is newline");
}

Yes, this is mostly old code. The macros do what you, er, expect: INTEXPECT() expects integer equality (or other scalars, to be honest), IDEXPECT() expects object equality. There are also some conveniences for nil, not nil, true and false, as well as a specialized one for floats that sets an acceptable range.

In theory, you can put these methods anywhere, but I tend to place them in a testing category at the bottom of the file.


...
@end

#import "DebugMacros.h"

@implementation MPWMASONParser(testing)

The DebugMacros.h header has the various EXPCECT() macros. The header is the only dependency in your code, you do not need to link anything.

Even more than not having a separate test bundle, not having a separate test class (-hierarchy) really simplifies things. A lot.

First, there is no question as to where to find the tests for a particular class: at the bottom of the file, just scroll down. Same for the class for some tests: scroll up. I find this incredibly useful, because the tests serve as specification, documentation and example code for class.

There is also no need to maintain parallel class hierarchies, which are widely regarded as a fairly serious code-smell, for the obvious reasons: the need to keep those hierarchies in sync along with the problems once they do get out of sync, which they will, etc.

Use

After the setup, you just build your projects, the tests will be run automatically as part of the build. If there are test failures, they are reported by Xcode as you would expect:

My steps tend to be:

add name of test to +testSelectors,
hit build to ensure tests are red,
while Xcode builds, add empty test method,
hit build again to ensure tests are now green,
either add an actual EXPECT() for the test,
or an EXPECTTRUE(false,@"impelemented") as placeholder

This may seem like a lot of steps, but it's really mostly just letting Xcode check things while I am doing the edits that need to be done anyhow. Hitting Cmd-B a couple of times while editing doesn't hurt.

The fact that tests run as part of every build, because you cannot build without running the tests, gives you a completely different level of confidence in your code, which translates to courage.

Running the tests all the time is also splendid motivation to keep those tests green, because if the tests fail, the build fails. And if the build fails, you cannot run the program. Last not least, running the tests on every build also is strong motivation to keep those tests fast. Testing just isn't this separate activity, it's as integral a part of the development process as writing code and compiling it.

Caveats

There are some drawbacks to this approach, one that the pretty Xcode unit test integration doesn't work, as when this was done Apple had already left the platform idea behind and was only focused on making an integrated solution.

As noted above, displaying test failures as errors and jumping to the line of the failed test-expectation does work. This hooks into the mechanism Xcode uses to get that information from compilers, which simply output the line number and error message on stdout. Any tool that formats its output the same way will work wth Xcode.

In the end, while I do enojoy the blinkenlights of Xcode's unit test integration, and being able to run tests individually with simple mouse-click, all this bling really just reinforces that idea of tests as a separate entity. If my tests are always run and are always green, and are always fast, then I don't need or even want UI for them, the UI is a distraction, the tests should fade into the background.

Another slightly more annoying issue is debugging: as the tests are run as part of the build, a test failure is a build failure and will block any executables from running. However, Xcode only debugs executables, so you can't actually get to a debuggable run session.

As I don't use debuggers all that much, and failure in TDD usually manifests itself in test failure rather than something you need the debugger to track, this hasn't been much of a problem. In the past, I would then just revert to the command line, for example with lldb testlogger MPWFoundation to debug my foundation framework, as you can't actually run a framewework. Or so I thought. Only receently did I find out that you can set an executable parameter in your target's build scheme. I now set that to testlogger and can debug the framework to my heart's content.

Leaving the problem of Xcode not actually letting me run the executable due to the build failing, and as far as I know having no facility for debugging build phases.

The workaround for that is temporarily disabling the Test build phase, which can be accomplished by misusing the "Run script only when installing" flag.

While these issues aren't actually all the significant, they are somewhat more jarring than you might expect because the experience is so buttery smooth the rest of the time.

Of course, if you want a pure test class, you can do that: just create a class that only has tests. Furthermore, each class is actually asked for a test fixture object for each test. The default is just to return the class object itself, but you can also return an instance, which can have setup and teardown methods the way you expect from xUnit.

The code to enumerate and probe all classes in the system in order to find tests is also interesting, if straightforward, and needs to be updated from time to time, as there are a few class in the system that do not like to be probed.

Outlook

I'd obviously be happy if people try out MPWTest and find it useful. Or find it not so useful and provide good feedback. I currently have no specific plans for Swift support. Objective-C compatible classes should probably work, the rest of the language probably isn't dynamic enough to support this kind of transparent integration, certainly not without more compiler work. But I am currently investigating Swift interop. more generally, and now that I am no longer restricted to C/Objective-C, more might be possible.

I will almost certainly use the lessons learned here to create linguistically integrated testing in Objective-Smalltalk. As with many other aspects of Objective-Smalltalk, the gap to be bridged for super-smooth is actually not that large.

Another takeaway is that unit testing is really, really simple. In fact, when I asked Kent Beck about it, his response was that everyone should build their own. So go and build wonderful things!

Bringing AWS to App Developers

swyx's site RSS Feed — Fri, 29 May 2020 15:35:53 GMT

Where Amplify fits in AWS' trajectory, and why I am joining

Double Friday roundup for 2020-05-22, 2020-05-29

infrequent oscillations — Fri, 29 May 2020 11:53:30 GMT

Various things, links, and excitement.

Why bother arguing?

Krishna's blog — Tue, 26 May 2020 03:00:00 GMT

The article discusses the influence of network science on how we form opinions and argues the importance of speaking out against harmful ideas.

Measuring Data Maturity

PREPEND — Fri, 22 May 2020 03:00:00 GMT

Lately I’ve been thinking about data maturity models within large organizations, and how to measure maturity and ability to use data. Specifically, what it means to be data literate or data fluent or whatever buzzword is used to mean “hip and with it like the cool kids” when it comes to being able to collect, sort, order, and use data to the greatest extent possible. To help think this through, I’m putting down a few thoughts here. My goal is to figure out ways to personally use data better and help the same in others.

What is Maturity?

I have not fond memories of CMMI maturity models for software engineering being applied by VCs to companies I worked for. I’m sure at some scale this model is helpful, but for the ~100 person dev shop in the ~$200M/year company it had a of effort trying to figure out stages of maturity to equate that to how well the company’s engineering was doing. It seems to measure the wrong things (lines of code, whether code review was performed, whether programmers had formal training, etc) whether they just existed instead of what those processes accomplished. So the joke was, among a few programmers, that they raised their score by just doing a code review when the person doing it wasn’t even familiar with code or underlying code base.

This isn’t intended to rehash programmer grunt work of yore, but to say that models can be helpful, but the aren’t what I’m trying to understand when I wonder how to improve my org’s capability to use data. I’m trying to figure out how to find ways to identify curiosity and potential ability and improve that. Also, I’m always curious as to whether training programs work, so I want to think of ways to measure training working.

So I do read through a lot of material on maturity models (CMMI comically, DAMA, US Data Governance Playbook, Gartner has lots, but this is all I could find publicly).

How About Competence Instead of Maturity?

So if my goal is really about doing more “good things” with data, maybe it’s better to measure competence or ability. Then I use that as a proxy for maturity.

Breaking this down further into types of competence:

Individual data competence means things that a person or group can do. Maybe this is data literacy. Being able to find a book to read, read it, and do the thing I learn from the book.
Enterprise data competence means things the organization needs to be able to do, and do. Maybe this is maturity. Building libraries, sponsoring the arts, using Dewey Decimal Classification.

Big, Imaginary Survey

So if I could survey everyone who matters in an organization, and have complete response (or at least enough to be statistically significant, although I’m no statistician), then I could have a proxy baseline of where we are. And maybe identify some areas to start improving.

“Everyone who matters” is hard to define, but philosophically would mean everyone who has the spark of potential to work with data (eg, read in raw data and calculate important values, link data together, predict values, etc) and everyone who needs to be able to accurately interpret and use data (eg, pick the right metric, use the right chart in their powerpoint, etc). So it’s already a really big percentage of most organizations, but I expect that not everyone really cares that much. But the idea is it’s not just the machine learning wizards who we need to know.

Organizational Questions

Do you know whose job it is to help you with any data related problems? No idea my current baseline, but think the target should be 100%
Who is that? Unknown baseline, but target should be 50% figuring some people won’t be able to list that on the spot.
Are you confident that they will help you? Unknown baseline, but think target is maybe 80% since there are lots of diverse data problems and it’s probably impossible to help with everything.
How many times have they helped you? Unknown baseline. Target should be 75% having one or more helps since we want everyone doing data stuff, even it’s just finding the report server.
How many times have they tried and failed to help you? Unknown baseline. Target should also be 75% having one or more fails, figuring that just as many failures are needed to have good helps. Next question tries to suss out healthy failures vs. malignant failures.
How many tries do you think were productive? A productive failure is one that creates validated learning, helps with future attempts, or otherwise benefits the workplace by being tried and known. Unknown baseline. Target is 100% where the answer is the sum of the tries question.
Please point to the source code, blog post, knowledge base, or other evidence of the most recent time they helped you. Unknown baseline. Target is 100% of the previous question. Providing an example helps encourage knowledge collaboration and filter out low effort responses, I think.

Individual Questions

Are you confident that you, personally, can gather and use enough data to either help make a decision, or to specify the statement of objectives for a consultant or service to do it for you? Unknown, guessing is 1%. Target is 100%.
Are there things you need that would increase the likelihood of you answering the previous question positively? Please list them… Unknown baseline, but guessing 100%. This one high is bad. True target is probably 25% since there are likely false positives of people being unaware of available training, services, tools, etc. Don’t want it to be zero, as there’s always new stuff coming out and we want to perpetually scan for new things using customer input.

Conclusion

This helps me a bit because the concept of data literacy as something everyone has was weird to me, since it’s hard to teach people to read if there are no libraries and printing presses. I don’t think everyone will be a viz swat team. There are some things the organization needs to be competent in, that enables literacy. We need both to thrive, but I worry that if we focus on getting all the individuals better, and then lack the organization infrastructure necessary, it’s not going to be as useful as doing both.

I also worry that we will create crappy organization stuff. Crappy is highly subjective, bu I think we can balance it by following human-centered design principles in a way that is auditable to customers. It’s weird how anti-HCD it is to say “I do that” and then not let anyone see the inputs and interstitial analyses and products that exist as part of building. Or even more kafka-esque, saying “we do that” but the only way to see the design material is to be lucky enough to know who is planning projects and lobby them to spend a bunch of time telling stories. I think a good measure of maturity is the visibility that comes from seeing skyscrapers going up for new, wonderful libraries and focusing literacy training in those neighborhoods.

Stuff I read while working on this post

Discriminating against left handers

Krishna's blog — Fri, 22 May 2020 03:00:00 GMT

This article discusses the importance of proportional representation and the effects of discrimination in different fields.

Understanding Facebook’s Planout A/B testing framework

Bytepawn - Marton Trencseni — Fri, 22 May 2020 01:00:00 GMT

PlanOut is a framework for online field experiments. It was created by Facebook in 2014 to make it easy to run and iterate on sophisticated experiments in a statistically sound manner.

Anzacs, updated

infrequent oscillations — Thu, 21 May 2020 15:06:21 GMT

In which we perfect the Anzac recipe.

Keto FAQ

Krishna's blog — Thu, 21 May 2020 03:00:00 GMT

A personal experience with the keto diet, including its benefits and downsides.

The ability to work remotely in Embedded is a sign of software engineering maturity

Linux Engineer's random thoughts — Thu, 21 May 2020 01:00:00 GMT

This has been brewing for a while, but I finally put it into words during this pandemic: if you're an embedded software engineer, the ability to work remotely (without the hardware next to you) is in fact a sign that you have reached a certain level of software engineering maturity …

The Third Age of JavaScript

swyx's site RSS Feed — Wed, 20 May 2020 17:57:23 GMT

A bunch of things are moving in JavaScript - it is quite feasible that the JS of 10 years from now will look totally unrecognizable

A new direction for The Road Chose Me

The Road Chose Me — Tue, 19 May 2020 17:00:08 GMT

Exactly eleven years ago today, on the 19th of May 2009, The Road Chose Me was born. We’ve come a very long way in that time. I vividly remember my brother Mike and I...

Working The Reverse 9 to 5

swyx's site RSS Feed — Mon, 18 May 2020 17:27:19 GMT

Having weird sleeping hours is working out... very well for me.

Learning from Board Game Design

Dan Palmer — Mon, 18 May 2020 02:00:00 GMT

Last year I bought a copy of Scythe from publisher Stonemaier Games, based in large part on the art. I was very happy with the art and enjoy playing the game, but what I found even more satisfying was the design of the rulebook, the iconography, and the use of physical tokens to re-inforce processes used throughout the game. This week I bought Wingspan from the same publisher, again based in large part on the artwork, and once again I’m finding the other aspects even more satisfying.

Slack is Fumbling Developers

swyx's site RSS Feed — Sun, 17 May 2020 19:43:08 GMT

Reflecting on the sudden switch away from Slack for Developer communities

RFCs and asynchronous-first culture

Notes on software development — Sat, 16 May 2020 03:00:00 GMT

I hated writing documentation before working on features. But after a while I realized I couldn't communicate well enough, even with folks I had a good connection with. It took me a number of mistaken deliveries to get the message.

Sketches and mockups

Designers solve this by producing low-fidelity sketches early on in the process, iterating on feedback to produce a high-fidelity mockup. I solve this by producing short RFC (request for comment) documents. This isn't an original idea, but I see it so rarely I wanted to share.

Now as soon as I begin thinking about a technical or organizational change, I write an RFC. My RFCs are typically a page or two long and typically take me 30-60 minutes for a good first draft. I make clear in the title that it is a proposal or draft. This allows me to make crazy suggestions without upsetting folks; a draft can be easily thrown away.

RFC process

My RFCs include three key sections:

What I think the problem is
Pros/cons of all the solutions I considered
Which solution I'm planning to go with if no one responds to the RFC

After I write the first draft I circulate it among a small group of peers I respect, my boss, etc. I request feedback at leisure and I check in every few days with a reminder. If no one responds after a while and there is little concern, I typically move forward with the proposed solution.

In addition to clarifying intent up front, this removes the need to schedule a meeting to discuss a problem. Discussion and decisions can be held asynchronously. I only schedule a meeting if there is disagreement that is unable to be resolved in writing.

After incorporating feedback, I either throw away the RFC and move on or feel reasonably confident about the proposal. I send it out to a wider group of relevant participants. Final meetings are held as needed.

The other option

In contrast, synchronous-first and undocumented proposals make some sense when you've got a small team in the same timezone with a similar schedule. Otherwise, you repeatedly reschedule meetings to accommodate everyone. You spend your first few meetings simply coming to understand and agree on the problem.

Spending 30-60 minutes to draft a proposal is almost always easier. It makes the decision-making process faster and produces more accurate results.

Spending 30-60 minutes to draft a technical (or organizational) proposal is almost always easier for discussion and action than just scheduling a meeting. Or "my asynchronous-first manifesto"https://t.co/gm4SUzBD2W
— Phil Eaton (@phil_eaton) May 16, 2020

Friday roundup for 2020-05-15

infrequent oscillations — Fri, 15 May 2020 12:34:53 GMT

Various things, links, and excitement.

This Saint Does Not Exist

infrequent oscillations — Fri, 15 May 2020 11:24:14 GMT

In which we make up some saints.

Trevors

infrequent oscillations — Thu, 14 May 2020 10:05:10 GMT

Making plans for Trevor.

IE11 Mainstream End Of Life in Oct 2020

swyx's site RSS Feed — Wed, 13 May 2020 03:00:00 GMT

Making the case for Mainstream EOL of IE 11.

Use a GIF as a Zoom Virtual Background

Donat Studios — Wed, 13 May 2020 00:23:38 GMT

Update August 17 2021: I've got the tool working in Chrome and Firefox by launching it a standalone new window instead of loading in an iframe. It is still broken in Safari for unknown reasons.

It's not ideal, but until the whole SharedArrayBuffer debacle settles down, it's probably the best I can do right now.

Update October 17 2022: The mp4's should now always be compatible with QuickTime. While they were always compatible with Zoom, occasionally not being compatible with QuickTime was confusing some people.

Stuck in boring Zoom meetings? Have the perfect gif you want to use as your "Zoom Virtual Background", but Zoom won't let you use animated gifs?

Here's a tool to convert GIFs to MP4's that Zoom will accept!

Previously, this page was just instructions for using ffmpeg yourself. The tool uses ffmpeg for you via WASM to make it easy.

There's an easy solution - using ffmpeg to convert your GIF to an MP4.

You can install ffmpeg with brew or other package manager if you're on a UNIX-like system. Otherwise downloads are available here.

Once you have ffmpeg installed just run the following in your terminal, replacing <gif-file.gif> with the path to your gif.
ffmpeg -i <gif-file.gif> -vf scale=w=640:h=360:force_original_aspect_ratio=increase output.mp4
The flags on the command specify a minimum size of 640 by 360, Zoom's minimum video size, while maintaining aspect ratio by increasing either dimension.

Writing a SQL database from scratch in Go: 4. a database/sql driver

Notes on software development — Sun, 10 May 2020 03:00:00 GMT

Previously in database basics: <! forgive me, for I have sinned >
1. SELECT, INSERT, CREATE and a REPL
2. binary expressions and WHERE filters
3. indexes

In this post, we'll extend gosql to implement the database/sql driver interface. This will allow us to interact with gosql the same way we would interact with any other database.

Here is an example familiar program (stored in cmd/sqlexample/main.go) we'll be able to run:

package main

import (
    "database/sql"
    "fmt"

    _ "github.com/eatonphil/gosql"
)

func main() {
    db, err := sql.Open("postgres", "")
    if err != nil {
        panic(err)
    }
    defer db.Close()

    _, err = db.Query("CREATE TABLE users (name TEXT, age INT);")
    if err != nil {
        panic(err)
    }

    _, err = db.Query("INSERT INTO users VALUES ('Terry', 45);")
    if err != nil {
        panic(err)
    }

    _, err = db.Query("INSERT INTO users VALUES ('Anette', 57);")
    if err != nil {
        panic(err)
    }

    rows, err := db.Query("SELECT name, age FROM users;")
    if err != nil {
        panic(err)
    }

    var name string
    var age uint64
    defer rows.Close()
    for rows.Next() {
        err := rows.Scan(&name, &age)
        if err != nil {
            panic(err)
        }

        fmt.Printf("Name: %s, Age: %d\n", name, age)
    }

    if err = rows.Err(); err != nil {
        panic(err)
    }
}

Our gosql driver will use a single instance of the Backend for all connections.

Aside from that, it is a simple matter of wrapping our existing APIs in structs that implement the database/sql/driver.Driver interface.

This post is largely a discussion of this commit.

Implementing the driver

A driver is registered by calling sql.Register with a driver instance.

We'll add the registration code to an init function in a new file, driver.go:

struct Driver {
    bkd Backend
}

func init() {
    sql.Register("postgres", &Driver{NewMemoryBackend()})
}

According to the Driver interface, we need only implement Open to return an connection instance that implements the database/sql/driver.Conn interface.

type Driver struct {
    bkd Backend
}

func (d *Driver) Open(name string) (driver.Conn, error) {
    return &Conn{d.bkd}, nil
}

func init() {
    sql.Register("postgres", &Driver{NewMemoryBackend()})
}

Implementing the connection

According to the Conn interface, we must implement:

Prepare(query string) (driver.Stmt, error) to handle prepared statements
Close to handle cleanup
and Begin to start a transaction

The connection can also optionally implement Query and Exec.

To simplify things we'll panic on Prepare and on Begin (we don't have transactions yet). There's no cleanup required so we'll do nothing in Close.

type Conn struct {
    bkd Backend
}

func (dc *Conn) Prepare(query string) (driver.Stmt, error) {
    panic("Prepare not implemented")
}

func (dc *Conn) Begin() (driver.Tx, error) {
    panic("Begin not implemented")
}

func (dc *Conn) Close() error {
    return nil
}

The only method we actually need, Query, is not required by the interface. It takes a query string and array of query parameters, returning an instance implementing the database/sql/driver.Rows interface.

To implement Query, we basically copy the logic we had in the cmd/main.go REPL. The only change is that when we return results when handling SELECT, we'll return a struct that implements the database/sql/driver.Rows interface.

database/sql/driver.Rows is not the same type as database/sql.Rows, which may sound more familiar. database/sql/driver.Rows is a simpler, lower-level interface.

If we receive parameterized query arguments, we'll ignore them for now. And if the query involves multiple statements, we'll process only the first statement.

func (dc *Conn) Query(query string, args []driver.Value) (driver.Rows, error) {
    if len(args) > 0 {
        // TODO: support parameterization
        panic("Parameterization not supported")
    }

    parser := Parser{}
    ast, err := parser.Parse(query)
    if err != nil {
        return nil, fmt.Errorf("Error while parsing: %s", err)
    }

    // NOTE: ignorning all but the first statement
    stmt := ast.Statements[0]
    switch stmt.Kind {
    case CreateIndexKind:
        err = dc.bkd.CreateIndex(stmt.CreateIndexStatement)
        if err != nil {
            return nil, fmt.Errorf("Error adding index on table: %s", err)
        }
    case CreateTableKind:
        err = dc.bkd.CreateTable(stmt.CreateTableStatement)
        if err != nil {
            return nil, fmt.Errorf("Error creating table: %s", err)
        }
    case DropTableKind:
        err = dc.bkd.DropTable(stmt.DropTableStatement)
        if err != nil {
            return nil, fmt.Errorf("Error dropping table: %s", err)
        }
    case InsertKind:
        err = dc.bkd.Insert(stmt.InsertStatement)
        if err != nil {
            return nil, fmt.Errorf("Error inserting values: %s", err)
        }
    case SelectKind:
        results, err := dc.bkd.Select(stmt.SelectStatement)
        if err != nil {
            return nil, err
        }

        return &Rows{
            rows:    results.Rows,
            columns: results.Columns,
            index:   0,
        }, nil
    }

    return nil, nil
}

Implementing results

According to the Rows interface we must implement:

Columns() []string to return an array of columns names
Next(dest []Value) error to populate an row array with the next row's worth of cells
and Close() error

Our Rows struct will contain the rows and colums as returned from Backend, and will also contain an index field we can use in Next to populate the next row of cells.

type Rows struct {
    columns []ResultColumn
    index   uint64
    rows    [][]Cell
}

func (r *Rows) Columns() []string {}

func (r *Rows) Close() error {}

func (r *Rows) Next(dest []driver.Value) error {}

For Columns we simply need to extract and return the column names from ResultColumn.

func (r *Rows) Columns() []string {
    columns := []string{}
    for _, c := range r.columns {
        columns = append(columns, c.Name)
    }

    return columns
}

For Next we need to iterate over each cell in the current row and retrieve its Go value, storing it in dest. The dest argument is simply a fixed-length array of interface{}, so we'll need no manual conversion.

Once we've reached the last row, the Next contract is to return an io.EOF.

func (r *Rows) Next(dest []driver.Value) error {
    if r.index >= uint64(len(r.rows)) {
        return io.EOF
    }

    row := r.rows[r.index]

    for idx, cell := range row {
        typ := r.columns[idx].Type
        switch typ {
        case IntType:
            i := cell.AsInt()
            if i == nil {
                dest[idx] = i
            } else {
                dest[idx] = *i
            }
        case TextType:
            s := cell.AsText()
            if s == nil {
                dest[idx] = s
            } else {
                dest[idx] = *s
            }
        case BoolType:
            b := cell.AsBool()
            if b == nil {
                dest[idx] = b
            } else {
                dest[idx] = b
            }
        }
    }

    r.index++
    return nil
}

Finally in Close we'll set index higher than the number of rows to force Next to only ever return io.EOF.

func (r *Rows) Close() error {
    r.index = uint64(len(r.rows))
    return nil
}

And that's all the changes needed to implement a database/sql driver! See here for driver.go in full.

Running the example

With the driver in place we can try out the example:

$ go build ./cmd/sqlexample/main.go
$ ./main
Name: Terry, Age: 45
Name: Anette, Age: 57

Next post in the database basics series, implementing a database/sql driver for more seamless interactions in Go.https://t.co/AUZfUByNGE
— Phil Eaton (@phil_eaton) May 10, 2020

Friday roundup for 2020-05-08

infrequent oscillations — Fri, 08 May 2020 21:09:17 GMT

Various things, links, and excitement.

Is this what modern web development is?

Dan Palmer — Fri, 08 May 2020 02:00:00 GMT

During GitHub’s annual product announcement on Wednesday, new features to edit code online were demoed. At one point a code snippet was shown from a toy web-app, written in Javascript using the Express server library.

Here’s the code sample…

After the announcement, David Heinemeier Hansson (DHH), the creator of Ruby-on-Rails gave his thoughts on Twitter.

Is this really what modern web app development looks like to people these days? We truly are living through the dark ages. The boiler plating, the low-level distractions, the raw pool handling + sql, the configuration situps. Lordy. pic.twitter.com/1sEhWV6il1

Geographical Spans

infrequent oscillations — Wed, 06 May 2020 07:25:32 GMT

In which we work out my geographical span.

Building a API rate limiter in Go for searchcode

Ben E. C. Boyter — Mon, 04 May 2020 03:00:00 GMT

So regular readers of this site might know that I run https://searchcode.com which I recently converted over to Go. While I had always run searchcode cheaply I never wanted to implement rate limits and instead operate as Bill and Ted by being “Excellent to each other”. Sadly the internet is not from the Bill and Ted universe and due to some level of abuse against the API I have been forced to implement a rate limiter.

Star Wars Beskar Viewing Order

Nemo's Home — Mon, 04 May 2020 03:00:00 GMT

I tweeted out my recommended viewing order for Star Wars recently¹:

Happy #StarWarsDay folks. If you've somehow managed to avoid watching Star Wars, here's my recommended viewing order (No Spoilers, 1/n) pic.twitter.com/K70O4ydCB7
— Nemo (@captn3m0) May 4, 2020

Thought I should expand a bit on the what and why. Spoilers towards the end (marked). I also went ahead and named it:

The Beskar Order

Rogue One: A Star Wars Story
Star Wars: Episode IV - A New Hope.
Star Wars: Episode V - The Empire Returns
Star Wars: Episode VI - Return of the Jedi
The Mandalorian (S01E01-04)
Star Wars: Episode VII - The Force Awakens
The Mandalorian (S01E05-08)

If you’ve enjoyed the above, you should pick between the following 2 next:

If you liked the core Skywalker Saga (Episodes IV-VI), and want to explore Anakin’s origins - Go and watch the prequel trilogy (Episode I, II, III).
If you liked Force Awakens, you should go finish the sequels (Star Wars Episode VIII, IX)

In either case, I want to leave this as a choice to the viewer. The prequel trilogy has a lot of flaws. The Machete order famously drops an entire film, and it wasn’t even trying to make room. Go to the prequels if you want to explore the lore. On the flip side, if you liked how Disney handled Episode VII, and want to see closing arcs for the major characters, try the sequels. I wouldn’t recommend interleaving them - it doesn’t get you much and makes it confusing.

If you’re still here after finishing both of these (that makes for a total of 10 films and 1 season of telly) - you ought to explore for yourselves. Here’s suggestions depending on what you’d like:

Clone Wars (TV Series, 7 seasons): for exploring the franchise at a less grand scale. There’s an Essential viewing order, which covers all the major arcs and best episodes.
The Mandalorian (Season 2, Oct 2020): To find out what happens to Baby Yoda
Star Wars: Rebels (TV Series, 4 Seasons): If you want to explore new characters and like something Firefly-esque.

There are boardgames, RPGs, and some really great books in the franchise as well. Pick what you’d like to explore.

Inspiration

The classic Machete Order which does a lot of great things, by skipping a film, preserving tension and plot-twists. Also of note are the various fan-edits, of which I’ve only ever tried The Phantom Edit.

Rationale

I tried to optimize for a few things:

Fun while watching the series. So good stuff comes first, paired films etc, and intentionally including The Mandalorian.
Easy stoppage. In case you don’t like the series, you should be able to stop midway, and still have seen the important/best bits.
Sticking to chronological order in the stuff I picked (as much as possible). Sticking to chronology makes it easier to consume.
Total time. I don’t want to prescribe a “complete-viewing-order”, but rather a “starting point”.

(1) is easy to optimize for. (2/3) results in things getting thrown around a bit, and (4) means I leave out stuff that you should pick for later.

Why not include __?

This is not meant to be an exhaustive order, and I was optimizing for total time. Important mentions:

Solo: Not really essential viewing.
Star Wars: The Clone Wars: The film is terrible (5.9 on IMDB), because it wasn’t meant to be one.
Star Wars: The Clone Wars TV Series: I wanted to stick to the main saga, and its just too damn long to recommend casually in any viewing order.
Star Wars: Holiday Special: I still can’t bring myself to finish it. It isn’t even canon any more (Life Day is)
Legends/Resistance: Again, not essential viewing.

FAQ

Who is this for?: Recommended for first-time viewers. If you are doing a rewatch, I recommend following The Beskar Machete order.
Have you watched everything Star Wars?: I can’t even claim to have watched all 12 films, because I couldn’t finish The Clone Wars (movie). I’m still watching Clone Wars (TV series).
Why Beskar?: I wanted something that would work well with Machete, for the hybrid order. It also makes a point about The Mandalorian ² belonging in the order.
I don’t have this much time!: I’ve tried to optimize for viewing time already. If you wanna trim further - you’ll be left with just the original trilogy (Episode IV, V, VI). Alternatively, just watch The Mandalorian - it stands very well by itself.
Did you backdate this post while publishing?: Yes. I wrote it just a few days after May 4th, and thought it would be nice.

More Rationale (SPOILERS AHEAD)

SPOILERS FROM HERE ONWARDS FOR ALL 12 FILMS. READ AT YOUR OWN PERIL

Why start with Rogue One?

Rogue One is a great film, and I love how well it segues into A New Hope. Watching them both back-to-back makes for a great experience. You have this ragged group that has laid down their lives for just a memory chip - and you get to see that bloom into an entire saga. Finishing the original trilogy from there makes sense. The Machete order strongly recommends against starting with Rogue One, but I’ve tried it and it works.

Why not stick with the Machete order as well?

The Machete order goes (IV, V, II, III, VI), deciding to leave out Episode I, and wedging Episodes II, III before you see Return of the Jedi. I was optimizing for time here a bit, and I had to leave the prequels as “for-later” in order to make space for the remaining. If you’re doing a rewatch, and aren’t short on time - you can totally follow it. This is what it morphs into:³

Beskar Machete Order

Rogue One: A Star Wars Story
Star Wars: Episode IV - A New Hope
Star Wars: Episode V - The Empire Returns
Star Wars: Episiode II - Attack of the Clones
Star Wars: Episode III - Revenge of the Sith
Star Wars: Episode VI - Return of the Jedi
The Mandalorian (S01E01-04)
Star Wars: Episode VII - The Force Awakens
The Mandalorian (S01E05-08)
Star Wars: Episode VIII - The Last Jedi
Star Wars: Episode IX - The Rise of Skywalker

Why add the Mandalorian at all?

Because frankly - it is both a piece of art, and the best entry into the Star Wars canon in a long time. It also fits into chronological order just after Return of the Jedi - you see how the New Republic has been incompetent, and the ashes of the Empire. You get to experience the power vaccuum in the galaxy, which hopefully makes sense before jumping into The Force Awakens and rise of the First Order.

Why jump to Episode VII (The Force Awakens) instead of finishing The Mandalorian?

We jump a bit ahead (before finishing The Mandalorian) to “A Force Awakens”, getting to just the start of Rey’s story. This is the only “chronology break” in the order, but has no side-effects⁴. The reason for the jump (as opposed to finishing The Mandalorian first) is to have a switch in pace. While I love The Mandalorian, I think pacing it out makes it better.

Why keep Episode VII (The Force Awakens) but not the other sequels?

The best and the worst thing about The Force Awakens is that it is very much “Star Wars”. It doesn’t take any risks, sticks to the tropes, and more importantly - it closes mostly as a self-contained film. Yes, there are a few plot-hooks (Rey’s parentage, Luke, Finn’s coma), but given how badly they are resolved in the following films - it seems Disney didn’t have any better idea to the answers than the viewers. It also gives you a “tasting experience” of the sequels. The sequels have always been polarizing, and watching it gives you a better heading to make the choice b/w Prequels/Sequels later on in the order.

We close with The Mandalorian finale. The Mandalorian isn’t, strictly speaking, essential viewing. While there are hooks, it doesn’t really change anything of consequence to the main saga (at least not in Season 1). But frankly, it is so well made - you deserve to enjoy it. Just look at the trailer:

If you have feedback, send me a tweet. If you’re reading this in the future, note that this was written in May 2020 and could not include media yet to be published.

Happy Star Wars Day! ↩
Beskar is the Star Wars universe’s Vibranium, and features majorly in The Mandalorian as a minor plot device. ↩
The arguments against starting with Rogue One don’t even apply to rewatches, so we ignore the Machete Guideline to keep it to the end. ↩
In other words, watching The Force Awakens can’t alter the experience of watching the last few episodes of The Mandalorian season 1. ↩

Anzacs

infrequent oscillations — Sun, 03 May 2020 19:26:21 GMT

Cooking experiments with ingredient substitutions.

Friday roundup for 2020-05-01

infrequent oscillations — Fri, 01 May 2020 16:25:59 GMT

Various things, links, and excitement.

Writing a SQL database from scratch in Go: 3. indexes

Notes on software development — Fri, 01 May 2020 03:00:00 GMT

Previously in database basics: <! forgive me, for I have sinned >
1. SELECT, INSERT, CREATE and a REPL
2. binary expressions and WHERE filters

Next in database basics:
4. a database/sql driver

In this post, we extend gosql to support indexes. We focus on the addition of PRIMARY KEY constraints on table creation and some easy optimizations during SELECT statements.

$ go run cmd/main.go
Welcome to gosql.
# CREATE TABLE users (id INT PRIMARY KEY, name TEXT, age INT);
ok
# \d users
Table "users"
Column |  Type   | Nullable
---------+---------+-----------
id     | integer | not null
name   | text    |
age    | integer |
Indexes:
        "users_pkey" PRIMARY KEY, rbtree ("id")

This post will broadly be a discussion of this commit.

What is an index?

An index is a mapping of a value to a row in a table. The value is often a column, but it can be many kinds of expressions. Databases typically store indexes in tree structures that provide O(log(n)) lookup time. When SELECTing and filtering on a column that is indexed, a database can greatly improve lookup time by filtering first on this index. Without an index, a database must do a linear scan for matching rows. Though sometimes if a condition is broad enough, even with an index, a database may still end up doing a linear scan.

While it may make sense initially to map a value to a row using a hash table for constant lookup times, hash tables don't provide ordering. So this would prevent an index from being applicable on anything but equality checks. For example, SELECT x FROM y WHERE x > 2 couldn't use a hash index on x.

Indexes in many SQL databases default to a B-Tree, which offers efficient ordering of elements. These indexes are thus not constant-time lookups even if filtering on a unique column for a single item. Some databases, like PostgreSQL, allow you to use a hash-based index instead of a tree. Here the previously listed restrictions apply (i.e. only equality checks will use the index).

Upgrading gosql

We proceed as follows:

Upgrade table creation to support specifying a primary key
- Pick a tree data structure for the index, adding it to the table
Upgrade INSERTs to let any indexes on the table process the new row
Upgrade SELECTs to make use of any indexes, if possible

Upgrading table creation

To allow the specification of a single column as the primary key when creating a table, we have to first modify the lexer and parser.

Lexing/parsing

Since we've covered this process a few times already suffice it so say we make the following key additions:

In-memory backend

Next we move on to handling a primary key during table creation.

Since there are many existing papers and blogs on implementing tree data structures, we will import an open-source implementation. And while most databases use a B-Tree, the most important properties of the tree for our purposes are 1) efficient ordering and 2) optionally duplicate keys. We go with a Red-Black Tree, GoLLRB.

The full definition of an index now includes:

A name
An expression (at first we only support this being an identifier referring to a column)
A unique flag
A type name (it will just be rbtree for now)
A primary key flag (so we know to apply null checks among other things)
And the actual tree itself

type index struct {
    name       string
    exp        expression
    unique     bool
    primaryKey bool
    tree       *llrb.LLRB
    typ        string
}

When we create a table, we add an index if one of the columns is a primary key. We call out to a new public method, CreateIndex, that will handle actually setting things up.

func (mb *MemoryBackend) CreateTable(crt *CreateTableStatement) error {
    if _, ok := mb.tables[crt.name.value]; ok {
        return ErrTableAlreadyExists
    }

    t := createTable()
    t.name = crt.name.value
    mb.tables[t.name] = t
    if crt.cols == nil {
        return nil
    }

    var primaryKey *expression = nil
    for _, col := range *crt.cols {
        t.columns = append(t.columns, col.name.value)

        var dt ColumnType
        switch col.datatype.value {
        case "int":
            dt = IntType
        case "text":
            dt = TextType
        case "boolean":
            dt = BoolType
        default:
            delete(mb.tables, t.name)
            return ErrInvalidDatatype
        }

        if col.primaryKey {
            if primaryKey != nil {
                delete(mb.tables, t.name)
                return ErrPrimaryKeyAlreadyExists
            }

            primaryKey = &expression{
                literal: &col.name,
                kind:    literalKind,
            }
        }

        t.columnTypes = append(t.columnTypes, dt)
    }

    if primaryKey != nil {
        err := mb.CreateIndex(&CreateIndexStatement{
            table:      crt.name,
            name:       token{value: t.name + "_pkey"},
            unique:     true,
            primaryKey: true,
            exp:        *primaryKey,
        })
        if err != nil {
            delete(mb.tables, t.name)
            return err
        }
    }

    return nil
}

Implementing CreateIndex is just a matter of adding a new index to the table.

func (mb *MemoryBackend) CreateIndex(ci *CreateIndexStatement) error {
    table, ok := mb.tables[ci.table.value]
    if !ok {
        return ErrTableDoesNotExist
    }

    for _, index := range table.indexes {
        if index.name == ci.name.value {
            return ErrIndexAlreadyExists
        }
    }

    index := &index{
        exp:        ci.exp,
        unique:     ci.unique,
        primaryKey: ci.primaryKey,
        name:       ci.name.value,
        tree:       llrb.New(),
        typ:        "rbtree",
    }
    table.indexes = append(table.indexes, index)
    return nil
}

And that's it for creation of tables and indexes! Table creation is also the last time we need to make changes to the gosql frontend. The rest of the changes simply wrap existing insertion and selection.

Upgrading INSERT

When a row is inserted into a table, each index on that table needs to process the row so it can add value-to-row mappings to the index.

In the project code, you'll notice logic in CreateIndex to also go back over all existing rows to add them to the new index. This post omits further discussing the case where an index is created after a table is created. After reading this post, that case should be easy to follow.

Adding a row to an index is a matter of evaluting the index expression against that row and storing the resulting value in the tree. Along with the value, we store the integer index of the row in the table.

If the index is required to be unique, we first check that the value does not yet exist.

func (i *index) addRow(t *table, rowIndex uint) error {
    indexValue, _, _, err := t.evaluateCell(rowIndex, i.exp)
    if err != nil {
        return err
    }

    if indexValue == nil {
        return ErrViolatesNotNullConstraint
    }

    if i.unique && i.tree.Has(treeItem{value: indexValue}) {
        return ErrViolatesUniqueConstraint
    }

    i.tree.InsertNoReplace(treeItem{
        value: indexValue,
        index: rowIndex,
    })
    return nil
}

And that's it for insertion!

Upgrading SELECT

Until now, the logic for selecting rows from a table is to pick the table and iterate over all rows. If the row does not match the WHERE filter, we pass the row.

If the table has an index and we are using the index in a recognized pattern in the WHERE AST (more on that later), we can pre-filter the table based on the index before iterating over each row. We can do this for each index and for each time a recognized pattern shows up.

This process is called query planning. We build a simplified version of what you may see in SQL databases, specifically focusing on index usage since we don't yet support JOINs. For further reading, SQLite has an excellent document on their query planner for index usage.

func (mb *MemoryBackend) Select(slct *SelectStatement) (*Results, error) {
    t := createTable()

    if slct.from != nil {
        var ok bool
        t, ok = mb.tables[slct.from.value]
        if !ok {
            return nil, ErrTableDoesNotExist
        }
    }

    if slct.item == nil || len(*slct.item) == 0 {
        return &Results{}, nil
    }

    results := [][]Cell{}
    columns := []ResultColumn{}

    if slct.from == nil {
        t = createTable()
        t.rows = [][]memoryCell{{}}
    }

    for _, iAndE := range t.getApplicableIndexes(slct.where) {
        index := iAndE.i
        exp := iAndE.e
        t = index.newTableFromSubset(t, exp)
    }

    for i := range t.rows {
        result := []Cell{}
        isFirstRow := len(results) == 0

        if slct.where != nil {
            val, _, _, err := t.evaluateCell(uint(i), *slct.where)
            if err != nil {
                return nil, err
            }

            if !*val.AsBool() {
                continue
            }
        }

        for _, col := range finalItems {
            value, columnName, columnType, err := t.evaluateCell(uint(i), *col.exp)
            if err != nil {
                return nil, err
            }

            if isFirstRow {
                columns = append(columns, ResultColumn{
                    Type: columnType,
                    Name: columnName,
                })
            }

            result = append(result, value)
        }

        results = append(results, result)
    }

    return &Results{
        Columns: columns,
        Rows:    results,
    }, nil
}

It's very simple and easy to miss, here is the change called out:

    for _, iAndE := range t.getApplicableIndexes(slct.where) {
        index := iAndE.i
        exp := iAndE.e
        t = index.newTableFromSubset(t, exp)
    }

getApplicableIndexes

There are probably a few very simple patterns we could look for, but for now we look for boolean expressions joined by AND that contain an index expression.

func (t *table) getApplicableIndexes(where *expression) []indexAndExpression {
    var linearizeExpressions func(where *expression, exps []expression) []expression
    linearizeExpressions = func(where *expression, exps []expression) []expression {
        if where == nil || where.kind != binaryKind {
            return exps
        }

        if where.binary.op.value == string(orKeyword) {
            return exps
        }

        if where.binary.op.value == string(andKeyword) {
            exps := linearizeExpressions(&where.binary.a, exps)
            return linearizeExpressions(&where.binary.b, exps)
        }

        return append(exps, *where)
    }

    exps := linearizeExpressions(where, []expression{})

    iAndE := []indexAndExpression{}
    for _, exp := range exps {
        for _, index := range t.indexes {
            if index.applicableValue(exp) != nil {
                iAndE = append(iAndE, indexAndExpression{
                    i: index,
                    e: exp,
                })
            }
        }
    }

    return iAndE
}

More specifically though, within binary operations we only support matching on an index if the following three conditions are met:

the operator is one of =, , >, <, >=, or <=


one of the operands is an identifier literal that matches the index's exp value
the other operand is a literal value



  This is a simpler, stricter matching of an index than PostgreSQL
  where you can index expressions more generally, not just identifer
  literals.
func (i *index) applicableValue(exp expression) *expression {
    if exp.kind != binaryKind {
        return nil
    }

    be := exp.binary
    // Find the column and the value in the boolean expression
    columnExp := be.a
    valueExp := be.b
    if columnExp.generateCode() != i.exp.generateCode() {
        columnExp = be.b
        valueExp = be.a
    }

    // Neither side is applicable, return nil
    if columnExp.generateCode() != i.exp.generateCode() {
        return nil
    }

    supportedChecks := []symbol{eqSymbol, neqSymbol, gtSymbol, gteSymbol, ltSymbol, lteSymbol}
    supported := false
    for _, sym := range supportedChecks {
        if string(sym) == be.op.value {
            supported = true
            break
        }
    }
    if !supported {
        return nil
    }

    if valueExp.kind != literalKind {
        fmt.Println("Only index checks on literals supported")
        return nil
    }

    return &valueExp
}

And that's it for finding applicable indexes.
newTableFromSubset
The last remaining piece is to go from a boolean expression in
a WHERE clause (where an index is applicable) to a subset
of rows in a table.
Since we are only working with patterns of the type
indexed-column OP literal-value, we grab the literal
using the previous applicableValue helper. Then we
look up that literal value in the index and return a new table with
every row in the index that meets the condition of the operator for the
literal value.
func (i *index) newTableFromSubset(t *table, exp expression) *table {
    valueExp := i.applicableValue(exp)
    if valueExp == nil {
        return t
    }

    value, _, _, err := createTable().evaluateCell(0, *valueExp)
    if err != nil {
        fmt.Println(err)
        return t
    }

    tiValue := treeItem{value: value}

    indexes := []uint{}
    switch symbol(exp.binary.op.value) {
    case eqSymbol:
        i.tree.AscendGreaterOrEqual(tiValue, func(i llrb.Item) bool {
            ti := i.(treeItem)

            if !bytes.Equal(ti.value, value) {
                return false
            }

            indexes = append(indexes, ti.index)
            return true
        })
    case neqSymbol:
        i.tree.AscendGreaterOrEqual(llrb.Inf(-1), func(i llrb.Item) bool {
            ti := i.(treeItem)
            if bytes.Equal(ti.value, value) {
                indexes = append(indexes, ti.index)
            }

            return true
        })
    case ltSymbol:
        i.tree.DescendLessOrEqual(tiValue, func(i llrb.Item) bool {
            ti := i.(treeItem)
            if bytes.Compare(ti.value, value) < 0 {
                indexes = append(indexes, ti.index)
            }

            return true
        })
    case lteSymbol:
        i.tree.DescendLessOrEqual(tiValue, func(i llrb.Item) bool {
            ti := i.(treeItem)
            if bytes.Compare(ti.value, value) <= 0 {
                indexes = append(indexes, ti.index)
            }

            return true
        })
    case gtSymbol:
        i.tree.AscendGreaterOrEqual(tiValue, func(i llrb.Item) bool {
            ti := i.(treeItem)
            if bytes.Compare(ti.value, value) > 0 {
                indexes = append(indexes, ti.index)
            }

            return true
        })
    case gteSymbol:
        i.tree.AscendGreaterOrEqual(tiValue, func(i llrb.Item) bool {
            ti := i.(treeItem)
            if bytes.Compare(ti.value, value) >= 0 {
                indexes = append(indexes, ti.index)
            }

            return true
        })
    }

    newT := createTable()
    newT.columns = t.columns
    newT.columnTypes = t.columnTypes
    newT.indexes = t.indexes
    newT.rows = [][]memoryCell{}

    for _, index := range indexes {
        newT.rows = append(newT.rows, t.rows[index])
    }

    return newT
}

As you can see, an index may not necessarily improve on a linear
search in some conditions. Imagine a table of 1 million rows indexed
on an autoincrementing column. Imagine filtering on col >
10. The index may be able to eliminate 10 items but still
return a pre-filtered table of around 1 million rows that must
be passed through the WHERE filter.
Additionally since we process each boolean expression one at a time,
we can't take advantage of knowledge that might seem obvious to a
human for two boolean expressions that together bound a range. For
example in x > 10 AND x < 20 we can see that only
integers from 11 to 19 are applicable. But the current logic would go
through each expression separately and find all rows that match either
before the final linear search through all pre-filtered rows would
eliminate the bulk.

  Thankfully real databases have decades of optimizations. But even
  then it can be difficult to know what index usages are being
  optimized without reading documentation, benchmarking, using
  EXPLAIN ANALYSE, or reading the source.
But that's it for changes needed to support basic indexes end-to-end!
Trialing an index
Since the addition of indexes is so seamless, it is difficult to tell
without trial that the index is effective. So we write a simple
program that inserts N rows with and without an index. Finally it will
query for the first and last items inserted. We show time and
memory used during both insertion and selection.
package main

import (
    "fmt"
    "os"
    "runtime"
    "strconv"
    "time"

    "github.com/eatonphil/gosql"
)

var inserts = 0
var lastId = 0
var firstId = 0

func doInsert(mb gosql.Backend) {
    parser := gosql.Parser{}
    for i := 0; i < inserts; i++ {
        lastId = i
        if i == 0 {
            firstId = lastId
        }
        ast, err := parser.Parse(fmt.Sprintf("INSERT INTO users VALUES (%d)", lastId))
        if err != nil {
            panic(err)
        }

        err = mb.Insert(ast.Statements[0].InsertStatement)
        if err != nil {
            panic(err)
        }
    }
}

func doSelect(mb gosql.Backend) {
    parser := gosql.Parser{}
    ast, err := parser.Parse(fmt.Sprintf("SELECT id FROM users WHERE id = %d", lastId))
    if err != nil {
        panic(err)
    }

    r, err := mb.Select(ast.Statements[0].SelectStatement)
    if err != nil {
        panic(err)
    }

    if len(r.Rows) != 1 {
        panic("Expected 1 row")
    }

    if int(*r.Rows[0][1].AsInt()) != inserts-1 {
        panic(fmt.Sprintf("Bad row, got: %d", r.Rows[0][1].AsInt()))
    }

    ast, err = parser.Parse(fmt.Sprintf("SELECT id FROM users WHERE id = %d", firstId))
    if err != nil {
        panic(err)
    }

    r, err = mb.Select(ast.Statements[0].SelectStatement)
    if err != nil {
        panic(err)
    }

    if len(r.Rows) != 1 {
        panic("Expected 1 row")
    }

    if int(*r.Rows[0][1].AsInt()) != 0 {
        panic(fmt.Sprintf("Bad row, got: %d", r.Rows[0][1].AsInt()))
    }
}

func perf(name string, b gosql.Backend, cb func(b gosql.Backend)) {
    start := time.Now()
    fmt.Println("Starting", name)
    cb(b)
    fmt.Printf("Finished %s: %f seconds\n", name, time.Since(start).Seconds())

    var m runtime.MemStats
    runtime.ReadMemStats(&m)
    fmt.Printf("Alloc = %d MiB\n\n", m.Alloc/1024/1024)
}

func main() {
    mb := gosql.NewMemoryBackend()

    index := false
    for i, arg := range os.Args {
        if arg == "--with-index" {
            index = true
        }

        if arg == "--inserts" {
            inserts, _ = strconv.Atoi(os.Args[i+1])
        }
    }

    primaryKey := ""
    if index {
        primaryKey = " PRIMARY KEY"
    }

    parser := gosql.Parser{}
    ast, err := parser.Parse(fmt.Sprintf("CREATE TABLE users (id INT%s)", primaryKey))
    if err != nil {
        panic(err)
    }

    err = mb.CreateTable(ast.Statements[0].CreateTableStatement)
    if err != nil {
        panic(err)
    }

    indexingString := " with indexing enabled"
    if !index {
        indexingString = ""
    }
    fmt.Printf("Inserting %d rows%s\n", inserts, indexingString)

    perf("INSERT", mb, doInsert)

    perf("SELECT", mb, doSelect)
}

Build and run once without an index:
$ go build cmd/indextest/main.go
./main --inserts 1000000
Inserting 1000000 rows
Starting INSERT
Finished INSERT: 76.175133 seconds
Alloc = 239 MiB

Starting SELECT
Finished SELECT: 1.301556 seconds
Alloc = 238 MiB

And run again with an index:
./main --inserts 1000000 --with-index
Inserting 1000000 rows with indexing enabled
Starting INSERT
Finished INSERT: 89.108121 seconds
Alloc = 341 MiB

Starting SELECT
Finished SELECT: 0.000137 seconds
Alloc = 341 MiB

The basic tradeoff that you can see is that for more memory and longer
insertion times, you get a significantly faster lookup.
Very excited to share the latest database basics post on implementing indexes in gosql.https://t.co/QHfjCe1XsC
— Phil Eaton (@phil_eaton) May 1, 2020



The Day I Became A Software Engineer
swyx's site RSS Feed — Mon, 27 Apr 2020 19:24:19 GMT
A single mindset shift that changed my entire attitude to software.


Building intuition for p-values and statistical significance
Bytepawn - Marton Trencseni — Sat, 25 Apr 2020 01:00:00 GMT
This is the transcript of a talk I did on experimentation and A/B testing to give the audience an intuitive understanding of p-values and statistical significance.

 


Friday roundup for 2020-04-24
infrequent oscillations — Fri, 24 Apr 2020 13:27:44 GMT
Various things, links, and excitement.


It's Time to Start Paying For Things Again
Tilting at Windmills — Fri, 24 Apr 2020 03:00:00 GMT
Today I'm launching my latest endeavor:
No Nonsense Recipes. With nearly 19,000 recipes
uploaded so far, the site is live and open for…


Managing Rust bloat with Github Actions
Tom Forbes — Fri, 24 Apr 2020 00:42:54 GMT
Cargo and crates.io are an amazing part of the Rust ecosystem and one of the things that makes Rust so pleasant to work with. However, it’s quite easy for your cargo dependencies to pile up without you noticing. For example you might blindly add the (rather awesome) reqwest crate, not realizing that...


Developer's Guide to Tech Strategy
swyx's site RSS Feed — Wed, 22 Apr 2020 20:09:50 GMT
This is a very high level overview of tech strategy; that is, the business of software rather than the art and science of creating software itself.


searchcode Rebuilt with Go
Ben E. C. Boyter — Wed, 22 Apr 2020 03:00:00 GMT
So as of about 12 hours a new version of https://searchcode.com/ has been rolled out. It marks the biggest update to the site in over 4 years since I moved over to version 3 which you can read about https://boyter.org/2016/07/searchcode-com-architecture-migration-3-0/
The reason for it was that searchcode has always been something I kept running as a showpiece. It’s something that I can point to publicly and say “I did that”. The idea being to ensure I remain employable. Recently though I was becoming ashamed of its look, performance and instability. I briefly considered selling it and had some interested parties although I think with COVID-19 rampaging that feel by the wayside.


Spokesperson certification
CRAIG KERSTIENS — Tue, 21 Apr 2020 23:55:56 GMT
One of my most fascinating work experiences was going through the spokesperson certification process at a large tech co. This isn’t some rubber stamp virtual training to not use profanity on stage type training. This is the training they would give to any executive before you were greenlit to talk to press. When I say press I mean Techcrunch, but also Bloomberg, or Jim Cramer, or any major big brand news outlet.
As a product manager over a specific product line I knew my product well. Put me in front of an unhappy customer and I could lay out our roadmap, listen to their questions, take product input, and get them to a happy place. But this wasn’t about my product (only). A person with the spokesperson stamp could be asked any question about an entirely other area fo the company. You had to know every recent product launch, all the key metrics, know where traps may lie, and you had to land the core company messages in addition to the ones you cared about. To study you received about 100 pages of a powerpoint presentation that had key releases from each product, key numbers, customer stories.
The certification itself was an interview. They flew in a former news reporter. You walked into a conference, the lights were off, except for a bright light focused on the seat you’d sit in with a camera rolling. It felt more like an intense interogation room than a big co tech conference room. To ensure light didn’t get in and no one stopped walking by the room they put up black paper to completely black out the room.
Oh, and the worst part of the process… I was told my our marketing person I could wear my hat… as I always do during the interview. (I was probably the only person not in a full suit they saw the entire day. And probably the only person that walked in with a ballcap on ever.) Well she said yes of course, turns out they couldn’t see my face under the lights, it was just a shadow so I had to take it off. From the outset I’d been tricked, but I digress.
The questions would start with a basic, tell me about yourself and your background. Can you tell whats new and exciting about product x. Then over time it would delve into the other product areas. I told some of the canned stories and some personal ones. I learned on the personal ones, it’s up to me but they coach their executives to not said spouses or childrens names, it can only get them in trouble. For each product line you were supposed to hit major 2 news and announcements, 2 customers, and 2 key stats (i.e. we crossed 1 billion mentions of my name).
But it wasn’t just softballs. There were traps. You were asked about an executive that recently left, and if the product line was okay or was it a sign of bad things to come during earnings. It’s fine though I reassured them they’ve made some great contributions to the company, and wanted to spend some time with their family and giving back to their local community. Of course internal speculation was they were cashed out and interested in running for public office.
As the camera turned off a few of the stories I’d told about customers and products the PR team wanted to dig deeper on. Remarking “That’s amazing, we power that I had no idea.” and “That’s awesome they’re able to do that at scale thanks to us.”
In the end I got a stamp of approval, I was cleared to talk to folks. The funny part was the comments from the PR team afterwards. They liked me, I seemed relatable, I nailed all the numbers. I wasn’t like any of their other spokespeople and they well weren’t quite sure what to do with that. I was used for some very particular media folks in the future that seemed to not want a cookie cutter. I’m good with this, hopefully it helped the company.
In the end it was fascinating experience. The ability to bridge, condense a lot of information (relatable story, customer brand validation, stats, and something quotable) into a single answer all from a question that was meant to be a trap for a juicy story has been easily one of the top work experiences I’ve encountered.


Cloud Distros
swyx's site RSS Feed — Tue, 21 Apr 2020 23:16:58 GMT
The next step in the evolution of the Cloud is specialized distros.


Friday roundup
infrequent oscillations — Fri, 17 Apr 2020 18:02:50 GMT
Various things, links, and excitement.


Validation checks for A/B tests
Bytepawn - Marton Trencseni — Thu, 16 Apr 2020 01:00:00 GMT
A/B tests go wrong all the time, even in sophisticated product teams. As this article shows, for a range of problems we can run automated validation checks to catch problems early, before they have too bad of an effect on customers or the business. These validation checks compare various statistical properties of the funnels A and B to catch likely problems. Large technology companies are running such validation checks automatically and continuously for their online experiments.

 


OSI Layers for Coding Careers
swyx's site RSS Feed — Wed, 15 Apr 2020 21:08:26 GMT
Let's think about the value chain of humans that code and how we interact.


Generating Worlds: Faking Realism
Ben Overmyer's Site — Tue, 14 Apr 2020 03:00:00 GMT

Note: this was a draft I never completed. However, it's useful enough as notes that I wanted to keep it around, so here it is.

Introduction: why I'm interested in this
From the ground up: beginning with the ground
Why I don't want to model plate tectonics

it's hard
it's computationally expensive (is this true?)
it's not necessary

Creating a heightmap

Diamond-Square vs Simplex noise
Start with Simplex noise (point out the difference between Simplex and Perlin noise)
Tweaking the noise to create more land-like patterns
Reworking this by placing areas that should be land, then applying the noise to make it land-like
Creating areas of mountains, rolling hills, and flat land
Deciding on sea level
Classifying cells based on altitude and whether it's underwater or not

Modelling air flow (reference Here Dragons Abound)

Hadley Cells and related concepts
Wind speed and direction

Generating climates based on air flow, humidity, temperature, and altitude

Precipitation as a function of air flow
Rain shadows

Generating biomes based on the results
Different types of biome models (e.g., Whittaker)

I love diversity of classification, so I choose the most complex model

Placing rivers and simulating erosion over time

River joins, why rivers never fork, and when that rule is broken

Below the surface: generating soil composition

Soil is complicated, so let's only go as far as we need to model interesting areas

Generating mineral resources

Meteor strikes are not part of the plan
Veins of ore and their size, composition, and placement
two primary methods of vein formation - open space filling and crack-seal growth
I'm not modelling plate tectonics, so modelling realistic veining is not happening
Instead, mimicking the frequency and shape of veins by random line generation and asymmetrical extrusion

Populating regions with flora and fauna

Predators and prey
How vegetation is chosen



How to Market Yourself
swyx's site RSS Feed — Mon, 13 Apr 2020 22:17:31 GMT
Assemble your Personal Brand, your Domain, and your Coding Skills/Business Value, then Market Yourself in Public + at Work.


Writing a SQL database from scratch in Go: 2. binary expressions and WHERE filters
Notes on software development — Sun, 12 Apr 2020 03:00:00 GMT

  Previously in database basics:
  <! forgive me, for I have sinned >
  

  1. SELECT, INSERT, CREATE and a REPL
  

  

  Next in database basics:
  

  3. indexes
  

  4. a database/sql driver
In this post, we'll extend gosql
to support binary expressions and very simple filtering on SELECT
results via WHERE. We'll introduce a general mechanism for
interpreting an expression on a row in a table. The expression may be
an identifier (where the result is the value of the cell corresponding
to that column in the row), a numeric literal, a combination via a
binary expression, etc.
The following interactions will be possible:
# CREATE TABLE users (name TEXT, age INT);
ok
#  INSERT INTO users VALUES ('Stephen', 16);
ok
# SELECT name, age FROM users;
name   | age
----------+------
Stephen |  16
(1 result)
ok
# INSERT INTO users VALUES ('Adrienne', 23);
ok
# SELECT age + 2, name FROM users WHERE age = 23;
age |   name
------+-----------
25 | Adrienne
(1 result)
ok
# SELECT name FROM users;
name
------------
Stephen
Adrienne
(2 results)
ok

The changes we'll make in this post are roughly a walk through of
this
commit.
Boilerplate updates
There are a few updates to pick up that I won't go into in this
post. Grab the following files from the main repo:

lexer.go
The big change here is to use the same keyword matching algorithm
for symbols. This allows us to support symbols that are longer
than one character.
This file also now includes the following keywords and symbols:
and, or, true,
false, =, ,
||, and +.



cmd/main.go
This file now uses a third-party table-rendering
library instead of the
hacky, handwritten original one.
This also uses a third-party readline
implementation so you get
history and useful cursor movement in the REPL.



Parsing boilerplate
We'll redefine three helper functions in parser.go before
going further:
parseToken, parseTokenKind, and
helpMessage.
The parseToken helper will consume a token if it matches
the one provided as an argument (ignoring location).
func parseToken(tokens []*token, initialCursor uint, t token) (*token, uint, bool) {
    cursor := initialCursor

    if cursor >= uint(len(tokens)) {
        return nil, initialCursor, false
    }

    if p := tokens[cursor]; t.equals(p) {
        return p, cursor + 1, true
    }

    return nil, initialCursor, false
}

The parseTokenKind helper will consume a token if it is
the same kind as an argument provided.
func parseTokenKind(tokens []*token, initialCursor uint, kind tokenKind) (*token, uint, bool) {
    cursor := initialCursor

    if cursor >= uint(len(tokens)) {
        return nil, initialCursor, false
    }

    current := tokens[cursor]
    if current.kind == kind {
        return current, cursor + 1, true
    }

    return nil, initialCursor, false
}

And the helpMessage helper will give an indication of
where in a program something happened.
func helpMessage(tokens []*token, cursor uint, msg string) {
    var c *token
    if cursor+1 < uint(len(tokens)) {
        c = tokens[cursor+1]
    } else {
        c = tokens[cursor]
    }

    fmt.Printf("[%d,%d]: %s, near: %s\n", c.loc.line, c.loc.col, msg, c.value)
}

Parsing binary expressions
Next we'll extend the AST structure in ast.go to
support a "binary kind" of expression. The binary expression will have
two sub-expressions and an operator.
const (
    literalKind expressionKind
    binaryKind
)

type binaryExpression struct {
    a  expression
    b  expression
    op token
}

type expression struct {
    literal *token
    binary  *binaryExpression
    kind    expressionKind
}

We'll use Pratt parsing to handle operator precedence. There is an
excellent introduction to this technique
here.
If at the beginning of parsing we see a left parenthesis, we'll
consume it and parse an expression within it. Then we'll look for a
right parenthesis. Otherwise we'll look for a non-binary expression
first (e.g. symbol, number).
func parseExpression(tokens []*token, initialCursor uint, delimiters []token, minBp uint) (*expression, uint, bool) {
    cursor := initialCursor

    var exp *expression
    _, newCursor, ok := parseToken(tokens, cursor, tokenFromSymbol(leftParenSymbol))
    if ok {
        cursor = newCursor
        rightParenToken := tokenFromSymbol(rightParenSymbol)

        exp, cursor, ok = parseExpression(tokens, cursor, append(delimiters, rightParenToken), minBp)
        if !ok {
            helpMessage(tokens, cursor, "Expected expression after opening paren")
            return nil, initialCursor, false
        }

        _, cursor, ok = parseToken(tokens, cursor, rightParenToken)
        if !ok {
            helpMessage(tokens, cursor, "Expected closing paren")
            return nil, initialCursor, false
        }
    } else {
        exp, cursor, ok = parseLiteralExpression(tokens, cursor)
        if !ok {
            return nil, initialCursor, false
        }
    }

    ...

    return exp, cursor, true
}

Then we'll look for a binary operator (e.g. =,
and) or delimiter. If we find an operator and it of
lesser "binding power" than the current minimum (minBp
passed as an argument to the parse function with a default value of
0), we'll return the current expression.
    ...

    lastCursor := cursor
outer:
    for cursor < uint(len(tokens)) {
        for _, d := range delimiters {
            _, _, ok = parseToken(tokens, cursor, d)
            if ok {
                break outer
            }
        }

        binOps := []token{
            tokenFromKeyword(andKeyword),
            tokenFromKeyword(orKeyword),
            tokenFromSymbol(eqSymbol),
            tokenFromSymbol(neqSymbol),
            tokenFromSymbol(concatSymbol),
            tokenFromSymbol(plusSymbol),
        }

        var op *token = nil
        for _, bo := range binOps {
            var t *token
            t, cursor, ok = parseToken(tokens, cursor, bo)
            if ok {
                op = t
                break
            }
        }

        if op == nil {
            helpMessage(tokens, cursor, "Expected binary operator")
            return nil, initialCursor, false
        }

        bp := op.bindingPower()
        if bp < minBp {
            cursor = lastCursor
            break
        }

        ...
    }

    return exp, cursor, true

The bindingPower function on tokens can be defined for
now such that sum and concatenation have the highest binding power,
followed by equality operations, then boolean operators, and then
everything else at zero.
func (t token) bindingPower() uint {
    switch t.kind {
    case keywordKind:
        switch keyword(t.value) {
        case andKeyword:
            fallthrough
        case orKeyword:
            return 1
        }
    case symbolKind:
        switch symbol(t.value) {
        case eqSymbol:
            fallthrough
        case neqSymbol:
            fallthrough
        case concatSymbol:
            fallthrough
        case plusSymbol:
            return 3
        }
    }

    return 0
}

Back in parseExpression, if the new operator has greater
binding power we'll parse the next operand expression (a recursive
call, passing the binding power of the new operator as the new
minBp).
Upon completion, the current expression (the return value of the
recursive call) is set to a new binary expression containing the
previously current expression on the left and the just-parsed
expression on the right.
        ...

        b, newCursor, ok := parseExpression(tokens, cursor, delimiters, bp)
        if !ok {
            helpMessage(tokens, cursor, "Expected right operand")
            return nil, initialCursor, false
        }
        exp = &expression{
            binary: &binaryExpression{
                *exp,
                *b,
                *op,
            },
            kind: binaryKind,
        }
        cursor = newCursor
        lastCursor = cursor
    }

    return exp, cursor, true
}

All together:
func parseExpression(tokens []*token, initialCursor uint, delimiters []token, minBp uint) (*expression, uint, bool) {
    cursor := initialCursor

    var exp *expression
    _, newCursor, ok := parseToken(tokens, cursor, tokenFromSymbol(leftParenSymbol))
    if ok {
        cursor = newCursor
        rightParenToken := tokenFromSymbol(rightParenSymbol)

        exp, cursor, ok = parseExpression(tokens, cursor, append(delimiters, rightParenToken), minBp)
        if !ok {
            helpMessage(tokens, cursor, "Expected expression after opening paren")
            return nil, initialCursor, false
        }

        _, cursor, ok = parseToken(tokens, cursor, rightParenToken)
        if !ok {
            helpMessage(tokens, cursor, "Expected closing paren")
            return nil, initialCursor, false
        }
    } else {
        exp, cursor, ok = parseLiteralExpression(tokens, cursor)
        if !ok {
            return nil, initialCursor, false
        }
    }

    lastCursor := cursor
outer:
    for cursor < uint(len(tokens)) {
        for _, d := range delimiters {
            _, _, ok = parseToken(tokens, cursor, d)
            if ok {
                break outer
            }
        }

        binOps := []token{
            tokenFromKeyword(andKeyword),
            tokenFromKeyword(orKeyword),
            tokenFromSymbol(eqSymbol),
            tokenFromSymbol(neqSymbol),
            tokenFromSymbol(concatSymbol),
            tokenFromSymbol(plusSymbol),
        }

        var op *token = nil
        for _, bo := range binOps {
            var t *token
            t, cursor, ok = parseToken(tokens, cursor, bo)
            if ok {
                op = t
                break
            }
        }

        if op == nil {
            helpMessage(tokens, cursor, "Expected binary operator")
            return nil, initialCursor, false
        }

        bp := op.bindingPower()
        if bp < minBp {
            cursor = lastCursor
            break
        }

        b, newCursor, ok := parseExpression(tokens, cursor, delimiters, bp)
        if !ok {
            helpMessage(tokens, cursor, "Expected right operand")
            return nil, initialCursor, false
        }
        exp = &expression{
            binary: &binaryExpression{
                *exp,
                *b,
                *op,
            },
            kind: binaryKind,
        }
        cursor = newCursor
        lastCursor = cursor
    }

    return exp, cursor, true
}

Now that we have this general parse expression helper in place, we can
add support for parsing WHERE in SELECT
statements.
Parsing WHERE
This part's pretty easy. We modify the existing
parseSelectStatement to search for an optional
WHERE token followed by an expression.
func parseSelectStatement(tokens []*token, initialCursor uint, delimiter token) (*SelectStatement, uint, bool) {
    var ok bool
    cursor := initialCursor
    _, cursor, ok = parseToken(tokens, cursor, tokenFromKeyword(selectKeyword))
    if !ok {
        return nil, initialCursor, false
    }

    slct := SelectStatement{}

    fromToken := tokenFromKeyword(fromKeyword)
    item, newCursor, ok := parseSelectItem(tokens, cursor, []token{fromToken, delimiter})
    if !ok {
        return nil, initialCursor, false
    }

    slct.item = item
    cursor = newCursor

    whereToken := tokenFromKeyword(whereKeyword)
    delimiters := []token{delimiter, whereToken}

    _, cursor, ok = parseToken(tokens, cursor, fromToken)
    if ok {
        from, newCursor, ok := parseFromItem(tokens, cursor, delimiters)
        if !ok {
            helpMessage(tokens, cursor, "Expected FROM item")
            return nil, initialCursor, false
        }

        slct.from = from
        cursor = newCursor
    }

    _, cursor, ok = parseToken(tokens, cursor, whereToken)
    if ok {
        where, newCursor, ok := parseExpression(tokens, cursor, []token{delimiter}, 0)
        if !ok {
            helpMessage(tokens, cursor, "Expected WHERE conditionals")
            return nil, initialCursor, false
        }

        slct.where = where
        cursor = newCursor
    }

    return &slct, cursor, true
}

Now we're all done with parsing binary expressions and
WHERE filters! If in doubt, refer to
parser.go
in the project.
Re-thinking query execution
In the first post in this series, we didn't establish any standard way
for interpreting an expression in any kind of statement. In SQL
though, every expression is always run in the context of a row in a
table. We'll handle cases like SELECT 1 and INSERT INTO
users VALUES (1) by creating a table with a single empty row to act
as the context.
This requires a bit of re-architecting. So we'll rewrite the
memory.go implementation in this post from scratch.
We'll also stop panic-ing when things go wrong. Instead
we'll print a message. This allows the REPL loop to keep going.
Memory cells
Again the fundamental blocks of memory in the table will be an untyped
array of bytes. We'll provide conversion methods from this memory cell
into integers, strings, and boolean Go values.
type MemoryCell []byte

func (mc MemoryCell) AsInt() int32 {
    var i int32
    err := binary.Read(bytes.NewBuffer(mc), binary.BigEndian, &i)
    if err != nil {
        fmt.Printf("Corrupted data [%s]: %s\n", mc, err)
        return 0
    }

    return i
}

func (mc MemoryCell) AsText() string {
    return string(mc)
}

func (mc MemoryCell) AsBool() bool {
    return len(mc) != 0
}

func (mc MemoryCell) equals(b MemoryCell) bool {
    // Seems verbose but need to make sure if one is nil, the
    // comparison still fails quickly
    if mc == nil || b == nil {
        return mc == nil && b == nil
    }

    return bytes.Compare(mc, b) == 0
}

We'll also extend the Cell interface in
backend.go to support the new boolean type.
package gosql

type ColumnType uint

const (
    TextType ColumnType = iota
    IntType
    BoolType
)

type Cell interface {
    AsText() string
    AsInt() int32
    AsBool() bool
}

...

Finally, we need a way for mapping a Go value into a memory
cell.
func literalToMemoryCell(t *token) MemoryCell {
    if t.kind == numericKind {
        buf := new(bytes.Buffer)
        i, err := strconv.Atoi(t.value)
        if err != nil {
            fmt.Printf("Corrupted data [%s]: %s\n", t.value, err)
            return MemoryCell(nil)
        }

        // TODO: handle bigint
        err = binary.Write(buf, binary.BigEndian, int32(i))
        if err != nil {
            fmt.Printf("Corrupted data [%s]: %s\n", string(buf.Bytes()), err)
            return MemoryCell(nil)
        }
        return MemoryCell(buf.Bytes())
    }

    if t.kind == stringKind {
        return MemoryCell(t.value)
    }

    if t.kind == boolKind {
        if t.value == "true" {
            return MemoryCell([]byte{1})
        } else {
            return MemoryCell(nil)
        }
    }

    return nil
}

And we'll provide global true and false
values:
var (
    trueToken  = token{kind: boolKind, value: "true"}
    falseToken = token{kind: boolKind, value: "false"}

    trueMemoryCell  = literalToMemoryCell(&trueToken)
    falseMemoryCell = literalToMemoryCell(&falseToken)
)

Tables
A table has a list of rows (an array of memory cells) and a list of
column names and types.
type table struct {
    columns     []string
    columnTypes []ColumnType
    rows        [][]MemoryCell
}

Finally we'll add a series of methods on table that,
given a row index, interprets an expression AST against that row in
the table.
Interpreting literals
First we'll implement evaluateLiteralCell that will look
up an identifier or return the value of integers, strings, and
booleans.
func (t *table) evaluateLiteralCell(rowIndex uint, exp expression) (MemoryCell, string, ColumnType, error) {
    if exp.kind != literalKind {
        return nil, "", 0, ErrInvalidCell
    }

    lit := exp.literal
    if lit.kind == identifierKind {
        for i, tableCol := range t.columns {
            if tableCol == lit.value {
                return t.rows[rowIndex][i], tableCol, t.columnTypes[i], nil
            }
        }

        return nil, "", 0, ErrColumnDoesNotExist
    }

    columnType := IntType
    if lit.kind == stringKind {
        columnType = TextType
    } else if lit.kind == boolKind {
        columnType = BoolType
    }

    return literalToMemoryCell(lit), "?column?", columnType, nil
}

Interpreting binary expressions
Now we can implement evaluateBinaryCell that will
evaluate it's two sub-expressions and combine them together according
to the operator. The SQL operators we have defined so far do no
coercion. So we'll fail immediately if the two sides of the operation
are not of the same type. Additionally, the concatenation and addition
operators require that their arguments are strings and numbers,
respectively.
func (t *table) evaluateBinaryCell(rowIndex uint, exp expression) (MemoryCell, string, ColumnType, error) {
    if exp.kind != binaryKind {
        return nil, "", 0, ErrInvalidCell
    }

    bexp := exp.binary

    l, _, lt, err := t.evaluateCell(rowIndex, bexp.a)
    if err != nil {
        return nil, "", 0, err
    }

    r, _, rt, err := t.evaluateCell(rowIndex, bexp.b)
    if err != nil {
        return nil, "", 0, err
    }

    switch bexp.op.kind {
    case symbolKind:
        switch symbol(bexp.op.value) {
        case eqSymbol:
            eq := l.equals(r)
            if lt == TextType && rt == TextType && eq {
                return trueMemoryCell, "?column?", BoolType, nil
            }

            if lt == IntType && rt == IntType && eq {
                return trueMemoryCell, "?column?", BoolType, nil
            }

            if lt == BoolType && rt == BoolType && eq {
                return trueMemoryCell, "?column?", BoolType, nil
            }

            return falseMemoryCell, "?column?", BoolType, nil
        case neqSymbol:
            if lt != rt || !l.equals(r) {
                return trueMemoryCell, "?column?", BoolType, nil
            }

            return falseMemoryCell, "?column?", BoolType, nil
        case concatSymbol:
            if lt != TextType || rt != TextType {
                return nil, "", 0, ErrInvalidOperands
            }

            return literalToMemoryCell(&token{kind: stringKind, value: l.AsText() + r.AsText()}), "?column?", TextType, nil
        case plusSymbol:
            if lt != IntType || rt != IntType {
                return nil, "", 0, ErrInvalidOperands
            }

            iValue := int(l.AsInt() + r.AsInt())
            return literalToMemoryCell(&token{kind: numericKind, value: strconv.Itoa(iValue)}), "?column?", IntType, nil
        default:
            // TODO
            break
        }
    case keywordKind:
        switch keyword(bexp.op.value) {
        case andKeyword:
            if lt != BoolType || rt != BoolType {
                return nil, "", 0, ErrInvalidOperands
            }

            res := falseMemoryCell
            if l.AsBool() && r.AsBool() {
                res = trueMemoryCell
            }

            return res, "?column?", BoolType, nil
        case orKeyword:
            if lt != BoolType || rt != BoolType {
                return nil, "", 0, ErrInvalidOperands
            }

            res := falseMemoryCell
            if l.AsBool() || r.AsBool() {
                res = trueMemoryCell
            }

            return res, "?column?", BoolType, nil
        default:
            // TODO
            break
        }
    }

    return nil, "", 0, ErrInvalidCell
}

Then we'll provide a generic evaluateCell method to wrap
these two correctly:
func (t *table) evaluateCell(rowIndex uint, exp expression) (MemoryCell, string, ColumnType, error) {
    switch exp.kind {
        case literalKind:
            return t.evaluateLiteralCell(rowIndex, exp)
        case binaryKind:
            return t.evaluateBinaryCell(rowIndex, exp)
        default:
            return nil, "", 0, ErrInvalidCell
    }
}

Implementing SELECT
As before, each statement will operate on a backend of tables.
type MemoryBackend struct {
    tables map[string]*table
}

func NewMemoryBackend() *MemoryBackend {
    return &MemoryBackend{
        tables: map[string]*table{},
    }
}

When we implement SELECT, we'll iterate over each row in
the table (we only support looking up one table for now).  If the
SELECT statement contains a WHERE block,
we'll evaluate the WHERE expression against the current
row and move on if the result is false.
Otherwise for each expression in the SELECT list of items
we'll evaluate it against the current row in the table.
If there is no table selected, we provide a fake table with a single
empty row.
func (mb *MemoryBackend) Select(slct *SelectStatement) (*Results, error) {
    t := &table{}

    if slct.from != nil && slct.from.table != nil {
        var ok bool
        t, ok = mb.tables[slct.from.table.value]
        if !ok {
            return nil, ErrTableDoesNotExist
        }
    }

    if slct.item == nil || len(*slct.item) == 0 {
        return &Results{}, nil
    }

    results := [][]Cell{}
    columns := []struct {
        Type ColumnType
        Name string
    }{}

    if slct.from == nil {
        t = &table{}
        t.rows = [][]MemoryCell{{}}
    }

    for i := range t.rows {
        result := []Cell{}
        isFirstRow := len(results) == 0

        if slct.where != nil {
            val, _, _, err := t.evaluateCell(uint(i), *slct.where)
            if err != nil {
                return nil, err
            }

            if !val.AsBool() {
                continue
            }
        }

        for _, col := range *slct.item {
            if col.asterisk {
                // TODO: handle asterisk
                fmt.Println("Skipping asterisk.")
                continue
            }

            value, columnName, columnType, err := t.evaluateCell(uint(i), *col.exp)
            if err != nil {
                return nil, err
            }

            if isFirstRow {
                columns = append(columns, struct {
                    Type ColumnType
                    Name string
                }{
                    Type: columnType,
                    Name: columnName,
                })
            }

            result = append(result, value)
        }

        results = append(results, result)
    }

    return &Results{
        Columns: columns,
        Rows:    results,
    }, nil
}

Implementing INSERT, CREATE
The INSERT and CREATE statements stay mostly
the same except for that we'll use the evaluateCell help
for every expression. Refer back to the first post if the
implementation is otherwise unclear.
func (mb *MemoryBackend) Insert(inst *InsertStatement) error {
    t, ok := mb.tables[inst.table.value]
    if !ok {
        return ErrTableDoesNotExist
    }

    if inst.values == nil {
        return nil
    }

    row := []MemoryCell{}

    if len(*inst.values) != len(t.columns) {
        return ErrMissingValues
    }

    for _, value := range *inst.values {
        if value.kind != literalKind {
            fmt.Println("Skipping non-literal.")
            continue
        }

        emptyTable := &table{}
        value, _, _, err := emptyTable.evaluateCell(0, *value)
        if err != nil {
            return err
        }

        row = append(row, value)
    }

    t.rows = append(t.rows, row)
    return nil
}

func (mb *MemoryBackend) CreateTable(crt *CreateTableStatement) error {
    t := table{}
    mb.tables[crt.name.value] = &t
    if crt.cols == nil {

        return nil
    }

    for _, col := range *crt.cols {
        t.columns = append(t.columns, col.name.value)

        var dt ColumnType
        switch col.datatype.value {
        case "int":
            dt = IntType
        case "text":
            dt = TextType
        default:
            return ErrInvalidDatatype
        }

        t.columnTypes = append(t.columnTypes, dt)
    }

    return nil
}

Back to the REPL
Putting it all together, we run the following session:
# CREATE TABLE users (name TEXT, age INT);
ok
#  INSERT INTO users VALUES ('Stephen', 16);
ok
# SELECT name, age FROM users;
name   | age
----------+------
Stephen |  16
(1 result)
ok
# INSERT INTO users VALUES ('Adrienne', 23);
ok
# SELECT age + 2, name FROM users WHERE age = 23;
age |   name
------+-----------
25 | Adrienne
(1 result)
ok
# SELECT name FROM users;
name
------------
Stephen
Adrienne
(2 results)
ok

And that's it for now! In future posts we'll get into indices, joining
tables, etc.
Latest post up in the database basics series: adding support for binary expressions and WHERE filtering in SELECTs.

Much nicer to have a real table rendering library and readline implementation in the REPL too.https://t.co/GYzn3FUNon
— Phil Eaton (@phil_eaton) April 12, 2020
 


I'm Writing A Book!
swyx's site RSS Feed — Fri, 10 Apr 2020 20:17:35 GMT
I'm writing a Dev Career Advice book... and I'm scared shitless!


How to Show a Range of Page Numbers using Django's Pagination
Tilting at Windmills — Wed, 08 Apr 2020 03:00:00 GMT
Posting this here in case anyone else has this problem and is searching for it. Django provides a very helpful way to break up long lists…


5 Things I Learned from The DynamoDB Book
swyx's site RSS Feed — Tue, 07 Apr 2020 20:40:45 GMT
Reviewing Alex DeBrie's new the DynamoDB Book


Running multiple A/B tests in parallel
Bytepawn - Marton Trencseni — Mon, 06 Apr 2020 01:00:00 GMT
I show using Monte Carlo simulations that randomizing user assignments into A/B test experiments makes it possible to run multiple A/B tests at once and measure accurate lifts on the same metric, assuming the experiments are independent.




How to Use class instead of className with Preact and TypeScript
swyx's site RSS Feed — Sun, 05 Apr 2020 09:08:05 GMT
Bottom Line Up Front


Want Better Design/CSS Skills? Spot the Difference!
swyx's site RSS Feed — Sat, 04 Apr 2020 18:03:06 GMT
A quick exercise to improve your attention to detail when implementing/designing with CSS.


How and Why to Un-Reset Tailwind's CSS Reset
swyx's site RSS Feed — Sat, 04 Apr 2020 16:04:49 GMT

2021 edit: note that tailwind typography now comes with a not-prose class that unsets typography styles



Studying foreign languages with inbox zero
Notes on software development — Sat, 04 Apr 2020 03:00:00 GMT
The only time I've been able to seriously, rapidly improve my ability
to speak a foreign language was through intensive language courses in
college. I was forced to actively speak, read, and write Chinese for
6-8 hours a week (1-2 hours every day). Then study another 5-10 hours
a week in preparation for the active sessions. I went three semesters
like this before I left school.
I've been trying to recreate that intensity since and mostly
failed. After marrying a Korean, I've redirected the little effort I
can muster to learning Korean. Aside from stints over the years
(mostly for a month or two before or after a trip to Korea), I haven't
been able to keep up any practice.
One thing I've tried over the years to commit myself to learning a
number of different topics is to set up recurring calendar invites:
"Study Linux", "Study TCP/IP", "Study Korean", etc.
This has mostly failed too. However, I do always look at the
invites as I get notified.
I keep inbox zero and I check my email many times a day, marking each
email read dilligently when I no longer need to think about it.
Tools like Quizlet, Anki, or even Duolingo let you self-learn
vocabulary when you feel like it. But basically no service
will try to keep giving you exposure to some set of topics whether you
spend time on it or not.
The most important thing I can think of is forced exposure to
vocabulary. So I've been planning for some time to hook up a list of
the one thousand most common Korean words to scheduled emails.
This weekend I finally got around to scripting the Google Calendar API
against the words list. I have an event for each word for the next
1000 days. Each day I receive a summary email including all events of
the day and the new word is part of it.
This is a pretty indirect approach but it's pretty simple to set
up. It's not very easy to reconfigure.
The code for doing this is available on
Github if you're
interested. And if you know a service that can build and manage
scheduled notifications against a spreadsheet or database I'd rather
be looking at that.
We'll see how this works out.
Daily new words in my inbox feels like the only way I can "force" myself to get exposed to new vocabulary. Wish there were a service for scheduling notifications from a spreadsheet. Finally got to scripting GCal's API populating daily events from 1000 most common Korean words
— Phil Eaton (@phil_eaton) April 4, 2020
 


Django + PostgreSQL + Docker-Compose: a few gotchas
Subclassed — Sat, 04 Apr 2020 00:15:34 GMT
A lot of the tutorials out there make it look like it's trivial to set up a development environment with Django, Postgres and Docker. That never quite matched my experience; you always end up knowing too much about docker, and there are a few gotchas that most people typically fail to mention. The following are a few specifically related to Postgres and Django, which I'm writing here because I tend to forget them every time I start a new project...

A running container is not a running database
Docker-compose will happily report a container as "up" even though it's busy doing init work. With postgres, this means that a container might look "up" when really it's still creating the actual db instance, so app connections might well fail.
A good workaround is to use pg_isready, like this :
#!/usr/bin/env sh

docker-compose up
until pg_isready -d your_pg_db -h your_pg_host \
                 -p your_pg_port -U your_pg_superuser
do
    echo "Waiting for db to be available..."
    sleep 2
done
# now we can do actual work, like db migrations
...


Don't run; exec

A lot of howtos state, more or less, "if you want to run something in an instance, use docker-compose run some_machine some_command". This is misleading. run will create a new ancillary container, which will run in parallel to any other container of the same type that might already be up. If you want to execute an ancillary process inside an already-running container, use docker-compose exec some_machine some_command instead. This will ensure you are "logged on" the running container.

While coding, don't copy; mount

Many will tell you that you need to ensure reproducibility; and as such, your code should be copied or checked out to the instance in Dockerfile, i.e. at build stage. That is a huge drag on development, since you need to rebuild the whole image on every minor change. It is annoying and slow even with multi-stage builds.

Instead, you can mount your actual source directory as a volume, and exploit all the goodies that make development tolerable, like Django's autoreload features. Make your docker-compose.ymllook like this instead:
services:
   your_app_machine:
      volumes:
         - type: bind
           source: /host/location/of/src
           target: /container/location/of/app
   ...

When you want to run tests or go to production, use a second Dockerfile that inherits from the first (with FROM) and actually copies data (or more likely checks it out via git), without the volume definition.

Know your tools

This is not really specific to docker! Master your tools in depth, it will help. I honestly didn't know that JetBrains PyCharm can now configure the interpreter running in a Docker container as the main one for the project, which makes a lot of things easier (debugging, REPL etc). Extremely helpful!


Instant GraphQL with OneGraph
swyx's site RSS Feed — Fri, 03 Apr 2020 09:23:06 GMT
Here's a recorded screenshare chat I had with Sean Grove (https://twitter.com/sgrove), Cofounder of OneGraph! We swapped out the Next.js API Routes that I manually set up in my livestreaming, to the premade GraphQL integration that OneGraph has with Dev.to. As a bonus, this takes care of user authentication so it doesn't rely on getting Dev.to API keys!


How to use Web Components with Next.js and TypeScript
swyx's site RSS Feed — Fri, 03 Apr 2020 04:07:42 GMT
In my livestream today I had the need to bring in a spinner component to show work in progress in my app. However found that existing React spinners were too heavy. That's when I had the idea to use web components in my Next.js (React/Preact) app for the first time ever!


Make your own Dev.to CMS livestream - Part 4
swyx's site RSS Feed — Fri, 03 Apr 2020 03:15:38 GMT
This is day 4 of my livecode Dev.to CMS.


A World Without Plugins
swyx's site RSS Feed — Thu, 02 Apr 2020 09:30:52 GMT
What happens if we did away with plugins altogether? The case for Imperative Recipes


10 Principles I Learned from Tiago Forte's Building a Second Brain
swyx's site RSS Feed — Wed, 01 Apr 2020 12:22:32 GMT
10 Principles I Learned


Make your own Dev.to CMS livestream - Part 3
swyx's site RSS Feed — Wed, 01 Apr 2020 08:29:01 GMT
This is day 3 of my livecode Dev.to CMS.


How To Add Monaco Editor to a Next.js app
swyx's site RSS Feed — Tue, 31 Mar 2020 02:29:25 GMT
Bottom Line Up Front


Bayesian A/B conversion tests
Bytepawn - Marton Trencseni — Tue, 31 Mar 2020 01:00:00 GMT
I compare probabilities from Bayesian A/B testing with Beta distributions to frequentist A/B tests using Monte Carlo simulations. Under a lot of circumstances, the bayesian probability of the action hypothesis being true and the frequentist p value are complementary.

 


Make your own Dev.to CMS livestream - Part 2
swyx's site RSS Feed — Sun, 29 Mar 2020 02:08:08 GMT
This is day 2 of my livecode Dev.to CMS. See Day 1 here


Make your own Dev.to CMS livestream - Part 1
swyx's site RSS Feed — Sat, 28 Mar 2020 00:41:36 GMT
Livecoding my own Dev.to CMS with Next.js + Preact + TypeScript + Tailwind UI


Cruising sailboat electronics setup with Signal K
Henri Bergius — Fri, 27 Mar 2020 02:00:00 GMT
I haven’t mentioned this on the blog earlier, but in the end of 2018 we bought a small cruising sailboat. After some looking, we went with a Van de Stadt designed Oceaan 25, a Dutch pocket cruiser from the early 1980s. S/Y Curiosity is an affordable and comfortable boat for cruising with 2-4 people, but also needed major maintenance work.



The refit has so far included osmosis repair, some fixes to the standing rigging, engine maintenance, and many structural improvements. But this post will focus on the electronics and navigation aspects of the project.

12V power

When we got it, the boat’s electrics setup was quite barebones. There was a small lead-acid battery, charged only when running the outboard. Light control was pretty much all-or-nothing, either we were running inside and navigation lights, or not. Everything was wired with 80s spec components, using energy-inefficient lightbulbs.

Looking at the state of the setup, it was also unclear when the electrics had been used for anything else than starting the engine last time.

Before going further with the electronics setup, all of this would have to be rebuilt. We made a plan, and scheduled two weekends in summer 2019 for rewiring and upgrading the electricity setup of the boat.

First step was to test all existing wiring with a multimeter, and label and document all of it. Surprisingly, there were only couple of bad connections from the main distribution panel to consumers, so for most part we decided to reuse that wiring, but just with a modern terminal block setup.



For most part we used a dymo label printer, with the labels covered with a transparent heat shrink.

We replaced the old main control panel with a modern one with the capability to power different parts of the boat separately, and added some 12V and USB sockets next to it.



All internal lighting was replaced with energy-efficient LEDs, and we added the option of using red lights all through the cabin for preserving night vision. A car charger was added to the system for easier battery charging while in harbour.

Next steps for power

With this, we had a workable lighting and power setup for overnight sailing. But next obvious step will be to increase the range of our boat.

For that, we’re adding a solar panel. We already have most parts for the setup, but are still waiting for the customized NOA mounting hardware to arrive. And of course the current COVID-19 curfews need to lift before we can install it.

Until we have actual data from our Victron MPPT charge controller, I’ve run some simulations using NASA’s insolation data for Berlin on how much the panel ought to increase our cruising range.



Navigation system

The basis for boat navigation is still the combination of a clock, a compass, and a paper chart (as well as a sextant on the open ocean). However, most modern cruising boats utilize some electrical tools to aid the process of running the boat. These typically come in form a chartplotter and a set of sensors to get things like GPS position, speed, and the water depth.

Commercial marine navigation equipment is a bit like computer networking in the 90s - everything is expensive, and you pretty much have to buy the whole kit from a single vendor to make it work. Standards like NMEA 0183 exist, but “embrace and extend” is typical vendor behaviour.

Signal K

Being open source hackerspace people, that was obviously not the way we wanted to do things. Instead of getting locked into an expensive proprietary single-vendor marine instrumentation setup, we decided to roll our own using off-the-shelf IoT components. To serve as the heart of the system, we picked Signal K.

Signal K is first of all a specification on how marine instruments can exchange data. It also has an open source implementation in Node.js. This allows piping in data from all of the relevant marine data buses, as well as setting up custom data providers. Signal K then harmonizes the data, and makes it available both via modern web APIs, and in traditional NMEA formats. This enables instruments like chartplotters also to utilize the Signal K enriched data.

We’re running Signal K on a Raspberry Pi 3B+ powered by the boat battery. With a GPS dongle, this was already enough to give some basic navigation capabilities like charts and anchor watch. We also added a WiFi hotspot with a LTE uplink to the boat.



To make the system robust, installation is automated via Ansible, and easy to reproduce. Our boat GitHub repo also has the needed functionality to run a clone of our boat’s setup on our laptops via Docker, which is great when developing new features.

Signal K has a very active developer community, which has been great for figuring out how the extend the capabilities of our system.

Chartplotter

We’re using regular tablets for navigation. The main chartplotter is a cheap old waterproof Samsung Galaxy Tab Active 8.0 tablet that can show both the Freeboard web-based chartplotter with OpenSeaMap charts, and run the Navionics Boating app to display commercial charts. Navionics is also able to receive some Signal K data over the boat WiFi to show things like AIS targets, and to utilize the boat GPS.



As a backup we have our personal smartphones and tablets.



Inside the cabin we also have an e-ink screen showing the primary statistics relevant to the current boat state.



Environmental sensing

Monitoring air pressure changes is important for dealing with the weather. For this, we added a cheap barometer-temperature-humidity sensor module wired to the Raspberry Pi, driven with the Signal K BME280 plugin. With this we were able to get all of this information from our cabin into Signal K.

However, there was more environmental information we wanted to get. For instance, the outdoor temperature, the humidity in our foul weather gear locker, and the temperature of our icebox. For these we found the Ruuvi tags produced by a Finnish startup. These are small weatherproofed Bluetooth environmental sensors that can run for years with a coin cell battery.



With Ruuvi tags and the Signal K Ruuvi tag plugin we were able to bring a rich set of environmental data from all around the boat into our dashboards.

Anchor watch

Like every cruising boat, we spend quite a lot of nights at anchor. One important safety measure with a shorthanded crew is to run an automated anchor watch. This monitors the boat’s distance to the anchor, and raises an alarm if we start dragging.

For this one, we’re using the Signal K anchor alarm plugin. We added a Bluetooth speaker to get these alarms in an audible way.

To make starting and stopping the anchor watch easier, I utilized a simple Bluetooth remote camera shutter button together with some scripts. This way the person dropping the anchor can also start the anchor watch immediately from the bow.



AIS

Automatic Identification System is a radio protocol used by most bigger vessels to tell others about their course and position. It can be used for collision avoidance. Having an active transponder on a small boat like Curiosity is a bit expensive, but we decided we’d at least want to see commercial traffic in our chartplotter in order to navigate safely.

For this we bought an RTL-SDR USB stick that can tune into the AIS frequency, and with the rtl_ais software, receive and forward all AIS data into Signal K.



This setup is still quite new, so we haven’t been able to test it live yet. But it should allow us to see all nearby bigger ships in our chartplotter in realtime, assuming that we have a good-enough antenna.

Putting it all together

All together this is quite a lot of hardware. To house all of it, we built a custom backing plate with 3D-printed brackets to hold the various components. The whole setup is called Voronoi-1 onboard computer. This is a setup that should be easy to duplicate on any small sailing vessel.



The total cost so far for the full boat navigation setup has been around 600€, which is less than just a commercial chartplotter would cost. And the system we have is both easy to extend, and to fix even on the go. And we get a set of capabilities that would normally require a whole suite of proprietary parts to put together.

Next steps for navigation setup

We of course have plenty of ideas on what to do next to improve the navigation setup. Here are some projects we’ll likely tackle over the coming year:


  Adding a timeseries database and some data visualization
  9 degrees of freedom sensor to track the compass course, as well as boat heel
  Instrumenting our outboard motor to get RPMs into Signal K and track the engine running time
  Wind sensor, either open source or commercial


If you have ideas for suitable components or projects, please get in touch!

Source code


  https://github.com/meri-imperiumi/curiosity contains the Ansible Signal K setup for Curiosity, as well as the CNC and 3D printing designs we’re using  on the boat
  https://github.com/meri-imperiumi/dashboard is the Python script we’re using to drive the e-ink dashboard in our cabin
  https://github.com/meri-imperiumi/signalk-autostate is a Signal K plugin for determining the boat state from sensor data
  https://github.com/meri-imperiumi/signalk-bluetooth-anchor-button contains the udev scripts for setting anchor watch from the Bluetooth remote camera shutter button
  https://github.com/meri-imperiumi/signalk-aws-iot is a Signal K plugin for transmitting our boat state to Amazon Web Services


Huge thanks to both the Signal K and Hackerfleet communities and the Curiosity crew for making all this happen.

Now we just wait for the curfews to lift so that we can get back to sailing!




The Computer History Museum YouTube Channel
swyx's site RSS Feed — Wed, 25 Mar 2020 04:10:46 GMT
I have been enjoying the CHM's YouTube Channel.


Every Public Engineering Career Ladder
swyx's site RSS Feed — Tue, 24 Mar 2020 06:42:35 GMT
A list of the public engineering career ladders I've found


A/B testing and the G-test
Bytepawn - Marton Trencseni — Mon, 23 Mar 2020 01:00:00 GMT
The G-test for conversion A/B tests is similar to the Chi-squared test. Monte-Carlo simulations show that the two are indistinguishable in practice.

 


Mise en Place Writing
swyx's site RSS Feed — Mon, 23 Mar 2020 00:05:45 GMT
How to write more, faster, and better by decoupling writing from pre-writing


6 Things Markdown Got Wrong
swyx's site RSS Feed — Sun, 22 Mar 2020 06:06:49 GMT
John Gruber's Markdown is almost a perfect content authoring format. Here are 6 things it got wrong.


A/B testing and networks effects
Bytepawn - Marton Trencseni — Sat, 21 Mar 2020 01:00:00 GMT
I use Monte Carlo simulations to explore how A/B testing on Watts–Strogatz random graphs depends on the degree distribution of the social network.

 


Farewell, Netlify
swyx's site RSS Feed — Fri, 20 Mar 2020 23:34:22 GMT
On leaving Netlify


What You Can Do with Netlify Build Plugins
swyx's site RSS Feed — Thu, 19 Mar 2020 22:24:55 GMT
Netlify Build Plugins are here! Here's a smattering of thoughts on what you can do with them.


Twitter as Universal Meta-Commentary Layer
swyx's site RSS Feed — Thu, 19 Mar 2020 05:04:54 GMT
Musing about how Twitter can enhance your Internet reading experience


Reviewing the Surface Book 2
Notes on software development — Wed, 18 Mar 2020 02:00:00 GMT
The first few paragraphs cover what I was looking for and what I
considered. Then the review.
Why the Surface Book 2
I used a Macbook throughout my professional career until I had the
choice a few years ago when I started my current job. Here, I ran
Gentoo, then FreeBSD, then Arch, and now Windows 10 on the Dell XPS
15.
I enjoy Windows and I think Microsoft is doing a better job on
hardware and software these days. At least, compared to Apple, they
appear to be trying. So when my personal 2015 Macbook Pro died this
year I decided to buy and run Windows at home.
On my Mac, I dealt with bad battery life for a while: running VMs,
running Docker, compiling Go, running Node.js kills any battery. So I
moved my development into the cloud and gained on battery life and
network speeds at the cost of memory (I am paying for 4GB of RAM).
My ideal replacement was a cheaper machine that felt as good as a 2015
Macbook Pro. (The build quality has not been good since.) I was
hoping not to pay more than $1000. My shortlist included the Surface
Book 2, the Surface Pro X, the Surface Laptop 3, the Lenovo Yoga 14,
and the LG Gram. So I went to Best Buy to try them out.
I was impressed by every Surface device. At first sight, I mistook the
Surface Book and Surface Laptop for an old Macbook Pro. They both have
a brushed aluminum body with a large trackpad and great
keyboards. Even the Surface Pro X, which is a tablet, has an addon
keyboard that is easy to type (that is, program) on.
I tried out the Lenovo Yoga 14 and it was solid, but I preferred the
brushed aluminum body of the Surface devices. I did not get a chance
to feel out the LG Gram.
I eliminated the Surface Laptop 3 because I like tablet mode. While
the Surface Laptop 3 is a touchscreen, it is not a 2-in-1 device and
does not have tablet mode.
And I eliminated the Surface Pro X because it is one of the first
mainstream Windows ARM devices. While Windows on ARM is now the same
operating system as Windows on a desktop, most consumer software ships
x86_64 (not ARM) binaries. Windows on ARM can emulate x86 but not yet
x86_64. I didn't feel like working around this on my primary personal
device.
I bought the 13.5", 7th generation i5 Surface Book 2 for $999. It
comes with 8GB DDR4 RAM and a 128GB SSD. I have had the device for two
weeks now and I use it at least 10 hours a day.
Keyboard
The keyboard layout is standard, easy to use. The control, shift,
caps, function, and alt keys are big enough that it is easy to program
without staring at the keyboard. The up and down arrow keys are
smaller than would be nice. But they are easier for me to find than on
a 2019 Macbook Pro.
The function key is modal by default (like a Caps key) and indicates
if function is enabled with a small LED. I have never seen a function
key like this. I find it annoying when I turn it on.
And while there is builtin volume controls and a play/pause button,
there is no media forward/back button. I assigned
Ctrl+Windows+Alt+Left/Right to be media forward/back.
There is also no right Ctrl key. Instead there is a "media key" which is
the equivalent of right-clicking... I guess. This is useless so I
mapped it back to another Ctrl key.
Unlike macOS, which needs an app like Spectacle, Windows default
window control shortcuts are great. Windows+Left to send to the left
half, Windows+Right to send to the right half, Windows+Up to make full
screen.
But macOS default swipe gestures are more intuitive: swipe left to go
backwards, swipe right to go forwards. So I mapped this back myself.
Here is my autohotkey
script.
Screen
The 13.5" screen feels top-heavy but may not actually weigh more than
the keyboard/body. The bevel is larger than it feels like it should
be. But the camera is in the right location: top and center.
Additionally, the default behavior when attaching/detaching the screen
is to prompt you to enter/exit tablet mode rather than doing it for
you. This prompt is easy to click out of and after doing so the option
to switch between disappears until you reattach and detach again.
The screen isn't flush with the body when you close it. Few marketing
pictures show you this, but here's
one. This
makes me worry something may snap if the laptop is ever slammed
against a wall for some reason.
And fully open, it only goes back 120 degrees. This makes it hard to
look at if it is on your legs and your legs are up higher than 90
degrees.
Finally, the headphone jack is not on the body but on the screen. This
makes sense since the screen is detachable. But the jack is on the
top-right corner, further away than usual. This requires me to be
closer to the screen to feel like I am not pulling the screen when I
am wearing headphones.
Pen
The Surface Pen is awesome and the screen's palm detection is too. I
have had a lot of fun drawing on it in Paint 3. And it has been useful
in annotating mockups for work too.
It costs $100 and comes with a AAAA battery. It is magnetized and
sticks to the left side of the screen.
Body
As mentioned, the body is a brushed aluminum. It feels great. The
power input is magnetic, which is helpful. But it uses a novel
Surface-specific input rather than USB-C, so that sucks. A new charger
from Microsoft costs $100.
The speakers are as good as Macbook speakers were 5 years ago.
They don't have much bass. Additionally, these speakers get a little
distorted at top volume.
The battery lasts 7-8 hours without charging. While this is as
advertised, it is still disappointing of a new laptop in 2020 that is
only running Chrome, Spotify, and Windows Terminal.
Tablet
To release the screen from the body, there is a key on the function
row. However, it is not a hardware release. So when I accidentally
killed the battery while the screen was flipped, I couldn't detach the
screen after booting (to turn it back into a laptop) until after 10-20
minutes of charging.
The screen isn't easy to detach. It requires both hands lifting up
from the base of the screen to get enough leverage. You cannot pull up
from the top of the screen.
Aside from drawing apps, tablet mode apps on Windows aren't
great. Kindle for Windows on tablet is terrible. I got stuck in
Kindle's full screen mode and couldn't adjust the page size or exit
full screen mode without reverting back to laptop mode first.
Tablet mode also throws away the standard Windows menu and shortcuts
to give you a desktop of application cards. However, these cards don't
adapt to recent or frequent applications. After I deleted Candy Crush
and other built in apps I will never use, this desktop is blank except
for Edge and Groove Music. It is incredible how bad the tablet desktop
is. You have to use the full application list view every time you want
to open a new program.
In summary
It's not a bad Windows machine for $1000. The body is great quality
and the pen/screen interaction is solid. But I'd like to see Windows
invest more in a useful tablet experience. And the detachable screen
comes at the cost of being a awkward. So I'd go with the Surface Pro X
or Surface Laptop 3 next time.
But above all I can't shake the expectation that a laptop built in
2020 running GMail and Slack in Chrome, Spotify, and a terminal
application should last at least 10 hours.
Wrote a short post reviewing Microsoft's Surface Book 2https://t.co/0n6K3y6FBC
— Phil Eaton (@phil_eaton) March 21, 2020
 


Lessons from college: Efficient meetings
CRAIG KERSTIENS — Tue, 17 Mar 2020 22:55:56 GMT
I think back to my time in college, and I learned some valuable things. I also learned some incredibly worthless things (i.e. don’t flip a car upside down and then backover… it’ll break the axle so you can’t roll it). Even in classes… the basic approach to a supply/demand curve to maximize profit is cute when done in a classroom vs. the complexities of how things actually work… I mean I get the idea behind it, but what you learn is so far being able to be translated into being usable. But what surprises me looking back was a couple of skills around running meetings that I find so rare in the workplace that have immense value.
I’ve always been fascinated at the intersection of business and technology. I’d been coding for a long time before college, and while interesting it was also a means to an end. When you combine technology with business you can solve things in entirely new and valuable ways. My major was management information systems, and all folks in my program came out with a computer science minor in addition to their business degree–something pretty rare for more MIS majors in other programs and well generally for anyone coming out of a business school. Perhaps I’ll get into the value of CS training even if you aren’t looking for a CS job some other time.
Within the program we would have a senior project that was actually a real world project for one of the large companies that sponsored part of the program. We’d have monthly reviews with the company stake holder. We’d also have weekly meetings, these were especially well run. There were really 3 items that made them especially efficient.


1. Agendas circulated out 24 hours ahead of time


Before each meeting there was a very explicit agenda. This was circulated out 24 hrs in advance, at almost exactly the 24 hr mark the professors would inquire into the delay in the agenda. This early circulation allowed for:

Time to review and prepare
Ability to make modifications

A sample agenda may look something like:

(5 minutes) - review last weeks action items
(10 minutes) - review blockers

blocker 1 - foo
blocker 2 - bar


(20 minutes) - feature design walk through of x
(15 minutes) - toubleshooting of
(10 minutes) - review action items and next steps



2. Explicit roles for the meeting


There were three roles explicitly set for each meeting:

Manager - This was essentially person responsible for setting up the agenda, ensuring the agenda was followed, and making sure everyone was involved and an active participant. I find that last piece is very key still today. Making sure folks that are remote have a chance to chime in, or ensuring very junior people are heard. Now days I often keep a small tally of how often various people within a meeting speak and make sure to give those that don’t a chance to.
Timekeeper - 75% of the meetings I’m in run long, and in the end there is a big scramble to figure out what the result of the meeting was and whats next. A timekeeper making sure that you spend the allocated amount of time is key. If you don’t do this that 10 mintues for reviewing action items and next steps gets squeezed and you lost a lot of the value of the meeting.
Scribe - A person whose sole job is to take notes ensured there were good quality notes. These would be kept within a sharepoint that was circulated around.



3. After the meeting


While the scribe was the one to take the notes, the meeting wasn’t done when the meeting was over. The notes were then circulated around. Everyone on the team would review and make comments/notes on things they felt were different or missed details. At the end in the review action items there would be very clear owners assigned and next steps laid out.
Within 24 hrs after the meeting while it was fresh everyone was required to review and acknowledge. This ensured there was a closed bookend and then this would be a clear transition for each of the roles to move on to the next meeting.


I’m sorry professors, you were right


At the time, it was annoying. I could take my own notes. I remembered the things I needed to do. If we went over on time for one item it was because it was important. But now… putting these things in place, any time I replicate this I get more time in my day and the team gets more done. Science/Math/History… sure, but running efficient meetings I never would have expected how basic but also challenging and how valuable.


Observability for Frontend Developers
swyx's site RSS Feed — Tue, 17 Mar 2020 16:43:24 GMT
Some thoughts on how frontend developers can also embrace instrumenting their apps for observability


Every Web Performance Test Tool
swyx's site RSS Feed — Tue, 17 Mar 2020 05:30:30 GMT
Check your site's speed quickly with a battery of tests


Essential Plugins for Gatsby Remark
swyx's site RSS Feed — Mon, 16 Mar 2020 05:43:23 GMT
Gatsby-Remark is one of those fun plugins that have their own plugins - but there are a lot of them. Here's a list I wrote down a few months ago of plugins I think everyone should use.


Formats over Functions
swyx's site RSS Feed — Sat, 14 Mar 2020 18:21:08 GMT
Why we should focus less on Implementation and more on Standardization


JavaScript: the First 20 Years by Allen Wirfs-Brock and Brendan Eich
swyx's site RSS Feed — Sat, 14 Mar 2020 05:00:44 GMT
A link to the 190 page history of JS by its original creator and the editor of ES6.


The Power of Lampshading
swyx's site RSS Feed — Fri, 13 Mar 2020 06:14:44 GMT
How to turn Ignorance into Power


Unit and Integration Testing for Plugin Authors
swyx's site RSS Feed — Thu, 12 Mar 2020 01:54:58 GMT
Some thoughts on how to set up testing with plugins


React Single File Components Are Here
swyx's site RSS Feed — Wed, 11 Mar 2020 05:02:23 GMT
React has long eschewed convention in favor of the extreme flexibility of JS. It is time for the next level in React authorship formats.


You May Not Need Controlled Form Components
swyx's site RSS Feed — Tue, 10 Mar 2020 01:29:43 GMT
A common design pattern for forms in React is using Controlled Components - but involves a lot of boilerplate code. Here's another way.


A/B testing on social networks
Bytepawn - Marton Trencseni — Mon, 09 Mar 2020 01:00:00 GMT
I use Monte Carlo simulations to show that experimentation on social networks is a beatiful statistical problem with unexpected nuances due to network effects.

 


The Coronavirus Recession and What it Means for Developers
swyx's site RSS Feed — Sat, 07 Mar 2020 07:28:29 GMT
The US is probably going into recession - here's why I'm talking about it now, what it could look like, what Devs can do to prepare, and why it's not the End of the World.


Guo Lai Ren (过来人)
swyx's site RSS Feed — Sat, 07 Mar 2020 02:00:00 GMT
One of the most powerful forms of persuasion is the argument from crossover people


Writing a SQL database from scratch in Go: 1. SELECT, INSERT, CREATE and a REPL
Notes on software development — Fri, 06 Mar 2020 02:00:00 GMT

  Next in database basics:
  <! forgive me, for I have sinned >
  

  2. binary expressions and WHERE filters
  

  3. indexes
  

  4. a database/sql driver
In this series we'll write a rudimentary database from
scratch in Go. Project source code is available on
Github.
In this first post we'll build enough of a parser to run some simple
CREATE, INSERT, and SELECT
queries. Then we'll build an in-memory backend
supporting TEXT and INT types and write a
basic REPL.
We'll be able to support the following interaction:
$ go run *.go
Welcome to gosql.
# CREATE TABLE users (id INT, name TEXT);
ok
# INSERT INTO users VALUES (1, 'Phil');
ok
# SELECT id, name FROM users;
| id | name |
====================
| 1 |  Phil |
ok
# INSERT INTO users VALUES (2, 'Kate');
ok
# SELECT name, id FROM users;
| name | id |
====================
| Phil |  1 |
| Kate |  2 |
ok

The first stage will be to map a SQL source into a list of tokens
(lexing). Then we'll call parse functions to find individual SQL
statements (such as SELECT). These parse functions will
in turn call their own helper functions to find patterns of
recursively parseable chunks, keywords, symbols (like parenthesis),
identifiers (like a table name), and numeric or string literals.
Then, we'll write an in-memory backend to do operations based on an
AST. Finally, we'll write a REPL to accept SQL from a CLI and pass it
to the in-memory backend.

  This post assumes a basic understanding of parsing concepts. We
  won't skip any code, but also won't go into great detail on why we
  structure the way we do.
  

  

  For a simpler introduction to parsing and parsing concepts,
  see this post on
  parsing JSON.
Lexing
The lexer is responsible for finding every distinct group of
characters in source code: tokens. This will consist primarily of
identifiers, numbers, strings, and symbols.

  What follows is a second, more orthodox pass at lexing. The first
  pass took a number of shortcuts and couldn't handle spaces in
  strings, for example.
  

  

  Here is the
  relevant pull request in gosql if you are curious.
The gist of the logic will be to pass control to a helper function for
each kind of token. If the helper function succeeds in finding a
token, it will return true and the location for the lexer to start at
next. It will continue doing this until it reaches the end of the
source.
First off, we'll define a few types and constants for use
in lexer.go:
package gosql

import (
    "fmt"
    "strings"
)

type location struct {
    line uint
    col  uint
}

type keyword string

const (
    selectKeyword keyword = "select"
    fromKeyword   keyword = "from"
    asKeyword     keyword = "as"
    tableKeyword  keyword = "table"
    createKeyword keyword = "create"
    insertKeyword keyword = "insert"
    intoKeyword   keyword = "into"
    valuesKeyword keyword = "values"
    intKeyword    keyword = "int"
    textKeyword   keyword = "text"
)

type symbol string

const (
    semicolonSymbol  symbol = ";"
    asteriskSymbol   symbol = "*"
    commaSymbol      symbol = ","
    leftparenSymbol  symbol = "("
    rightparenSymbol symbol = ")"
)

type tokenKind uint

const (
    keywordKind tokenKind = iota
    symbolKind
    identifierKind
    stringKind
    numericKind
)

type token struct {
    value string
    kind  tokenKind
    loc   location
}

type cursor struct {
    pointer uint
    loc     location
}

func (t *token) equals(other *token) bool {
    return t.value == other.value && t.kind == other.kind
}

type lexer func(string, cursor) (*token, cursor, bool)

Next we'll write out the main loop:
func lex(source string) ([]*token, error) {
    tokens := []*token{}
    cur := cursor{}

lex:
    for cur.pointer < uint(len(source)) {
        lexers := []lexer{lexKeyword, lexSymbol, lexString, lexNumeric, lexIdentifier}
        for _, l := range lexers {
            if token, newCursor, ok := l(source, cur); ok {
                cur = newCursor

                // Omit nil tokens for valid, but empty syntax like newlines
                if token != nil {
                    tokens = append(tokens, token)
                }

                continue lex
            }
        }

        hint := ""
        if len(tokens) > 0 {
            hint = " after " + tokens[len(tokens)-1].value
        }
        return nil, fmt.Errorf("Unable to lex token%s, at %d:%d", hint, cur.loc.line, cur.loc.col)
    }

    return tokens, nil
}

Then we'll write a helper for each kind of fundemental token.
Analyzing numbers
Numbers are the most complex. So we'll refer to the PostgreSQL
documentation (section
4.1.2.6)
for what constitutes a valid number.
func lexNumeric(source string, ic cursor) (*token, cursor, bool) {
    cur := ic

    periodFound := false
    expMarkerFound := false

    for ; cur.pointer < uint(len(source)); cur.pointer++ {
        c := source[cur.pointer]
        cur.loc.col++

        isDigit := c >= '0' && c <= '9'
        isPeriod := c == '.'
        isExpMarker := c == 'e'

        // Must start with a digit or period
        if cur.pointer == ic.pointer {
            if !isDigit && !isPeriod {
                return nil, ic, false
            }

            periodFound = isPeriod
            continue
        }

        if isPeriod {
            if periodFound {
                return nil, ic, false
            }

            periodFound = true
            continue
        }

        if isExpMarker {
            if expMarkerFound {
                return nil, ic, false
            }

            // No periods allowed after expMarker
            periodFound = true
            expMarkerFound = true

            // expMarker must be followed by digits
            if cur.pointer == uint(len(source)-1) {
                return nil, ic, false
            }

            cNext := source[cur.pointer+1]
            if cNext == '-' || cNext == '+' {
                cur.pointer++
                cur.loc.col++
            }

            continue
        }

        if !isDigit {
            break
        }
    }

    // No characters accumulated
    if cur.pointer == ic.pointer {
        return nil, ic, false
    }

    return &token{
        value: source[ic.pointer:cur.pointer],
        loc:   ic.loc,
        kind:  numericKind,
    }, cur, true
}

Analyzing strings
Strings must start and end with a single apostrophe. They can contain
a single apostophe if it is followed by another single
apostrophe. We'll put this kind of character delimited lexing logic
into a helper function so we can use it again when analyzing
identifiers.
func lexCharacterDelimited(source string, ic cursor, delimiter byte) (*token, cursor, bool) {
    cur := ic

    if len(source[cur.pointer:]) == 0 {
        return nil, ic, false
    }

    if source[cur.pointer] != delimiter {
        return nil, ic, false
    }

    cur.loc.col++
    cur.pointer++

    var value []byte
    for ; cur.pointer < uint(len(source)); cur.pointer++ {
        c := source[cur.pointer]

        if c == delimiter {
            // SQL escapes are via double characters, not backslash.
            if cur.pointer+1 >= uint(len(source)) || source[cur.pointer+1] != delimiter {
                return &token{
                    value: string(value),
                    loc:   ic.loc,
                    kind:  stringKind,
                }, cur, true
            } else {
                value = append(value, delimiter)
                cur.pointer++
                cur.loc.col++
            }
        }

        value = append(value, c)
        cur.loc.col++
    }

    return nil, ic, false
}

func lexString(source string, ic cursor) (*token, cursor, bool) {
    return lexCharacterDelimited(source, ic, '\'')
}

Analyzing symbols and keywords
Symbols come from a fixed set of strings, so they're easy
to compare against. Whitespace should be thrown away.
func lexSymbol(source string, ic cursor) (*token, cursor, bool) {
    c := source[ic.pointer]
    cur := ic
    // Will get overwritten later if not an ignored syntax
    cur.pointer++
    cur.loc.col++

    switch c {
    // Syntax that should be thrown away
    case '\n':
        cur.loc.line++
        cur.loc.col = 0
        fallthrough
    case '\t':
        fallthrough
    case ' ':
        return nil, cur, true
    }

    // Syntax that should be kept
    symbols := []symbol{
        commaSymbol,
        leftParenSymbol,
        rightParenSymbol,
        semicolonSymbol,
        asteriskSymbol,
    }

    var options []string
    for _, s := range symbols {
        options = append(options, string(s))
    }

    // Use `ic`, not `cur`
    match := longestMatch(source, ic, options)
    // Unknown character
    if match == "" {
        return nil, ic, false
    }

    cur.pointer = ic.pointer + uint(len(match))
    cur.loc.col = ic.loc.col + uint(len(match))

    return &token{
        value: match,
        loc:   ic.loc,
        kind:  symbolKind,
    }, cur, true
}

Keywords are even simpler, and use the same longestMatch
helper.
func lexKeyword(source string, ic cursor) (*token, cursor, bool) {
    cur := ic
    keywords := []keyword{
        selectKeyword,
        insertKeyword,
        valuesKeyword,
        tableKeyword,
        createKeyword,
        whereKeyword,
        fromKeyword,
        intoKeyword,
        textKeyword,
    }

    var options []string
    for _, k := range keywords {
        options = append(options, string(k))
    }

    match := longestMatch(source, ic, options)
    if match == "" {
        return nil, ic, false
    }

    cur.pointer = ic.pointer + uint(len(match))
    cur.loc.col = ic.loc.col + uint(len(match))

    return &token{
        value: match,
        kind:  kind,
        loc:   ic.loc,
    }, cur, true
}

And finally we implement the longestMatch helper:
// longestMatch iterates through a source string starting at the given
// cursor to find the longest matching substring among the provided
// options
func longestMatch(source string, ic cursor, options []string) string {
    var value []byte
    var skipList []int
    var match string

    cur := ic

    for cur.pointer < uint(len(source)) {

        value = append(value, strings.ToLower(string(source[cur.pointer]))...)
        cur.pointer++

    match:
        for i, option := range options {
            for _, skip := range skipList {
                if i == skip {
                    continue match
                }
            }

            // Deal with cases like INT vs INTO
            if option == string(value) {
                skipList = append(skipList, i)
                if len(option) > len(match) {
                    match = option
                }

                continue
            }

            sharesPrefix := string(value) == option[:cur.pointer-ic.pointer]
            tooLong := len(value) > len(option)
            if tooLong || !sharesPrefix {
                skipList = append(skipList, i)
            }
        }

        if len(skipList) == len(options) {
            break
        }
    }

    return match
}

Analyzing identifiers
An identifier is either a double-quoted string or a group of
characters starting with an alphabetical character and possibly
containing numbers and underscores.
func lexIdentifier(source string, ic cursor) (*token, cursor, bool) {
    // Handle separately if is a double-quoted identifier
    if token, newCursor, ok := lexCharacterDelimited(source, ic, '"'); ok {
        return token, newCursor, true
    }

    cur := ic

    c := source[cur.pointer]
    // Other characters count too, big ignoring non-ascii for now
    isAlphabetical := (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')
    if !isAlphabetical {
        return nil, ic, false
    }
    cur.pointer++
    cur.loc.col++

    value := []byte{c}
    for ; cur.pointer < uint(len(source)); cur.pointer++ {
        c = source[cur.pointer]

        // Other characters count too, big ignoring non-ascii for now
        isAlphabetical := (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')
        isNumeric := c >= '0' && c <= '9'
        if isAlphabetical || isNumeric || c == '$' || c == '_' {
            value = append(value, c)
            cur.loc.col++
            continue
        }

        break
    }

    if len(value) == 0 {
        return nil, ic, false
    }

    return &token{
        // Unquoted dentifiers are case-insensitive
        value: strings.ToLower(string(value)),
        loc:   ic.loc,
        kind:  identifierKind,
    }, cur, true
}

And that's it for the lexer! If you copy
lexer_test.go
from the main project, the tests should now pass.
AST model
At the highest level, an AST is a collection of statements:
package main

type Ast struct {
    Statements []*Statement
}

A statement, for now, is one of INSERT,
CREATE, or SELECT:
type AstKind uint

const (
    SelectKind AstKind = iota
    CreateTableKind
    InsertKind
)

type Statement struct {
    SelectStatement      *SelectStatement
    CreateTableStatement *CreateTableStatement
    InsertStatement      *InsertStatement
    Kind                 AstKind
}

INSERT
An insert statement, for now, has a table name and a list of values to
insert:
type InsertStatement struct {
    table  token
    values *[]*expression
}

An expression is a literal token or (in the future) a function call or
inline operation:
type expressionKind uint

const (
    literalKind expressionKind = iota
)

type expression struct {
    literal *token
    kind    expressionKind
}

CREATE
A create statement, for now, has a table name and a list of column
names and types:
type columnDefinition struct {
    name     token
    datatype token
}

type CreateTableStatement struct {
    name token
    cols *[]*columnDefinition
}

SELECT
A select statement, for now, has a table name and a list of column
names:
type SelectStatement struct {
    item []*expression
    from token
}

And that's it for the AST.
Parsing
The Parse entrypoint will take a list of tokens and
attempt to parse statements, separated by a semi-colon, until it
reaches the last token.
In general our strategy will be to increment and pass around a cursor
containing the current position of unparsed tokens. Each helper will
return the new cursor that the caller should start from.
package main

import (
    "errors"
    "fmt"
)

func tokenFromKeyword(k keyword) token {
    return token{
        kind:  keywordKind,
        value: string(k),
    }
}

func tokenFromSymbol(s symbol) token {
    return token{
        kind:  symbolKind,
        value: string(s),
    }
}

func expectToken(tokens []*token, cursor uint, t token) bool {
    if cursor >= uint(len(tokens)) {
        return false
    }

    return t.equals(tokens[cursor])
}

func helpMessage(tokens []*token, cursor uint, msg string) {
    var c *token
    if cursor < uint(len(tokens)) {
        c = tokens[cursor]
    } else {
        c = tokens[cursor-1]
    }

    fmt.Printf("[%d,%d]: %s, got: %s\n", c.loc.line, c.loc.col, msg, c.value)
}

func Parse(source string) (*Ast, error) {
    tokens, err := lex(source)
    if err != nil {
        return nil, err
    }

    a := Ast{}
    cursor := uint(0)
    for cursor < uint(len(tokens)) {
        stmt, newCursor, ok := parseStatement(tokens, cursor, tokenFromSymbol(semicolonSymbol))
        if !ok {
            helpMessage(tokens, cursor, "Expected statement")
            return nil, errors.New("Failed to parse, expected statement")
        }
        cursor = newCursor

        a.Statements = append(a.Statements, stmt)

        atLeastOneSemicolon := false
        for expectToken(tokens, cursor, tokenFromSymbol(semicolonSymbol)) {
            cursor++
            atLeastOneSemicolon = true
        }

        if !atLeastOneSemicolon {
            helpMessage(tokens, cursor, "Expected semi-colon delimiter between statements")
            return nil, errors.New("Missing semi-colon between statements")
        }
    }

    return &a, nil
}

Parsing statements
Each statement will be one of INSERT,
CREATE, or SELECT. The
parseStatement helper will call a helper on each of these
statement types and return true if one of them succeeds
in parsing.
func parseStatement(tokens []*token, initialCursor uint, delimiter token) (*Statement, uint, bool) {
    cursor := initialCursor

    // Look for a SELECT statement
    semicolonToken := tokenFromSymbol(semicolonSymbol)
    slct, newCursor, ok := parseSelectStatement(tokens, cursor, semicolonToken)
    if ok {
        return &Statement{
            Kind:            SelectKind,
            SelectStatement: slct,
        }, newCursor, true
    }

    // Look for a INSERT statement
    inst, newCursor, ok := parseInsertStatement(tokens, cursor, semicolonToken)
    if ok {
        return &Statement{
            Kind:            InsertKind,
            InsertStatement: inst,
        }, newCursor, true
    }

    // Look for a CREATE statement
    crtTbl, newCursor, ok := parseCreateTableStatement(tokens, cursor, semicolonToken)
    if ok {
        return &Statement{
            Kind:                 CreateTableKind,
            CreateTableStatement: crtTbl,
        }, newCursor, true
    }

    return nil, initialCursor, false
}

Parsing select statements
Parsing SELECT statements is easy. We'll look for the
following token pattern:

SELECT
$expression [, ...]
FROM
$table-name

Sketching that out we get:
func parseSelectStatement(tokens []*token, initialCursor uint, delimiter token) (*SelectStatement, uint, bool) {
    cursor := initialCursor
    if !expectToken(tokens, cursor, tokenFromKeyword(selectKeyword)) {
        return nil, initialCursor, false
    }
    cursor++

    slct := SelectStatement{}

    exps, newCursor, ok := parseExpressions(tokens, cursor, []token{tokenFromKeyword(fromKeyword), delimiter})
    if !ok {
        return nil, initialCursor, false
    }

    slct.item = *exps
    cursor = newCursor

    if expectToken(tokens, cursor, tokenFromKeyword(fromKeyword)) {
        cursor++

        from, newCursor, ok := parseToken(tokens, cursor, identifierKind)
        if !ok {
            helpMessage(tokens, cursor, "Expected FROM token")
            return nil, initialCursor, false
        }

        slct.from = *from
        cursor = newCursor
    }

    return &slct, cursor, true
}

The parseToken helper will look for a token of a
particular token kind.
func parseToken(tokens []*token, initialCursor uint, kind tokenKind) (*token, uint, bool) {
    cursor := initialCursor

    if cursor >= uint(len(tokens)) {
        return nil, initialCursor, false
    }

    current := tokens[cursor]
    if current.kind == kind {
        return current, cursor + 1, true
    }

    return nil, initialCursor, false
}

The parseExpressions helper will look for tokens
separated by a comma until a delimiter is found. It will use existing
helpers plus parseExpression.
func parseExpressions(tokens []*token, initialCursor uint, delimiters []token) (*[]*expression, uint, bool) {
    cursor := initialCursor

    exps := []*expression{}
outer:
    for {
        if cursor >= uint(len(tokens)) {
            return nil, initialCursor, false
        }

        // Look for delimiter
        current := tokens[cursor]
        for _, delimiter := range delimiters {
            if delimiter.equals(current) {
                break outer
            }
        }

        // Look for comma
        if len(exps) > 0 {
            if !expectToken(tokens, cursor, tokenFromSymbol(commaSymbol)) {
                helpMessage(tokens, cursor, "Expected comma")
                return nil, initialCursor, false
            }

            cursor++
        }

        // Look for expression
        exp, newCursor, ok := parseExpression(tokens, cursor, tokenFromSymbol(commaSymbol))
        if !ok {
            helpMessage(tokens, cursor, "Expected expression")
            return nil, initialCursor, false
        }
        cursor = newCursor

        exps = append(exps, exp)
    }

    return &exps, cursor, true
}

The parseExpression helper (for now) will look for a
numeric, string, or identifier token.
func parseExpression(tokens []*token, initialCursor uint, _ token) (*expression, uint, bool) {
    cursor := initialCursor

    kinds := []tokenKind{identifierKind, numericKind, stringKind}
    for _, kind := range kinds {
        t, newCursor, ok := parseToken(tokens, cursor, kind)
        if ok {
            return &expression{
                literal: t,
                kind:    literalKind,
            }, newCursor, true
        }
    }

    return nil, initialCursor, false
}

And that's it for parsing a SELECT statement!
Parsing insert statements
We'll look for the following token pattern:

INSERT
INTO
$table-name
VALUES
(
$expression [, ...]
)

With the existing helpers, this is straightforward to sketch out:
func parseInsertStatement(tokens []*token, initialCursor uint, delimiter token) (*InsertStatement, uint, bool) {
    cursor := initialCursor

    // Look for INSERT
    if !expectToken(tokens, cursor, tokenFromKeyword(insertKeyword)) {
        return nil, initialCursor, false
    }
    cursor++

    // Look for INTO
    if !expectToken(tokens, cursor, tokenFromKeyword(intoKeyword)) {
        helpMessage(tokens, cursor, "Expected into")
        return nil, initialCursor, false
    }
    cursor++

    // Look for table name
    table, newCursor, ok := parseToken(tokens, cursor, identifierKind)
    if !ok {
        helpMessage(tokens, cursor, "Expected table name")
        return nil, initialCursor, false
    }
    cursor = newCursor

    // Look for VALUES
    if !expectToken(tokens, cursor, tokenFromKeyword(valuesKeyword)) {
        helpMessage(tokens, cursor, "Expected VALUES")
        return nil, initialCursor, false
    }
    cursor++

    // Look for left paren
    if !expectToken(tokens, cursor, tokenFromSymbol(leftparenSymbol)) {
        helpMessage(tokens, cursor, "Expected left paren")
        return nil, initialCursor, false
    }
    cursor++

    // Look for expression list
    values, newCursor, ok := parseExpressions(tokens, cursor, []token{tokenFromSymbol(rightparenSymbol)})
    if !ok {
        return nil, initialCursor, false
    }
    cursor = newCursor

    // Look for right paren
    if !expectToken(tokens, cursor, tokenFromSymbol(rightparenSymbol)) {
        helpMessage(tokens, cursor, "Expected right paren")
        return nil, initialCursor, false
    }
    cursor++

    return &InsertStatement{
        table:  *table,
        values: values,
    }, cursor, true
}

And that's it for parsing an INSERT statement!
Parsing create statements
Finally, for create statements we'll look for the following token
pattern:

CREATE
$table-name
(
[$column-name $column-type [, ...]]
)

Sketching that out with a new parseColumnDefinitions
helper we get:
func parseCreateTableStatement(tokens []*token, initialCursor uint, delimiter token) (*CreateTableStatement, uint, bool) {
    cursor := initialCursor

    if !expectToken(tokens, cursor, tokenFromKeyword(createKeyword)) {
        return nil, initialCursor, false
    }
    cursor++

    if !expectToken(tokens, cursor, tokenFromKeyword(tableKeyword)) {
        return nil, initialCursor, false
    }
    cursor++

    name, newCursor, ok := parseToken(tokens, cursor, identifierKind)
    if !ok {
        helpMessage(tokens, cursor, "Expected table name")
        return nil, initialCursor, false
    }
    cursor = newCursor

    if !expectToken(tokens, cursor, tokenFromSymbol(leftparenSymbol)) {
        helpMessage(tokens, cursor, "Expected left parenthesis")
        return nil, initialCursor, false
    }
    cursor++

    cols, newCursor, ok := parseColumnDefinitions(tokens, cursor, tokenFromSymbol(rightparenSymbol))
    if !ok {
        return nil, initialCursor, false
    }
    cursor = newCursor

    if !expectToken(tokens, cursor, tokenFromSymbol(rightparenSymbol)) {
        helpMessage(tokens, cursor, "Expected right parenthesis")
        return nil, initialCursor, false
    }
    cursor++

    return &CreateTableStatement{
        name: *name,
        cols: cols,
    }, cursor, true
}

The parseColumnDefinitions helper will look column names
followed by column types separated by a comma and ending with some
delimiter:
func parseColumnDefinitions(tokens []*token, initialCursor uint, delimiter token) (*[]*columnDefinition, uint, bool) {
    cursor := initialCursor

    cds := []*columnDefinition{}
    for {
        if cursor >= uint(len(tokens)) {
            return nil, initialCursor, false
        }

        // Look for a delimiter
        current := tokens[cursor]
        if delimiter.equals(current) {
            break
        }

        // Look for a comma
        if len(cds) > 0 {
            if !expectToken(tokens, cursor, tokenFromSymbol(commaSymbol)) {
                helpMessage(tokens, cursor, "Expected comma")
                return nil, initialCursor, false
            }

            cursor++
        }

        // Look for a column name
        id, newCursor, ok := parseToken(tokens, cursor, identifierKind)
        if !ok {
            helpMessage(tokens, cursor, "Expected column name")
            return nil, initialCursor, false
        }
        cursor = newCursor

        // Look for a column type
        ty, newCursor, ok := parseToken(tokens, cursor, keywordKind)
        if !ok {
            helpMessage(tokens, cursor, "Expected column type")
            return nil, initialCursor, false
        }
        cursor = newCursor

        cds = append(cds, &columnDefinition{
            name:     *id,
            datatype: *ty,
        })
    }

    return &cds, cursor, true
}

And that's it for parsing! If you copy
parser_test.go
from the main project, the tests should now pass.
An in-memory backend
Our in-memory backend should conform to a general backend interface
that allows a user to create, select, and insert data:
package main

import "errors"

type ColumnType uint

const (
    TextType ColumnType = iota
    IntType
)

type Cell interface {
    AsText() string
    AsInt() int32
}

type Results struct {
    Columns []struct {
        Type ColumnType
        Name string
    }
    Rows [][]Cell
}

var (
    ErrTableDoesNotExist  = errors.New("Table does not exist")
    ErrColumnDoesNotExist = errors.New("Column does not exist")
    ErrInvalidSelectItem  = errors.New("Select item is not valid")
    ErrInvalidDatatype    = errors.New("Invalid datatype")
    ErrMissingValues      = errors.New("Missing values")
)

type Backend interface {
    CreateTable(*CreateTableStatement) error
    Insert(*InsertStatement) error
    Select(*SelectStatement) (*Results, error)
}

This leaves us room in the future for a disk-backed backend.
Memory layout
Our in-memory backend should store a list of tables. Each table
will have a list of columns and rows. Each column will have a name and
type. Each row will have a list of byte arrays.
package main

import (
    "bytes"
    "encoding/binary"
    "fmt"
    "strconv"
)

type MemoryCell []byte

func (mc MemoryCell) AsInt() int32 {
    var i int32
    err := binary.Read(bytes.NewBuffer(mc), binary.BigEndian, &i)
    if err != nil {
        panic(err)
    }

    return i
}

func (mc MemoryCell) AsText() string {
    return string(mc)
}

type table struct {
    columns     []string
    columnTypes []ColumnType
    rows        [][]MemoryCell
}

type MemoryBackend struct {
    tables map[string]*table
}

func NewMemoryBackend() *MemoryBackend {
    return &MemoryBackend{
        tables: map[string]*table{},
    }
}

Implementing create table support
When creating a table, we'll make a new entry in the backend tables
map. Then we'll create columns as specified by the AST.
func (mb *MemoryBackend) CreateTable(crt *CreateTableStatement) error {
    t := table{}
    mb.tables[crt.name.value] = &t
    if crt.cols == nil {

        return nil
    }

    for _, col := range *crt.cols {
        t.columns = append(t.columns, col.name.value)

        var dt ColumnType
        switch col.datatype.value {
        case "int":
            dt = IntType
        case "text":
            dt = TextType
        default:
            return ErrInvalidDatatype
        }

        t.columnTypes = append(t.columnTypes, dt)
    }

    return nil
}

Implementing insert support
Keeping things simple, we'll assume the value passed can be correctly
mapped to the type of the column specified.
We'll reference a helper for mapper values to internal storage,
tokenToCell.
func (mb *MemoryBackend) Insert(inst *InsertStatement) error {
    table, ok := mb.tables[inst.table.value]
    if !ok {
        return ErrTableDoesNotExist
    }

    if inst.values == nil {
        return nil
    }

    row := []MemoryCell{}

    if len(*inst.values) != len(table.columns) {
        return ErrMissingValues
    }

    for _, value := range *inst.values {
        if value.kind != literalKind {
            fmt.Println("Skipping non-literal.")
            continue
        }

        row = append(row, mb.tokenToCell(value.literal))
    }

    table.rows = append(table.rows, row)
    return nil
}

The tokenToCell helper will write numbers as binary bytes
and will write strings as bytes:
func (mb *MemoryBackend) tokenToCell(t *token) MemoryCell {
    if t.kind == numericKind {
        buf := new(bytes.Buffer)
        i, err := strconv.Atoi(t.value)
        if err != nil {
            panic(err)
        }

        err = binary.Write(buf, binary.BigEndian, int32(i))
        if err != nil {
            panic(err)
        }
        return MemoryCell(buf.Bytes())
    }

    if t.kind == stringKind {
        return MemoryCell(t.value)
    }

    return nil
}

Implementing select support
Finally, for select we'll iterate over each row in the table and
return the cells according to the columns specified by the AST.
func (mb *MemoryBackend) Select(slct *SelectStatement) (*Results, error) {
    table, ok := mb.tables[slct.from.table]
    if !ok {
        return nil, ErrTableDoesNotExist
    }

    results := [][]Cell{}
    columns := []struct {
        Type ColumnType
        Name string
    }{}

    for i, row := range table.rows {
        result := []Cell{}
        isFirstRow := i == 0

        for _, exp := range slct.item {
            if exp.kind != literalKind {
                // Unsupported, doesn't currently exist, ignore.
                fmt.Println("Skipping non-literal expression.")
                continue
            }

            lit := exp.literal
            if lit.kind == identifierKind {
                found := false
                for i, tableCol := range table.columns {
                    if tableCol == lit.value {
                        if isFirstRow {
                            columns = append(columns, struct {
                                Type ColumnType
                                Name string
                            }{
                                Type: table.columnTypes[i],
                                Name: lit.value,
                            })
                        }

                        result = append(result, row[i])
                        found = true
                        break
                    }
                }

                if !found {
                    return nil, ErrColumnDoesNotExist
                }

                continue
            }

            return nil, ErrColumnDoesNotExist
        }

        results = append(results, result)
    }

    return &Results{
        Columns: columns,
        Rows:    results,
    }, nil
}

The REPL
At last, we're ready to wrap the parser and in-memory backend in a
REPL. The most complex part is displaying the table of results from a
select query.
package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"

    "github.com/eatonphil/gosql"
)

func main() {
    mb := gosql.NewMemoryBackend()

    reader := bufio.NewReader(os.Stdin)
    fmt.Println("Welcome to gosql.")
    for {
        fmt.Print("# ")
        text, err := reader.ReadString('\n')
        text = strings.Replace(text, "\n", "", -1)

        ast, err := gosql.Parse(text)
        if err != nil {
            panic(err)
        }

        for _, stmt := range ast.Statements {
            switch stmt.Kind {
            case gosql.CreateTableKind:
                err = mb.CreateTable(ast.Statements[0].CreateTableStatement)
                if err != nil {
                    panic(err)
                }
                fmt.Println("ok")
            case gosql.InsertKind:
                err = mb.Insert(stmt.InsertStatement)
                if err != nil {
                    panic(err)
                }

                fmt.Println("ok")
            case gosql.SelectKind:
                results, err := mb.Select(stmt.SelectStatement)
                if err != nil {
                    panic(err)
                }

                for _, col := range results.Columns {
                    fmt.Printf("| %s ", col.Name)
                }
                fmt.Println("|")

                for i := 0; i < 20; i++ {
                    fmt.Printf("=")
                }
                fmt.Println()

                for _, result := range results.Rows {
                    fmt.Printf("|")

                    for i, cell := range result {
                        typ := results.Columns[i].Type
                        s := ""
                        switch typ {
                        case gosql.IntType:
                            s = fmt.Sprintf("%d", cell.AsInt())
                        case gosql.TextType:
                            s = cell.AsText()
                        }

                        fmt.Printf(" %s | ", s)
                    }

                    fmt.Println()
                }

                fmt.Println("ok")
            }
        }
    }
}

Putting it all together:
$ go run *.go
Welcome to gosql.
# CREATE TABLE users (id INT, name TEXT);
ok
# INSERT INTO users VALUES (1, 'Phil');
ok
# SELECT id, name FROM users;
| id | name |
====================
| 1 |  Phil |
ok
# INSERT INTO users VALUES (2, 'Kate');
ok
# SELECT name, id FROM users;
| name | id |
====================
| Phil |  1 |
| Kate |  2 |
ok

And we've got a very simple SQL database!
Next up we'll get into filtering, sorting, and indexing.
Further reading

Writing a simple JSON parser
This post goes into a little more detail about the theory and basics of parsing.


Database Systems: A Practical Approach to Design, Implementation and Management
A giant book, but an excellent and very easy introduction to database theory.



Latest blog post: writing a simple SQL database from scratch in Go https://t.co/csQmNhWIEf
— Phil Eaton (@phil_eaton) March 10, 2020
 


Feedback Ladders
swyx's site RSS Feed — Fri, 06 Mar 2020 02:00:00 GMT
A post for the Netlify blog on how we did Code Reviews during my recent product rotation.


Early stopping in A/B testing
Bytepawn - Marton Trencseni — Thu, 05 Mar 2020 01:00:00 GMT
Increased false positive rate due to early stopping is beautiful nuance of statistical testing. It is equivalent to running at an overall higher alpha. Data scientists need to be aware of this phenomenon so they can control it and keep their organizations honest about their experimental results.

 


Working at a Startup From Series B to C
swyx's site RSS Feed — Wed, 04 Mar 2020 02:00:00 GMT
Some careful thoughts from joining a startup from Series B to C.


Why Svelte? (The Short Version)
swyx's site RSS Feed — Wed, 04 Mar 2020 02:00:00 GMT
The short version of Why Svelte


A/B testing and Fisher's exact test
Bytepawn - Marton Trencseni — Tue, 03 Mar 2020 01:00:00 GMT
Fisher’s exact test directly computes the same p value as the Chi-squared test, so it does not rely on the Central Limit Theorem to hold.

 


Compile Svelte in Your Head by Tan Li Hau
swyx's site RSS Feed — Mon, 02 Mar 2020 02:00:00 GMT
Linking to a blogpost I was too lazy to do


Three Jobs of Containers
swyx's site RSS Feed — Sun, 01 Mar 2020 02:00:00 GMT
A great nugget I learned from watching Brian Holt's Intro to Containers workshop


Starting Svelte Society
swyx's site RSS Feed — Sat, 29 Feb 2020 02:00:00 GMT
Thoughts on how I am doing with 4 months of Svelte Society in the bag.


Switching to Android after 13 years of iOS
swyx's site RSS Feed — Fri, 28 Feb 2020 05:16:47 GMT
I have used iPhone/iOS for the longest time. Last month I switched to OnePlus 6T/Android. Here are my impressions.


Mocking and Using CRUD APIs with MirageJS and React-Query
swyx's site RSS Feed — Fri, 28 Feb 2020 02:00:00 GMT
This is how to pair two newcomers on the React scene for mocking and using CRUD APIs, for a great developer experience.


How I Approach First Principles Thinking
swyx's site RSS Feed — Fri, 28 Feb 2020 02:00:00 GMT
An explanation of First Principles via comparing Inductive vs Deductive Reasoning, and thoughts on Applications


A/B testing and the Chi-squared test
Bytepawn - Marton Trencseni — Fri, 28 Feb 2020 01:00:00 GMT
In an ealier post, I wrote about A/B testing conversion data with the Z-test. The Chi-squared test is a more general test for conversion data, because it can work with multiple conversion events and multiple funnels being tested (A/B/C/D/..).

 


Kickstarter: The Road Chose Me Volume 2 !
The Road Chose Me — Wed, 26 Feb 2020 01:45:48 GMT
I’ve been extremely busy this winter writing my next book – ‘The Road Chose Me Volume 2: Three years and 54,000 miles around Africa’. It’s almost finished, and I’ve launched a Kickstarter to get...


Digital Garden Terms of Service
swyx's site RSS Feed — Tue, 25 Feb 2020 02:00:00 GMT
This is my attempt to explicitly define a not-legally-binding "terms of service" for people who peruse Digital Gardens, and the people who Learn in Public with them.


Metrics, Logs, and Traces in JavaScript Tools
swyx's site RSS Feed — Mon, 24 Feb 2020 06:40:36 GMT
The DX of JavaScript Developer Tools could be better if we added Metrics, Logs, and Traces


Fixing Up the Svelte Community Site
swyx's site RSS Feed — Mon, 24 Feb 2020 02:00:00 GMT
Adding GitHub Actions and Updating Data Dependencies


A/B testing and the t-test
Bytepawn - Marton Trencseni — Sun, 23 Feb 2020 01:00:00 GMT
The t-test is better than the z-test for timespent A/B tests, because it explicitly models the uncertainty of the variance due to sampling. Using Monte-Carlo simulations I show that around N=100, the t-test becomes the z-test.

 


Using DEV.to as a CMS
swyx's site RSS Feed — Sat, 22 Feb 2020 02:00:00 GMT
Blog on DEV.to, publish on your own domain, using the DEV.to API!


Serverless Functions are Stateful
swyx's site RSS Feed — Fri, 21 Feb 2020 08:10:18 GMT
A reminder that serverless functions actually have a lot of state, and how the "function" analogy breaks down when you look through the abstraction


Good Enough
swyx's site RSS Feed — Fri, 21 Feb 2020 02:00:00 GMT
In general, you move faster and feel a lot less stress once you realize - You don't need "the best", you just need "good enough".


React Distros
swyx's site RSS Feed — Wed, 19 Feb 2020 02:00:00 GMT
Why we no longer have frontend framework wars, and musing on the state of React metaframeworks today and tomorrow


The Platinum Rule
swyx's site RSS Feed — Tue, 18 Feb 2020 02:00:00 GMT
You've heard of the Golden Rule? "Treat others as you want to be treated." Introducing the Platinum and Silver Rules.


Requirements change for the better
Dan Palmer — Mon, 17 Feb 2020 02:00:00 GMT
I’m an armchair space enthusiast – I like to watch new launches but I know very
little about rockets. Recently there’s been a lot of renewed interest in landing
on the moon which is very exciting, and also a lot of press coverage of NASA’s
Commercial Crew programme returning manned spaceflight capability to the United
States.
Between these two advances, there have been many pointing out the decades where
we as a society, and the US in particular, were going backwards. The Moon
landing was in the 60s, but we stopped going in the 70s, and the US lost the
ability to launch humans into space in 2011 at the end of the Shuttle programme.


Avoiding Flash of Unthemed Code
swyx's site RSS Feed — Mon, 17 Feb 2020 02:00:00 GMT
If your site has a dark mode or custom theme, you might have a flash of the default theme before JavaScript loads. Here is the solution.


Branding Bundles
swyx's site RSS Feed — Sun, 16 Feb 2020 02:00:00 GMT
The best way to communicate a group of benefits is to slap a label on it.


How to Contribute to Open Source Frameworks
swyx's site RSS Feed — Sat, 15 Feb 2020 02:00:00 GMT
One of the best ways to level up in programming is to contribute to a high quality open source project, especially a framework.


A/B testing and the Z-test
Bytepawn - Marton Trencseni — Sat, 15 Feb 2020 01:00:00 GMT
I discuss the Z-test for A/B testing and show how to compute parameters such as sample size from first principles. I use Monte Carlo simulations to validate significance level and statistical power, and visualize parameter scaling behaviour.

 


Google Sheets v4 API with Netlify Dev
swyx's site RSS Feed — Fri, 14 Feb 2020 02:00:00 GMT
wiring up Google Sheets via a Netlify Function


2019 /r/ReactJS Survey Results
swyx's site RSS Feed — Thu, 13 Feb 2020 02:00:00 GMT



Clientside Webmentions
swyx's site RSS Feed — Wed, 12 Feb 2020 04:48:02 GMT
How you can enhance your blog with webmentions without adding heavy build times.


To Understand Concurrent React, Look Outside React
swyx's site RSS Feed — Wed, 12 Feb 2020 02:00:00 GMT
Table of Contents


Big L Notation
swyx's site RSS Feed — Mon, 10 Feb 2020 22:43:29 GMT
In this post I sketch out Big L notation, which plots your learning as a function of N years of experience, with P peers.


"No Code" vs RPA
swyx's site RSS Feed — Sun, 09 Feb 2020 02:00:00 GMT
RPA seems to be NoCode Enterprise Edition™ and the pricing disparity is wild. But there are real differences and I am trying to note them down here.


Intro to Node-RED
swyx's site RSS Feed — Sat, 08 Feb 2020 02:00:00 GMT
I dipped into my automation repo today and explored Huginn. It didn't really appeal so I looked for alternatives and found Node-RED. It uses a much more familiar toolchain (JS) and is draggy-droppy which I likey!


How To Deploy a Django App to Render.com
swyx's site RSS Feed — Fri, 07 Feb 2020 02:00:00 GMT
Messing around learning Django and deploying


Intelligence
Krishna's blog — Thu, 06 Feb 2020 02:00:00 GMT
This article challenges the idea of a 'general intelligence' and argues that success in various skills is determined by a combination of factors.


Scrollbar Shenanigans
swyx's site RSS Feed — Thu, 06 Feb 2020 02:00:00 GMT
Styling scrollbars for my space on the Internet


Beyond the Central Limit Theorem
Bytepawn - Marton Trencseni — Thu, 06 Feb 2020 01:00:00 GMT
In the previous post, I talked about the importance of the Central Limit Theorem (CLT) to A/B testing. Here we will explore cases when we cannot rely on the CLT to hold.

 


Cleaning up Tableau workbook files to remove the inline thumbnails
PREPEND — Wed, 05 Feb 2020 02:00:00 GMT
I like to track my Tableau visualization projects in git to protect against screwups and for curiosity of what previous versions looked like. Theoretically, it might help with collaborators working on the same workbook, but I’ve never actually done that or wanted to do that. I use live updates for my data sets so the .twb files don’t contain sensitive data and the xml is at least possible to get a sense of what changed. But Tableau stores thumbnail images of worksheets and dashboards as base64 encoded graphics inline the xml. This makes the files really big and I got tired of manually removing these so I came up with this script to clean out the unneeded elements.

Update 2020-02-25: fixed the shell script to add quotes around my my find parameter

Why use git with tableau

I’m trying to figure out better ways to increase reuse of data visualizations within my employer, so I thought that if I stored by Tableau workbooks in git it would be easier to share techniques and data sources used. I’m not sure how useful that is, but it is convenient to version control my workbooks on the off chance that I screw up harder than command+Z will fix. Or if I revisit an old project and want to see previous versions before the final version. And in general, just all the basic development use cases where version control comes in handy. Perhaps one day, I’ll live the dream to get a Pull Request with new functionality added by some helpful contributor. But for now, I’ll be happy with being able to email links to projects instead of sending file attachments.

Tableau added versioning back in 2016’s 9.3 but that doesn’t help me if I accidentally delete the file or want to diff between versions.

I don’t check in twbx files as they are huge and typically the data are not something I share easily. So I’m happy with just the xml-version twbx file checked in and then other users have to get the data themselves if they want to reproduce. All of my tableau work is internal, so space isn’t an issue for our internal GitLab servers, but it seems just wrong to waste that much space. I already feel bad about giant xml files, not to mention all my jupyter notebooks.

Finding a problem

A few months ago, I was about to check in a day’s work and I noticed my workbook was really big. Instead of 100kb, it was 9MB. I thought I had accidentally extracted the data instead of pulling directly from my data sources, but it was actually lots and lots of base64 encoded, inline xml images for each of my worksheets crammed into a <thumbnails> element with an individual <thumbnail> element for each worksheet. I had a lot of worksheets, maybe 30, so there was lots of xml. I tried deleting them from the workbook file and everything still opened up again, Tableau just regenerated the thumbnails. So they didn’t seem to be required unless I eventually published the workbook to Tableau Public or to a server.

Here’s an excerpt from superstore.twb, but mine looked pretty similar…

<thumbnails>
    <thumbnail height='192' name='Commission Model' width='192'>
      iVBORw0KGgoAAAANSUhEUgAAAMAAAADACAYAAABS3GwHAAAACXBIWXMAAA7DAAAOwwHHb6hk
      AAAgAElEQVR4nO29yZMbeZbn9/EV7nCsDiCA2BkLd+bC3FOd1dMzUxqp1XPpvsxIpoM0d+kg
      M5l0bB1kMtOY/gSZTtMHmWlMKs1oeqY6LbMrK6syycqFyeQWEYwIRCCAwOLYHDt80SEyWcnk
      FgxGMEiGf040BPjwfoA/99/3997v/QTf930CAk4o4nE7EBBwnAQBEHCi2VcA+L6PZVlH7cux
      4Y77tDuD+15zBl2aP3st4NXjXgB4gya/+n//HVevfMn69u797/J9vrxy5YnG8ivX+fSzz7l6
      9SuG7sEc8lyH69/feOTfx50qN9aLT7QzqK/z3/2P/wtjH3xvzP/2P/333CraD33vsFXi5t37
      x9yvbvLdevnpnA946ZB//Ic77NIZy7z97vuIAlSKefI7VXpjjz/94B0AWuU8364W8RH403cv
      8PFv/kA8nuTye5dR8Pl+dZu/+Iv/DABv3OfvP/sSRRJYuPgOt/7w9yTTWUrlKpPZNGNBI+q2
      sAjTrpSYmpqk2ujwyz/9gF6vT+HuLfJVm6SZYSLsciNfI5lMcX4uyqA3ZDe/xt2ShevAh+9d
      4N/+20+YmU6jRHO8cX4RgA/eWOT6Sol5rcbMmbcAuH39a9p9BzEU4c2zs/z2i68J+V188wK3
      r39NqzciFEmxFHveP0XAcXDvCaDEp/mLf/Aav/30Y76+sYbggw9s3F279+a//+z3hHWV6uq3
...10000 lines...
    </thumbnail>
</thumbnails>


In the superstore workbook included in Tableau Desktop 2019.4, thumbnails runs from lines 6452-8996 and increases the size on disk of the workbook from 463KB to 607KB. And since any change to the workbook completely regenerates the thumbnail image, leaving these in just really increases the size of the repo quite a bit, and quickly.

Looking for a solution

I don’t use server and rarely use public so just I was happy with smaller git projects. I deleted the thumbnails, checked in my project and forgot until the next time I was working on a tableau project. Then I remembered it all again, and became sad because of how boring it will be to do this forever.

I googled around a bit and read about others (like cmtoomey and sechilds)who were using tableau and github and automatically handling the stripping out of the thumbnails element. These approaches used python to run as a pre-commit hook. I set this up for my project and was happy. For a while.

The downside of this is that I had to have that python script included in every repo and I had to have an environment set up for python to run it. It wasn’t that big of a deal as most of my projects already have some python along with tableau, but I didn’t want to introduce python everywhere just to clean up these files.

An easier, quick hack

So today, I finally got around to writing a small shell script to strip out these elements. I do all my development either on a Mac or on Windows using the git bash shell that comes with git for windows. So a shell script would run even if I didn’t have python set up on that machine.

It’s a quick hack that uses awk instead of actually parsing the xml. I’m afraid it might break with future versions of Tableau, so I’m not comfortable setting it up as a pre-commit git hook. But one day, I’ll rewrite it. But until someone bugs me about it, this should be fine to run before I rebase and push to my org’s repos.

/bin/find . -name "*.twb" | while read fname; do awk '/<thumbnails>+$/,/<\/thumbnails>+$/{next}1' "$fname" > temp.tmp && mv temp.tmp "$fname"; done


It’s just a simple one liner that cuts out the entire thumbnails element. It runs it for every single workbook, but if there were no changes then the output will be the same so it won’t register with git. Comically, I do have to adjust it between mac and windows because gitbash finds the windows “find” command instead of the unix find.

Reusability

I’m not sure what the right solution is to having reusable visualizations that can reliably run on multiple users. It’s interesting seeing software engineering concepts come into new worlds like data analysts and visualization designers. I wish tool designers would think about this more when designing their tool. One of my pet peeves with PowerBI is that everything is a giant binary workbook so version tracking and sharing is pretty hard. I’m not sure what I’d do if I had to have my methods audited and I was using PowerBI, pretty hard to show what changed or didn’t change.

If I really want something to be reusable, or move it to production, I try to recreate it in html+javascript so it can be widely used and clearly show the line of site between data, cleaning and processing, and visualization. But I still use tableau quite a bit for quick explorations. This script at least helps me organize projects a little better.

Future work


  Do a proper xml parsing
  Write a git pre-commit hook to only run this on changed workbooks


Related writing


  Chimed Altandush’s “Tableau Version Control is here to make your life easier”
  Abhishek Singh’s Version Control thread on tableau forums
  Chris Toomey’s awesome “Tableau & Github”



Jan 2020 Recap
swyx's site RSS Feed — Wed, 05 Feb 2020 02:00:00 GMT
a belated recap, highlighting my best writing over the past couple months


A/B testing and the Central Limit Theorem
Bytepawn - Marton Trencseni — Wed, 05 Feb 2020 01:00:00 GMT
When working with hypothesis testing, the desciptions of the statistical method often has normality assumptions. For example, the Wikipedia page for the z-test starts like this: "A Z-test is any statistical test for which the distribution of the test statistic under the null hypothesis can be approximated by a normal distribution". What does this mean? How do I know it’s a valid assumption for my data?

 


Scaling Coding Communities
swyx's site RSS Feed — Tue, 04 Feb 2020 02:00:00 GMT
People who can scale coding communities are 🔥 🔥 🔥


Friendcatchers
swyx's site RSS Feed — Mon, 03 Feb 2020 02:00:00 GMT
Patrick McKenzie's simple concept for making friends on the Internet.


A beginner hacker's guide to IPv6
Linux Engineer's random thoughts — Sun, 02 Feb 2020 21:22:02 GMT
I noticed recently how little I knew about IPv6. For someone working on broadband gateways, that's not something I'm most proud of. But I've learned a little in the past few months, and I thought I'd share it here.
How to read an IPv6 address: the zeroes are hidden
An …


My Favorite Podcasts
swyx's site RSS Feed — Sun, 02 Feb 2020 02:00:00 GMT
250 Podcasts I enjoy listening to


A minimal REST API in Java
Notes on software development — Sat, 01 Feb 2020 02:00:00 GMT
There's a style of Java that is a joy to write. This post will cover
how to set up a basic PostgreSQL-integrated REST API using
Jersey and
JOOQ in a style not dissimilar to Flask and
SQLAlchemy in Python.
In particular, we'll try to avoid as much runtime
reflection/class-loading as possible. This will make the application
less flexible but easier to debug and understand.
I'd appreciate pointers in email if you see anything weird or can fix
any of my bugs.
Dependencies
Install Maven, a recent
JDK, and PostgreSQL.
Copy the following into pom.xml to tell Maven about Java
dependencies:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>api</groupId>
  <artifactId>api</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <maven.compiler.source>13</maven.compiler.source>
    <maven.compiler.target>13</maven.compiler.target>
  </properties>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <compilerArgs>
            <arg>-Xlint:all,-options,-path</arg>
          </compilerArgs>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.6.0</version>
        <configuration>
          <mainClass>api.Main</mainClass>
        </configuration>
      </plugin>
    </plugins>
  </build>

  <dependencies>
    <dependency>
      <groupId>org.glassfish.jersey.containers</groupId>
      <artifactId>jersey-container-jetty-http</artifactId>
      <version>2.30</version>
    </dependency>

    <dependency>
      <groupId>org.jooq</groupId>
      <artifactId>jooq</artifactId>
      <version>3.12.3</version>
    </dependency>

    <dependency>
      <groupId>org.jooq</groupId>
      <artifactId>jooq-meta</artifactId>
      <version>3.12.3</version>
    </dependency>

    <dependency>
      <groupId>org.postgresql</groupId>
      <artifactId>postgresql</artifactId>
      <version>42.2.9</version>
    </dependency>

    <dependency>
      <groupId>org.glassfish.jersey.inject</groupId>
      <artifactId>jersey-hk2</artifactId>
      <version>2.30</version>
    </dependency>

    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-core</artifactId>
      <version>1.2.3</version>
    </dependency>

    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.30</version>
    </dependency>

    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.2.3</version>
    </dependency>

    <dependency>
      <groupId>org.glassfish.jersey.media</groupId>
      <artifactId>jersey-media-json-jackson</artifactId>
      <version>2.30</version>
    </dependency>

    <dependency>
      <groupId>javax.persistence</groupId>
      <artifactId>javax.persistence-api</artifactId>
      <version>2.2</version>
    </dependency>

    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>1.18.10</version>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>com.fasterxml.jackson</groupId>
      <artifactId>jackson-bom</artifactId>
      <version>2.10.2</version>
      <type>pom</type>
    </dependency>
  </dependencies>
</project>

Now run mvn install to download and configure all dependencies.
Project setup
The Main class will be our entrypoint
within src/main/java/api/Main.java.
It will handle loading configuration, setting up the application
server, and starting it.
package api;

import java.io.InputStream;

import api.app.Application;
import api.app.Config;

public class Main {
  public static void main(String[] args) {
    try {
      var cfg = new Config();
      var server = new Application(cfg);
      server.start();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

The Config class in
src/main/java/api/app/Config.java will contain a few
hard-coded settings for now. In the future it could be read from a
file.
package api.app;

import java.io.InputStream;
import java.time.Duration;
import java.util.Properties;

public final class Config {
  public final String server_address = "http://localhost";
  public final int server_port = 7780;

  public final String db_connection = "jdbc:postgresql://localhost/todo";
  public final String db_username = "todo";
  public final String db_password = "todo";
}

And finally the Application class
in src/main/java/api/app/Application.java will handle
loading a PostgreSQL connection, registering the class path to look
for Jersey routes/controllers, registering the PostgreSQL connection
in the dependency injection controller and starting the Jersey
controller.
package api.app;

import javax.ws.rs.core.UriBuilder;

import org.glassfish.jersey.internal.inject.AbstractBinder;
import org.glassfish.jersey.jetty.JettyHttpContainerFactory;
import org.glassfish.jersey.server.ResourceConfig;
import org.slf4j.LoggerFactory;

import api.dao.Dao;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;

public class Application {
  private static final Logger logger = (Logger) LoggerFactory.getLogger(Application.class);
  private static final Logger rootLogger = (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
  static {
    rootLogger.setLevel(Level.INFO);
  }

  Config cfg;

  public Application(final Config _cfg) {
    cfg = _cfg;
  }

  private void addShutdownHook(final Runnable hook) {
    Runtime.getRuntime().addShutdownHook(new Thread(hook));
  }

  public void start() {
    var dao = new Dao(cfg.db_connection, cfg.db_username, cfg.db_password);
    try {
      dao.initialize();
    } catch (Exception e) {
      e.printStackTrace();
      return;
    }
    addShutdownHook(() -> {
      try {
        dao.close();
      } catch (java.sql.SQLException e) {
        e.printStackTrace();
      }
    });

    var resourceConfig = new ResourceConfig();
    resourceConfig.packages("api.controller");
    resourceConfig.register(new AbstractBinder() {
      @Override
      protected void configure() {
        bind(dao).to(Dao.class);
      }
    });

    var baseUri = UriBuilder.fromUri(cfg.server_address).port(cfg.server_port).build();
    var server = JettyHttpContainerFactory.createServer(baseUri, resourceConfig);
    logger.info("Started listening on {}:{}", cfg.server_address, cfg.server_port);
  }
}


  I couldn't figure out a reasonable way to avoid the class path
  registration for routes.
  

  

  It's also important to note that the AbstractBinder
  appears to search the class path implicitly for any available
  dependency injection controller. I'd rather we had specified it
  explicitly but I'm not sure how. It will succeed because we
  installed
  HK2 as a dependency
  (see pom.xml).
With the Application code finished, we'll need to build
out the referenced Dao and controller classes.
Dao
The Dao class in
src/main/java/api/dao/Dao.java will enclose the
connection to PostgreSQL via JOOQ.
package api.dao;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

import org.jooq.DSLContext;
import org.jooq.SQLDialect;
import org.jooq.impl.DSL;

public class Dao {
  private Connection conn;
  private String url;
  private String username;
  private String password;

  public Dao(final String _url, final String _username, final String _password) {
    url = _url;
    username = _username;
    password = _password;
  }

  public void initialize() throws SQLException {
    conn = DriverManager.getConnection(url, username, password);
  }

  public void close() throws SQLException {
    conn.close();
  }

  public DSLContext getDSLContext() {
    return DSL.using(conn, SQLDialect.POSTGRES);
  }
}

And this will be enough to use in our controller. But let's take a
moment to talk about the data model.
Data
This API will return results from a TODO list. The database should
store each TODO item and a timestamp of completion, if completed.
We'll start by creating a database and user for the application:
$ sudo su postgres
postgres $ psql
postgres=# CREATE DATABASE todo;
postgres=# CREATE USER todo WITH PASSWORD 'todo';
postgres=# GRANT ALL ON DATABASE todo TO todo;

Then we'll write an initial migration:
$ cat migrations/1_init.sql
CREATE TABLE todo_item (
  id BIGSERIAL NOT NULL PRIMARY KEY,
  item TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at TIMESTAMPTZ
);

And a helper script for running migrations:
$ cat scripts/migrate.sh
#!/usr/bin/env bash

set -e

export PGPASSWORD=todo

for file in $(ls migrations); do
    echo "Running migration: $file"
    psql -U todo -f "migrations/$file"
done

Run it:
$ chmod +x ./scripts/migrate.sh
$ ./scripts/migrate.sh
Running migration: 1_init.sql
CREATE TABLE

And let's add some data:
$ sudo su postgres
postgres $ psql -U todo
todo=# INSERT INTO todo_item (item) VALUES ('My note');

Now we're ready to model the data in Java.
Models
While it's possible to have JOOQ generate Java data
classes (or
POJOs) by reading the database schema, the generated class cannot be
directly serialized to a JSON string.
So for each table (there's only one) we'll write a class with fields
for each column. We'll use the Java Persistence
API (JPA)
to annotate the class and fields so JOOQ will know how to deserialize
query results into an instance of the model.
We'll use Lombok to label the whole
object as Data so that getter and setter methods are
generated automatically for each private field. And we'll use a
Jackson annotation to label
the JSON field name of each column.
This is the TodoItem class in
src/main/java/api/model/TodoItem.java:
package api.model;

import java.time.OffsetDateTime;

import javax.persistence.Column;
import javax.persistence.Id;
import javax.persistence.Table;

import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonProperty;

import lombok.Data;

@Data
@Table(name = "todo_item")
public class TodoItem {
  @Column(name = "id")
  @JsonProperty("id")
  @Id
  private long id;

  @Column(name = "name")
  @JsonProperty("name")
  private String name;

  @Column(name = "created_at")
  @JsonProperty("createdAt")
  @JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ssZ")
  private OffsetDateTime createdAt;

  @Column(name = "completed_at")
  @JsonProperty("completedAt")
  @JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ssZ")
  private OffsetDateTime completedAt;
}


  The JSON format specifications for the timestamp fields aren't
  actually working. The formatted JSON returns a giant object and I
  haven't figured out how to get it to serialize to the RFC3339 string
  yet.
We're set! The last step is a simple controller to return a list of
TODO items.
The /items controller
In the ItemsController class in
src/main/java/api/model/ItemsController.java we'll inject
the Dao object and use it to return a page of TODO items
as JSON.
package api.controller;

import java.util.List;

import javax.inject.Inject;
import javax.persistence.Table;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import org.jooq.DSLContext;

import api.dao.Dao;
import api.model.TodoItem;

@Path("items")
public class ItemsController {
  @Inject
  Dao dao;

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public List<TodoItem> getServers() {
    DSLContext dslCtx = dao.getDSLContext();
    Table table = TodoItem.class.getAnnotation(Table.class);
    return dslCtx.select().from(table.name()).fetch().into(TodoItem.class);
  }
}


  There's some more implicit magic here when we return a list of
  TodoItems. Since we marked the endpoint as producing
  JSON, and since Jackson is in our class path, Jersey will
  automatically use Jackson to serialize the list to JSON.
  

  

  The API is quite nice but I could do without the automatic
  class-loading magic.
Now we're ready to build, run and test.
Building and running
$ mvn clean compile
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------< api:api >-------------------------------
[INFO] Building api 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ api ---
[INFO] Deleting /Users/philipeaton/tmp/test/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ api ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/philipeaton/tmp/test/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ api ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 6 source files to /Users/philipeaton/tmp/test/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.198 s
[INFO] Finished at: 2020-02-01T17:07:14-05:00
[INFO] ------------------------------------------------------------------------
$ mvn exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------< api:api >-------------------------------
[INFO] Building api 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ api ---
17:06:53.793 [api.Main.main()] INFO org.eclipse.jetty.util.log - Logging initialized @2017ms to org.eclipse.jetty.util.log.Slf4jLog
17:06:54.378 [api.Main.main()] INFO org.eclipse.jetty.server.Server - jetty-9.4.17.v20190418; built: 2019-04-18T19:45:35.259Z; git: aa1c656c315c011c01e7b21aabb04066635b9f67; jvm 13+33
17:06:54.425 [api.Main.main()] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@3943a159{HTTP/1.1,[http/1.1]}{0.0.0.0:7780}
17:06:54.425 [api.Main.main()] INFO org.eclipse.jetty.server.Server - Started @2651ms
17:06:54.425 [api.Main.main()] INFO api.app.Application - Started listening on http://localhost:7780

In a new terminal curl the endpoint:
$ curl localhost:7780/items | jq
[
  {
    "id": 1,
    "name": null,
    "createdAt": {
      "offset": {
        "totalSeconds": -18000,
        "id": "-05:00",
        "rules": {
          "transitions": [],
          "transitionRules": [],
          "fixedOffset": true
        }
      },
      "dayOfWeek": "SATURDAY",
      "dayOfYear": 32,
      "nano": 594440000,
      "year": 2020,
      "monthValue": 2,
      "dayOfMonth": 1,
      "hour": 17,
      "minute": 8,
      "second": 0,
      "month": "FEBRUARY"
    },
    "completedAt": null
  }
]

And we're done!
I really enjoy using Java for REST APIs, avoiding Spring and Play. Use simple but mature libraries that are no more difficult to cobble together than everything you must do in Go or Flask for a REST API. vs Go you get generics and vs python you get safetyhttps://t.co/twmjZprow6
— Phil Eaton (@phil_eaton) February 1, 2020
 


Svelte as an Eleventy Template Engine
swyx's site RSS Feed — Sat, 01 Feb 2020 02:00:00 GMT
Svelte is a really nice authoring format for HTML components. I wanted to explore if I could extend Eleventy to use it.


Optimizing waits in Airflow
Bytepawn - Marton Trencseni — Sat, 01 Feb 2020 01:00:00 GMT
Sometimes I get to put on my Data Engineering hat for a few days. I enjoy this because I like to move up and down the Data Science stack and I try to keep myself sharp technically. Recently I was able to spend a few days optimizing our Airflow ETL for speed. 

 


How rollup-plugin-svelte Works
swyx's site RSS Feed — Fri, 31 Jan 2020 02:00:00 GMT
Svelte is often thought of as a compiler. But really it is a compiler within a bundler.


Line Lengths
swyx's site RSS Feed — Thu, 30 Jan 2020 02:00:00 GMT
Reflecting on what I have learned about CSS Units and Line Lengths


Documentation Levels
swyx's site RSS Feed — Wed, 29 Jan 2020 02:00:00 GMT
People can't use your code without docs. People might get overwhelmed with too many docs. How can we match the maturity of docs to the maturity of the project?


Failing to use HSL to do Palette Theming
swyx's site RSS Feed — Tue, 28 Jan 2020 02:00:00 GMT
Learning from Slack and Refactoring UI to create color palettes, then using HSL colors to do customizable theming with CSS variables


Eponymous Laws
swyx's site RSS Feed — Mon, 27 Jan 2020 02:00:00 GMT
Interesting Two Word ideas with names of people on them


Why I Enjoy Svelte
swyx's site RSS Feed — Sun, 26 Jan 2020 02:00:00 GMT
Reasons I enjoy Svelte, despite not using it for work.


SQL best practices for Data Scientists and Analysts
Bytepawn - Marton Trencseni — Sun, 26 Jan 2020 01:00:00 GMT
My list of SQL best practices for Data Scientists and Analysts, or, how I personally write SQL code. I picked this up at Facebook, and later improved it at Fetchr. 

 


Two Words
swyx's site RSS Feed — Sat, 25 Jan 2020 02:00:00 GMT
The best way to sell to, brand, persuade, or inspire people is to condense your idea down to Two Words.


Svelte Zen Garden
swyx's site RSS Feed — Fri, 24 Jan 2020 02:00:00 GMT
reigniting the Zen Garden era with modern technology


How I write SQL code
Bytepawn - Marton Trencseni — Fri, 24 Jan 2020 01:00:00 GMT
This is a simple post about SQL code formatting. Most of this comes from my time as a Data Engineer at Facebook. 

 


Optimistic, Offline-First Apps
swyx's site RSS Feed — Thu, 23 Jan 2020 02:00:00 GMT
This is my attempt adapting an Amplify DataStore tutorial to Svelte


Writing an Authentication Store in Svelte
swyx's site RSS Feed — Wed, 22 Jan 2020 02:00:00 GMT
This is my attempt today wrapping an Authentication workflow into a Svelte Store


Automated Data Scraping with Github Actions
swyx's site RSS Feed — Tue, 21 Jan 2020 02:00:00 GMT
A neat trick I discovered from Mikeal Rogers


Collapsing Layers
swyx's site RSS Feed — Mon, 20 Jan 2020 02:00:00 GMT
As Moore's Law ends, devices multiply, and software becomes critical to life, we must take another look at our full stack for continued gains in efficiency, reliability and security.


Best Practice Open Source Repo Setup
swyx's site RSS Feed — Sun, 19 Jan 2020 02:00:00 GMT
Quick and simple ways to set up Open Source Repos with Best Practices


writing Advice
swyx's site RSS Feed — Sat, 18 Jan 2020 02:00:00 GMT
Some thoughts on writing your first few writings


CFP Advice
swyx's site RSS Feed — Sat, 18 Jan 2020 02:00:00 GMT
Some thoughts on writing your first few CFPs


How To Learn In Private
swyx's site RSS Feed — Fri, 17 Jan 2020 02:00:00 GMT
Of course I don't think that everything should be public. I don't even think everyone should Learn In Public. The majority of the time you are still learning in private. Here are some thoughts on how to do it well.


Enron: The Smartest Guys in the Room
Dan Palmer — Tue, 14 Jan 2020 02:00:00 GMT
I’ve been reading this extensive breakdown by Bethany McLean and Peter
Elkind of Enron’s collapse after a colleague’s recommendation
(based on my enjoyment reading Bad Blood). I found it
fascinating how much of the classic image I have of corporate greed stems from
the relatively recent collapse of Enron in 2001. Since I just missed the Enron
collapse, being about ten years old at the time, I had assumed that these ideas
had existed for much longer, but during the bull market of the 90s the image had
yet to fully form.


How to identify yourself, when yourself changes? Neal Stephenson's PURDAH.
PREPEND — Mon, 13 Jan 2020 02:00:00 GMT
I try to read every book by Neal Stephenson because he writes characters that I wish I could be, or perhaps are the closest to from any other characters in fiction. I was excited when Fall; or, Dodge in Hell came out. So excited that I accidentally ordered two hard copies, received one as a gift, and bought the audible version to listen to on my commute. The book is great in general, but specifically it introduces something called a “Personal Unseperable Registered Designator for Anonymous Holography” that all the characters call by its acronym, PURDAH. I’m curious as to how close this is to existing.

The PURDAH is a method in the novel that uniquely identifies a person using “AI magic” of all a person’s inputs to identify them based on what they do, how they type, everything they do digitally. More importantly, it doesn’t require anything conscious, and because the AI magic self updates its code all the time and replicated across a blockchain or something to prevent bad acting, a person’s PURDAH persists across consciousnesses through death and rebirth. That would be convenient.

I’m interested in proof systems for identifying myself, or identifying others. I’ve always liked reading about them, playing with keys, messing around with cryptographic systems. There’s just something neat about proving that I am me, and being able to rely on someone else being someone else. And so many programming problems rely on sound authentication to enable authorization to enable everything not going crazy.

The simplest identity is the password, or just some secret that you and only you know. There’s quite a few problems with passwords today, not to mention when our consciousnesses are moving between worlds, in the order I think they are important.


  They are hard to remember - I think this is the root problem for many other weaknesses: writing down passwords, sharing passwords, reusing passwords, weak passwords
  They are shared and people have to know a password to work - a system has to be able to check a password. Theoretically, every competent system should hash passwords and never store them and promise never to peak, but at the end of the day, if you send a password to a site you have to trust that they don’t do something bad with it. And you have to trust that their security is good enough to not get compromised. Public Key infrastructure gets around this somewhat, but you have to eventually trust someone or limit interactions to trusted out of band key exchanges and then how do you prove that the stranger you meet in the parking lot is really the strange you expect?
  They are hard to protect - assuming I eventually store the password somewhere/ Chrome, Firefox, MacOS have free password managers. Sites like 1password.com make money from managing password sets). Open source software like KeePass encrypts lots of passwords with a single password. Hardware devices like YubiKey rely on a physical thing. Protection seems to be gotten better, or at least more expensive, once crypto currency wallets made losing passwords or passwords stolen a lot more expensive to screw up.
  They don’t reliably work. I like biometric devices like thumb readers and face scanners, but they don’t always work and that would be a big problem if my digital life depended on it. Or even if just a few thousand dollars in a wallet didn’t work.


When I read analysis of password breaches (like Scheiner, haveibeenpwned]) it seems like the reason passwords are so weak boil down to very reasonable human weaknesses. I don’t think the reason the four most common passwords (at least according to the 1995 modern film classic, Hackers) isn’t because people are stupid, but probably because it’s harder and harder to make unique, secure passwords as we use more and more sites. I just looked at my Chrome saved passwords, and I have 182. And I don’t even save passwords for real stuff in Chrome.

I have friends who use password managers, but they get breached and then you start over (lastpass in 2015 and 2011, OneLogin in 2017).

So the idea of a magical way to identify me just based on what I do online is really appealing. It’s sort of like gait analysis and there’s software to authenticate based on typing patterns (which interestingly was part of epiphyte2’s PKI-based authentication scheme in Stephenson’s Cryptonomicon).

I like reading about the future early in science fiction, and Stephenson has been pretty good I think about predicting stuff years before it’s mainstream (multiverse, optical computers, living in international shipping units, vagina darts, glass knives, garbage gyres, cryptocurrencies, artificial sovereign hosting companies, cryonics, kindles, drones). I hope he’s right on this as well.

Related topics

Fred Saberhagen’s short story “What Do You Want Me to Do to Prove I’m Human Stop” in the Battlefields Beyond Tomorrow and in Analog October 1974 - short story about spaceships where a pilot must prove that he’s a human and not a robot pretending to be a human. I guess this was originally titled “Inhuman Error.”
Inhuman Conditions - is a game where one player pretends to be a cop trying to figure out if the other player is a human or robot. By the creators of secret hitler. Neat that it’s print and play for free, but I funded this back in October, 2018 and was supposed to arrive in September, 2019.

Related writings on PURDAHs


  http://xditao.blogspot.com/2019/06/introducing-purdah.html
  https://lareviewofbooks.org/article/nonexistence-seems-preferable-post-truth-feed-identity-and-the-npc-afterlife-in-neal-stephensons-fall-or-dodge-in-hell/



Oracle Hyperion EPM 11.2 - first impressions
Subclassed — Fri, 10 Jan 2020 19:37:53 GMT
Last month, Oracle finally released the long-awaited on-premises release of EPM 11.2, rumoured to be the last hurrah for the venerable suite. After a couple of test installations, I have a few thoughts from an infrastructure perspective.
 First, the bad news:

This is very much a .0 release, in many ways the first since 11.1.2.0 (the last time so much of the infrastructure had substantially changed). It is rough in quite a few places, and configuration is not for the faint of heart.
I have encountered what, to me, looks like a blocking bug in multi-server configurations. I filed a support request with Oracle and will see how it goes. At the moment I simply wouldn't be able to recommend it in situations where high-availability is necessary.
Even after configuration, it is harder to operate by non-techies than in the past. Even the "start" and "stop" links provided, which used to be ok for single-machine use at least, are not sufficient anymore - some extra scripting will be necessary to make it useable even in the most basic scenarios.
Documentation has not fully caught up with 12c, particularly in the area of additional configuration tasks for Financial Close Manager (which was already pretty lacking). Some of the additions, to deal with the now-necessary Repository Creation Utility, are ambiguous.
Weblogic seems hungrier for memory than in previous releases. This might be due to 12c, or to the fact that it's now using Java 8 under the hood - I had already measured an increase in this area when replacing Java 6 with 7 on 11.1.2.4.
Essbase seems to have been shipped almost unmodified from the latest 11.1.2.4 version. Among other things, this makes it currently impossible to do a full-SSL configuration if Essbase is in the picture. It's disappointing, particularly considering how Essbase customers are typically among the few who often require fully-encrypted solutions.
The documentation seems to state that running any "Configure Database" task more than once will end in failure. I suspect this is true only for certain components, but I have not had the time yet to explore this area. This makes it even harder than before to migrate databases after configuration, so make sure you take design choices that will stand the test of time.
EPMA lives! Well, not really - the product is gone, but somehow it's still avilable in the Provisioning screen and you can assign roles for it. 🤷🏻‍♂️
This is the end of the line for IIS stalwarts. Using IIS as front-end webserver will simply not work (according to docs).
It's not particularly clear whether Oracle 18c and 19c are officially supported as database. In practice they seem to work (after all, they are really 12.2.0.2 and .03 rebranded), but it would be nice if the matrix was clarified.

But there are also some good news!

The products themselves are basically the same, with the occasional extra feature. Users shouldn't need any substantial retraining, with the notable exception of any EPMA or Interactive Reporting power-users still around.
The 12c stack, in theory, unlocks a number of attractive possibilities for system administrators and implementors - containers, no-downtime patching, and so on. It will take some time to fully unpack how EPM can take advantage of these goodies.
Recent Windows Server releases are supported.
In my test, everything seems to work even with the free editions or Oracle and SQLServer (XE 18c and Express 2017), which is nice for consultants and other professionals.
Both 12c and Java 8 support modern crypto standards, which should make things less awkward when risk-assessing and pentesting.
Java 8 brings a lot of improvements to the language and the ecosystem. Whole classes of problems have been removed (MaxPermGen, anyone?), which should make services more reliable and performant. Custom integrations will also be easier and faster to develop and run.

In the past, I would have expected such a release to be followed by a quick stream of fixes, necessary to make it actually fit for production use. We will have to see if it is going to be the case here, considering how long it took to debut and the overall tone of the Oracle roadmap (mostly cloud-focused). Should Oracle really commit to improving the on-premise, this release would soon prove itself as a substantial improvement over 11.1.2.4.
If you'd like to know more and discuss your options in the EPM space, make sure to get in touch with us.


Small team planning
Bytepawn - Marton Trencseni — Fri, 10 Jan 2020 01:00:00 GMT
I’ve worked at 5-10 different organizations, most of them were startups or startuppy companies. I’ve done a lot of planning in small teams, and also taken part in company-wide leadership planning. Here I will describe what has worked well for me in small team settings, focusing on time estimation. 

 


The Many Jobs of JS Build Tools
swyx's site RSS Feed — Mon, 06 Jan 2020 02:00:00 GMT
A discussion of why JS developers use build tools like Webpack and what we do with them, for new JS developers.


My Setup: Passwords, 2FA, and Yubikeys
Nemo's Home — Sat, 04 Jan 2020 02:00:00 GMT
I upgraded my encryption setup recently, so I thought I should write about it,
just in case it is helpful to someone else. As a security professional, I have a
different threat model from most folks, and as such my setup does involve a bit
more complexity than what I’d recommend to everyone. But if you are an at-risk
individual (journalist, person holding hundreds of bitcoins or other digital
assets, activist) or if you are a linux user with a lot of free time - you might
consider duplicating some of this.



I’ll discuss some of the other approaches I’ve considered, and my thought
process around each choice I made. There are general recommendations at the
bottom of the post.

Passwords

I used to be on LastPass till 2017¹, when I migrated to [pass][pass] (“the
standard unix password manager”). With pass, each password lives inside of a gpg
encrypted file whose filename is the title of the website or resource that
requires the password.

pass automatically manages a git repository for you, which I sync against
GitLab. The one downside of using pass is that the list of my domains is
visible to my hosting provider. ² In the past, I’ve set this up against
Keybase Encrypted Git (Keybase doesn’t get to see even the file list), and my
own git-server (only I get to see it).

I don’t push it to GitHub, since most of my stuff lives on GitHub anyway, and I
didn’t want to add my passwords there as well. GitLab uptime is decent enough
for my usecase³. Finally, my GitLab account is fairly locked down with:


  zero integrations or third-party apps
  no active personal tokens
  social signin disabled
  only yubikey SSH key configured


Mobile Passwords

There are 2 primary considerations I have:


  Using pass involves GPG keys, and I can’t use hardware GPG keys on my
current device (iPhone SE).
  I don’t want to sync all my passwords to my phone. I have a limited number of
applications on my device, and syncing all passwords doesn’t make sense.


For the first issue, I am forced to use a PGP key in software on
[passforios][passforios]. If you are on Android, take a look at
[OpenKeychain][okc], and Fidesmo/Yubikey NFC.

For the second issue, I use pass cp, and the .gpg-id file, which allows me
to maintain a mobile-sync directory inside my pass git repository encrypted
against a different key. From the pass documentation:

~/.password-store/.gpg-id


  Contains the default gpg key identification used for encryption and
decryption. Multiple gpg keys may be specified in this file, one per line. If
this file exists in any sub directories, passwords inside those sub
directories are encrypted using those keys. This should be set using the init
command.


My ~/.password-store/mobile-sync/.gpg-id file holds 2 keys: My main encryption
key, and the key I’ve configured on my phone.

Unfortunately, I haven’t gotten it working well as a git submodule, so I have a
helper script that copies the encrypted password files from mobile-sync
subdirectory to a different repository (mobile-passwords.git). The script is
just 2 lines:

cp -r ~/.password-store/mobile-sync/*.gpg .
git-sync


It updates the git repository, and runs a sync to push any local changes to my
mobile-passwords repository. I can pull that on my passforios application. I
could also clone the entire repo, but the iOS app doesn’t work nicely with a
single-subdirectory approach.

GPG

pass relies on GPG, and as such I require a strong key setup. I have the
following:


  2xYubikey 4 (Doesn’t have NFC)
  Fidesmo Smartcard, currently unused


Both the Yubikeys are configured against my GPG Encryption key. I carry one of
the Yubikeys on my keyring with me. The backup Yubikey stays at my home.

I followed this guide
while configuring the same. As of now, switching between keys is not very
user-friendly, but future
GnuPG versions plan to fix it. The Yubikey holds:


  An encryption key
  A signing key
  An authentication key


I keep a copy of all these keys using
paperkey as per the same guide.
I have a subkey backup as well, since Yubikeys are known to fail⁴.

SSH

The Authentication key in my Yubikey is configured for SSH. I just need to
ensure that my GPG agent is configured for SSH as well:

export GPG_TTY="$(tty)"
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent


U2F

U2F lets me use a physical key as my second-factor on supported websites (as an
alternative to SMS/TOTP). I configure both of my Yubikeys for U2F wherever
possible (Twitter/AWS are notable exceptions and only support a single key). U2F
support for [OpenSSH][ssh-u2f] is coming soon. So you can soon authenticate to
your server with the Yubikey+PIN, and finish the 2FA with U2F by tapping the
key⁵.

Root-of-Identity

For most people, the root of identity comes down to ownership of their email
address. As such, it is very often the juiciest target for most attackers. I run
my mail against Migadu, a privacy friendly swiss
email-hosting service. They provide me a management layer for managing my
domains which uses my GMail account. (See FAQ for why). I also have 2FA (TOTP
only) configured on the Migadu management setup.

The domain is currently registered at a Indian registrar (which doesn’t offer
U2F, but I do have TOTP configured). I would have moved this to CloudFlare, but
CloudFlare doesn’t support the .in TLD yet.⁶

The email address used for registring the domain again is my GMail. My DNS is
configured on CloudFlare, which again uses my GMail and has appropriate 2FA
configured. (It doesn’t support U2F). So here’s the list of critical providers:


  
    
      Provider
      What can an attacker do
      Auth
      2FA
    
  
  
    
      Regisrar/Mitsu
      Change my nameserver, read any future email, reset passwords
      GMail/Password
      TOTP
    
    
      DNS/CloudFlare
      Change my MX records, read any future emails, reset passwords
      GMail/Password
      TOTP
    
    
      Email/Migadu
      Reset my email password, read my current emails, reset passwords
      GMail/Password
      TOTP
    
    
      GMail
      Reset passwords for the above 3 accounts
      Email/Password
      U2F
    
  


As you can see, I end up trusting GMail a lot here.

Recovery/Backup codes and Security Questions

I use randomly generated UUIDs as answers to security questions. Currently, I’m
storing these for various services within the same password store. As it stands,
I can’t get access to my password OR recovery token without my Yubikey and PIN.
The access Matrix looks like this:


  
    
      What
      Physical Access
      Additional Authentication
    
  
  
    
      Password
      Yubikey
      PIN
    
    
      U2F-2FA
      Yubikey
      Physical touch
    
    
      TOTP
      Phone
      TouchID
    
    
      Recovery Code
      Yubikey
      PIN
    
    
      Security Question answers
      Yubikey
      PIN
    
  


Failure Scenarios

There are lots of failure scenarios with such a setup, and while I’ve got a
pretty spotless record of not getting hacked - I’m not immune to screwups.
Here’s all the bad things that can happen:

Yubikey failures

If my Yubikey fails (or if I forget its PIN), I can’t access passwords on my
device. I still have access to commonly used passwords and my mail on my phone.
My backup Yubikey is kept safely at my home. If I lose both, I have the paperkey
backup at home (which I should store elsewhere).

Device failures

I have a current version of the password repository against 3 PCs, and a partial
version in my mobile. If all these 4 devices fail at once, I can still clone a
fresh version of the repository with my YubiKey (I would still have GitLab SSH
access). I might need to prepare better for this though, since configuring
GPG-SSH might not always be easy during an incident.

As a alternate scenario, my phone GPG key does have my GitLab password, so I can
clone the repo over HTTPS (with password) if needed.

Circular dependency against GitLab

My GitLab password is randomly generated and stored in the same password store
on GitLab. That is not too big of an issue, because I don’t need the GitLab
password anymore to clone my passwords repo, just my Yubikey (for SSH).

Lost Key

If I lose my key, the GPG card contains public info, including my email address,
which can be used to contact me. I have a [Tile bluetooth tracker][tile] on my
keychain to make it easier for me to find it.

Malware

A hardware key doesn’t protect you from all attacks. At the end of the day, my
passwords must be decrypted by the key and passed unencrypted back to my browser
(or editor). pass for eg, doesn’t protect against memory scraping attacks. If
I edit a password on an infected machine, it gets that password.

If my browser has a malicious extension, it already has keys to the kingdom. But
if I then log into a website, it does get access to that password additionaly.

xkcd 1200 famously illustrates this:



A password vault protected by a hardware key protects against some attacks:


  A malicious extension can’t sniff my vault passphrase, since I don’t have one
  The key can’t be exfiltrated from hardware.


However, a malware can connect to my authenticated GPG socket, and start
decrypting things. To prevent against that, I run my Yubikey in “touch-only”
mode, so it requires a “physical touch” before it actually does anything, even
if the PIN is cached. Customizability is
dependent on your Yubikey model.
But remember the xkcd warning - if I have a malware running on my device, it is
pretty much game over anyway. pass doesn’t prevent against memory scraping
attacks, and actually uses /dev/shm to store the temporary plain-text files
containing passwords. Ultimately, your identity is as secure as the device you
trust it with.

Improvements

If you have any suggestions for any of the below, I’m
happy to hear them.

Travel Plans

Since my backup key stays at home, how do I deal with long-term travel? This is
something I’m still figuring out. Do I take my backup Yubikey on my
longer-travels? Or should I setup a third-key before I do that? Chances of me
losing both the keys together are quite high, so I’m trying to avoid that.

Domain Ownership

I’d like to transfer my domain to
a registrar that supports U2F, likely
Namecheap since I already own some domains there⁷. If you use CloudFlare,
they should roll out
U2F support soon.

2FA Recovery Guides

I wish more organizations published what they consider as valid 2FA recovery
mechanisms. GitHub supports 2FA recovery by proof of SSH keys or Personal
Tokens; Migadu just needs a few domain names from your account, and lots of
services require proof-of-identity.

A lot of this is undocumented, and I wish organizations were more public about
this so users can take appropriate measures and understand their risk better.

Fidesmo Card

I’m planning to configure my Fidesmo card against my existing GPG/SSH key, so it
stays in my wallet to improve redundancy. Unfortunately, it is not supported on
iOS, so I plan to get a NFC reader/writer and test that out. This also helps
with travel plans a bit, since I’m less likely to lose my wallet
anecdotally(which also has [a bluetooth tracker][slim]).

U2F on iPhone

U2F support on Mobile Safari is non-existent. Brave recently added support for
the upcoming Yubikey 5Ci, which supports both USB-C and lightning. However, this
requires a special Yubikey SDK, which breaks the idea of U2F being
interoperable. The 5Ci is also quite costly at $70. I don’t know of any
application that is actually supporting GPG-over-Yubikey-over-lightning.

Compare this to Android where NFC based smartcards or Yubikeys just work. I’d
like that to happen with iPhones.

Full Disk Encryption using a Yubikey

It is possible to configure
Full Disk Encryption with Yubikeys,
but I haven’t tried it yet.

2FA on my email account

Migadu currently does not support 2FA on webmail access, just on
manage.migadu.com. This is very unfortunate, but I’m told this is planned soon
(January 2020).

Recovery/Backup codes and Security Questions

My current setup of saving recovery codes alongside passwords isn’t optimal, but
I don’t have a better way either. I’ve considered keeping my recovery codes on a
alternate password store (such as bitwarden, or keypassX), but I’ll have to
memorize the password, and setup a separate 2FA for it to be truly
fault-tolerant.

FAQ

Why do you have stuff configured on your GMail? Aren’t you anti-Google?

Despite all the flak that Google gets for privacy, their security team is pretty
awesome. Your account is pretty much unhackable once you are enrolled into their
Advanced Protection Program. The few security-sensitive places where I use it
are:


  Domain Registration
  Email Management
  DNS Configuration


Everywhere else, I use my actual domain (captnemo.in) to ensure nothing else
routes over GMail. Using GMail for the above 3 ensures that I don’t have a
circular dependency. If I were to lose my main email password, I can recover via
multiple ways:


  Change DNS to another email provider.
  Reset password via migadu admin panel.


Ensuring that either of these workflows do not rely on the same email account
I’ve just lost access to is vital. Another alternative is to use a
trusted-friend (ideally someone more paranoid than me) as a proxy for these
emails, and use their domain for managing these 2 services. Might get around to
it someday.

My GMail recovery email is set to my main account, so it creates a circular
dependency, but one that I actually want.

What do you recommend I use?


  Bitwarden for password management.
  2xHyperFIDO Mini U2F Keys configured for second
factor against as many accounts as possible.
U2F is not only safer, but much more convenient than TOTP/SMS based 2FA. For
iPhone/USB-C users, see
the Yubico website.
If you don’t like to pay the USB-C tax, there are cheap
USB-C to miniUSB
adapters that can work with the HyperFIDO key and fit on your keychain. If you
aren’t convinced on why this is a good idea, see
this guide. The
second key is just a backup key, and could be the primary key used by your
spouse, friends or co-workers.
  A PIN configured on all your SIMs. Instructions for
iPhone,
Android.
  Full-Disk-Encryption on all your devices. Instructions for
Windows,
Mac,
ArchLinux,
Fedora,
Ubuntu.
  Use randomly generated passwords everywhere. Trust your password manager on
this.
  Setup a PIN on your WhatsApp.
  If you own a lot of cryptocurrency, use a hardware wallet and put it in a bank
safe. Have a backup one, in another safe. You can put the PIN for those in
your password store. I haven’t researched enough to suggest you which
wallet(s).
  Get a SIM without an Aadhaar, to make SIM-Jacking attackes harder (applies in
India).
  Go through securityplanner.org, which gives
you personalized recommendations customized for our risk profile. I agree with
most of their recommendations⁸
  Signup for breach notifications against your email at
https://haveibeenpwned.com/.
  If you’d like to get off GMail, pay for FastMail.
Alternatively, if I know you in real-life, I’m happy to host your mail in my
Migadu account. (Only works if you know me well enough to trust me)


Why are you so paranoid?

I work in infosec. Breaking things comes naturally to me, and I plan for
defense-in-depth. Plus, I’d be a terrible security person if I got hacked.

Why not recommend open source keys instead?

Availability is a pain point, especially if you aren’t in the US. Even getting
my hands on a SoloKey was hard, despite backing it on KickStarter.


  OnlyKey also makes some claims regarding open source,
but I can’t find their schematics anywhere.
  SoloKey is great, and what I’d recommend, but
it doesn’t support OpenPGP yet.
  [NitroKey Start][nitrokey-start] is
apparently completely FOSS,
so you might wanna check that.


The HyperFIDO keys are compliant to the U2F/FIDO standards, and I’ve not faced
any issues while using them. They’re cheap and widely available. Unless you need
GPG, go for it.



Thanks to Giridharan, Santosh, and Akshay for reviewing drafts of this
and offering valuable suggesions. If you have any suggestions, happy to hear
them



[passforios]:
  https://mssun.github.io/passforios/
  “Open Source, no-network, minimalist pass client for iOS”
[okc]: https://www.openkeychain.org/
[pass]: https://passwordstore.org
[tile]: https://www.thetileapp.com/en-us/
[slim]:
  https://www.thetileapp.com/en-us/store/tiles/slim
  “I have the older version of the Tile Slim”
[ssh-u2f]:
  https://www.undeadly.org/cgi?action=article;sid=20191115064850
  “Does it really count as 2FA if both your SSH and U2F is the same device?”
[nitrokey-start]: https://shop.nitrokey.com/shop/product/nitrokey-start-6


  
    

      I moved away from Lastpass after Tavis Ormandy reported a RCE vulnerability
on their browser extension. Their wikipedia page mentions 2 breaches, and 3
security incidents. It has never undergone a security audit (unlike
bitwarden) and is not something I recommend anymore. ↩
    
    

      The pass-tomb extension bypasses
this limit and encrypts your filenames as well. ↩
    
    

      I have my own git server configured
as a fallback if it goes down. I ensure the same controls on my Git server
as Gitea, and it runs in my living room. ↩
    
    
      I lost my previous GPG key because my Yubikey stopped working ↩
    
    

      The jury is still out on whether this counts as an “independent second
factor”. ↩
    
    

      The domain is stuck in a legal limbo, because of an
ongoing case between my registrar and NIXI
(which runs the .in registry). If you have any suggestions/ideas, please
reach out. ↩
    
    

      Namecheap announced
U2F support
in April 2019, and while it was buggy at first, it has definitely improved. ↩
    
    
      The one major exception is lastpass, which I no longer recommend. ↩
    
  



Predictions for 2030
Ben Overmyer's Site — Fri, 03 Jan 2020 02:00:00 GMT
This post is inspired by a thread from Hacker News. I posted a comment there
with ten predictions for 2030. This is an extrapolation of that.
Global Trends and Events


Populism reaches a crescendo globally. Nascent populist governments become more entrenched, especially in North and South America.


China supersedes the United States as the preeminent global superpower.


A conflict between Russia and China comes seemingly out of nowhere, but is resolved quickly.


Electric vehicles become more mainstream, but still do not overtake ICE vehicles in sales or ownership.


Cannabis is de-scheduled in the United States. A minor economic boom results.


Tech hubs in the Midwest begin to overtake the old tech hubs in activity and population.


Housing shortages worsen globally.


Australia suffers a series of natural disasters that results in heavy depopulation.


Battery technology will see a significant breakthrough that results in much greater life and much lower weight, but it will not be affordable. Yet.


An economic "adjustment" occurs as trade between the United States and China is heavily curtailed.


Personal Trends and Events


I finally achieve financial independence.


I publish a book, but it's not the one I expect to write.


My health improves dramatically. My hobbies change as a result.


I start another hobby business, and this time it sticks around.


I make the switch to electric vehicle ownership and never look back.


My remaining hair disappears. My beard grows to compensate.


New friends and stronger relationships with existing ones enrich my life.


A personal tragedy changes my outlook on life.


My political views change as a result of world events.


I buy a house.




Books I read in 2019
Krishna's blog — Thu, 02 Jan 2020 02:00:00 GMT
A list of exceptional and good books read in 2019, along with a warning about a book to skip.


Pick Up What They Put Down
swyx's site RSS Feed — Wed, 01 Jan 2020 02:00:00 GMT

Translations welcome! (Português)

Provider	What can an attacker do	Auth	2FA
Regisrar/Mitsu	Change my nameserver, read any future email, reset passwords	GMail/Password	TOTP
DNS/CloudFlare	Change my MX records, read any future emails, reset passwords	GMail/Password	TOTP
Email/Migadu	Reset my email password, read my current emails, reset passwords	GMail/Password	TOTP
GMail	Reset passwords for the above 3 accounts	Email/Password	U2F

What	Physical Access	Additional Authentication
Password	Yubikey	PIN
U2F-2FA	Yubikey	Physical touch
TOTP	Phone	TouchID
Recovery Code	Yubikey	PIN
Security Question answers	Yubikey	PIN