Hacker News Personal Blogs 2019 | Top 100 Blogs

36th Chaos Communication Congress

Linux Engineer's random thoughts — Sat, 28 Dec 2019 14:00:00 GMT

I’m at CCC for the first time this year ! Here my notes for a few of the talks.

Tamago

In an ideal world, one could pick the programming language of their choice and always generate the optimal machine code. But we live in the real world, and it isn't …

In Defense of Hammers

swyx's site RSS Feed — Thu, 26 Dec 2019 02:00:00 GMT

Let's say you needed a multipurpose tool. Which of these would you pick?

Personal goaling

Bytepawn - Marton Trencseni — Sun, 22 Dec 2019 01:00:00 GMT

The meta-goal of goaling is to stretch yourself to achieve more, and to feel good about what you’ve achieved. Whatever happened this year, it’s always possible to achieve a lot more and feel better about yourself next year. To hijack a Feynman quote, there is plenty of room at the top.

The modern web is becoming an unusable, user-hostile wasteland

Omar Abid - Personal Blog — Fri, 20 Dec 2019 02:00:00 GMT

Some thoughts about the modern web

Put All Your Pelican Posts in One Org File

Beetle Space — Thu, 19 Dec 2019 10:00:00 GMT

Pelican is a static site generator. Out of the box it supports two formats: Restructured Text and Markdown. There exist plugins to support the Org format. For a while, I would write my posts in org mode, with one file per post. But I longed for a way to have …

Distributing malware with Google images, service workers and vegan Twitter bots

Tom Forbes — Sun, 15 Dec 2019 12:19:01 GMT

What started with a simple image search for “Skippers hat” quickly turned into investigating a clever malware distribution network utilizing service workers, which is somehow linked to a group of hardcore vegan twitter bots. The hat at the entrance of the rabbit hole was innocent enough: However, wh...

Pytorch in 2019

Bytepawn - Marton Trencseni — Thu, 12 Dec 2019 01:00:00 GMT

2019 was another big year for Pytorch, one of the most popular Deep Learning libraries out there. Pytorch has become the de facto deep learning library used for research thanks to it’s dynamic graph model which allows fast model experimentation. It’s also become production ready, with support for mobile and infrastructure tooling such as Tensorboard.

BHAGs

swyx's site RSS Feed — Wed, 11 Dec 2019 02:00:00 GMT

Recently, I asked tweeple:

Learn to drive the Pan-American Highway

The Road Chose Me — Mon, 09 Dec 2019 18:00:33 GMT

As many of you know, long ago – in fact before I had even heard the word “Overlanding” – I set out to drive my little Jeep Wrangler from the top of Alaska to...

Writing a lisp compiler from scratch in JavaScript: 6. an x86 upgrade

Notes on software development — Sun, 08 Dec 2019 02:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
5. LLVM system calls

This post upgrades the ulisp x86 backend from using a limited set of registers (with no spilling support) to solely using the stack to pass values between expressions.

This is a slightly longer post since we've got a lot of catchup to do to get to feature parity with the LLVM backend. Namely:

"Infinite" locals, parameters
Function definitions
Variable references
Arithmetic and logical operations
If
Syscalls

We'll tackle the first four points first and finish up with the last two. This way we can support the same fibonacci program that prints integers to stdout that we support in the LLVM backend.

As always the code is available on Github.

But first a digression into how this is suddenly easy for us to do with x86 and one-pass (sorta) code generation.

Stack-based languages

Stack-based languages have the extremely convenient attribute that values are (by default) stored on the stack, which allows a code generator targeting a stack-based language the option to omit handling register allocation. And as it happens, x86 has enough support to make it easy to treat as a stack machine.

As we build out the code generator for x86 as a stack machine we need to keep two commitments in mind:

Every expression must pop all arguments/operands
Every expression must store one result back on the stack

In the future, we may replace the second commitment. But for now it is more than enough.

Boilerplate

We'll start with the existing x86 backend code and strip all the implementation code:

const cp = require('child_process');
const fs = require('fs');
const os = require('os');

let GLOBAL_COUNTER = 0;

const SYSCALL_MAP = {
  darwin: {
    exit: '0x2000001',
    write: '0x2000004',
  },
  linux: {
    exit: 60,
    write: 1,
  },
}[os.platform()];

class Scope {}

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      if: this.compileIf.bind(this),
      ...this.prepareArithmeticWrappers(),
      ...this.prepareLogicalWrappers(),
      ...this.prepareSyscallWrappers(),
    };
  }

  prepareArithmeticWrappers() {}

  prepareLogicalWrappers() {}

  prepareSyscallWrappers() {}

  emit(depth, args) {
    if (depth === undefined || args === undefined) {
      throw new Error('Invalid call to emit');
    }

    const indent = new Array(depth + 1).join('  ');
    this.outBuffer.push(indent + args);
  }

  compileExpression(arg, scope, depth) {}

  compileIf([test, then, els], scope, depth) {}

  compileBegin(body, scope, depth, topLevel = false) {}

  compileDefine([name, params, ...body], scope, depth) {}

  compileCall(fun, args, scope, depth) {}

  emitPrefix() {
    this.emit(1, '.global _main\n');

    this.emit(1, '.text\n');
  }

  emitPostfix() {
    this.emit(0, '_main:');
    this.emit(1, 'CALL main');
    this.emit(1, 'MOV RDI, RAX'); // Set exit arg
    this.emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`);
    this.emit(1, 'SYSCALL');
  }

  getOutput() {
    const output = this.outBuffer.join('\n');

    // Leave at most one empty line
    return output.replace(/\n\n\n+/g, '\n\n');
  }
}

module.exports.compile = function(ast) {
  const c = new Compiler();
  c.emitPrefix();
  const s = new Scope();
  c.compileBegin(ast, s, 1, true);
  c.emitPostfix();
  return c.getOutput();
};

module.exports.build = function(buildDir, program) {
  const prog = 'prog';
  fs.writeFileSync(`${buildDir}/${prog}.s`, program);
  cp.execSync(
    `gcc -mstackrealign -masm=intel -o ${buildDir}/${prog} ${buildDir}/${prog}.s`,
  );
};

The prefix and postfix stays mostly the same as the original implementation. But we'll assume a couple of new helpers to get us in parity with the LLVM backend:

compileDefine
compileBegin
compileIf
compileCall
prepareArithmeticWrappers
prepareLogicalWrappers
prepareSyscallWrappers

The prepareArithmeticWrappers helper will define wrappers for arithmetic instructions. The prepareLogicalWrappers helper will define wrappers for logical instructions. And the prepareSyscallWrappers helper will define a wrapper for syscalls and generate builtins based on the SYSCALL_MAP entries.

Scope

Similar to our LLVM backend's Context and Scope helpers we'll define our own for the x86 backend. Since we're placing all locals on the stack, the two most important things Scope will do for us are:

Map identifiers to escaped strings
Store and increment the location of the local on the stack

Here's what it will look like:

class Scope {
  constructor() {
    this.localOffset = 1;
    this.map = {};
  }

  assign(name) {
    const safe = name.replace('-', '_');
    this.map[safe] = this.localOffset++;
    return safe;
  }

  symbol() {
    return this.localOffset++;
  }

  lookup(name) {
    const safe = name.replace('-', '_');
    if (this.map[safe]) {
      return { name: safe, offset: this.map[safe] };
    }

    return null;
  }

  copy() {
    const s = new Scope();
    // In the future we may need to store s.scopeOffset = this.scopeOffset + 1
    // so we can read outer-scoped values at runtime.
    s.map = { ...this.map };
    return s;
  }
}

compileExpression

An expression will be one of:

A function call (possibly a builtin like def or +)
A literal value (e.g. 29)
A reference (e.g. &c)
An identifier (e.g. my-var)

We'll handle compiling an expression in that order. If the AST argument passed to compileExpression is an array, we will call compileCall and return.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }
  }

If the AST is a number, we will push the number onto the stack and return.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }
  }

If the AST is a string that starts with & we will look up the location of the identifier after the &, push its location onto the stack and return.

We count on the Scope storing its offset from the "frame pointer", which we will later set up to be stored in RBP.

Locals will be stored after the frame pointer and parameters will be stored before it. So we'll need to add or subtract from the frame pointer depending on if we need a positive or negative offset from it.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }

    if (arg.startsWith('&')) {
      const { offset } = scope.lookup(arg.substring(1));
      // Copy the frame pointer so we can return an offset from it
      this.emit(depth, `MOV RAX, RBP`);
      const operation = offset < 0 ? 'ADD' : 'SUB';
      this.emit(depth, `${operation} RAX, ${Math.abs(offset * 8)} # ${arg}`);
      this.emit(depth, `PUSH RAX`);
      return;
    }
  }

Finally, we'll look up the identifier and copy the value (in its offset on the stack) to the top of the stack.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }

    if (arg.startsWith('&')) {
      const { offset } = scope.lookup(arg.substring(1));
      // Copy the frame pointer so we can return an offset from it
      this.emit(depth, `MOV RAX, RBP`);
      const operation = offset < 0 ? 'ADD' : 'SUB';
      this.emit(depth, `${operation} RAX, ${Math.abs(offset * 8)} # ${arg}`);
      this.emit(depth, `PUSH RAX`);
      return;
    }

    // Variable lookup
    const { offset } = scope.lookup(arg);
    if (offset) {
      const operation = offset < 0 ? '+' : '-';
      this.emit(
        depth,
        `PUSH [RBP ${operation} ${Math.abs(offset * 8)}] # ${arg}`,
      );
    } else {
      throw new Error(
        'Attempt to reference undefined variable or unsupported literal: ' +
          arg,
      );
    }
  }

And that's it for handling expression! Let's add compileCall support now that we referenced it.

compileCall

A call will first check if the call is a builtin. If so, it will immediately pass control to the builtin.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }
  }

Otherwise it will compile every argument to the call (which will leave all the resulting values on the stack.)

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));
  }

Then we will check that function is defined and call it.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));

    const fn = scope.lookup(fun);
    if (fn) {
      this.emit(depth, `CALL ${fn.name}`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }
  }

Then we'll reset the stack pointer (to maintain our commitment) based on the number of arguments and push RAX (where the return result of the function call will be stored) onto the stack. We'll make two minor optimizations for when there is exactly zero or one argument to the function.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));

    const fn = scope.lookup(fun);
    if (fn) {
      this.emit(depth, `CALL ${fn.name}`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }

    if (args.length > 1) {
      // Drop the args
      this.emit(depth, `ADD RSP, ${args.length * 8}`);
    }

    if (args.length === 1) {
      this.emit(depth, `MOV [RSP], RAX\n`);
    } else {
      this.emit(depth, 'PUSH RAX\n');
    }
  }

When there is only one argument, we can just set the top value on the stack to be the return result of the call rather than resetting the stack pointer just to push onto it.

And that's it for compileCall! Now that we've got a feel for expressions and function calls, let's add some simple arithmetic operations.

prepareArithmeticWrappers

There are two major kind of arithmetic instructions we'll wrap for now:

"General" instructions that operate on two arguments, putting the return result in the first argument
"RAX" instructions that operate on RAX and the first argument, putting the return result in RAX and possibly RDX

prepareGeneral

This helper will compile its two arguments and pop the second argument into RAX. This is because x86 instructions typically require one argument to be a register if one argument is allowed to be a memory address.

We'll use the stack address as the first argument so 1) that non-commutative operations are correct and 2) the result is stored right back onto the stack in the right location.

    const prepareGeneral = (instruction) => (arg, scope, depth) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);
      this.emit(depth, `POP RAX`);

      // Compile operation
      this.emit(depth, `${instruction.toUpperCase()} [RSP], RAX`);

      this.emit(depth, `# End ${instruction.toUpperCase()}`);
    };

prepareRax

This helper will similarly compile its two arguments and pop the second argument into RAX. But the RAX-implicit instructions require the argument to be stored in a register so we'll use the XCHG instruction to swap RAX with the value on the top of the stack (the first argument).

    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

This may seem roundabout but remember that we must pop all arguments to the instruction to maintain our commitment.

Next we'll zero out the RDX register if the operation is DIV, perform the operation, and store the result on the top of the stack.

    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

      // Reset RDX for DIV
      if (instruction.toUpperCase() === 'DIV') {
        this.emit(depth, `XOR RDX, RDX`);
      }

      // Compiler operation
      this.emit(depth, `${instruction.toUpperCase()} QWORD PTR [RSP]`);

      // Swap the top of the stack
      this.emit(depth, `MOV [RSP], ${outRegister}`);
    };

We parameterize the out register because the % wrapper will call DIV but need RDX rather than RAX after the operation.

prepareArithmeticWrappers

Putting everything together we get:

  prepareArithmeticWrappers() {
    // General operatations
    const prepareGeneral = (instruction) => (arg, scope, depth) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);
      this.emit(depth, `POP RAX`);

      // Compile operation
      this.emit(depth, `${instruction.toUpperCase()} [RSP], RAX`);

      this.emit(depth, `# End ${instruction.toUpperCase()}`);
    };

    // Operations that use RAX implicitly
    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

      // Reset RDX for DIV
      if (instruction.toUpperCase() === 'DIV') {
        this.emit(depth, `XOR RDX, RDX`);
      }

      // Compiler operation
      this.emit(depth, `${instruction.toUpperCase()} QWORD PTR [RSP]`);

      // Swap the top of the stack
      this.emit(depth, `MOV [RSP], ${outRegister}`);
    };

    return {
      '+': prepareGeneral('add'),
      '-': prepareGeneral('sub'),
      '&': prepareGeneral('and'),
      '|': prepareGeneral('or'),
      '=': prepareGeneral('mov'),
      '*': prepareRax('mul'),
      '/': prepareRax('div'),
      '%': prepareRax('div', 'RDX'),
    };
  }

Next we'll tackle compileBegin and compileDefine.

compileBegin

A begin form is an expression made up of a series of expressions where all expression values are thrown away and the last expression value is the result of the begin form.

To compile this form we will compile each expression passed in and pop from the stack to throw its value away. If the expression is the last in the list we will not pop since it is the result of the begin form.

We will add one exception to this popping logic: if the begin is called from the top-level we will omit the popping.

  compileBegin(body, scope, depth, topLevel = false) {
    body.forEach((expression, i) => {
      this.compileExpression(expression, scope, depth);
      if (!topLevel && i < body.length - 1) {
        this.emit(depth, `POP RAX # Ignore non-final expression`);
      }
    });
  }

That's it for compileBegin!

compileDefine

The prelude for a function definition will add its name to scope, push the current frame pointer (RBP) onto the stack and store the current stack pointer (RSP) as the new frame pointer (RBP).

Remember that we use the frame pointer as a point of reference when setting and getting local and parameter values. It works out entirely by convention.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);
  }

Next we copy the parameters into local scope at their negative (from the frame pointer) location. In the future we may decide to actually copy in the parameter values into the local stack but for now there's no benefit.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });
  }

Next we'll compile the body of the function within a begin block.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });

    // Pass childScope in for reference when body is compiled.
    this.compileBegin(body, childScope, depth);
  }

Then in the postlude we'll pop the stack (for the return result of the begin form), save it in RAX, pop the previous frame pointer back to restore the calling frame, and return.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });

    // Pass childScope in for reference when body is compiled.
    this.compileBegin(body, childScope, depth);

    // Save the return value
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `POP RBP\n`);

    this.emit(depth, 'RET\n');
  }

And now we're ready to compile a simple program!

Our first program

Here's a simple one we can support:

$ cat tests/meaning-of-life.lisp
(def main ()
  (+ 8 (* 2 17)))

We'll compile this program without the ulisp kernel (which contains a lisp library we cannot currently compile):

$ node src/ulisp.js tests/meaning-of-life.lisp --no-kernel --backend x86
$ ./build/prog
$ echo $?
42

Not bad. Let's finish up with support for prepareLogicalWrappers, prepareSyscallWrappers, and compileIf.

prepareLogicalWrappers

Storing logical results as values is a bit of pain. Most of the internet wants you to use branching. And a better compiler may optimize an idiom like (if (> 5 2) ...) into a single branch.

But we're going to resort to an instruction I just learned about called CMOV. This allows us to conditionally assign a value based on flags, similar to how you can conditionally branch.

Otherwise we'll follow a pattern similar to our arithmetic wrappers. At the end of the procedure we will have a 0 or a 1 on the top of the stack.

  prepareLogicalWrappers() {
    const prepareComparison = (operator) => {
      return {
        [operator]: (arg, scope, depth) => {
          depth++;
          this.emit(depth, `# ${operator}`);

          // Compile first argument, store in RAX
          this.compileExpression(arg[0], scope, depth);

          // Compile second argument
          this.compileExpression(arg[1], scope, depth);
          this.emit(depth, `POP RAX`);

          // Compile operation
          this.emit(depth, `CMP [RSP], RAX`);

          // Reset RAX to serve as CMOV* dest, MOV to keep flags (vs. XOR)
          this.emit(depth, `MOV RAX, 0`);

          // Conditional set [RSP]
          const operation = {
            '>': 'CMOVA',
            '>=': 'CMOVAE',
            '<': 'CMOVB',
            '<=': 'CMOVBE',
            '==': 'CMOVE',
            '!=': 'CMOVNE',
          }[operator];
          // CMOV* requires the source to be memory or register
          this.emit(depth, `MOV DWORD PTR [RSP], 1`);
          // CMOV* requires the dest to be a register
          this.emit(depth, `${operation} RAX, [RSP]`);
          this.emit(depth, `MOV [RSP], RAX`);
          this.emit(depth, `# End ${operator}`);
        },
      };
    };

    return {
      ...prepareComparison('>'),
      ...prepareComparison('>='),
      ...prepareComparison('<'),
      ...prepareComparison('<='),
      ...prepareComparison('=='),
      ...prepareComparison('!='),
    }
  }

prepareSyscallWrappers

This helper is similar to compileCall except for that it needs to follow the SYS V ABI and use the SYSCALL instruction rather than CALL.

  prepareSyscallWrappers() {
    const registers = ['RDI', 'RSI', 'RDX', 'R10', 'R8', 'R9'];

    const wrappers = {};
    Object.keys(SYSCALL_MAP).forEach((key, obj) => {
      wrappers[`syscall/${key}`] = (args, scope, depth) => {
        if (args.length > registers.length) {
          throw new Error(`Too many arguments to syscall/${key}`);
        }

        // Compile first
        args.forEach((arg) => this.compileExpression(arg, scope, depth));

        // Then pop to avoid possible register contention
        args.forEach((_, i) =>
          this.emit(depth, `POP ${registers[args.length - i - 1]}`),
        );

        this.emit(depth, `MOV RAX, ${SYSCALL_MAP[key]}`);
        this.emit(depth, 'SYSCALL');
        this.emit(depth, `PUSH RAX`);
      };
    });

    return wrappers;
  }

And we're set! Last up is compileIf.

compileIf

This is standard code generation but gets a little tricky due to our stack commitments. Testing must pop the test value off the stack. And then/else blocks must push a value onto the stack (even if there is no else block).

Here is an example we'd like to support:

(if (foo)
  (do-bar))

We compile the test and branch:

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);
  }

Then we compile the then block and jump to after the else block afterward.

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);

    // Compile then section
    this.emit(depth, `# If then`);
    this.compileExpression(then, scope, depth);
    this.emit(depth, `JMP .after_${branch}\n`);
  }

Finally we compile the else block if it exists, and otherwise we push a zero onto the stack (possibly to represent null).

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);

    // Compile then section
    this.emit(depth, `# If then`);
    this.compileExpression(then, scope, depth);
    this.emit(depth, `JMP .after_${branch}\n`);

    // Compile else section
    this.emit(depth, `# If else`);
    this.emit(0, `.${branch}:`);
    if (els) {
      this.compileExpression(els, scope, depth);
    } else {
      this.emit(1, 'PUSH 0 # Null else branch');
    }
    this.emit(0, `.after_${branch}:`);
    this.emit(depth, '# End if');
  }

And we're ready for an interesting program! Let's print (to stdout) the result of fib(20).

Fibonacci

$ cat ./tests/fib.lisp
(def fib (n)
     (if (< n 2)
       n
       (+ (fib (- n 1)) (fib (- n 2)))))

(def main ()
     (print (fib 20)))

And check out the kernel:

$ cat ./lib/kernel.lisp
(def print-char (c)
     (syscall/write 1 &c 1))

(def print (n)
     (if (> n 9)
       (print (/ n 10)))

     (print-char (+ 48 (% n 10))))

Compile and run it:

$ node src/ulisp.js tests/fib.lisp --backend x86
$ ./build/prog
6765

And we're in business!

Latest post in the compiler basics series: an x86 upgrade. We've got basic syscall support, "infinite" locals and parameters, and if/else. More than enough to handle printing integers to stdout and recursive fibonacci. https://t.co/B3OV0vEX1V
— Phil Eaton (@phil_eaton) December 8, 2019

Nov 2019 Recap

swyx's site RSS Feed — Tue, 03 Dec 2019 02:00:00 GMT

Hello mailing list friends!

The Great British Baking Show

Tilting at Windmills — Mon, 02 Dec 2019 02:00:00 GMT

Just what is it about The Great British Baking Show? (or The Great British Bake-Off as it's called in its homeland.) In case you haven't…

Learning to use emacs

Ben Overmyer's Site — Mon, 02 Dec 2019 02:00:00 GMT

I’ve been a vim user for somewhere around 10 years. While I’m comfortable enough in it, my knowledge and practice of it are still relatively amateurish. For a long time now I have heard others sing the praises of emacs, and since I’m trying to expand my horizons, it sounded like a good idea to try and pick it up.

I’m writing this blog post in emacs. Where possible, emacs is the first editor that I reach for, even though I know I could accomplish my goals faster and with less pain if I stuck to what I know. It’s an interesting experience.

I like that emacs is more natural-feeling than vim. The key incantations to do things are relatively intuitive. However, there is a lot of memorization I’m going to need to do in order to be effective with it.

This month will be my trial period. By the end of December, I’ll have figured out whether I want to continue using emacs, or switch back to vim.

Warren Buffett style fundamental metrics of long-term company performance

Bytepawn - Marton Trencseni — Mon, 02 Dec 2019 01:00:00 GMT

I look at some fundamental charts of Apple, Activision Blizzard and Intel.

Confusion and disengagement in meetings

Notes on software development — Sat, 30 Nov 2019 02:00:00 GMT

The quickest way to cut through confusion or disagreement among otherwise amiable and honest folks is to ask questions.

Ask early so you don't waste time. But it's not enough to just ask clarifying questions because the answers won't always be clear.

Sounds like Human Interaction 101, and maybe it is. These techniques show up more when discussing outcomes and very rarely when discussing assumptions.

Meetings are called to discuss outcomes, not assumptions. But assumptions almost always need to be called into question too.

If you have clarity personally but you observe confusion and disengagement, ask questions and summarize. Someone must be aware of the group and be willing to sound dumb.

If you aren't aware of confusion or disengagement, start paying attention. Addressing doesn't need to be hard and is personally meaningful.

Address confusion and disengagement in meetings by asking questions and summarizing, whether you're confused or not. Question outcomes _and_ assumptions. https://t.co/2OPifEBSq5
— Phil Eaton (@phil_eaton) December 1, 2019

Calibration curves for delivery prediction with Scikit-Learn

Bytepawn - Marton Trencseni — Thu, 21 Nov 2019 01:00:00 GMT

I show calibration curves for four different binary classification Scikit-Learn models we built for delivery prediction at Fetchr, trained using real-world data: LogisticRegression, DecisionTree, RandomForest and GradientBoosting.

Using simulated self-play to solve all OpenAI Gym classic control problems with Pytorch

Bytepawn - Marton Trencseni — Thu, 14 Nov 2019 01:00:00 GMT

I use simulated self-play by ranking episodes by summed reward. Game outcomes are divided in two by cutting at the median, winners are assigned +1 rewards, losers are assigned -1 rewards, like in games like Go and Chess. Unlike naive policy gradient descent used in previous posts, this version solves all OpenAI classic control problems, albeit slowly.

An interview on what makes Postgres unique (extensions)

CRAIG KERSTIENS — Wed, 13 Nov 2019 22:55:56 GMT

I’ve been at dinners before with developers that admitted developers, themselves included, can be a bit opinionated. In one case one said for example, “I love Postgres, but I have no idea why.” They were sitting at the wrong table to use Postgres as an example… But it is quite often that I am asked Why Postgres.

In fact a little over a year ago good friend Dimitri Fontaine asked if he could interview me for a book he’s working on for Postgres. I’ve long said their is a shortage of good books about Postgres and he’s done a great job with his in providing a guide targetted at developers, not just DBAs, that want to become better with their database. What follows is the excerpt of the interview from the book. And if you’re interested in picking up a copy he was friendly enough to share a discount code you can find below.

Intro

Craig heads up the Cloud team @citusdata now running product for Azure Postgres since being acquired by Microsoft. Citus extends Postgres to be a horizontally scalable distributed database. If you have a database, especially Postgres, that needs to scale beyond a single node (typically at 100GB and up) Craig is always happy to chat and see if Citus can help.

Previously Craig has spent a number of years @heroku, a platform-as-a-service, which takes much of the overhead out of IT and lets developers focus on building features and adding value. The bulk of Craig’s time at Heroku was spent running product and marketing for Heroku Data.

In your opinion, how important are extensions for the PostgreSQL open source project and ecosystem?

To me the extension APIs and growing ecosystem of extensions are the biggest advancement to Postgres in probably the last 10 years. Extensions have allowed Postgres to extend beyond a traditional relational database to much more of a data platform. Whether it’s the initial NoSQL datatypes (if we exclude XML that is) in hstore, to the rich feature set in geospatial with GIS, or approximation algorithms such as HyperLogLog or TopN you have extensions that now by themselves take Postgres into a new frontier.

Extensions allow the core to move at a slower pace, which makes sense. Each new feature in core means it has to be thoroughly tested and safe. That’s not to say that extensions don’t, but extensions that can exist outside core, then become part of the contrib provide a great on ramp for things to move much faster.

What are your favorite PostgreSQL extensions, and why?

My favorite three extensions are:

pg_stat_statements

Citus

HyperLogLog

pg_stat_statements is easily the most powerful extension for an application developer without having to understand deep database internals to get insights to optimize their database. For many application developers the database is a black box, but pg_stat_statements is a great foundation for AI for your database that I only expect to be improved upon in time.

Citus: I’m of course biased because I work there, but I followed Citus and pg_shard for 3 years prior to joining. Citus turns Postgres into a horizontally scalable database. Under the covers it’s sharded, but application developers don’t have to think or know about that complexity. With Citus Postgres is equipped to tackle larger workloads than ever before as previously Postgres was constrained to a single box or overly complicated architectures.

HyperLogLog: I have a confession to make. In part I just love saying it, but it also makes you seem uber-intelligent when you read about the algorithm itself. “K minimum value, bit observable patterns, stochastic averaging, harmonic averaging.” I mean who doesn’t want to use something with all those things in it? In simpler terms, it’s close enough approximate uniques that are compose-able with a really small footprint on storage. If you’re building something like a web analytics tool HyperLogLog is an obvious go to.

How do you typically find any extension you might need? Well, how do you know you might need a PostgreSQL extension in the first place?

pgxn.org and github are my two go-tos. Though Google also tends to work pretty well. And of course I stay up to date on new ones via PostgresWeekly.com.

Though in reality I often don’t always realize I need one. I search for the problem I’m trying to solve and discover it. I would likely never search for HyperLogLog, but a search for Postgres approximate count or approximate distincts would yield it pretty quickly.

Is there any downside you could think of when your application code base now relies on some PostgreSQL extension to run? I could think of extension’s availability in cloud and SaaS offerings, for instance.

It really depends. There are extensions that are much more bleeding edge, and ones that are more mature. Many of the major cloud providers support a range of extensions, but they won’t support any extension. If they do support it there isn’t a big downside to leveraging it. If they don’t you need to weigh the cost of running and managing Postgres yourself vs. how much value that particular extension would provide. As with all things managed vs. not, there is a trade-off there and you need to decide which one is right for you.

Though if something is supported and easy to leverage wherever you run, by all means, go for it.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Applying policy gradient to OpenAI Gym classic control problems with Pytorch

Bytepawn - Marton Trencseni — Tue, 12 Nov 2019 01:00:00 GMT

I try to generalize the policy gradient algorithm as introduced earlier to solve all the OpenAI classic control problems. It works for CartPole and Acrobot, but not for Pendulum and MountainCar environments.

Swyx Faves: Software Engineering Daily

swyx's site RSS Feed — Mon, 11 Nov 2019 02:00:00 GMT

"How to become a 10x engineer: listen to @softwaredaily everyday." - Amjad Masad

Narrow Waists

swyx's site RSS Feed — Sun, 10 Nov 2019 02:00:00 GMT

When talking about "Narrow Waists" I should clarify that I'm only referring to the term from the somewhat obscure "Internet Architecture" model of the different technology layers…

The Perfect Restaurant

swyx's site RSS Feed — Tue, 05 Nov 2019 02:00:00 GMT

This is a rant of uninformed thoughts and theories about running a restaurant.

The Three F's of Open Source Development

Ben E. C. Boyter — Mon, 04 Nov 2019 02:00:00 GMT

I write and publish open source code. I quite often get bug fixes, patches, requests and questions regarding the things I have published. Most of the feedback is respectful and helpful. However occasionally there are other requests. This page is where I plan to redirect them as a final warning before I block and ignore them.

The three F’s to open source development.

Fix it, Fork it, Fuck off

Fix it

If you find a bug, notice missing, incorrect or misleading documentation, or want an enhancement, your first option is to make a request to fix it. Raise the issue in the bug/request tracker of choice. Heck even a tweet at the owner might be good enough. However that is not your only option. You can choose submit a patch after you fix it yourself, or if you lack the skills, pay someone else to do so. You could also offer to pay the maintainer to resolve this for you. If there is enough value for you to raise the issue, perhaps there is enough value to pay to resolve it?

Oct 2019 Recap

swyx's site RSS Feed — Sat, 02 Nov 2019 02:00:00 GMT

Hello mailing list friends!

Rust 2020: A better ecosystem

Omar Abid - Personal Blog — Fri, 01 Nov 2019 02:00:00 GMT

Suggestions for Rust 2020

Linux Security Summit Europe 2019

Linux Engineer's random thoughts — Thu, 31 Oct 2019 17:00:00 GMT

Following my ELCE notes, you'll find below my notes for some of the talks the Linux Security Summit Europe 2019.

Exploiting race conditions using the scheduler — Jann Horn

The first bug Jann talked about is a race between mremap and fallocate. mremap allows moving a memory mapping from an address …

Sloc Cloc and Code - Can a crusty Go program outperform a well written Rust Project?

Ben E. C. Boyter — Thu, 31 Oct 2019 02:00:00 GMT

For those that wish to follow along with the presentation use the below link,

https://boyter.org/static/gophercon-syd-presentation/

You can press S to see the speaker notes as you go along.

ELCE, OS Summit and KVM Forum 2019 notes

Linux Engineer's random thoughts — Thu, 31 Oct 2019 01:00:00 GMT

After a two years hiatus, I couldn't miss this year's ELCE happening in Lyon, the first in France since 10 years ago in Grenoble, which coincidentally was one of my first conferences. OS Summit and KVM forum events are co-located and with the same ticket, which is nice, and why …

Interesting Upcoming pgDays

CRAIG KERSTIENS — Tue, 29 Oct 2019 22:55:56 GMT

I’ve been to a lot of conferences over the years. PgConf EU, PostgresOpen, too many pgDays to count, and even more none Postgres conferences (OSCON, Strangeloop, Railsconf, PyCon, LessConf, and many more). I’ve always found Postgres conferences one of the best places to get training and learn about what’s new with Postgres (in addition to Dimitri’s recent book, more on that below). They’re my regular stop to catch up on all the new features of a release before it comes out, and often there is a talk highlighting what is new with a simple easy to understand summary once released.

I just got back from PGConf EU a little over a week ago and it was a great time. I’m sure we’ll see some rundowns of it start appearing on Postgres planet. But, as far as I’m concerned PGConf EU is in the past (unless your counting next year which is in Berlin-in which case I’ll see you there). For me it’s time to look to the future and there are a number of upcoming pgDays I’m looking forward to.

The first two I want to highlight are separate events, but you’ll notice they’re scheduled nicely for you to easily attend both. With a day in between for travel you’ll find that many speakers and attendees depart one and head straight to the other. It makes for an easy opportunity to visit two cities, see two different communities, and yet not have to spend too much time traveling. The first is Nordic pgDay in Helsinki. It’s coming up on March 24. The second is pgDay Paris on March 26. Both of these are great single track conferences. If you’re in Europe of fancy a trip to Europe I recommend giving them a look, and even better the CFPs are open so consider submitting.

Another pgDay I have to mention is right in my backyard. pgDay SF I’m particularly excited for a few reasons:

San Francisco is very much a central tech hub, which means a great chance of learning from folks at many many interesting tech companies in attendance
Just like Nordic and Paris this is a single track conference, which I’m a personal fan of because you can have continuity between talks and shared conversation with other attendees
The venue! If you’re not from the bay area you may not be aware, but Swedish American Hall is well known music venue within SF. It’s had many famous artists over the years and now pgDay SF joins the ranks.

This isn’t an exhaustive list of course, just a few on my personal list that I hope to make it to. If you’re there and see me make sure to say hi!

Machine Learning at Fetchr

Bytepawn - Marton Trencseni — Tue, 29 Oct 2019 01:00:00 GMT

Opportunities for automating, optimizing and enabling processes with ML at a delivery company such as Fetchr are plentiful. We put three families of ML models into production. These 3 areas are: Scheduling, Notifications and Operational choice.

Private Conversations are Private

swyx's site RSS Feed — Sun, 27 Oct 2019 03:00:00 GMT

I recently made a mistake. I make many, but this involved someone important to me and to people I know, so it stands out among the general cacophony of my many other failures. I…

Ensembles vs Committees

swyx's site RSS Feed — Sat, 26 Oct 2019 03:00:00 GMT

A useful dichotomy for team dynamics.

Solving the CartPole Reinforcement Learning problem with Pytorch

Bytepawn - Marton Trencseni — Tue, 22 Oct 2019 01:00:00 GMT

The CartPole problem is the Hello World of Reinforcement Learning, originally described in 1985 by Sutton et al. The environment is a pole balanced on a cart. CartPole is one of the environments in OpenAI Gym, so we don't have to code up the physics. Here I walk through a simple solution using Pytorch.

Getting started with security keys

Paul Stamatiou — Mon, 21 Oct 2019 03:00:00 GMT

A USB-C YubiKey 5C security key plugged into my ThinkPad X1 Carbon

Every week I come across another headline about how someone got hacked and within moments many of their online accounts had become compromised. These aren't simple cases of bad actors using account credentials from large public data breaches and the unfortunate result of people using the same password across many websites.

These hacks, horror stories rather, are all the result of increasingly common attacks that gain access to a person's phone number by exploiting vulnerabilities in phone carriers. More advanced attacks prey on SS7, the old protocol underlying our cellular phone networks that lacks authentication. We more commonly hear about SIM card porting and SIM swapping attacks. In general, these attacks work by socially engineering your phone carrier to convince them you need a new SIM card as you lost your device or are upgrading it, or that you just want to switch carriers.

Today, many online accounts and services request your phone number to use as a form of account recovery as well as playing a role as a second factor in two-factor authentication via SMS text messages.¹ These attacks aim to get access to your phone number so the attackers can reset whatever accounts are tied to the number.

Attackers especially want to reset important accounts like your email account. From there they are often able to reset other accounts that might not be linked to your phone number. It's a dangerous and all-too-quick sequence of events that can turn your life upside down in moments.

Increasingly, the end goal is to gain access to a cryptocurrency account like Coinbase or to steal a high-profile Instagram account handle; methods that can quickly be used to get money.

Just take a look at these headlines:

A technology that can give you everything you want is a technology that can take away everything that you have.

—Dan Geer, Security analyst

I get freaked out every time I read one of those. As someone with a decent web presence, it feels like it’s only a matter of time before I become a victim of such an attack. While phone carriers have been working on making authentication better with efforts like Zenkey (née Project Verify) and the Mobile Authentication Taskforce, I think it’s best to take things into your own hands now, especially when technologies like Zenkey rely on mass adoption from third parties.

I've been a bit paranoid about security for a while now. It began as a curiosity for interesting new tools and services from installing PGP WDE in 2008 to encrypt my Mac (before FileVault 2 offered full-disk encryption) to exploring newer tools and services like VeraCrypt and Tarsnap.

SIM attacks like these, often the result of social engineering, are not the only thing to be worried about these days. Online phishing has become incredibly advanced, be it from official-looking texts asking for account info (smishing), malicious phone calls scams (vishing), email spoofing, brand impersonation, website spoofing and more. Take a look at this one:

Quick phishing demo. Would you fall for something like this? pic.twitter.com/phONMKHBle

— Mustafa Al-Bassam (@musalbas) September 9, 2018

A Tweet embed was not displayed. Read this article in a browser to see it as intended.

Paranoia is not entirely unwarranted with the plethora of these attacks today. It was with this mix of curiosity and fear that I began using hardware security keys two years ago. In a nutshell, security keys are little devices (typically USB, NFC or Bluetooth) that compute cryptography keys and serve as a more secure second factor² that only you can physically possess.

My assortment of Bluetooth, USB-A, USB-C, Lightning, NFC security keys

My evangelism of the benefits of these security keys amongst my friends has come up so often that I decided to write this beginner's guide (albeit rather detailed) to security keys—a topic generally left to very tech savvy people—to accomplish a few things:

Encourage people to take a closer look at their personal online security.
Explore what security keys are, understand the benefits and trade-offs, and see if security keys make sense for you.

Updates (Jan 2020)

There have been quite a few updates—really good updates—in the security key world since I published this article in October 2019. I have updated the respective sections throughout this article, but also wanted to note them in one place here:

Jun 2020

Google has added WebAuthn support on iOS! This is huge news and something I've been waiting for. Now you can use USB, Lightning and NFC security keys in all Google login flows. Previously you could only use Bluetooth security keys via the Smart Lock app. The Yubico post has additional context.

Jan 2020

Google has updated the Smart Lock app for iOS to utilize the built-in security key present on modern iPhones, the Secure Enclave. This new functionality is equivalent to what modern Android devices like the Pixel 3 and 4 can do with their integrated Titan security chips. Unfortunately, we're still waiting on Google to update their apps to fully support WebAuthn and allow logging in with USB, Lightning and NFC keys.

Dec 2019

iOS 13.3 officially released and Safari gets FIDO2 WebAuthn support. This long-awaited update finally brings better WebAuthn support to iOS, starting with support in Safari for NFC, USB and Lightning security keys.
Yubico Authenticator for iOS now supports NFC security keys.

Nov 2019

iOS 13.3 Safari introduces FIDO2 security key support! Wow, I didn't expect to see this so soon. While still in beta and noted in the release notes, iOS 13.3 "supports NFC, USB, and Lightning FIDO2-compliant security keys in Safari" (and related Safari-powered view controllers that might be found inside apps). I can't understate how great this news is for iOS users. Lack of proper iOS support was the largest issue facing security key adoption and convenience.
YubiKey Bio: Yubico announced they are working on a FIDO2 security key with an integrated fingerprint reader.
Yubico Authenticator for iOS released. A lovely surprise with the release of Yubico Authenticator for iOS. This app lets you use your iPhone to access TOTP codes (not FIDO2) stored on your Lightning connector or NFC YubiKeys.
OpenSSH to get U2F support. For those interested in easy native security key support with OpenSSH.

Security best practices

A few basics for improving your online security right now

Before I dive right into talking about security keys, I thought it would be best to discuss a few security best practices: low-hanging fruit that go a long way for improving your security online. There's a wealth of information on this topic across the web, so this is in no way meant to be an exhaustive list.

In addition, it really depends on a few things like what you are trying to protect, who might want it and where along the spectrum from convenience to security you would feel comfortable. Security folks would take this opportunity to talk about your personal threat model to help assess what level of security paranoia and defenses are warranted for you. That could probably be it's own article describing various routes to take and is fairly outside the scope of what I want to do with this article.

You can be on the supremely paranoid end of the spectrum but it would just make simple daily tasks like checking email and browsing the web a chore, and most likely just increase the likelihood of locking yourself out of your accounts.

Authentication is always a trade off between security and usability.

—Bruce Schneier, "Click here to kill yourself"

Use a password manager, religiously. Make sure the sync works or opt for a hosted cloud solution. Pick a very strong master password for the password manager and remember it. Don't use it for any other website or service. Backup the recovery key. Go through and change every online account password to something unique if it's your first time setting up the password manager.

I use and love 1Password and pay for the cloud account but there are plenty of alternatives out there including DashLane, LastPass (some people avoid it due to a history of security incidents and their acquisition by LogMeIn), Enpass and open source options like Bitwarden.

Set up Two-Factor Auth for your password manager if available. This is separate from 2FA for your accounts it stores and is just 2FA for the password manager itself. For example, a 2FA-enabled cloud-hosted 1Password account will ask you for your second factor when adding 1Password to any new device.

Regularly take advantage of advanced password manager functionality that evaluates your accounts to see if they have been involved in any data breaches (1P calls this Watchtower) as well as informs you when you reuse passwords.
Enable Two-Factor Authentication (2FA) wherever supported. Don’t enable SMS for your second factor if TOTP authenticator apps are an option.³ Enable 2FA using an authenticator app for all supported sites (we’ll get to security keys later).

You can use an authenticator app like Google Authenticator, Microsoft Authenticator, Yubico Authenticator⁴, FreeOTP (open sourced by Red Hat), AndOTP (open source and has encrypted backups), Duo Mobile and Authy⁵.
Backup account recovery codes if they exist, especially for important accounts and email providers like 1Password, Google and Apple accounts.
Remove your phone number as a recovery method for your email account. Ideally from all of your accounts, but we’ll get to that later.
Add a passphrase/PIN for your mobile phone carrier. This passphrase would be requested from the carrier to authorize any SIM port. Or at least that’s the idea. Unfortunately, phone carriers have been known to be socially engineered into not asking for it and attackers have been known to have insiders at telephone companies to bypass this.
Be a savvy web surfer. Much easier said than done, but lots of attack vectors are much closer to home than you might think. Keep your software up to date. Use an adblocker and/or DNS provider that can help keep you away from malware and phishing. (I have been testing out NextDNS.io and liking it so far. It's like a cloud-hosted Pi Hole if you don't want to bother setting that up yourself.) More popular options include Google Public DNS (while not an explicit privacy/adblocker type of DNS provider it does protect against several types of attacks), Cloudflare 1.1.1.1 and Quad9.

Consider using a VPN when you are on an untrusted connection, but know that a VPN merely changes who you are trusting but doesn't instantly make everything secure or private. Be skeptical of any browser extensions you install, links you click in emails, be careful when opening attachments and so on. Look for authentic website SSL certificates and correct domains when putting your credentials in a site. If your password manager can usually auto-fill in the details for a site but doesn't one day, it could be that you're not actually on the authentic site. So be sure to be suspicious if your password manager doesn't seem to be working as intended.
Revoke permissions for unused authenticated services. Go through your accounts like Google, Facebook and Twitter and revoke authenticated apps/integrations you no longer use or don't recognize that have any kind of access to your account. There’s probably way more than you recognize due to past single sign on uses.
Audit the authenticated devices linked to your Apple ID. Make sure you don’t have any old devices—maybe that you don’t use or no longer own—listed as devices in your account. And while we’re on the topic, make sure Find My iPhone is enabled so you can remotely disable a lost device in a pinch.

Getting more advanced

More tips for the increasingly paranoid

Secure your account with your domain name registrar. If you’re using email on a custom domain, ensure your domain name registrar is locked down with a strong password and two-factor auth using either an authenticator app or security keys. Remove any recovery phone number as well.

I’d recommend migrating your domains to something like Google Domains or AWS Route 53. Moving domains is easy. They offer strong 2FA solutions out of the box and I would trust them more than any other random registrar selling $5 domains.

This step is important because it’s another easy attack vector. Someone can socially engineer your registrar or mobile phone carrier (if linked), get access to your domain, change DNS MX records and then be able to receive password reset emails and take over other accounts.
Consider how and if you want to backup your authenticator codes. After you’ve got all your accounts set up with two-factor auth using an authenticator app, you need to think about what might happen if you lose your phone. With an app like Google Authenticator, you will lose the codes if you have to get a new phone and restore from an iCloud backup. There are convenient options like Authy and 1Password that let you easily backup these codes and access from any device. There's also AndOTP with it's encrypted backup capability.

1Password has a really lovely integration to also store these one-time password codes (and automatically paste them in when needed, depending on your app/extension/OS), but if you’re storing your second factor alongside your first, then you don’t really have two-factor authentication.

This solution is fine for most people, but this section is about being a bit more paranoid, so I would recommend not using the 1Password integration for your one-time password codes. Sure, you’re probably very screwed already in the unlikely event that someone gets access to your 1P database, but it’s even better if your most vital accounts require one more step by not having the OTP codes there.

The more extreme option is to manually keep track of the QR code or setup key provided when setting up 2FA for a TOTP authenticator on each account. Backing up these setup codes is a bit controversial and not recommended by the more hardcore security folks as it introduces another avenue by which you could be compromised if not securely stored. If you opt to backup your QR codes, you may want to store them outside of your password manager and in an encrypted manner. This could be on a service like Tarsnap, an encrypted Mac sparse bundle image or VeraCrypt volume. But now you have a new problem: how to securely store the passphrase or key for that service or volume.
Don’t set a recovery email for your email accounts. Depending on your level of paranoia, you may opt to remove any recovery email addresses linked to your email accounts. The logic here is that every linked recovery email is another potential attack vector that could be used to gain access to your primary email account.

If this sounds like a bit too much to you and you’re worried about just locking yourself out of your account, you should at the least ensure that the recovery email account has a strong password and two factor authentication enabled. And ideally this account is with different email provider, not added or logged in on any of your devices and is only used for recovery.

Switch your carrier to Google Fi

If you want another safety net

Even if you have done all you can to not link your online accounts to your phone number, there are always those services that just do not have two-factor authentication options beyond SMS. And that will always be a weak link in your security. You can do something like get a separate phone number that you only use for securing those accounts and don't use for anything else. You could also create a separate Google Voice number for that purpose; the benefit being that it's protected by your Google account's 2FA.

Or you can move switch your mobile phone carrier to Google Fi. I find this to be the simplest solution and have been using Fi for almost 4 years now. The security benefit here is that your phone number is now protected by your Google account's two-factor authentication, and is significantly less susceptible to social engineering attacks to gain account access.

I asked Google for more clarity on Fi security and was told the following. Basically, nothing is possible without access to the Google account already.

[...] we check for the authentication of their account and if a user contacts us with an unregistered email ID then we ask them to confirm their identity by sending secret codes to the email ID which they claim that have been registered with Google Fi.

Enable Google Advanced Protection

Consider Advanced Protection for your Google accounts

Google began offering something called Advanced Protection in late 2017. It offers enhanced security for those that might be at increased risk of targeted attacks. According to Google that includes journalists, celebrities, activists, business leaders, political campaign teams, firms dealing with cryptocurrencies and law firms. Fortunately, this functionality is available to anyone with a Google account and it brings some significant changes.

Targeted attacks could be low volume, carefully crafted, phishing attacks, often personalized to individuals, and can be hard to distinguish from legitimate activity. This makes targeted attacks the hardest to protect against. The Advanced Protection Program is specifically designed to thwart targeted online attacks on Google accounts.

—Google

First, Advanced Protection requires that you use security keys. It won't let you enable it until you pair two security keys to ensure you have a backup key. The main benefit here being that security keys prevent phishing. Google accounts with AP enabled also bring stringent changes like:

Very strict account access rules for your emails and Google Drive files. This means you won't be able to use your Google account outside of native Google apps, except for a select list of trusted third-party applications like Apple Mail, Apple Calendar, Apple Contacts and Mozilla Thunderbird.
App passwords are no longer supported
Exhaustive Gmail scans to catch phishing messages and attachments containing malware.
No recovery code. That’s right. If you don't have access to your password and one of your security keys, there is no way back in. You must go through the recovery process, which is much more rigorous and includes various delays on purpose. They may ask recovery questions like what month and year you first opened the Google account.
Enhanced web browsing safety when signed into Google Chrome. Google will alert you if it detects you downloading a file it thinks is unsafe according to its Safe Browsing policies.
And finally, I feel like Advanced Protection is more suspicious of activities and will aggressively prevent logging in if it thinks something is astray. I recall trying to test it out and logged in and out of my account on various devices and different IPs on VPNs and it was quick to prevent access.

Google Advanced Protection may be overkill for some people but it's a good segue into security keys and how they can help enhance your online security.

What are security keys?

The small hardware devices to enhance your online security.

Security keys are inexpensive, little hardware devices that use public-key cryptography to securely authenticate the website to the key. Maybe you've seen them around; these security tokens tend to look similar to regular USB memory sticks. Compared to using an authenticator app (also known as a TOTP code generator) as a second factor for 2FA, security keys (also called external or roaming authenticators) are a preferred and more secure option.

USB-C security key plugged into a Pixel 3 Android phone.

When you go to log into a website where you have registered a security key as your second factor, you are first prompted for your username and password and then for the key. Usually this means plugging in and touching a special touch sensitive area of the key (for USB keys), but could also mean bringing the key closer (NFC) or pairing and pressing a button on the key in the case of Bluetooth security keys.

Okay, sounds easy enough.. but a bit more of a hassle to keep another device handy, right? So what’s so great about these security keys?

It’s no coincidence that Google requires hardware security keys as the second factor method for Google accounts with Advanced Protection enabled. Google themselves pushed for their employees to use security keys and they saw that it entirely neutralized phishing attacks on all of their 85,000+ employees.

Compared to TOTP authenticators, the key benefit of using a security key is that they are immune to phishing attacks.⁶ This is partially due to the fact that the origin/domain of the website is taken into account when you register your key. If you fall for a credential phishing email and went to log into a look-alike fake website, verification would fail, even if you thought the website was real.

YubiKey 5C Nano plugged into my ThinkPad X1 Carbon

Security key history

FIDO, U2F, CTAP, WebAuthn.. where did it come from?

But first I wanted to briefly dive into the history of security keys over the last few years. For the purpose of this article, I want to refer to these hardware authenticators and security keys as just security keys. You may have seen them referred to as FIDO U2F, CTAP, FIDO2 and WebAuthn. It’s all very confusing and even the newer overarching and correct term (FIDO2) doesn’t quite roll off the tongue.

You may be thinking, wait a minute, I used hardware tokens years before this. And you’re not wrong. Code generating hardware tokens like RSA SecurID have been around since the early 2000s. They would continually display newly generated codes on their LCD display: codes that would often be appended to a password when logging in.

Then there were USB security keys in 2007 that would identify themselves to the computer as standard USB HID keyboard input devices and as such, not require any special drivers. When pressed, they would input a one-time password string comprised of things like counters, timers and random values. The one-time password would then be verified by the server based on a shared state (like the usage counter and device id) between the server and key.

While those keys sound a lot like the keys we have today, their implementation is quite a bit different and they are still susceptible to phishing and man-in-the-middle attacks.

Fast-forward to 2012. Google and Yubico collaborated on a hardware security key based on public-key cryptography. This eventually became known as U2F (Universal 2nd Factor) and became part of the FIDO Alliance (Fast IDentity Online), an organization initially founded by PayPal, Lenovo and other companies to work on a passwordless authentication protocol.

In 2014, Google officially shipped FIDO U2F support in their Chrome browser. The next few years brought better security key hardware, increased support from large, mainstream websites like GitHub, Facebook and Dropbox, as well as increased support from browser vendors.

Now, in 2019, we have the FIDO2 project. It recognizes the growing need to enable easy and secure multi-factor authentication experiences across both desktop and mobile devices to further the FIDO Alliance's goal while making strong authentication more accessible. FIDO2 consists of two distinct parts: a client API called WebAuthn (an official W3C standard created in collaboration with the FIDO Alliance) and an API for hardware authenticators.

FIDO2 = WebAuthn + CTAP

The first part, WebAuthn (short for Web Authentication), is essentially the client/browser JavaScript API that allows websites to create and use credentials based on public keys, which may come from hardware authenticators. It’s huge news that this is now a W3C standard so that every browser manufacturer can begin developing and supporting it. WebAuthn is the core link between the website server—frequently referred to as the Relying Party—and the browser.

The second part helps define the link between the browser and the external authenticator device. This protocol is called the Client to Authenticator Protocol, or just CTAP.⁷ This aids in all the particulars of how various hardware authenticators communicate with the browser, such as via USB, Bluetooth Low Energy and NFC.

What is passwordless?

What is this passwordless thing all about?

One thing I keep hearing about with respect to authentication these days is passwordless auth. Honestly, it just seems like a buzzword.. so what exactly is passwordless? I'll briefly describe the premise here and why it's not the focus of this article.

One of the primary goals of the FIDO Alliance at its inception was to provide "open and free authentication standards to help reduce the world's reliance on passwords." So far in this article when talking about FIDO1 I have only mentioned Universal 2nd Factor (U2F), which eponymously refers to enhancing two-factor authentication by using hardware security keys as a second factor.

FIDO U2F became well-known, in no small part to strong support from Google and subsequent integration into the Chrome browser. And well, U2F is comparatively easy to understand for people already familiar with two-factor auth... you just use a key you plug in as your second factor. Easy enough.

FIDO1 also contained a spec for something called UAF, Universal Authentication Framework. Its original goal was to provide a passwordless authentication solution employing biometrics. Similar to U2F, UAF utilizes public-key cryptography to authenticate with each service.

The difference is that UAF does not require any password. The browser asks the user to complete their previously registered authentication gesture, such as tapping their finger on a fingerprint reader or typing in a PIN.⁸ As such, it's called passwordless since no password is actually sent across the network. The authentication is done with the standard challenge-response auth as part of public-key cryptography.

Okay, so what about FIDO2? FIDO2 brings WebAuthn, which essentially merges the capabilities of U2F and UAF into one API that can be accessed by any supported browser or OS. People utilizing FIDO2 for two-factor auth with a security key as their second factor now have the ability to use a broader set of authenticators, such as internal platform authenticators like fingerprint readers and facial biometrics built into their devices. And finally, FIDO2 provides various passwordless authentication formats into the same system: passwordless single-factor or multi-factor.⁹

Android supports built-in platform authenticators like fingerprint readers. Here's an example flow.

All of this work is done to make passwords obsolete...why?

Passwords have several notable issues. Strong passwords are hard to memorize, they take time to type, many people still use the same password for all of their accounts and it's easier to be phished with just a password. And of course, should anyone get access your password, they can just use it right away to log in.

But these seem like relatively solved issues today, right? We have robust password managers and we have two-factor authentication. The thinking behind passwordless is that those are just workarounds to an ancient authentication system that should be re-imagined with the latest technology we have at our disposal today.

We already have a taste of what FIDO2 brings today with enhanced authenticator support: you can use Touch ID on your MacBook Pro in Chrome, the fingerprint reader on your Windows 10 device via Windows Hello or even use the Pixel 3 Android phone itself as an authenticator, storing credentials inside its Titan M security chip and more. And you can imagine that in the future smart watches could begin to function as authenticators too: Apple already lets you use your Apple Watch as a proximity-based authenticator to unlock your Mac.

TouchID working with WebAuthn in Google Chrome

The passwordless vision is compelling but feels quite a few steps away. First, barely any websites even support passwordless authentication flows.¹⁰ Second, passwordless functionality requires storing data like display name and relying party name on the security key itself—something that doesn't happen when simply using your security key as a second factor. This is called a resident key credential and this data takes up space. Modern YubiKeys only have storage room for 25 resident keys¹¹ for use with such passwordless auth. Sure that might be a moot point if we all end up using internal platform biometric authenticators¹² for everything, but biometrics aren't a flawless¹³ solution:

Biometrics are easy, convenient, and when used properly, very secure; they're just not a panacea. Understanding how they work and fail is critical to understanding when they improve security and when they don't.

—Bruce Schneier, Security expert

The very compelling counter-argument is that, well, people just know how to use passwords and not everyone understands (or cares!) about security. We've finally made some headway on getting people to use two-factor authentication.

It's for those reasons that I would like to focus on the here and now: learning about the benefits of hardware security keys and how you can begin using them today with many services you already use.

Security keys vs. auth apps

Are security keys really any better than TOTP code generator apps for 2FA?

Before I go into detail about how security keys work and their benefits, it might make sense to talk a bit about the issues with using authenticator apps for two-factor auth.

TOTP risks

Why authenticator apps aren't perfect

Even though they are a vastly preferred second factor compared to SMS, authentication with TOTP (Time-based One-Time Password) has some risks and inconveniences compared to security keys employing public-key cryptography.

Lack of phishing prevention. A huge issue with TOTP is that there is no inherent replay attack protection. You could still fall victim to a fake website (or real one being proxied via man-in-the-middle like with Evilginx 2 and Modlishka) looking exactly like a website you visit and supply your password and TOTP codes to them. The attacker or bot can then log into the real website as you and still be within the TOTP time limit.
TOTP employs a shared secret. With TOTP, the website has a secret key it provides to you. This is the key your authenticator app uses for initial setup. The TOTP authenticator then generates a hash with the secret key and the current time. The website does the same thing on its end, using the same secret key it has, and compares the two.

The issue is that the user and the relying party website share and store the same key. Unless the website followed the security recommendation in the TOTP spec to encrypt the secret key and only decrypt when in use to limit exposure in memory, it’s possible an attacker could generate TOTP codes for any account.¹⁴
Manual code entry. A significant and frequent annoyance with TOTP codes is that you have to manually type in the code each time: not to mention open your phone, then open the authenticator app. There are, however, methods of simplifying this with desktop authenticator apps, especially with password managers like 1Password automatically copying and pasting it in for you, but you don't really want to have your password manager also store OTPs.
Backing up TOTP authenticators is a hassle. Lose your phone where you had the only setup of Google Authenticator? You will need to proceed to account recovery for every account you had in there, which may go through an SMS (hopefully you’ve disabled this already!) or email reset flow. That’s annoying.

There are some modern solutions to backing up and syncing your authenticator app in the cloud like Authy, using the OTP integration in 1Password or managing andOTP encrypted backups. But that’s up to you to decide if those routes are for you as they have some downsides. You can also decide if you prefer to backup the TOTP setup secret key or QR code, but it would need to be stored securely as to not introduce another attack vector.

But I digress, my point here is that backing up your authenticator is yet another thing to think and worry about with TOTP.

Security key benefits

No shared secret, phishing prevention and much more

On the other hand, security keys work in a very different manner and avoid many of the issues inherent with TOTP authenticators. Security keys are based on public key cryptography, meaning there is no shared secret between the user and the website. The benefit here is that even if a website was breached and an attacker gained access to a user's public key, they would not be able to do anything with it.

My 90mm macro lens was a great investment and was used thoroughly in this post.

In addition, security keys are designed to make it impossible to get the private key out of the security key itself as cryptographic functions are computed locally on the device and never exposed to the computer.

Bulletproof phishing protection. The largest benefit of security keys is that they entirely thwart phishing attempts unlike TOTP authenticators. The exact domain name of the website is taken into account when the security key is registered. Even if the user willingly tried to log into a fake phishing site, the security key authentication would not work as the domain would differ. There's also support for token binding which furthers this protection, though I'm not sure it is widely supported yet.
Security keys prevent replay attacks. Security keys are designed to never expose private keys outside of the hardware. However, should they somehow be cloned, they contain a signature counter. The WebAuthn spec says the security key should provide an incremental counter to the relying party on each successful authentication. Should the value not match the last known value, the authentication should fail. I say should because this is not a strict requirement for WebAuthn and it doesn't appear to be common yet.

In addition, security keys and the relying parties utilize a challenge-response authentication flow to ensure single use and further prevent replay attacks.
Built-in user presence required. One of the core features of security keys from the beginning has been that they require user presence before the key can do any cryptographic operations. All security keys require you to prove your presence by physically touching a capacitive touch sensor (or button for Bluetooth keys). This prevents remote takeovers as the user has to prove their physical presence to the device. Newer FIDO2 keys support user verification where you need to provide a PIN or a biometric gesture like a fingerprint tap (like with the upcoming YubiKey Bio with integrated fingerprint reader) if configured as such.
No typing in codes. And after you tap the key, that’s it. There’s no more work to do, you’re already logged in. However, this assumes you already have the key plugged in your machine and don’t have to fish it out of your pocket or bag.
Easy to backup. Backing up a security key is easy—just buy spare security keys and register them on each service!
Privacy as default. And finally, since security keys generate unique public/private key pairs for each website, it is impossible for anyone to know what sites are registered to your security key. Even if your key is stolen, there is no way to know what websites it is used with, or any associated data.¹⁵

Types of security keys

How many keys do you need? How much do they cost?

That’s a long way of saying that security keys are a terrific option to increase your online security. Now we can go into some more actionable info: basics about the security keys themselves, how many you need and what kind.

You’ll need to buy some security keys of course before you can get started. FIDO2 and WebAuthn paint a vivid picture of the future of authentication from supporting more types of authenticators—from numerous internal biometric sensors to more types of mobile devices—to supporting passwordless authentication flows.

As I mentioned earlier, external security key authenticators are the easiest to incorporate into your daily life today compared to passwordless auth. It won't be perfect though: there are still some inconveniences and support issues associated with using security keys for your two-factor auth that I’ll dive into later.

How many do I need?

Yes, you need to purchase your security keys and they are not free. This is quite different than past solutions like SMS and TOTP authenticator apps. But it’s a small price to pay for enhanced security and phishing prevention. Security keys come in many varieties (USB-A, USB-C, NFC, Bluetooth LE and most recently, Lightning connector) to interface with computers and mobile devices.

There are benefits to using only one security key, but having multiple keys to rely on as your second factor is ideal.

Purchase two or more keys if you want to rely entirely on security keys—and fully disable SMS and TOTP authenticator apps where possible—you will want at least 2 keys so that you can have a backup key. That being said, most services let you program many keys so there is not much harm in having a few more (unless they are too easily misplaced or haphazardly stored and fall into the wrong hands). You may also find it handy to have small "nano" keys that always stay plugged in your computer.

You will need at least two keys if you want to consider enabling Google Advanced Protection on your account.
Purchase one key if you want to test it out. There is still a strong benefit to just using one security key as a second factor in addition to a TOTP authenticator app as your backup. By relying on your security key to authenticate as much as possible, you’ll still receive the strong phishing protection that comes with security key usage.

YubiKeys are pretty durable. I've had this one on my keychain for a year and a half now. I have heard people say the USB-C version is less durable than the USB-A version but I haven't had any issues myself.

How much do they cost?

Depending on feature set and durability, security keys range in price from around $10 all the way to $100+.

Entry-level: At the entry-level, we have security keys that cost something like $12 HyperFIDO Mini, $20 for a Yubico Security Key and the open source SoloKey.

These cheaper keys tend to be strictly for authentication via one protocol only and won't support additional protocols and software customization that pricier keys have—which is totally fine. Some keys in this price range may only support FIDO U2F and not the latest FIDO2 spec (aside from the model by Yubico). This means you can't use them for new WebAuthn passwordless functionality if that matters to you.
Mid to top tier: The main players in this space are Google (with the Google Titan Security Key bundle as well as the just-released USB-C version that is similar to the Yubico 5C) and Yubico with their latest YubiKey 5 series offerings like the flagship $70 YubiKey 5Ci featuring dual USB-C and Apple Lightning connectors.

At this level you start to see more durable keys with better build quality, as well as support for many applications in addition to FIDO U2F and FIDO2¹⁶, such as Yubico OTP, Smart Card, OATH and OpenPGP. But most of these probably mean very little to you unless you're a developer.

Left: YubiKey 5Ci (Lightning connector) with iPhone 11 Pro Right: Titan Security Key bundle

Advanced & specialized: At the high-end there are devices that do much more than just U2F authentication and have robust, integrated password managers and cryptocurrency wallets. Trezor and Ledger both offer models in the $50 to $120+ range.

I should also mention that there is a slowly growing category of devices not listed above that will become more important in the future as passwordless auth expands: external FIDO2 biometric authenticators. These devices are full-fledged FIDO2 security keys but also contain a biometric sensor like a fingerprint reader.

The fingerprint reader comes in handy for local user verification (instead of a PIN) for multi-factor passwordless auth. For example, like this Feitian BioPass authenticator. But I can't vouch for that brand and don't know if it requires special fingerprint drivers like similar models do. But I digress, this doesn't matter right now, especially as I'm focusing on two-factor auth security key uses.

Update (Nov 2019): Yubico has since announced that they are working on YubiKey Bio, a security key featuring an integrated fingerprint reader that has no battery and requires no additional drivers or software to function. Unfortunately, it appears that this first biometric security key from Yubico is only USB-A.

Compatibility issues

A look at what supports WebAuthn

There are two pieces to successfully adopting security keys: support for FIDO U2F or FIDO2 WebAuthn from websites you use and the same support on the browsers and devices you wish to use to access those websites or apps.

Both have been a bit confusing for various reasons. Take website support of FIDO U2F or FIDO2 WebAuthn for example. A website may support one of those and let you register your security key, but it's up to the site's specific two-factor auth implementation to determine whether you will be allowed to register multiple security keys and/or use security keys as your only type of second factor. One reason is that that website's accompanying mobile app may not support security key login and they don't want to have to explain this trade-off to users.

Spotty browser support has been a big issue over the years. This really comes down to the devices you use, as well as their current state of support for security keys. While security keys come in many flavors spanning USB-A, USB-C, NFC, BLE and Apple Lightning connector, just because the port fits into your device doesn't mean it will work everywhere on your device.

Support for security keys is by no means ubiquitous across devices, operating systems and browsers and websites. Things have gotten much, much better over the years and really, the only real rough patch lies with iOS. I'll dive into a brief description of the current support landscape right now, but one way to keep up to date on WebAuthn support status is by checking CanIUse.

Windows - Great support

You can use any USB security key on Windows 10 with Google Chrome, Mozilla Firefox, Brave and Microsoft Edge.

Windows 10's Hello authentication system also supports FIDO2 security keys and currently the Edge browser has WebAuthn integration to access additional Hello-supported authentication methods such as PIN and biometric readers.

Linux - Great support

USB security keys should be fully supported by any modern Linux distro as well as a browser like Chrome, Chromium and Firefox. However, certain distros may require you to manually tinker with the udev device management subsystem to enable browsers to interact with security keys.

ChromeOS - Great support

Supports USB security keys. Certain Chromebooks also feature their own integrated Google H1 CR50 security chip so you don't even need an external security key, but I believe they only operate as a FIDO U2F+CTAP1 key.

macOS - Great support

While security keys are fully supported on Chrome and Firefox for macOS, Safari has lagged behind and not had support for a while. Fortunately, Safari 13 shipping with macOS 10.15 Catalina brings FIDO U2F and FIDO2 WebAuthn support.

If you're not on Catalina yet, you're able to put it to use with the Safari Technology Preview now, but it does appear to entirely lack basic UI to show when a site is requesting key access. Chrome also supports biometrics with WebAuthn, letting you access TouchID on TouchID-sporting hardware like the MacBook Pro and MacBook Air. No FaceID support and no word if that will happen.

Android - Great support

Android support for security keys is in a great state, and that's no surprise with Google's push for U2F, and now WebAuthn, since the beginning. Chrome and Firefox enjoy full FIDO2 support and this means you can easily opt to use USB, Bluetooth and NFC security keys, along with any integrated biometrics such as your device's fingerprint reader. (However, Brave browser and Firefox Focus did not appear to have functioning security key support at the time I wrote this.)

In addition, the Android OS itself is FIDO2 certified. The OS-level integration means that browsers and other apps only need to make an API call to trigger Android's robust auth flow that handles everything.

Using a USB-C security key on a Pixel 3

iOS - Great support (Updated 2024)

Update 2024: Things have come a long way and you're no longer at a disadvantage using a security key with iOS.

The state of security key support on iOS used to be utterly horrible. The native Safari browser didn't support any kind of web authentication. The only option was to use a special Lightning connector security key and a different browser that had security key support baked in (Brave for iOS).

The only other way to work around it was with custom solutions employing Bluetooth security keys. Google lets people use security keys on iOS by redirecting to a separate app called Google Smart Lock that can pair with Bluetooth security keys—an annoying, one-off and clunky solution only for Google services.

Fortunately, things have changed for the better. They're still not great, but they're on the right track. As of December 2019, Apple released iOS 13.3 which brought Safari support for WebAuthn using USB, NFC and Lightning connector FIDO2 security keys.

Even though Safari supports WebAuthn now, this does not mean that everything on iOS will. Many apps are in need of software updates to let you use security keys. Since iOS had no security key support for ages, it's likely that many apps just incorrectly assume that iOS as a whole can't interface with security keys and won't even let iOS users provide their security key. Gmail is one such example: both the iOS app and website in Safari won't let you use a USB, NFC or Lightning connector security key yet. Update (Jun 2020): Google now supports USB, NFC and Lightning connector security keys for WebAuthn on iOS!

This is also the current state of support for the aforementioned YubiKey 5Ci Lightning connector security key. It only works with a small set of apps and browsers like 1Password, LastPass, Safari and Brave browser.

A note about NFC keys Don't modern iPhones have NFC? Can't we use that along with NFC security keys for authentication?

Fortunately, the short answer is yes! Unfortunately, this was literally only possible until late in 2019 with iOS 13 bringing full read/write NFC support to devices like the iPhone 7 and newer. And then it wasn't available to consumers until iOS 13.3 brought NFC key support into Safari in December 2019.

In the past iOS provided read-only NFC access on iPhones. You might have seen some examples of NFC security key authentication on iPhones in the past (in case you've seen the LastPass integration advertised). Those solutions have also been proprietary (sound familiar?) and required custom integration with the YubiKey Mobile SDK as they did not use FIDO U2F or FIDO2/WebAuthn, but instead used the proprietary YubiKey OTP. That was the only option at the time because FIDO U2F and FIDO2 require NFC read-write access to function, which iOS did not allow at the time.

Safari does now support the use of NFC keys and it's a great and easy experience. App and other browser developers (looking at you Chrome!) will still need to manually add support for this functionality. Yubico Authenticator for iOS was recently updated to support NFC keys so you can also use the key for TOTP code generation purposes in addition to WebAuthn. However, NFC is not a perfect solution with mixed support: current iPads and older iPhones don't have the necessary NFC hardware.

Update (Jun 2020): Google now supports NFC security keys for WebAuthn on iOS, so if you're primarily using a security key for Google products and services, now is a great time to consider an NFC key.

Left: iPhone 11 Pro + YubiKey 5Ci Lightning connector security key Right: Using the 5Ci withBrave browser for iOS

Buying your keys

NFC, BLE, USB-C, oh my! What to choose?

Hopefully you now have enough context about the compatibility landscape to help you make an informed decision about what security keys to purchase given your mobile and desktop computing needs today.

If every type of security key was well-supported everywhere, you would probably just want a USB-C key for your computer and NFC key for your mobile devices. Holding your NFC key up against your phone for a moment is such an easy task compared to plugging anything into the USB-C port—dramatically easier than using a Bluetooth key that constantly needs to be charged, uses an archaic Micro-USB port and has to be paired.

But that would be too easy. Unfortunately, we're stuck with a not-so-great matrix of varied security key support across devices, operating systems and browsers. Below I've put together some personal security key recommendations depending on what mobile device you use. I focused on mobile devices as the differentiator as desktop support is already fantastic for any USB key.

New users

For those that want to test the waters first

If you're not sure if you want to fully commit to security keys right now but want to give it a try, I'd recommend starting off with an affordable, no-frills USB-A key like the Security Key by Yubico. It's perfect for letting you try out web authentication flows on your desktop computer easily. Even having one key would still provide benefit of protecting you from phishing attempts while serving as just an extra second factor authentication method alongside your current TOTP/authenticator app.

Just note that this model cannot be used with desktop and mobile Yubico Authenticator apps to store TOTP secrets and function as an authenticator app for services that don't support security keys yet. Those are reserved for higher-end "YubiKey" series keys from Yubico.

Security Key: The basic $20 Yubico key — Amazon · Yubico

Recommendation for iOS users

Lightning, NFC, USB keys & limitations

If you use iOS today, your best bet it to use either an NFC key or a Lightning connector key. Unfortunately, while Yubico has announced that they will release an NFC key that is also USB-C in Q1 2020, it's not out yet. Until then, I would recommend the YubiKey 5Ci that is both Lightning connector and USB-C.

A dual USB-C and Lightning connector YubiKey will also be handy if you need to use this on an iPad as current iPads do not have any NFC hardware. An NFC key may prove to be more convenient if you only plan to use it with your iPhone.

YubiKey 5Ci: Dual Lightning connector and USB-C FIDO2 security key. — Amazon · Yubico
YubiKey 5 NFC: NFC/USB-A key. FIDO2 & multi-protocol — Amazon · Yubico
YubiKey 5C: USB-C FIDO2 & multi-protocol security key — Amazon · Yubico

For Google users

Of course, it's not that easy. If you use Google services, you will need a Bluetooth key¹⁷ as they have not yet updated to take advantage of Safari's WebAuthn support nor implement their own into Chrome for iOS. And then your best bet is to also get a USB-C key for desktop use.

Unfortunately, no great Bluetooth keys exist. They all seem mediocre¹⁹ in terms of design, durability, usability and lacking FIDO2 support. And then there's the fact that many of these keys, like ones by Feitian and Thetis, are made in China. Paranoid security folks avoid them due to concerns around untrustworthy firmware.

The Bluetooth key I'm linking to here is Google's Bluetooth Titan Security Key. These keys are essentially rebranded Feitian keys loaded with Google's own firmware. These keys seem to be only FIDO U2F keys, not the latest FIDO2, so there is no WebAuthn passwordless support.

Up until recently, the Titan Key was only sold in a bundle, but Google recently began selling them separately, along when they announced their new USB-C key. Google's new USB-C Titan key is created by Yubico but is neutered and lacks the flagship Yubico features I would have expected: support for multiple protocols (mainly to be able to use Yubico Authenticator) and full FIDO2 support. As such, I recommend the YubiKey 5C²⁰ instead.

If these options are limiting for you and the services you wish to use won't work with a Bluetooth key on iOS (at this time that's most things not Google), there is one fallback: adding an authenticator app as another second factor to use for these scenarios.

Update (Jun 2020): Google now supports WebAuthn for iOS with USB, NFC and Lightning connector security keys! This is huge news and means you no longer have to rely on sub-par Bluetooth security keys. As such I am removing my recommendation for Bluetooth security keys.

That being said, Chrome for iOS does not yet support WebAuthn but Google has been making great progress lately between this update and their recent Google Smart Lock update, so a Chrome for iOS update can't be too far away.

There's no way around it: Bluetooth security keys are annoying.

Recommendation for Android users

Use whatever you like, Android is the best.

If you use Android, you can basically do whatever you want: support for all sorts of keys across Android is great. Personally, I would just recommend going for multiple USB-C keys (assuming your desktop computer is also USB-C).

I tend to prefer the smaller form factor keys over the slight convenience benefit of a larger NFC key, but that really comes down to personal preference. When the NFC version of the YubiKey 5C comes out, I would recommend that over the USB-A version shown below.

YubiKey 5C: USB-C key. FIDO2 & multi-protocol — Amazon · Yubico
YubiKey 5 NFC: NFC/USB-A key. FIDO2 & multi-protocol — Amazon · Yubico

About nano-sized keys

Tiny keys for frequent use

Nano-sized security keys are made to be remarkably low-profile and meant to remain in a USB port. They're so small that they're even a bit annoying to remove—and you're likely to lose them if you do.

They're intended for keeping in computers, or even servers, where security key functionality will be used frequently and ease-of-access to touch the key is important. I find having one plugged into the USB port on my desktop's monitor to be very handy: only a short distance to tap it when I need to login. But this will especially be handy for those doing more with their security keys than just two-factor auth, like using it for PGP and SSH keys.

It's up to you to decide if you would feel comfortable using a nano key in a laptop you travel with where there is a risk of the two being stolen together, compared to a stationary desktop computer.

YubiKey 5 Nano: Extremely compact USB-A FIDO2 key — Amazon · Yubico
YubiKey 5C Nano: Extremely compact USB-C FIDO2 key — Amazon · Yubico

Nano-sized keys live in the USB port of your computer, ideal for cases where they will be used frequently.

An ode to Yubico

I really love Yubico security keys. It's no surprise that one of the original companies pushing for strong authentication online knows the importance of high quality, durable hardware. I’ve never had any issues with Yubico build quality and their higher-end YubiKey series supports a robust feature set that's customizable with software. It costs a premium but you can trust it and they are known to do recalls if they suspect there are any issues that could affect security (they replaced one of my older 4C keys).

If I had to nitpick, their software (YubiKey Personalization Tool, YubiKey Manager, Yubico Authenticator) isn't simple or well-designed.

There is, however, one significant criticism about Yubico keys from security folks in the industry. It's that Yubico's keys are closed source, preventing developer community-sourced verification of secure code along with the ability to update and modify the device however the owner wishes. For customers that fall in that camp, there are open source options like SoloKeys.

Hopefully that all made some sense! If you're still not sure what you need, it might be worth checking out this quiz from Yubico to help you find the right security key for your needs.

Registering your keys

Using security keys for two-factor auth

So you've purchased a security key or a few and are ready to set things up.. great! Each security key you own must be registered with each service you wish to use it on such as Google, 1Password, Facebook, GitHub, et cetera.

You'll register each key one by one. Usually navigating to the website's settings page related to two-factor auth and finding a button to add a key. This assumes you've already setup two-factor auth. Then when prompted you plug your key in and tap it. Depending on the website you may be prompted to type in a name for each key. This step is vital as it helps you identify which key to remove from the service in the event you lose that key and have multiple registered on the same service.²²

And that's pretty much it. Once a key is registered with a service, the next time you log in ²³ it's as easy as touching your security key when prompted after entering your username and password.

That's generally how two-factor authentication with security keys works but these things differ from website to website depending on how they've implemented things. For example, you may be prompted for a code from an authenticator app instead (if you have one added as a second factor option) and are in a browser that does not support FIDO U2F or FIDO2 WebAuthn.

YubiKey 5Ci plugged into a MacBook Pro

You can check to see if the websites and services you use support security keys by taking a look at DongleAuth.info. It's also easy enough to search the settings pages of the websites you use for any mention of security keys for two-factor authentication.²⁴

Different browsers will display varying prompts requesting you to touch your security key. Left: Windows 10 Firefox Right: Chrome for macOS Not pictured: Safari 13 for macOS Catalina (while it has experimental WebAuthn support, it shows no UI to prompt for a key yet)

Setting up Google Advanced Protection

Google's strongest, security key-required security program

I mentioned the benefits of Google Advanced Protection earlier in this article. It is designed to help thwart targeted attacks by requiring the use of security keys, in addition to numerous enhanced security measures.

Signing up for Advanced Protection follows the same process I mentioned above: enable two-factor auth, register and name two security keys and you're ready to go.

Chrome will ask to see detailed info about your key during the key registration process.

Advanced Protection has some aggressive requirements: you must register at least two security keys, you can't add any TOTP authenticator apps as a second factor and there's no recovery code. You can however add an approved Android device as an authenticator. Google requires having two keys with Advanced Protection so that one can operate as a spare should you lose one.

Use G Suite? You'll have to enable it first.

If you use the enterprise G Suite for your email, you may not have the option to even enable the Advanced Protection Program until your administrator has turned it on. If you run G Suite for yourself, you can read about enabling the beta here.

What about regular Google accounts?

If you're not ready to make the jump to Advanced Protection

If you just want to test the waters a bit with security keys and don't think enabling Advanced Protection is for you at this time, you can always just register keys as part of any Google account's regular 2-Step Verification settings.

The process for adding a key is similar to the Advanced Protection flow from above but you just navigate to Google Account → Security → 2-Step Verification → Add Security Key.

With regular Google accounts, you're free to mix second factor authentication types from SMS to authenticator apps and even other phones using Google 2SV phone prompts. Of course, I strongly advise against using SMS at all.

1Password

If you use the hosted option for 1Password password manager, adding your keys is a simple process as well. Just log into 1Password.com, click your name in the corner then navigate to My Profile → More Actions → Manage Two-Factor Authentication → Add a Security Key.

You can add as many keys as you like. However, the current 1Password implementation does not yet allow you to remove the authenticator app as a second factor. Since people often use 1Password on their mobile devices in addition to their desktop computers, they would likely only support security key-only authentication once every 1Password mobile app offered security key support.

Only the 1Password iOS app supports physical security keys at this time and via a custom Yubico Mobile SDK integration only supporting the new and expensive Lightning connector YubiKey 5Ci.. so definitely not for everyone just yet.

Update (May 2020): 1Password for iOS now supports NFC security keys, though only the YubiKey 5 NFC for now it seems.

Will 1Password ask for my security key every time?

Adding a security key does not mean that 1Password will ask for it every single time you need to open 1Password to log into an account: something that likely happen many times throughout the day for you. 1Password only really asks for your security key when adding your 1Password account to an untrusted device like a new browser or desktop app.

Fastmail

Fastmail is a popular non-Google email service that has supported security keys for years now. It also lets you entirely remove SMS, recovery codes and authenticator apps if you so wish. Setup is trivial as well: Settings → Password & Security → Two-Step Verification → Add Verification Device → Set Up Security Key.

Facebook

Of course, you can register your security keys with your social accounts like Facebook as well. The process is the same: find the two-factor auth section in the account settings then add and name each of your security keys.

Registering keys on other services

If you have any questions about how to set up your security keys on other services, Yubico maintains a well-documented section on their site for just about every supported service.

What about mobile?

It's easy on Android, less so on iOS..

By now you've registered your security keys and figured out how to log back in with them on your desktop browser. But how do you login on your mobile devices with security keys?

As I mentioned in the Android section earlier, it's a breeze. The operating system itself (well, Google Play Services technically) supports FIDO2 so any website or app should trigger this authentication activity view, giving you numerous authentication options such as with NFC, USB and Bluetooth keys.

Using your security key on Android just works. (Pictured: Pixel 3 + YubiKey 5C USB-C key)

It's not the same with iOS. You're out of luck if your main browser is Chrome for iOS as it does not have support yet. While iOS 13.3 brought support for USB, NFC and Lightning keys to Safari, not all apps and services have taken note yet.

If you use a Google account on your iPhone or iPad, however, you do have another option. Google has a separate authentication app called Google Smart Lock for this purpose. You log into all of your Google accounts in the Smart Lock app—it supports Bluetooth security keys—and it can redirect you back to the app you came from to complete your authentication. This works for native Google apps as well as apps utilizing Google login.

Using a Bluetooth security key on iOS with Google Smart Lock.

It's not a perfect solution but you don't have much of a choice for Google apps and services on iOS right now. Even attempting to log into Gmail in Safari does not let you use a security key directly in Safari; instead it tries to redirect you to Google Smart Lock.

Having to install a separate app just to use your security key is a hassle. Then you add on to that some general Bluetooth security key woes: having to pair the key, having to keep your security key charged at all times, having to carry around a MicroUSB cable around to keep the key charged, having the key's button always accidentally get pressed in your bag and drain the battery. Annoying.

However, I'm hoping that in the first few months of 2020 we'll begin to see more apps and services update to support security keys, at least via utilizing SFSafariViewController to take advantage of the existing WebAuthn implementation.

Bonus: Pixel 3 as a security key

Take advantage of the integrated Titan M security chip

This probably isn't what you have in mind right now when you think about a security key. For most of this article I've been talking about external hardware authenticators that you plug into your computer or mobile device. But they don't always have to be separate from your device.

More and more devices have integrated security chips, some Google Chromebooks even have an integrated "H1" chip that can be used for FIDO U2F auth. These integrated security chips may go by various names like secure element, TPM and Apple's Secure Enclave, but the gist is generally the same. They are secure cryptoprocessors that have a secure boot process and microkernel to completely isolate itself from the rest of the hardware. Much like USB security keys, the system cannot access the secrets stored on the security chip: the chip itself just runs computations. Now, with WebAuthn and initiatives like this Google solution I'm about to describe, there are more ways for these chips to be used for web authentication.

But I digress... the Pixel 3 has a "Titan M" security chip that can be used to store your cryptographic keys for second factor authentication. This is similar to how an external security key is used to generate a unique public/private key pair for auth. When you go to Google account settings to add another second factor for 2-Step Verification, it will display a prompt to add your Android device if it recognizes that your Google account is logged in and supported. While it supports any Android device running Android 7+, it's only the Pixel 3 (and I'm guessing the new Pixel 4 can do the same) that features this enhanced protection. (Update January 2020: Google has updated the Smart Lock app for iOS to enable similar functionality for iOS users.)

Adding the Pixel 3 as a second factor to a Google account.

It's important to note that this is not the same type of phone prompt you may have seen in the past where your phone receives a login request notification via the cloud. This new implementation is quite novel: it's local proximity based, and as such it requires Bluetooth to be enabled on both devices. Also, Google was able to get this working entirely without requiring Bluetooth pairing, which is always a hassle. They developed an extension to WebAuthn that introduces pairing-less secure communication between your computer and the mobile device.

When you log into your Google account, it will communicate with your phone locally via Bluetooth. Similar to an external security key, user presence is required in the form of a physical press. Google had the security chip hardwired to the Pixel 3's volume down button. As such, not even malicious software running on the phone would be able to control the Titan M chip on its own.

Why does it ask for a PIN?

Set a PIN to enhance user verification on your FIDO2 key

If you're using a FIDO2 security key you may have been prompted at some point during the registration process or elsewhere to set or enter a PIN for the key. This was confusing to me the first time it happened, especially when I was using my key as a second factor.

Of course, there is a reason for this. But first, I'll go ahead and say that for the purpose of using the key as a second factor only, you don't need to set a PIN.

Left: Setting a FIDO2 key PIN in YubiKey Manager Right: Chrome for macOS prompting for the PIN.

The PIN is there to enable a form of multi-factor authentication for passwordless authentication flows. We're not quite there yet as not many services support passwordless functionality, but the modern security keys support it. FIDO2 keys have the ability to store username and password credentials directly on the key, via FIDO2 resident credentials.

This is a significant change to how things were done with FIDO U2F where it was not possible to know which services had been registered with the key. Now with resident credentials it's possible for someone to ask the key to list out what domains it has been registered on or just what accounts exist on the key for a given domain. As such, you need a way to add a stronger user verification step to limit that access. In this passwordless world, your FIDO2 hardware key can require user verification with a PIN that is local and stored directly on the key itself, or it can require something like a biometric gesture if the device has such sensors/readers and was configured that way.²⁵

But wait, I thought this was supposed to be passwordless.. why do I need to set and remember a PIN? Yubico has the reasoning here:

A PIN is actually different than a password. The purpose of the PIN is to unlock the Security Key so it can perform its role. A PIN is stored locally on the device, and is never sent across the network. In contrast, a password is sent across a network to the service for validation, and that can be phished. In addition, since the PIN is not part of the security context for remotely authenticating the user, the PIN does not need the same security requirements as passwords that are sent across the network for verification. This means that a PIN can be much simpler, shorter and does not need to change often, which reduces concerns and IT support loads for reset and recovery. Therefore, the hardware authenticator with a PIN provides a passwordless, phishing-resistant solution for authentication.

—Yubico

Final words of advice

Keeping track of your keys

If you're like me you probably have quite a few websites you will be using your security keys with. If you have more than a few security keys it may be handy to make a spreadsheet of each key you own and each service it is registered on (or in your password manager, add a note for each login about what keys are registered with each service.

Done. Now I’ll just have to wait for banking hours when I lose my other u2f keys and lock myself out of my accounts 😂 pic.twitter.com/kC2NmX8TTa

— Paul Stamatiou 📷 (@Stammy) August 15, 2018

This is particularly handy if you store any of your keys at separate locations for safekeeping and need to know which ones to register on new services when you get a hold of them next.

Which brings me to my next point, if you have more than one security key, you should keep the spare somewhere safe. That could be a trusted friend or family member's house, it could be your place of work, a bank safe deposit box, et cetera. Unless you think you could be targeted, this is likely just to prevent you from misplacing the spare security keys at your place.

Pro tip: Disable Yubico OTP mode

There's one common annoyance with the default YubiKey configuration: accidentally tapping the button on a YubiKey will make it begin spewing out a large string of random characters each time. Contrary to popular belief, this is not related to actual FIDO/FIDO2 functionality. It's a mode specific to YubiKeys for another type of authentication that you probably won't be using: Yubico OTP.

This is distinct from the TOTP you already know about for generating app codes, and different from using your YubiKey to store TOTP secrets. Here's more detail about Yubico OTP.²⁶

The point here is that having a long string accidentally get inputted into whatever application or document you were working on will drive you crazy. It happened to me one time when I was working in Adobe Lightroom as I reached over to plug in a USB SD card reader and accidentally tapped the YubiKey in the other USB port.. and let me tell you, Lightroom went crazy and interpreted every keystroke as a command. It took me a while to undo that mayhem.

Needless to say, I disable Yubico OTP mode on every YubiKey I own. Fortunately, it's very easy. Install the YubiKey Manager app and uncheck OTP mode:

On the Interfaces tab uncheck OTP and save.

FAQ

Because this is entirely more complex than it should be

What is a security key?

Security keys, security tokens, U2F keys, roaming authenticators, platform authenticators—whatever you want to call them—are hardware authenticators that run public-key cryptography operations in a manner that is entirely isolated from the rest of mobile device or computer. There are two general classes of security keys: internal platform authenticators and external roaming authenticators.

Platform authenticators are integrated inside of a device. They may take the form of an integrated security chip accessible by software for the purpose of web authentication, such as the security chip in the Pixel 3, or biometrics devices like fingerprint readers and facial recognition sensors.

This article has primarily been about other variant: roaming authenticators. These are the little external hardware authenticators you connect to any device (via Bluetooth LE, NFC or USB) to carry out your authentication needs. They communicate with your computer via FIDO CTAP (Client to Authenticator Protocol).

In one sentence, why is a security key better than an authenticator app for two-factor authentication?

There are several reasons (view the keys vs TOTP section above), but the main one is that security keys prevent phishing by using public-key cryptography that verifies the domain you're attempting to authenticate with is the same one you originally registered your key with.

What services and browsers support WebAuthn?

See the compatibility section above. For the most part, support is great across Windows, Linux, ChromeOS, macOS and Android. iOS is getting there with recent Safari support for USB, NFC and Lightning connector security keys. However, many iOS apps and browsers still need to catch up.

As for what websites support FIDO2 WebAuthn (and the earlier FIDO U2F) for two-factor auth, it's probably easiest to just log into the website and poke around the security and two-factor auth sections. You can also check DongleAuth.info or the Yubico catalog. FIDO2 WebAuthn also supports passwordless functionality but the vast majority of websites do not support that at this time.

How often do I need use my security key?

This depends on the exact implementation for each website, but for the most part: not that often. Most services only prompt for your security key if you are using an untrusted browser or device. For example, after logging in the service will mark your browser as trusted for 30 days, or until you log out. However, if the service thinks something is weird like you are logging in from a different IP in another country or you're in a new browser, you will need to do the full two-factor auth login with the security key.

For mobile apps like Gmail, it seems to not automatically expire the trusted client status. You only need to present the security key the first time you add your Google account to the phone.

Whether this means you should carry around your security key with you at all times is up to you. But if you're traveling, I would definitely take it. And a second key as a backup (assuming you still have a spare somewhere else safe) that you store in a different place while traveling. This is similar to how I travel with SD cards when on photography trips. I keep a set of SD cards on me and a duplicate set of SD cards (or imported to my computer if traveling with one) at my lodging, until I'm able to get home and load it on my NAS.

What happens if I lose my security key?

You just log in with your spare security key and remove the security key you lost from the account. Easy.

To get more specific: This depends on the service, but it's almost certainly the case that you have several other second factors you can use to login with (be it backup security keys or TOTP authenticator apps) or that you have a recovery method configured (email reset or recovery code). Since you are still able to log into your services, you just need remove the lost security key from your account. That's why naming the key something specific, like the model name of the key, is handy to help you figure out which key to remove from the account.

Removing the lost security key also protects you in case you happen to lose your remaining second factor key(s) or do something like accidentally wipe a phone that had your only TOTP authenticator app that wasn't backed up. If that happened, the service would still think you have your second factor handy and would prevent login. Of course, depending on the service and how you configured it, it might still allow recovery.

What happens if someone else gets access to my security key?

The short answer here is that you're still protected. Someone can't use your security key to log in as you wherever you have it registered as your second factor. For one, it's a second factor. They would still have to know your username and password. Second, they would have to know what websites you have your key registered with. And they cannot get that from the security key—it doesn't even exist in the key. This gives you enough time to remove the key from your account when you notice it's lost or stolen.

The more complicated answer emerges if we begin talking about newer FIDO2 functionality intended for passwordless authentication. FIDO2 resident key credentials are a way of storing the username along with other data on the actual security key itself. When the server requests a credential from the client and only provides the origin to the key, the key has to know which resident (on the key) credential to provide. If there are multiple resident credentials for that website, it must provide a way for the user to select which credential to use.

Needless to say, the security key itself needs to have this info locally for this functionality and as such, there is a way for someone with access to your security key to list out the saved resident credentials.

However, it's important to note that this can be safeguarded with an optional user verification step requiring a PIN or biometric (PIN is more well-supported today). This only applies to services utilizing resident key credentials, which seems like it's only going to be single factor passwordless and not multi-factor passwordless configurations that have that extra verification step.

And finally, it seems like the ability to list out all resident key credentials is nascent, with only Yubico starting to offer this ability.

But for the most part, unless part of a highly targeted attack, chances are anyone that found the key would think it's a broken USB memory stick. Or if they know what it is, they could start using it themselves for their own U2F/etc needs and it would work just fine. You should be more concerned about attackers stealing your info online. Using a security key as intended is your best defense there.

Can I log into my Mac, Windows PC or Linux computer with my security key?

Support for using your security key to log into your computer is varied and in most cases will require a Yubico brand YubiKey, not a regular security key from another provider that lacks certain functionality.

Support for Windows login has had some ups and downs over the years but it just became officially supported with the new Yubico Login for Windows. Unfortunately, setting it up seems quite involved. It used to work with YubiKey 4 series keys alongside a Windows program called YubiKey for Windows Hello but that was deprecated and Yubico then pointed customers to a much older implementation called YubiKey Windows Logon.

MacOS users can use certain YubiKeys to log into their computers, but installation is definitely left to advanced users. And the story is the same with Linux: it's possible but challenging to configure the pam_u2f module properly.

Why does my YubiKey print out a bunch of random characters when I tap it?

If you tap your YubiKey—this only applies to YubiKeys— while your cursor is focused on any sort of input field, it may spew out a ton of seemingly-random characters. It's triggering a proprietary feature called Yubico OTP that is unrelated to standardized FIDO2/WebAuthn stuff. You can easily disable Yubico OTP mode on your YubiKey.

What else can security keys do?

Yubico brand YubiKeys feature a variety of technologies beyond FIDO U2F and FIDO2 capabilities, including: PIV/Smart Card, OpenPGP, OATH-HOTP/OATH-TOTP and Yubico OTP. You can manually configure your key to store ssh keys, GPG private keys, sign your git commits and more. It was also recently announced that OpenSSH will be receiving support for U2F, enabling your security key to be your keypair.

Personally, I find one of the more intriguing YubiKey uses is using it for TOTP code generation. This is handy for websites that don't yet support FIDO U2F or FIDO2 security keys but do support TOTP authenticator apps. Once configured, the YubiKey can store up to 32 credentials. They are accessible by plugging in your key and opening up the desktop or Android Yubico Authenticator app. Yubico has hinted that iOS app is on the way (Update: Yubico Authenticator for iOS is out now and supports the YubiKey 5Ci and NFC keys).

What's next?

A recap and look at the future of web authentication

Today we learned about improving your online security through several facets from using a password manager, removing your phone number from your online accounts and then migrating to security keys to protect against phishing attacks. However, being diligent about your online security and using security keys won’t protect you from every attack vector. If everyone began using security keys overnight, attackers would focus all their efforts on other attack vectors.

Not everyone needs the most paranoid level of security. It's always a trade-off between security and convenience. The more paranoid you get, the more likely you are to just lock yourself out of accounts.

On top of that, we need more sites to treat security keys as first class citizens. I can't even use multiple security keys on Amazon AWS yet and that's the place you would think would need it the absolute most. Why can't I opt out of SMS entirely on every service? Why don't more services let me only allow security keys as my only type of second factor?

As much as I want to evangelize how great security keys are today, lets face it— they're kinda annoying. You have this new device—probably several—to keep track of now. I don't expect everyone reading this post to go out and start using security keys right away. We shouldn't have to go through this much trouble to be a bit safer online.

The goal post with FIDO2 lies one step beyond what I've talked about here today about using these security keys as a second factor for two-factor authentication. The vision is around using devices you already have—phones with face scanning cameras, laptops with fingerprint readers, maybe even smart watches one day—to help authenticate you on websites securely.

The hope is that authentication may happen as passwordless but I really think that is a hard concept for people to grok after decades of being primed on how passwords work. I'll be keeping a keen eye on this space over the next decade.

Recap

Because this was long and complex

Seemingly everyone is getting hacked today. SIM port and SIM swap attacks involve someone taking over your phone number, typically by socially engineering your mobile phone carrier. With access to your phone number, attackers can rely on your various online accounts having your phone number added as a two-factor auth method with SMS. Attackers can then begin gaining access to every such account (or via your email) and wreak havoc on your digital and financial assets.

These attacks are becoming more common. If part of a targeted attack, it may not even matter if you have a PIN/passphrase set with your mobile carrier.
Don't use SMS for two-factor auth. Google has shown that utilizing SMS as a second factor is still beneficial (compared to not having two-factor authentication at all) for automated attacks using stolen credentials, such as from data breaches. However, SMS is horrible for targeted attacks.

But, in an abundance of caution, it is highly recommended to remove phone numbers and SMS recovery methods from every account where possible. Enable two-factor authentication everywhere and start with using an authenticator app/TOTP code generator for the second factor instead of SMS.
You can increase your online security in a number of ways. There are numerous other security precautions and best practices one can follow depending on their threat model and comfort with increased inconvenience for enhanced security. The first and most highly-recommended is using a password manager, like 1Password. Further down the line, you may want to switch your mobile carrier to Google Fi and enable Google Advanced Protection for your Google accounts.
Using a security key for two-factor auth prevents phishing. These tiny little devices are secure cryptoprocessors that utilize public-key cryptography to generate unique key pairs for every website. The website stores the public key while the private key remains isolated on the security key, inaccessible to even malware running on the machine.

Security keys take the origin/domain of the website into account and will not allow successful authentication, even if the user is tricked into using it on a look-alike phishing site. This is something that authenticator/code generator apps cannot do. There are other benefits too.
Security keys are just one part of the larger FIDO2 vision. Security keys live under an umbrella called FIDO U2F and a newer (and backwards compatible with FIDO) FIDO2 moniker. FIDO2 introduces a browser Web Authentication API along with a modern protocol for communicating with hardware called CTAP2. FIDO2 leads the way to a simpler and stronger online authentication with support for more devices, including biometrics already integrated into devices, as well as strong passwordless single-factor and multi-factor auth.
Security key support is getting better—both websites that support it as well as browsers/devices—but it's not ubiquitous. It's not the best user experience today. You shouldn't have to carry around a separate device just to be safe online. WebAuthn aims to change that. But for now, security keys—combined with strong online security best practices—are a great way to fortify your regular online activities.

Footnotes

1 Of course, there are other ways to get your SMS two-factor codes—such as being phished via an official-looking text message to relay an SMS code.

2 And recently, much more with modern "passwordless" functionality that I'll dive into later.

3 Despite my entire pre-amble about SMS, it largely does help thwart mass attacks, as reported by Google. SMS as 2FA can still prove beneficial for automated bots trying to log into sites with your password from a data breach, but does not help much in the face of targeted attacks. This act of trying out stolen credentials on various sites is called credential stuffing.

4 This is a separate functionality that YubiKeys have in addition to typical U2F usage. It can function as a TOTP code generator once a security key is plugged into the device (they have desktop and mobile apps).

5And about authy: Authy brings convenient multi-device support to the table. Accounts in Authy are encrypted with a passphrase and should be safe even if access is gained to your Authy account. Accessing the app can also be protected via PIN/fingerprint/FaceID. They have, however, had security incidents in the past that scared a lot of folks away.

6 Well.. when it comes to security, it's wise to never say never. DNS hijacking is the one attack vector that comes to mind. Token binding solves this but does not seem widely implemented. The same goes for security key counter checking..

7 FIDO U2F keys were part of the previous version of CTAP, so now they are labeled as CTAP1 under FIDO2. CTAP1 is backwards compatible with the latest FIDO2 CTAP2 protocol. Yea, there’s a lot of nuance in here..

8 I say authentication but this part is user verification, not authentication. The difference is the user is proving itself to the particular device instead of the website.

9 If you want to see an example passwordless auth flow, look at this slide deck from Yubico/Google (Start at slide 30).

10 Microsoft.com is the only one that comes to mind.

11 SoloKey, the first open source FIDO2 security key, supports up to 50 resident keys.

12 And if your computer doesn't have internal biometrics you can use for this purpose, you don't have many options. There aren't really any good CTAP2 devices on the market yet that include fingerprint readers. Update (Nov 2019): Yubico has announced that they are working on YubiKey Bio, the first biometric YubiKey that integrates a fingerprint reader for multi-factor auth purposes and does not require any batteries or custom drivers. This makes it ideal for enhancing user verification with biometrics (you can do old PIN-based too) when using passwordless auth. Unfortunately, it seems like this first device is only USB-A, not USB-C.

13 And for United States residents, you could be compelled to provide your biometrics to unlock a device and not invoke the Fifth Amendment, unlike a password.

14 They would still need your password but could try credential stuffing from any public data breach and hope to get lucky. Or if you had any malicious software running on your machine when you register a new authenticator, the provided secret key could be captured. I’m really getting into the weeds here, but in general having any shared secret can be seen as another potential attack vector.

15However, this gets complicated as FIDO2 brings new functionality called resident key credentials. This makes it possible to locally store the additional user data required to make passwordless authentication function, but also introduces the ability to list the stored credentials on the security key. Fortunately, there are ways to protect against this with additional user verification.

16 Unfortunately, Google's Titan keys do not support FIDO2 functionality at this time.

17 I have yet to come across another service that has an iOS app that allows it to utilize paired Bluetooth security keys like Google Smart Lock does.

18 Unless you are okay with using Brave browser along with the YubiKey 5Ci.

19 The Google Titan Bluetooth key is frustrating to use as there's no way to turn it off completely and it always turns itself on and blinks forever when in my bag or pocket, eventually draining itself so you need to have a Micro-USB cable handy.

20 Yubico also has FIPS versions of their keys that meet certain compliance requirements for special needs but they do not support FIDO2 (yet) so I would not recommend unless you know you need a FIPS 140-2 key.

21 Technically the Google Titan Bluetooth key also has NFC but it's not FIDO2.

22 I go into more detail about this "what happens if I lose my key" scenario in the FAQ below. This assumes you were able to log back into the website with your backup/spare key (or other form of second factor.. went through an account recovery flow) and want to prevent it from being used by an unauthorized person that finds it (this would be hard as they wouldn't know what account/username was registered to it.. but I go into detail about this in the FAQ too) or prevent you from getting locked out in the future.

23 A few caveats: it may not be the next time you log in if the service has you marked as using a trusted client. This depends on the service and how they categorize trusted clients. It could depend on a time period, different IP/location, new browser, et cetera.

24 Most websites that support security keys today tend to only support FIDO U2F and not FIDO2 WebAuthn. But that's fine for two-factor auth purposes. You're still accomplishing what we came here to do with this article: enhance your online security, prevent phishing.

25 Adding an enhanced, local user verification step like this to your passwordless single factor makes it passwordless MFA (multi-factor authentication).

26 There's also some, arguably paranoid, concern that you can be tracked across multiple accounts and websites by using Yubico OTP mode as part of the string used contains the device identifier of your key.

Donating $10k to FreeCodeCamp

swyx's site RSS Feed — Sun, 20 Oct 2019 03:00:00 GMT

I benefited from FreeCodeCamp and would like to give back

What are you working on ?

Linux Engineer's random thoughts — Sun, 20 Oct 2019 01:00:00 GMT

This was previously published on LinkedIn

Until very recently I couldn’t really answer this question. But it has shipped, so now I can tell you: I've been working on Edge Computing.

To be more precise, this is the first compute service available to ISP subscribers directly on their ISP …

Django Docker Box is now an official Django project!

Tom Forbes — Fri, 18 Oct 2019 00:40:54 GMT

When I was working on adding queryset.explain() to Django I got annoyed by how complex it was to set up a local Django environment with multiple databases and versions. The traditional way of handling this was to use django-box which utilizes Vagrant to spin up a VM and install different types of da...

Testing my dotfiles with Github actions

Tom Forbes — Sat, 12 Oct 2019 21:01:55 GMT

I went through a phase that involved reinstalling MacOS way, way too often. At a certain point I got fed up with installing things over and over again and decided to version my dotfiles. And so https://github.com/orf/dotfiles was born. This reolved around a bootstrap.sh script that clones your dotfi...

Elasticsearch mappings - How to add Facets and Aggregations

Ben E. C. Boyter — Sat, 12 Oct 2019 03:00:00 GMT

Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub

In elasticsearch mappings define type and values in documents. You use them to specify that fields within your document should be treated as numbers, dates, geo-locations and whatever other types elasticsearch supports. You can also define the stemming algorithm used and other useful index fields.

Interpreting Go

Notes on software development — Sat, 12 Oct 2019 03:00:00 GMT

After spending some time at work on tooling for keeping documentation in sync with Go struct definitions I had enough exposure to Go's built-in parsing package that next steps were clear: write an interpreter. It's a great way to get more comfortable with a language's AST.

In this post we'll use the Go parser package to interpret the AST directly (as opposed to compiling to a bytecode VM) with enough to support a recursive implementation of the fibonacci algorithm:

package main

func fib(a int) int {
  if a == 1 {
    return 0
  }

  if a == 2 {
    return 1
  }

  return fib(a-1) + fib(a-2)
}

func main() {
  println(fib(15))
}

You'll note this isn't actually valid Go because we are using an undefined function println. We'll provide that for the runtime to make things easier on ourselves.

The fibonacci algorithm is my goto minimal program that forces us to deal with basic aspects of:

Function definitions
Function calls
Function arguments
Function return values
If/else
Assignment
Arithmetic and boolean operators

We'll do this in around 200 LoC. Project code is available on Github.

A follow-up post will cover support for an iterative fibonacci implementation with support for basic aspects of:

Local variables
Loops

First steps

I always start exploring an AST by practicing error-driven development. It's helpful to have the Go AST, parser, and token package docs handy as well.

We'll focus on single-file programs and start with parser.ParseFile. This function will return an *ast.File. This in turn contains a list of Decls. Unfortunately Go stops being helpful at this point because we have no clue what is going to implement this Decl interface. So we'll switch on the concrete type and error until we know what we need to know.

package main

import (
  "go/ast"
  "go/parser"
  "go/token"
  "io/ioutil"
  "log"
  "os"
  "reflect"
)

func interpret(f *ast.File) {
  for _, decl := range f.Decls {
    switch d := decl.(type) {
    default:
      log.Fatalf("Unknown decl type (%s): %+v", reflect.TypeOf(d), d)
    }
  }
}

func main() {
  fset := token.NewFileSet() // positions are relative to fset

  src, err := ioutil.ReadFile(os.Args[1])
  if err != nil {
    log.Fatalf("Unable to read file: %s", err.Error())
  }

  f, err := parser.ParseFile(fset, os.Args[1], src, 0)
  if err != nil {
    log.Fatalf("Unable to parse file: %s", err.Error())
  }

  interpret(f)
}

Build and run:

$ go build goi.go
$ ./goi fib.go
2019/10/12 09:43:48 Unknown decl type (*ast.FuncDecl): &{Doc:<nil> Recv:<nil> Name:fib Type:0xc000096320 Body:0xc00009a3c0}

Cool! This is the declaration of the fib function and its type is *ast.FuncDecl.

Interpreting ast.FuncDecl

A function declaration is going to need to add its name to a context map, mapped to a function reference for use in function calls. Since Go throws everything into the same context namespace this we can simply pass around a map of strings to values where a value can be any Go value. To facilitate this, we'll define a value struct to hold an integer to represent "kind" and an empty interface "value". When a value is referenced it will have to switch on the "kind" and then cast the "value".

Additionally, and unlike a value-oriented language like Scheme, we'll need to track a set of return values at all stages through interpretation so, when set, we can short circuit execution.

type kind uint

const (
  i64 kind = iota
  fn
  bl
)

type value struct {
  kind  kind
  value interface{}
}

type context map[string]value

func (c context) copy() context {
  cpy := context{}
  for key, value := range c {
    cpy[key] = value
  }

  return cpy
}

type ret struct {
  set bool
  vs  []value
}

func (r *ret) setValues(vs []value) {
  r.vs = vs
  r.set = true
}

func interpretFuncDecl(ctx context, r *ret, fd *ast.FuncDecl) {
  ctx[fd.Name.String()] = value{
    fn,
    func(ctx context, r *ret, args []value) {},
  }
}

func interpret(ctx context, f *ast.File) {
  for _, decl := range f.Decls {
    switch d := decl.(type) {
    case *ast.FuncDecl:
      interpretFuncDecl(ctx, nil, d)
    default:
      log.Fatalf("Unknown decl type (%s): %+v", reflect.TypeOf(d), d)
    }
  }
}

Now that we have the idea of return management and contexts set out, let's fill out the actual function declaration callback. Inside we'll need to copy the context so variables declared inside the function are not visible outside. Then we'll iterate over the parameters and map them in context to the associated argument. Finally we'll interpret the body.

func interpretBlockStmt(ctx context, r *ret, fd *ast.BlockStmt) {}

func interpretFuncDecl(ctx context, r *ret, fd *ast.FuncDecl) {
  ctx[fd.Name.String()] = value{
    fn,
    func(ctx context, r *ret, args []value) {
      childCtx := ctx.copy()
      for i, param := range fd.Type.Params.List {
        childCtx[param.Names[0].String()] = args[i]
      }

      interpretBlockStmt(childCtx, r, fd.Body)
    },
  }
}

And we'll add a call to the interpreted main to the end of the interpreter's main:

func main() {
  fset := token.NewFileSet() // positions are relative to fset

  src, err := ioutil.ReadFile(os.Args[1])
  if err != nil {
    log.Fatalf("Unable to read file: %s", err.Error())
  }

  f, err := parser.ParseFile(fset, os.Args[1], src, 0)
  if err != nil {
    log.Fatalf("Unable to parse file: %s", err.Error())
  }

  ctx := context{}
  interpret(ctx, f)
  var r ret
  ctx["main"].value.(func(context, *ret, []value))(ctx, &r, []value{})
}

Next step!

Interpreting ast.BlockStmt

For this AST node, we'll iterate over each statement and interpret it. If the return value has been set we'll execute the loop to short circuit execution.

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {}

func interpretBlockStmt(ctx context, r *ret, bs *ast.BlockStmt) {
  for _, stmt := range bs.List {
    interpretStmt(ctx, r, stmt)
    if r.set {
      return
    }
  }
}

Next step!

Interpreting ast.Stmt

Since ast.Stmt is another interface, we're back to error-driven development.

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  switch s := stmt.(type) {
  default:
    log.Fatalf("Unknown stmt type (%s): %+v", reflect.TypeOf(s), s)
  }
}

And the trigger:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:15:14 Unknown stmt type (*ast.ExprStmt): &{X:0xc0000a02c0}

Great! Checking the docs on ast.ExprStmt we'll just skip directly to a call to a new function interpretExpr:

func interpretExpr(ctx context, r *ret, expr ast.Expr) {}

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  switch s := stmt.(type) {
  case *ast.ExprStmt:
    interpretExpr(ctx, r, s.X)
  default:
    log.Fatalf("Unknown stmt type (%s): %+v", reflect.TypeOf(s), s)
  }
}

Moving on!

Interpreting ast.Expr

We've got another interface. Let's error!

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

And the trigger:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:19:16 Unknown expr type (*ast.CallExpr): &{Fun:println Lparen:146 Args:[0xc0000a2280] Ellipsis:0 Rparen:154}

Cool! For a call we'll evaluate the arguments, evaluate the function expression itself, cast the resulting value to a function, and call it.

func interpretCallExpr(ctx context, r *ret, ce *ast.CallExpr) {
  var fnr ret
  interpretExpr(ctx, &fnr, ce.Fun)
  fn := fnr.values[0]

  values := []value{}
  for _, arg := range ce.Args {
    var vr ret
    interpretExpr(ctx, &vr, arg)
    values = append(values, vr.values[0])
  }

  fn.value.(func(context, *ret, []value))(ctx, r, values)
}

All of this casting is unsafe because we aren't doing a type-checking stage. But we can ignore this because if a type-checking stage were introduced (which it need to be at some point), it would prevent bad casts from happening.

Handling more ast.Expr implementations

Let's give the interpreter a shot again:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:28:17 Unknown expr type (*ast.Ident): println

We'll need to add ast.Ident support to interpretCallExpr. This will be a simple lookup in context. We'll also add a setValue helper since the ret value is serving double-duty as a value passing mechanism and also a function's return value (solely where multiple value are a thing).

...


func (r *ret) setValue(v value) {
  r.values = []value{v}
  r.set = true
}

...

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.CallExpr:
    interpretCallExpr(ctx, r, e)
  case *ast.Ident:
    r.setValue(ctx[e.Name])
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

This is also a good time to add the println builtin to our top-level context.

func main() {
  ...

  ctx := context{}
  interpret(ctx, f)
  ctx["println"] = value{
    fn,
    func(ctx context, r *ret, args []value) {
      var values []interface{}
      for _, arg := range args {
        values = append(values, arg.value)
      }

      fmt.Println(values...)
    },
  }

  var r ret
  ctx["main"].value.(func(context, *ret, []value))(ctx, &r, []value{})
}

More ast.Expr implementations

Running the interpreter again we get:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:41:59 Unknown expr type (*ast.BasicLit): &{ValuePos:151 Kind:INT Value:15}

Easy enough: we'll switch on the "kind" and parse a string int to an int and wrap it in our value type.

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.CallExpr:
    interpretCallExpr(ctx, r, e)
  case *ast.Ident:
    r.setValue(ctx[e.Name])
  case *ast.BasicLit:
    switch e.Kind {
    case token.INT:
      i, _ := strconv.ParseInt(e.Value, 10, 64)
      r.setValue(value{i64, i})
    default:
      log.Fatalf("Unknown basiclit type: %+v", e)
    }
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

Now we run again:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:48:46 Unknown stmt type (*ast.IfStmt): &{If:38 Init:<nil> Cond:0xc0000ac150 Body:0xc0000ac1b0 Else:<nil>}

Cool, more control flow!

Interpreting ast.IfStmt

For ast.IfStmt we interpret the condition and, depending on the condition, interpret the body or the else node. In order to make empty else interpreting easier, we'll also add a nil short-circuit to interpretStmt.

func interpretIfStmt(ctx context, r *ret, is *ast.IfStmt) {
  interpretStmt(ctx, nil, is.Init)

  var cr ret
  interpretExpr(ctx, &cr, is.Cond)
  c := cr.valus[0]

  if c.value.(bool) {
    interpretBlockStmt(ctx, r, is.Body)
    return
  }

  interpretStmt(ctx, r, is.Else)
}


func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  if stmt == nil {
    return
  }

  switch s := stmt.(type) {
  case *ast.IfStmt:
    interpretIfStmt(ctx, r, s)

  ...

Let's try it out:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:56:28 Unknown expr type (*ast.BinaryExpr): &{X:a OpPos:43 Op:== Y:0xc00008a120}

Great!

Interpreting ast.BinaryExpr

An ast.BinaryExpr has an Op field that we'll switch on to decide what operations to do. We'll interpret the left side and then the right side and finally perform the operation and return the result. The three binary operations we use in this program are ==, + and -. We'll look these up in go/token docs to discover the associated constants.

func interpretBinaryExpr(ctx context, r *ret, bexpr *ast.BinaryExpr) {
  var xr, yr ret
  interpretExpr(ctx, &xr, bexpr.X)
  x := xr.values[0]
  interpretExpr(ctx, &yr, bexpr.Y)
  y := yr.values[0]

  switch bexpr.Op {
  case token.ADD:
    r.setValue(value{i64, x.value.(int64) + y.value.(int64)})
  case token.SUB:
    r.setValue(value{i64, x.value.(int64) - y.value.(int64)})
  case token.EQL:
    r.setValue(value{bl, x.value.(int64) == y.value.(int64)})
  default:
    log.Fatalf("Unknown binary expression type: %+v", bexpr)
  }
}

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.BinaryExpr:
    interpretBinaryExpr(ctx, r, e)

  ...

Let's try one more time!

$ go build goi.go
$ ./goi fib.go
2019/10/12 11:06:19 Unknown stmt type (*ast.ReturnStmt): &{Return:94 Results:[0xc000070540]}

Awesome, last step.

Interpreting ast.ReturnStmt

Based on the ast.ReturnStmt definition we'll have to interpret each expression and set all of them to the ret value.

func interpretReturnStmt(ctx context, r *ret, s *ast.ReturnStmt) {
  var values []value
  for _, expr := range s.Results {
    var r ret
    interpretExpr(ctx, &r, expr)
    values = append(values, r.values[0])
  }

  r.setValues(values)

  return
}

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  if stmt == nil {
    return
  }

  switch s := stmt.(type) {
  case *ast.ReturnStmt:
    interpretReturnStmt(ctx, r, s)

  ...

And let's try one last time:

$ go build goi.go
$ ./goi fib.go
377

Looking good. :) Let's try with another input:

$ cat fib.go
package main

func fib(a int) int {
  if a == 1 {
    return 0
  }

  if a == 2 {
    return 1
  }

  return fib(a-1) + fib(a-2)
}

func main() {
  println(fib(14))
}
$ ./goi fib.go
233

We've got the basics of an interpreter for Golang.

Here's a blog post on building a simple AST interpreter for Go to support running a recursive fibonacci implementation https://t.co/5Zz388d8ZN
— Phil Eaton (@phil_eaton) October 12, 2019

"No Code" Is A Lie

swyx's site RSS Feed — Fri, 11 Oct 2019 03:00:00 GMT

Originally published on the Webflow blog.

Guidance for conducting offsites

CRAIG KERSTIENS — Thu, 10 Oct 2019 23:55:56 GMT

Offsites an invaluable tool in getting a team aligned. I’ve been a part of organizations where offsites never happened, and then when they happened at a regular interval. Just because offsites happened it didn’t mean they had the same significant impact to alignment and ability to execute moving forward. What follows is a few key principles around conducting an impactful offsite.

Get out of the office

An offsite isn’t reserving a conference room for a full day and just sitting there meeting on special topics. A change of scenery is important, and nice scenery will impact your abiltiy to collaborate. One long time friend and colleague communicated to me about 10 years into his working career, that the thing he most values in a workplace is natural light. He remarked younger him wanted free food and drinks, now he finds him much happier and more productive with natural light above all other amenities. A similar environment for an offsite is very helpful.

The best option isn’t a hotel with a rented conference room, but rather an AirBnB with each person their own room, a large kitchen, and large living room. More on the why for each of those later.

From working with some teams that are often local and a portion of them remote, hosting the offsite within a 2 hr drive of your primary office can be good. You don’t waste the full day traveling, for remote team members they can also budget 1-2 days in the office to see other team members in addition to the time spent at the offsite.

Collaboration is key

Get rid of powerpoint presentations and someone presenting on a screen. That isn’t collaboration, that is a presentation. If you have a team of 8 and each person presents for an hour you haven’t collaborated for 8 hrs, you’ve been presented to for 7 and presented for 1. I personally don’t like team building exercises, but I’ll admit they can be useful here.

A presentation by the leadership at the offsite for a few minutes to frame things can be helpful. It’s key to not have a projector and folks staring at a screen. Instead having a few prepared notes and ability interactively discuss is great. Collaborating on a large notepad, or using sticky notes for ideas and brainstorming is great. One of my favorite things to do at an offsite is a gridding exercise.

2-3 days is ideal

You don’t need a full week offsite, but a 1 day offsite doesn’t really provide ideas to evolve and change over time. The ideal to me is 3 days where you can make some statements about ideas on day 1, and then drill deeper into those on day 2 or 3. Having the ability to do research on data and evaluate any plans you make is helpful. You also then can have good collaborative time, but also set aside time for contentious topics. If you cover a contentious topic and then jump right into collaboration without time to change your mindset you’re going to get a very different outcome. Breaking things up across days makes this way easier.

2-3 days also allows time for non work activities. Coming together as a team will let you work better in the future. Boardgames are a common favorite, video games are an occasional activity. Sometimes sitting by a fireplace and enjoying a nice glass of wine and just catching up.

Those moments can live on for a very long time. Many colleagues from years ago will recall how when playing “We didn’t play test this at all”, we brought back online an archived server intentionally to cause a page so an engineer would pick up his phone, only for one of us to play a card causing him to lose the game due to holding his phone. Those stories create a foundation and shared experience to build on in the future for how you work together

This takes me onto the next point…

Food

Too often offsites are a day of meetings in a conference room, followed by dinner out at a nice restaurant. The exact opposite works better. We’ve already talked about location. But food should be collaborative as well. Over 2-3 days you may do one dinner out, but even that isn’t the norm. Instead make a meal list ahead of time, and people spend time in the kitchen together. You’ll typically find you have 1-2 more cooks in the team than others, but then you have a lot of helpers.

It actually reminds me a bit of my wedding reception back home. I smoked about 80 lbs of meat for the reception. As we were taking the meat off, I walked back in to 4 generations of in-laws, the older ones teaching the younger ones how to pull pork.

Cooking together is another form of collaborating. You can easily learn about cultures and backgrounds. I didn’t realize melon with proscuitto was always supposed to be server with port if you’re in France, but now I can eat it no other way. Splitting up the duties so that 2-3 help cook a meal means you don’t spend all your time cooking, but you get people working together.

Don’t steal a weekend

I mentioned 2-3 days is ideal. Please don’t use one of those days as a weekend. As the leader or coordinator of such you may not have weekend obligations and happy to give up a day of it. But many on your team may have other commitements and such, an offsite is valuable, but asking people to give up their weekend to essentially do more work isn’t a great look.

Offsites for everyone

Offsites aren’t just for management. The folks in boots on the ground doing the work has as much need and value as the leadership team sitting there determining the next 5 year strategy. Allowing engineers that are building things, supporting customers, answering pages to brainstorm and collaborate on how to more effectively design the systems and move more quickly will pay huge dividends. You’ll find experiments, and then new tools and systems that allow you to ship things faster with higher quality. Collaborating with the product team is key as well, an engineering offsite in isolation leaves out the voice of the customer. A product offsite without engineering leaves you daydreaming of solutions without being grounded in the reality of what is possible.

How Elasticsearch Stores Documents

Ben E. C. Boyter — Thu, 10 Oct 2019 03:00:00 GMT

Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub

It is useful to understand is how elasticsearch stores documents. Documents that are indexed need to go into an index and have a type. Indexes can contain one or more types. This may sound limiting but you can search over all indexes or all types within an index or just one type within an index if you require.

A brief introduction to Rust

Omar Abid - Personal Blog — Thu, 10 Oct 2019 03:00:00 GMT

A brief introduction to Rust

API Design: Modifying Defaults

swyx's site RSS Feed — Thu, 10 Oct 2019 03:00:00 GMT

This is a quick note on a API Design. I hope to make this an ongoing series.

My list of useful command line tools

Ben E. C. Boyter — Wed, 09 Oct 2019 03:00:00 GMT

A post that is constantly a work in progress.

ripgrep - Similar to grep, but a drop in replacement for large file scanning without unicode penalities. It’s also similar to Ag and Ack and optimised for searching code, ignoring .git directories and the like. https://github.com/burntsushi/ripgrep
tmux - Terminal multiplexer. I wrote a 5 min guide to this. Use it to have resumable sessions after closing ssh sessions and to have split pane windows.
bat - Replacement for cat but provides syntax highlighting and line numbers. Can work as a drop in replacement for cat as well as it detects redirects and removes the fancy output. https://github.com/sharkdp/bat
fd - Better than find. Recursively looks for whatever it is you need using regular expressions and like ripgrep ignores .git directories and the like. https://github.com/sharkdp/fd
fzf - Fuzzy finder. Hard to explain why this is so useful. Suggest checking it out https://github.com/junegunn/fzf
gron - Makes JSON files grepable by flattening the structure. Very useful when trying to find something in JSON quickly. https://github.com/TomNomNom/gron
hexyl - Hex view of files with highlighting and the like. Find BOM in files quickly! https://github.com/sharkdp/hexyl
hyperfine - A command line benchmark tool. Give it two commands hyperfine 'scc' 'tokei' to determine which one is faster. https://github.com/sharkdp/hyperfine
z - A shell script that enables quick jumping to your most recently used directories. For example I work with scc a lot and its directory is /c/Users/bboyter/Documents/Go/src/github.com/boyter/scc which is a pain to remember. Having changed to it a few times I can now type z scc and it takes me right there. A massive time saver. https://github.com/rupa/z/
zoxide - Similar to z above, but I prefer it over z these days https://github.com/ajeetdsouza/zoxide
stu - A small TUI application explorer for AWS S3 or any other S3 compatible storage. Very useful when used with localstack stu --endpoint-url http://localhost:4566 https://github.com/lusingander/stu

Ejectable Defaults

swyx's site RSS Feed — Sat, 05 Oct 2019 03:00:00 GMT

Borrowing heavily from Gatsby Themes to improve the Sapper developer experience, and a discussion of how Ejectable Defaults fits with the Zero Config movement

Sept 2019 Recap

swyx's site RSS Feed — Tue, 01 Oct 2019 03:00:00 GMT

Hello mailing list friends!

Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100

Ben E. C. Boyter — Mon, 30 Sep 2019 03:00:00 GMT

The command line tool I created Sloc Cloc and Code (scc) (which is now modified and maintained by many other excellent people) counts lines of code, comments and makes a complexity estimate for files inside a directory. The latter is something you need a good sample size to make good use of. The way it works is that it counts branch statements in code. However what does that actually mean? For example “This file has a complexity of 10” is not very useful without some context. To solve this issue I thought I would try to run scc at all the source code I could get my hands on. This would also allow me to see if there are any edge cases I didn’t consider in the tool itself. A brute force Q/A trial by fire.

Administering Kubernetes is hard

Notes on software development — Mon, 30 Sep 2019 03:00:00 GMT

Kubernetes is easy to use after some exposure; it's pretty convenient too. But it is super hard to set up.

eksctl is a good tool for folks who don't want to spend hours/days/weeks debugging VPC configuration in 1000s of lines of CloudFormation. None of the other tools seem to be that much easier to use (kops, kubeadm, etc.).

But even with EKS and eksctl you are constrained to Amazon Linux worker nodes. AMIs are practically impossible to discover.

I haven't spent much time with GKE.

And while eksctl operates on the right level for developers needing to administrate small/medium-sized systems, it... doesn't exist outside EKS.

It is unfortunate the only major container orchestration system is this complex to administer. The user-facing APIs are pretty solid and guide toward sustainable system design. It is really hard to see the value for most companies with medium-sized deployments tasked with administration. Among serious proprietary alternatives, sure, there's ECS and Google App Engine. But there's little advantage in existing Kubernetes user knowledge. The OSS alternatives don't have the adoption to seem like a good investment.

OpenStack's magnum or OpenShift seem like possible high-level providers for a generic environment. But neither are particularly known for stability.

In all, the ecosystem has gotten friendlier. There will probably be a time in the future (3-5 years from now?) when Kubernetes is fairly easy to administer.

I'd love to hear your thoughts and experiences administering Kubernetes.

I hope I hate this code one day

Tom Forbes — Sat, 28 Sep 2019 18:25:53 GMT

I remember the first program I built fully on my own: a music downloader. There was a site that would basically index random music files found on Google and present you with a way to search them. I guess they would periodically search google for things like intitle:index.of mp3 (which you can still ...

Friday Links 2019-09-27

infrequent oscillations — Fri, 27 Sep 2019 22:52:33 GMT

In which we copy an idea and post some links.

Git-Centric Workflow

swyx's site RSS Feed — Thu, 26 Sep 2019 03:00:00 GMT

Published on the Netlify Blog

Why I Won't Choose To Be A Manager

swyx's site RSS Feed — Thu, 26 Sep 2019 03:00:00 GMT

I recently decided that I will not choose to be a manager. This is how I arrived at that conclusion.

The World's Greatest Netlify Demo 2019

swyx's site RSS Feed — Tue, 24 Sep 2019 03:00:00 GMT

Note: this was published as the Complete Intro to Netlify on the Netlify Blog and the FreeCodeCamp Channel

Adverse Status Updates and Trust

swyx's site RSS Feed — Mon, 23 Sep 2019 03:00:00 GMT

I noticed an asymmetry between adverse status updates and trust that I figured it would be worth a quick comment on.

Europython 2019 Talk - Rewriting the Django autoreloader

Tom Forbes — Sun, 22 Sep 2019 22:38:42 GMT

Over two years ago I set out to try and improve Django’s autoreloader implementation. My PR was merged and the feature released in Django 2.2, and I thought I should at least mention it on my blog. The role of an autoreloader is conceptually pretty simple: When a developer changes a file your softwa...

Waterfall

swyx's site RSS Feed — Sun, 22 Sep 2019 03:00:00 GMT

The worlds of software, business, and music use the word "Waterfall" incredibly differently and they are completely ignorant of each other. I figured I would make a quick note to…

The Checklist Manifesto – Atul Gawande

Dan Palmer — Sun, 22 Sep 2019 02:00:00 GMT

Not long after a recent one to one with my manager, discussing how we could improve our incident response process in engineering at Thread, I returned to my desk to find a copy of The Checklist Manifesto that he had kindly got for me.

This is less of a book review and more of some highlights that I wanted to pull out from the book. Going into it, I had already read about the effectiveness of checklists in preventing human error, particularly in commercial aviation and medicine, but the book still had some great points to make. I won’t mention on any of the evidence as the book goes into plenty of detail, I’m more interested in how checklists can be used and the effect they have had on various professions.

Redesigning my blog

Tom Forbes — Sat, 21 Sep 2019 23:28:03 GMT

I like Hugo a lot. It’s really fast and really simple. However it’s very much still evolving, and the theme I was using had not been updated in nearly a year. This meant that it broke with the new release of Hugo, and I was generally getting a bit sick of the design. Now I’m not a designer by any me...

Friday Links 20

infrequent oscillations — Fri, 20 Sep 2019 15:49:19 GMT

In which we copy an idea and post some links.

Your Site's Calling Card

swyx's site RSS Feed — Fri, 20 Sep 2019 03:00:00 GMT

Note: this tutorial is now out of date - I don't pre-generate og:image cards anymore as it added too much time to my builds

A Better Way to Get Data

swyx's site RSS Feed — Sun, 15 Sep 2019 03:00:00 GMT

Note: this was an unfinished draft, i published it anyway but it is incomplete

Why JavaScript Tooling Sucks

swyx's site RSS Feed — Sun, 15 Sep 2019 03:00:00 GMT

JavaScript Tooling is just too hard to use, and it's not your fault.

Bad Ways to Get Data

swyx's site RSS Feed — Sat, 14 Sep 2019 03:00:00 GMT

In working on my Sapper export library, I ran into a very peculiar problem: my pages were being generated a lot more slowly than expected:

Friday Links 2019-09-13

infrequent oscillations — Fri, 13 Sep 2019 17:07:35 GMT

In which we copy an idea and post some links.

Static Svelte: JavaScript Blogging with 93% less JavaScript

swyx's site RSS Feed — Fri, 13 Sep 2019 03:00:00 GMT

This blog now uses Svelte & Sapper as a static site generator, where it previously used React & Gatsby. This is achieved through Sapper's sapper export feature.

Using Next.js as a Static Site Generator for Netlify

swyx's site RSS Feed — Wed, 11 Sep 2019 03:00:00 GMT

How to deploy a Static Next.js Site to Netlify, and then upgrade it to use Dynamic Route Segments to become a full fledged Static Site Generator!

Building Authenticated Serverless Jamstack Apps with Gatsby and Netlify

swyx's site RSS Feed — Wed, 11 Sep 2019 03:00:00 GMT

Gatsby is more than a simple static site generator. It uses JavaScript to rehydrate Markup into a fully dynamic React app - which means you can use APIs to do all sorts of…

GraphQL Variables and Persisted Queries

swyx's site RSS Feed — Mon, 09 Sep 2019 03:00:00 GMT

My mistakes and insights on GraphQL Variables and Persisted Queries.

r2wars 2019

Linux Engineer's random thoughts — Sun, 08 Sep 2019 01:00:00 GMT

You might have noticed I was at r2con 2019. This is my writeup for the r2wars 2019 challenge.

r2wars primer: a Core Wars-like game

In radare2, there's an intermediate language and VM called ESIL. It is used to emulate code, and supports many architectures. r2wars is built on top of …

Friday Links

infrequent oscillations — Fri, 06 Sep 2019 21:36:01 GMT

In which we copy an idea and post some links.

r2con 2019

Linux Engineer's random thoughts — Fri, 06 Sep 2019 01:00:00 GMT

I'm back in Barcelona for this year's edition of r2con. You can read my r2con 2018 report.

radare2 is debugging and reverse engineering toolkit. It's mostly used from the command line or through a programming interface (r2pipe) that is identical to the command line one.

Cutter

Cutter is the official …

Keyboard First

swyx's site RSS Feed — Thu, 05 Sep 2019 03:00:00 GMT

I enjoyed listening to Des and Paul's discussion of keyboard-first apps on their podcast today. They discussed the rise of the cmd+K UI paradigm in apps like:

Compilers Copout

swyx's site RSS Feed — Wed, 04 Sep 2019 03:00:00 GMT

Today I dropped out of the Compilers course I accidentally registered myself for. I'm not proud of it but there's too much I want to do and I know I'm just going to do a shit job…

JAMstack for Indie Hackers

swyx's site RSS Feed — Mon, 02 Sep 2019 03:00:00 GMT

This article was published on Indie Hackers!

Book Poll 2019

swyx's site RSS Feed — Mon, 02 Sep 2019 03:00:00 GMT

I've been feeling like I've skewed too much on empty calories recently. So I ran this poll:

Unit testing C code with gtest

Notes on software development — Sat, 31 Aug 2019 03:00:00 GMT

This post covers building and testing a minimal, but still useful, C project. We'll use Google's gtest and CMake for testing C code. This will serve as a foundation for some upcoming posts/projects on programming Linux, userland networking and interpreters.

The first version of this post only included one module to test. The test/CMakeLists.txt would also only expose a single pass-fail status for all modules. The second version of this post extends the test/CMakeLists.txt to expose each test/*.cpp file as its own CMake test so that results are displayed by ctest per file. The second version also splits the original src/testy.c and include/testy/testy.h module into a widget and customer module to demonstrate the changes to the CMake configuration.

The "testy" sample project

In this project, we'll put source code in src/ and publicly exported symbols (functions, structs, etc.) in header files in include/testy/. There will be a main.c in the src/ directory. Tests are written in C++ (since gtest is a C++ testing framework) and are in the test/ directory.

Here's an overview of the source and test code.

This file has some library code that we should be able to test.

#include "testy/widget.h"

int private_ok_value = 2;

int widget_ok(int a, int b) {
  return a + b == private_ok_value;
}

This file handles exported symbols for widget code.

#ifndef _WIDGET_H_
#define _WIDGET_H_

int widget_ok(int, int);

#endif

src/customer.c

This file has some more library code that we should be able to test.

#include "testy/customer.h"

int customer_check(int a) {
  return a == 5;
}

include/testy/customer.h

This file handles exported symbols for customer code.

#ifndef _CUSTOMER_H_
#define _CUSTOMER_H_

int customer_check(int);

#endif

src/main.c

This is the entrypoint to a program built around libtesty.

#include "testy/customer.h"
#include "testy/widget.h"

int main() {
  if (widget_ok(1, 1)) {
    return customer_check(5);
  }

  return 0;
}

This is one of our test files. It registers test cases and uses gtest to make assertions. We need to wrap the testy/widget.h include in an extern "C" to stop C++ from name-mangling.

#include "gtest/gtest.h"

extern "C" {
#include "testy/widget.h"
}

TEST(widget, ok) {
  ASSERT_EQ(widget_ok(1, 1), 1);
}

TEST(testy, not_ok) {
  ASSERT_EQ(widget_ok(1, 2), 0);
}

You can see a good high-level overview of gtest testing utilities like ASSERT_EQ and TEST here.

test/customer.cpp

This is another one of our test files.

#include "gtest/gtest.h"

extern "C" {
#include "testy/customer.h"
}

TEST(customer, ok) {
  ASSERT_EQ(customer_check(5), 1);
}

TEST(testy, not_ok) {
  ASSERT_EQ(customer_check(0), 0);  
}

test/main.cpp

This is a standard entrypoint for the test runner.

#include "gtest/gtest.h"

int main(int argc, char **argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Building with CMake

CMake is a build tool that (among other things) produces a Makefile we can run to build our code. We will also use it for dependency management. But fundementally we use it because gtest requires it.

CMake options/rules are defined in a CMakeLists.txt file. We'll have one in the root directory, one in the test directory, and a template for one that will handle the gtest dependency.

A first draft of the top-level CMakeLists.txt might look like this:

cmake_minimum_required(VERSION 3.1)

project(testy)

##
### Source definitions ###
##

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources "${PROJECT_SOURCE_DIR}/src/*.c")

add_executable(testy ${sources})

Using include_directory will make sure we compile with the -I flag set up correctly for our include directory.

Using add_executable sets up the binary name to produce from the given sources. And we're using the file helper to get a glob match of C files rather than listing them all out verbatim in the add_executable call.

Building and running

CMake pollutes the current directory, and is fine running in a different directory, so we'll make a build/ directory so we don't pollute root. Then we'll build a Makefile with CMake, run Make, and run our program.

$ mkdir build
$ cd build
$ cmake ..
-- The C compiler identification is AppleClang 10.0.1.10010046
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build
$ make
[ 25%] Building C object CMakeFiles/testy.dir/src/customer.c.o
[ 50%] Building C object CMakeFiles/testy.dir/src/widget.c.o
[ 75%] Building C object CMakeFiles/testy.dir/src/main.c.o
[100%] Linking C executable testy
[100%] Built target testy
$ ./testy
$ echo $?
1

CMakeLists.txt.in

This template file handles downloading the gtest dependency from github.com pinned to a release. It will be copied into a subdirectory during the cmake .. step.

cmake_minimum_required(VERSION 3.1)

project(googletest-download NONE)

include(ExternalProject)
ExternalProject_Add(googletest
  GIT_REPOSITORY    https://github.com/google/googletest.git
  GIT_TAG           release-1.8.1
  SOURCE_DIR        "${CMAKE_BINARY_DIR}/googletest-src"
  BINARY_DIR        "${CMAKE_BINARY_DIR}/googletest-build"
  CONFIGURE_COMMAND ""
  BUILD_COMMAND     ""
  INSTALL_COMMAND   ""
  TEST_COMMAND      ""
)

Now we can tell CMake about it and how to build, within the top-level CMakeLists.txt file.

cmake_minimum_required(VERSION 3.1)

project(testy)

##
### Test definitions ###
##

configure_file(CMakeLists.txt.in
        googletest-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )
execute_process(COMMAND ${CMAKE_COMMAND} --build .
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )

add_subdirectory(${CMAKE_BINARY_DIR}/googletest-src
        ${CMAKE_BINARY_DIR}/googletest-build)

enable_testing()
add_subdirectory(test)

##
### Source definitions ###
##

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources
  "${PROJECT_SOURCE_DIR}/include/testy/*.h"
  "${PROJECT_SOURCE_DIR}/src/*.c")

add_executable(testy ${sources})

The add_subdirectory calls register a directory that contains its own CMakeLists.txt. It would fail now without a CMakeLists.txt file in the test/ directory.

test/CMakeLists.txt

This final file registers a unit_test executable compiling against the source and test code, and includes the project header files.

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources "${PROJECT_SOURCE_DIR}/src/*.c")
list(REMOVE_ITEM sources "${PROJECT_SOURCE_DIR}/src/main.c")

file(GLOB tests "${PROJECT_SOURCE_DIR}/test/*.cpp")
list(REMOVE_ITEM tests "${PROJECT_SOURCE_DIR}/test/main.cpp")

foreach(file ${tests})
  set(name)
  get_filename_component(name ${file} NAME_WE)
  add_executable("${name}_tests"
    ${sources}
    ${file}
    "${PROJECT_SOURCE_DIR}/test/main.cpp")
  target_link_libraries("${name}_tests" gtest_main)
  add_test(NAME ${name} COMMAND "${name}_tests")
endforeach()

We have to register a test for each file otherwise each file's tests won't show up by default (i.e. without a --verbose flag).

Building and running tests

Similar to building and running the source, we run CMake in a subdirectory but run make test or ctest after building all sources and tests with make.

$ cd build
$ cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build/googletest-download
Scanning dependencies of target googletest
[ 11%] Creating directories for 'googletest'
[ 22%] Performing download step (git clone) for 'googletest'
Cloning into 'googletest-src'...
Note: checking out 'release-1.8.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2fe3bd99 Merge pull request #1433 from dsacre/fix-clang-warnings
[ 33%] No patch step for 'googletest'
[ 44%] Performing update step for 'googletest'
[ 55%] No configure step for 'googletest'
[ 66%] No build step for 'googletest'
[ 77%] No install step for 'googletest'
[ 88%] No test step for 'googletest'
[100%] Completed 'googletest'
[100%] Built target googletest
-- Found PythonInterp: /usr/local/bin/python (found version "2.7.16")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build
$ make
[  4%] Building C object CMakeFiles/testy.dir/src/customer.c.o
[  9%] Building C object CMakeFiles/testy.dir/src/widget.c.o
[ 13%] Building C object CMakeFiles/testy.dir/src/main.c.o
[ 18%] Linking C executable testy
[ 18%] Built target testy
[ 22%] Building CXX object googletest-build/googlemock/gtest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 27%] Linking CXX static library libgtest.a
[ 27%] Built target gtest
[ 31%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 36%] Linking CXX static library libgmock.a
[ 36%] Built target gmock
[ 40%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 45%] Linking CXX static library libgmock_main.a
[ 45%] Built target gmock_main
[ 50%] Building CXX object googletest-build/googlemock/gtest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 54%] Linking CXX static library libgtest_main.a
[ 54%] Built target gtest_main
[ 59%] Building C object test/CMakeFiles/customer_tests.dir/__/src/customer.c.o
[ 63%] Building C object test/CMakeFiles/customer_tests.dir/__/src/widget.c.o
[ 68%] Building CXX object test/CMakeFiles/customer_tests.dir/customer.cpp.o
[ 72%] Building CXX object test/CMakeFiles/customer_tests.dir/main.cpp.o
[ 77%] Linking CXX executable customer_tests
[ 77%] Built target customer_tests
Scanning dependencies of target widget_tests
[ 81%] Building C object test/CMakeFiles/widget_tests.dir/__/src/customer.c.o
[ 86%] Building C object test/CMakeFiles/widget_tests.dir/__/src/widget.c.o
[ 90%] Building CXX object test/CMakeFiles/widget_tests.dir/widget.cpp.o
[ 95%] Building CXX object test/CMakeFiles/widget_tests.dir/main.cpp.o
[100%] Linking CXX executable widget_tests
[100%] Built target widget_tests

After running cmake and make, we're finally ready to run ctest.

$ ctest
Test project /Users/philipeaton/tmp/testy/build
    Start 1: customer
1/2 Test #1: customer ..........................   Passed    0.01 sec
    Start 2: widget
2/2 Test #2: widget ............................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec

Now we're in a good place with most of the challenges of unit testing C code (i.e. ignoring mocks) past us.

In preparation for a couple new articles on some C projects, here's a foundational post on building C code and writing/running unit tests with gtest and cmake https://t.co/aMVyr7LO73
— Phil Eaton (@phil_eaton) August 31, 2019

Netlify Year One - 360 Review

swyx's site RSS Feed — Sat, 31 Aug 2019 03:00:00 GMT

I thought I would write a quick coda to my Netlify Year One Recap. That post was entirely me looking back at myself. It's helpful to see what others said about me at my review.

(Incomplete) The introduction to TypeScript Generics you've been missing

swyx's site RSS Feed — Fri, 30 Aug 2019 03:00:00 GMT

Note: this is an unfinished draft

The Case for the React Native Web Singularity

swyx's site RSS Feed — Fri, 30 Aug 2019 03:00:00 GMT

There is a possible "React Native Web Singularity", when it starts being a better standalone choice for developing for the mobile web than react-dom. If this speculation comes…

You Already Use Types

swyx's site RSS Feed — Thu, 29 Aug 2019 03:00:00 GMT

Published on Freecodecamp

Metrics Atlas

Bytepawn - Marton Trencseni — Thu, 29 Aug 2019 01:00:00 GMT

The idea is simple: write a document which helps new and existing people—both managers and individual contributors—get an objective, metrics-based picture of the business. This is helpful when new people join, when people start working in new segments of the business, and to understand other parts of the company.

Notes on 'Metaphysics and JavaScript'

swyx's site RSS Feed — Wed, 28 Aug 2019 03:00:00 GMT

Slides here

How to run Python and Portray on Netlify

swyx's site RSS Feed — Tue, 27 Aug 2019 03:00:00 GMT

How to deploy Python projects to Netlify

Playing Go with supervised learning in Pytorch

Bytepawn - Marton Trencseni — Sun, 25 Aug 2019 01:00:00 GMT

Using historic gameplay between strong Go players as training data, a CNN model is built to predict good Go moves on a standard 19x19 Go board.

Brave New Blog

Tilting at Windmills — Wed, 21 Aug 2019 03:00:00 GMT

Whew. It's been a while since I blogged. Astute readers (if there are any readers) might notice that the site looks a little bit different…

JAMstack or 'Pilha de Atolamento'? i18n in Gatsby + Netlify

swyx's site RSS Feed — Wed, 21 Aug 2019 03:00:00 GMT

Originally published on Scotch.io

Slowmad Q&A

swyx's site RSS Feed — Mon, 19 Aug 2019 03:00:00 GMT

The other thing about Netlify Year One that people are generally interested in is my experience of remote work. As I put it recently:

10,000 Loose Fans

swyx's site RSS Feed — Sun, 18 Aug 2019 03:00:00 GMT

I broke past 10,000 Twitter followers on Saturday. Obviously this is a completely arbitrary milestone and pretty minor in the hierarchy of needs. I'm not celebrating.

Why Naked Promises are Not Safe For Work

swyx's site RSS Feed — Wed, 14 Aug 2019 03:00:00 GMT

Published on FreeCodeCamp's Developer News. A followup for this was published in Errors are not Exceptions

Downgrading my iPad 2 to iOS 8

Nemo's Home — Sun, 11 Aug 2019 03:00:00 GMT

Update: Apple is no longer signing iOS6 for the iPad 2, so this is no longer feasible.

I own a iPad 2 (GSM), which is rarely used these days because it is too slow with the latest iOS 9 upgrades. It is a 8 year old device, but I can’t just install Linux on it and make it usable, which is what I do with most other devices.

The next best thing was to downgrade the iOS version. The device is anyway un-supported at this point, so I might as well go there. Apple has restrictions on which iOS releases are installable at any point on any device, ~but thankfully they are still signing iOS6 for my device for some legal reasons~.

Steps:

Download the iOS firmware for your device from https://ipsw.me/#platform
Launch iTunes
Option+Click on the Restore button in iTunes
Select the file you just downloaded.
Disable iCloud on the device
Upgrade to iOS8

This is possible as long as Apple is signing the IPSW for your device. The case of iOS6 being signed seems to be true for:

iPhone 4S
iPad 2

Security Notes

Running an unsupported OS is not something I take lightly. Here’s a list of defensive measures I took to ensure that I’m not at risk while doing so:

Try to keep the device always in Airplane mode.
Keep sensitive data off the device. No photos/keychain sync for eg. Don’t enable Calendar/Contact/Media sync.
Enable Restrictions on the device:
- Restrict Safari to limited websites.
- Disable application installs.
- Disable iTunes store/iBooks store etc.
- Disable GPS/Bluetooth.
- Limit Background Refresh to very few trusted applications.
Limit number of applications (I only have Kybooks installed).
Disable Javascript on Safari.

If possible, I’d recommend using a separate Apple Account on the device.

Netlify Environment Variables

swyx's site RSS Feed — Fri, 09 Aug 2019 03:00:00 GMT

We usually use Environment Variables as (global) variables, but did you know they can literally configure your environments as well?

Netlify Year One

swyx's site RSS Feed — Fri, 09 Aug 2019 03:00:00 GMT

Update: I have since left Netlify. Alex has published his own writeup on The Mo.

Camels Dressed As Unicorns

smcleod.net — Thu, 08 Aug 2019 11:10:00 GMT

Stop trying to hire with titles like ‘DevOps Engineer’ or ‘Cloud Engineer’#

“DevOps is … not a job title”

“It’s more of a cultural practice, like innovation, and it makes just as little sense hiring “innovation engineers” and expecting your organisation to be innovative, without also creating the culture to foster innovation.” (Joel Shea when responding to this posted on LinkedIn)

In most cases it’s clear the organisation doesn’t truly know what they want or need and likely don’t understand the nuances of the aspects of engineering.

Looking For New Opportunities

smcleod.net — Tue, 06 Aug 2019 13:30:00 GMT

As stated in my announcement post - after 7~ years, I resigned from Infoxchange several months ago.

I planned on taking a holiday, relaxing, decompressing, dipping my toes in some off-the-shelf tech I haven’t spent a lot of time with and performing some personal growth through reading a number of books that have been sitting on my ’to read’ shelf.

This is exactly what I’ve done.

I’m now open to new opportunities#

I’d rather find a workplace that I can add great value too and be happy to work with than rush into a new role quickly, obviously if something great turns up right away - I’ll pursue it.

Using Novela by Narative (updated)

swyx's site RSS Feed — Sun, 04 Aug 2019 03:00:00 GMT

With the growing community interest in Gatsby, we hope to create more resources that make it easier for anyone to grasp the power of this incredible tool.

Arabic name classification with Scikit-Learn and Pytorch

Bytepawn - Marton Trencseni — Fri, 02 Aug 2019 01:00:00 GMT

While working on arabic-vs-rest classification, I was curious how good out-of-the-box models perform with publicly available data, and then compare that with what we can achieve with internal data / features derived from millions of deliveries. We train Scikit-learn and Pytorch models for this classification task and achieve 90% prediction accuracy on publicly available data and out-of-the-box models, while internally 99% is achievable.

Fight Link Rot with Server- and Client-side Redirects (Netlify and Gatsby)

swyx's site RSS Feed — Mon, 22 Jul 2019 18:55:15 GMT

Why you need redirects on clientside and serverside and how to set that up with Gatsby and Netlify

Hiatus

infrequent oscillations — Sat, 20 Jul 2019 23:36:01 GMT

In which there is a temporary hiatus.

Writing an x86 emulator from scratch in JavaScript: 2. system calls

Notes on software development — Sat, 20 Jul 2019 03:00:00 GMT

Previously in emulator basics: <! forgive me, for I have sinned >
1. a stack and register machine

In this post we'll extend x86e to support the exit and write Linux system calls, or syscalls. A syscall is a function handled by the kernel that allows the process to interact with data outside of its memory. The SYSCALL instruction takes arguments in the same order that the regular CALL instruction does. But SYSCALL additionally requires the RAX register to contain the integer number of the syscall.

Historically, there have been a number of different ways to make syscalls. All methods perform variations on a software interrupt. Before AMD64, on x86 processors, there was the SYSENTER instruction. And before that there was only INT 80h to trigger the interrupt with the syscall handler (since interrupts can be used for more than just syscalls). The various instructions around interrupts have been added for efficiency as the processors and use by operating systems evolved.

Since this is a general need and AMD64 processors are among the most common today, you'll see similar code in every modern operating system such as FreeBSD, OpenBSD, NetBSD, macOS, and Linux. (I have no background in Windows.) The calling convention may differ (e.g. which arguments are in which registers) and the syscall numbers differ. Even within Linux both the calling convention and the syscall numbers differ between x86 (32-bit) and AMD64/x86_64 (64-bit) versions.

See this StackOverflow post for some more detail.

Code for this post in full is available as a Gist.

Exit

The exit syscall is how a child process communicates with the process that spawned it (its parent) when the child is finished running. Exit takes one argument, called the exit code or status code. It is an arbitrary signed 8-bit integer. If the high bit is set (i.e. the number is negative), this is interpreted to mean the process exited abnormally such as due to a segfault. Shells additionally interpret any non-zero exit code as a "failure". Otherwise, and ignoring these two common conventions, it can be used to mean anything the programmer wants.

The wait syscall is how the parent process can block until exit is called by the child and receive its exit code.

On AMD64 Linux the syscall number is 60. For example:

  MOV RDI, 0
  MOV RAX, 60
  SYSCALL

This calls exit with a status code of 0.

Write

The write syscall is how a process can send data to file descriptors, which are integers representing some file-like object. By default, a Linux process is given access to three file descriptors with consistent integer values: stdin is 0, stdout is 1, and stderr is 2. Write takes three arguments: the file descriptor integer to write to, a starting address to memory that is interpreted as a byte array, and the number of bytes to write to the file descriptor beginning at the start address.

On AMD64 Linux the syscall number is 1. For example:

  MOV RDI, 1   ; stdout
  MOV RSI, R12 ; address of string
  MOV RDX, 8   ; 8 bytes to write
  MOV RAX, 1   ; write
  SYSCALL

This writes 8 bytes to stdout starting from the string whose address is in R12.

Implementing syscalls

Our emulator is simplistic and is currently only implementing process emulation, not full CPU emulation. So the syscalls themselves will be handled in JavaScript. First we'll write out stubs for the two syscalls we are adding. And we'll provide a map from syscall id to the syscall.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {},
  60: function sys_exit(process) {},
};

We need to add an instruction handler to our instruction switch. In doing so we must convert the value in RAX from a BigInt to a regular Number so we can look it up in the syscall map.

      case 'syscall': {
        const idNumber = Number(process.registers.RAX);
        SYSCALLS_BY_ID[idNumber](process);
        process.registers.RIP++;
        break;
      }

Exit

Exit is really simple. It will be implemented by calling Node's global.process.exit(). Again we'll need to convert the register's BigInt value to a Number.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {},
  60: function sys_exit(process) {
    global.process.exit(Number(process.registers.RDI));
  },
};

Write

Write will be implemented by iterating over the process memory as bytes and by calling write() on the relevant file descriptor. We'll store a map of these on the process object and supply stdout, stderr, and stdin proxies on startup.

function main(file) {
  ...

  const process = {
    registers,
    memory,
    instructions,
    labels,
    fd: {
      // stdout
      1: global.process.stdout,
    }
  };

  ...
}

The base address is stored in RSI, the number of bytes to write are stored in RDX. And the file descriptor to write to is stored in RDI.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {
    const msg = BigInt(process.registers.RSI);
    const bytes = Number(process.registers.RDX);
    for (let i = 0; i < bytes; i++) {
      const byte = readMemoryBytes(process, msg + BigInt(i), 1);
      const char = String.fromCharCode(Number(byte));
      process.fd[Number(process.registers.RDI)].write(char);
    }
  },
  ...

All together

$ cat exit3.asm
main:
  MOV RDI, 1
  MOV RSI, 2
  ADD RDI, RSI

  MOV RAX, 60 ; exit
  SYSCALL
$ node emulator.js exit3.asm
$ echo $?
3

$ cat hello.asm
main:
  PUSH 10  ; \n
  PUSH 33  ; !
  PUSH 111 ; o
  PUSH 108 ; l
  PUSH 108 ; l
  PUSH 101 ; e
  PUSH 72  ; H

  MOV RDI, 1   ; stdout
  MOV RSI, RSP ; address of string
  MOV RDX, 56  ; 7 8-bit characters in the string but PUSH acts on 64-bit integers
  MOV RAX, 1   ; write
  SYSCALL

  MOV RDI, 0
  MOV RAX, 60
  SYSCALL
$ node emulator.js hello.asm
Hello!
$

Next steps

We still aren't setting flags appropriately to support conditionals, so that's low-hanging fruit. There are some other fun syscalls to implement that would also give us access to an emulated VGA card so we could render graphics. Syntactic support for string constants would also be convenient and more efficient.

Latest post in the emulator basics series up: implementing some syscalls starting with sys_exit and sys_write so we can print a nice hello message. https://t.co/NEfId0lnJx #javascript #x86
— Phil Eaton (@phil_eaton) July 20, 2019

Go Binary Sizes Have Been Pretty Stable

Donat Studios — Thu, 18 Jul 2019 03:55:21 GMT

Update October 2nd 2024: This was updated to include Go versions all the way up to 1.23

In 2013 I wrote an unintentionally inflammatory post: Go Binary Sizes Are Growing out of Control. I had imagined no one would read it, but it took off. I wrote it because I noticed my binaries getting larger as I upgraded versions of Go and found it curious.

Much has changed since then; biggest of all there have a handful of major releases of Go in that time.

I wanted to find out how much this had changed since my post went up. I used Docker to automate the process of compiling against major releases of Go. The exact code I used is on GitHub.

I hit a couple of hurdles. The oldest official version of Go on Docker Hub is 1.2 - which happens to be the version my previous post left off. Also, many of the programs from the previous comparison no longer compile in such old versions of Go.

Thus, consider this a standalone comparison rather than a direct continuation of the previous post.

I ended up going through most of my Go projects to find what would actually compile in ancient versions of Go.

The tools I compiled for the comparison are in order

hello world using fmt
hello world using println
Hookah - a tool for GitHub automation
imgavg - a tool for averaging a collection of images
sqlread - a tool for querying MySQL dumps without loading them into a database.

The Results

Without further ado, the raw data in megabytes.

binary	1.2	1.3	1.4	1.5	1.6	1.7	1.8	1.9	1.10	1.11	1.12	1.13	1.14	1.15	1.16	1.17	1.18	1.19	1.20	1.21	1.22	1.23
hello-nofmt	0.577	0.475	0.635	1.076	1.101	1.003	0.960	1.035	1.086	1.068	1.127	1.149	1.234	1.224	1.216	1.159	1.155	1.203	1.236	1.285	1.377	1.519
hello	2.240	1.824	1.941	2.367	2.288	1.634	1.552	1.860	2.012	1.907	1.997	2.009	2.068	2.035	1.937	1.772	1.764	1.814	1.852	1.806	1.894	2.129
hookah	8.352	6.737	6.868	7.684	8.739	6.391	6.553	6.805	7.311	7.282	8.234	8.289	8.315	7.191	6.907	6.848	7.014	7.260	7.477	7.620	7.892	8.464
imgavg	3.793	3.040	3.089	3.607	3.560	2.736	2.529	2.656	2.854	2.766	2.891	2.920	2.978	2.887	2.802	2.569	2.558	2.629	2.676	2.619	2.748	2.970
sqlread	4.730	3.648	3.739	4.242	4.192	3.170	3.043	3.091	3.284	3.144	3.302	3.316	3.373	3.265	3.149	2.989	2.968	3.057	3.110	3.171	3.291	3.538

Summary

The trendlines show the majority of the projects have a small downtick in size. The Go 1.5 and Go 1.6 releases caused a pretty universal uptick which 1.7 seems to have largely mitigated.

Hookah shows by far the most fluctuation in size by a pretty large margin. This is interesting because comparing dependencies the only builtin packages Hookah uses that sqlread does not are regexp and regexp/syntax. sqlread itself actually utilizes a pretty large number more. This could indicates the variation is deeper than the changes to the built in libraries, and potentially more to do with the compilation output and optimization.

Everything except for the simplest "Hello World" hello-nofmt has shown an overall decrease. hello-nofmt boils down to println("Hello, 世界") has shown an overall 658kb increase. This is likely the purest illustratration the growth in size of the runtime.

Conclusion

Go binaries are not "growing out of control". They're pretty stable and for the most part have shown a decrease.

I'm still hopeful future optimizations will help improve sizes. Given WASM becoming a major compilation target, size matters more than ever as it directly affect load times.

Tests or Types: Why Not Both?

swyx's site RSS Feed — Wed, 10 Jul 2019 03:00:00 GMT

Published on CSS Tricks as Types or Tests: Why Not Both?

Exploring prior beliefs with MCMC

Bytepawn - Marton Trencseni — Sat, 06 Jul 2019 01:00:00 GMT

I use PyMC3 to solve the food delivery toy problem and explore some alternative priors.

Design Issues of Sign in with Apple

Dan Palmer — Fri, 05 Jul 2019 02:00:00 GMT

Last month at their annual Worldwide Developers Conference (WWDC), most interesting announcements was Sign in with Apple. Built to compete with Facebook and Google’s single-sign-on (or social sign-on, SSO) offerings, Apple’s SSO will eschew control over the data and analytics that its competitors seek in favour of a privacy preserving design intended to advance Apple’s pro-privacy stance and ultimately to sell more devices by bringing more value to the Apple ecosystem.

Deploying a Create-React-App Website to Netlify with HTTPS

swyx's site RSS Feed — Wed, 03 Jul 2019 17:51:34 GMT

Tutorial on Deploying a Create-React-App Website to Netlify with HTTPS

How AWS and Go Transformed a Public Broadcaster

Ben E. C. Boyter — Wed, 03 Jul 2019 03:00:00 GMT

Today I presented at the Sydney Dev Warriors meet-up about a project I was working on from the start to end of 2018. In a nutshell it was a self serve digital archive for video, audio, images and metadata.

Since I am in the habit of collecting everything I do into one place I chose to do the presentation using reveal.js which means I can also include it here. How AWS and Go Transformed a Public Broadcasters Digital Archive.

Pass the SALT 2019 live report (part 3/3)

Linux Engineer's random thoughts — Wed, 03 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my part 1 and part 2 of my notes here.

Configurations, Do you prove yours ?

by Alexandre Brianceau (slides)

Alexandre went a bit more into details on the core concepts behind DevSecOps, and why continous configuration and observability …

Pass the SALT 2019 live report (part 2/3)

Linux Engineer's random thoughts — Tue, 02 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my part 1 of my notes and part 2 here.

Time-efficient assessment of open-source projects for Red Teamers

by Thomas Chauchefoin and Julien Szlamowicz (slides)

In their pentest team, Julien often does red team assessments with big scopes …

Pass the SALT 2019 live report (part 1/3)

Linux Engineer's random thoughts — Mon, 01 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my notes updated in real time here.

Kill MD5

by Ange Albertini (slides)

Ange starts by saying that he does not know much about crypto, and he had help from Marc Stevens, a hash-collision cryptographer. This talk should …

A/B tests: Moving Fast vs Being Sure

Bytepawn - Marton Trencseni — Mon, 01 Jul 2019 01:00:00 GMT

Most A/B testing tools default to α=0.05, meaning the expected false positive rate is 5%. In this post I explore the trade-offs between moving fast, ie. using higher α, versus being sure, ie. using lower α.

The Engineering Manager/Product Manager Marriage

CRAIG KERSTIENS — Sun, 30 Jun 2019 23:55:56 GMT

I’ve worked as a PM at a number of size companies for a few years now. At a startup and then as a part of a larger company once startups were acquired. I’ve been the first PM for a team as well as first for a company. I’ve written at times about product management, and today I’d like to drill into one aspect that doesn’t seem to get talked about enough and that is the pairing of product manager and engineering manager.

Mom vs. Dad

As parents my partner and I have learned very quickly that we need to have a consistent voice and unified view of things. I care that our son watches less power rangers otherwise he’s going to use his megazord powers on our TV and we’ll be watching a lot more of nothing. My partner cares that when we’re visiting family in the south they drink enough water so they don’t get dehydrated.

Meanwhile my kids are experts at leveraging us to get what they want. My daughter came to me last night asking if she could play on her iPad some. Not knowing if she had or if she’d already given an answer my safest question was have you asked your mom? In the absense of knowing my default isn’t a yes or no, it’s a “let me learn more.” and then potentially discuss it.

EM vs. PM

As a PM I want us to build a rich and powerful product, but there is a strong balance to doing too little vs. too much. It isn’t always a question of doing more, we need to make sure the product is well built. In order to do that we need to say no at times. Saying no, as well as yes, needs to come from a unified front, both engineering and product. If one side is agreeing without being aligned with the other half you’re going to wind up with a confused and frustrated team.

There are a number of ways engineering managers and product managers can stay aligned. The first starts with being explicit with each other, so if you’re struggling with one side communicating things the other don’t agree with… sit down and have a conversation about it. From an ongoing perspective you can get to a better aligned position to not have to discuss every question before you give an answer. My partner knows to make sure we don’t watch more than 2 episodes of power rangers and when we’re too amped up on it we shorten that.

Are you having regularly 1:1s with your counterpart?
Do you review emails out to the team ahead of time to capture feedback?
Do you have conversations where you’re not on the same page in an offline fashion as opposed to in team meetings?
Are you treating your EM/PM just like your partner at home? (I guess that does assume you treat your partner well, but that’s a whole other blog post)

Leaving Infoxchange

smcleod.net — Sat, 29 Jun 2019 12:00:00 GMT

After 7~ years, I have resigned from Infoxchange.

In 2012 I was hired to work with Infoxchange to “shake things up” (in the Systems Operations team) and stabilise the hosting environment.

As Team Lead of Systems Operations I was tasked with melding the team and then for us to work to provide a stable, robust, modern and scalable platform for Infoxchange developed application hosting and product delivery and in a short time, with a small budget - we did just that.

Betwixt Reason and Result

swyx's site RSS Feed — Sat, 29 Jun 2019 03:00:00 GMT

I'm writing to you from a cheap hostel in downtown Toronto (Hostelling International, highly recommend for solo travel, I have stayed in SF, LA, NY, PHL, TO, NZ and more I…

Solve CORS once and for all with Netlify Dev

swyx's site RSS Feed — Fri, 28 Jun 2019 03:00:00 GMT

Published on Alligator.io

Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab

Ben E. C. Boyter — Thu, 27 Jun 2019 03:00:00 GMT

This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

A very brief update in the world of scc. I noticed that tokei.rs which used to offer badges seemed to be having issues. Since I had wanted an excuse to hook up a AWS Lambda behind an ALB I thought I would add scc as a ‘service’ with badges.

How I traded my first software project 15 years ago

Linux Engineer's random thoughts — Wed, 26 Jun 2019 01:00:00 GMT

2004

Gmail recently celebrated its 15 years anniversary, which led me down the path of how I got my first gmail invite. Back then, the private Gmail beta was all the rage. In the first few months, gmail invites were scarce. I got one by doing the most popular thing …

Writing a lisp compiler from scratch in JavaScript: 6. LLVM system calls

Notes on software development — Sat, 22 Jun 2019 03:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
Next in compiler basics:
5. an x86 upgrade

In this post we'll extend the ulisp compiler's LLVM backend to support printing integers to stdout.

Exit code limitations

Until now we've validated program state by setting the exit code to the result of the program computation. But the exit code is an eight bit integer. What if we want to validate a computation that produces a result larger than 255?

To do this we need a way to print integers. This is challenging because printing normally deals with byte arrays. libc's printf, for example, takes a byte array as its first argument.

The shortest path forward is to add support for system calls so we can print one character at a time. Here's a version of a print form that hacks around not having arrays to send each integer in a number to stdout.

(def print-char (c)
     ; First argument is stdout
     ; Second argument is a pointer to a char array (of length one)
     ; Third argument is the length of the char array
     (syscall/sys_write 1 &c 1))

(def print (n)
     (if (> n 9)
         (print (/ n 10)))

     ; 48 is the ASCII code for '0'
     (print-char (+ 48 (% n 10))))

In order to support this we need to add the syscall/sys_write, >, %, and / builtin forms. We'll also need to add support for taking the address of a variable.

All code is available on Github as is the particular commit related to this post.

References

The sys_write syscall requires us to pass the memory address of the byte array to write. We don't support arrays, but we can treat an individual variable as an array of length one by passing the variable's address.

If we were compiling to C we could just pass the address of a local variable. But LLVM doesn't allow us to take the address of variables directly. We need to push the variable onto the LLVM stack to get an address.

Under the hood LLVM will likely optimize this into a local variable reference instead of first pushing to the stack.

Since LLVM IR is typed, the value representing the address of a local variable will be a pointer type. We'll need to refer to all uses of this value as a pointer. So we'll need to modify ulisp to track local types rather than hard-coding i64 everywhere.

Scope

To begin we'll modify the Scope class to track types. We only need to do this on registration. Afterward, we'll have to find all uses of local variables to make sure they use the local's value and type fields appropriately.

class Scope {

  ...

  register(local) {
    let copy = local.replace('-', '_');
    let n = 1;
    while (this.locals[copy]) {
      copy = local + n++;
    }

    this.locals[local] = {
      value: copy,
      type: 'i64',
    };
    return this.locals[local];
  }

  ...

}

We won't go through every use of a Scope variable in this post, but you can find it in the related commit to ulisp.

Reference

The long-term approach for handling a reference syntactically is probably to rewrite &x to (& x) in the parser. The lazy approach we'll take for now is to handle a reference as a special kind of identifier in compileExpression.

We'll use the LLVM alloca instruction to create space on the stack. This will return a pointer and will turn the destination variable into a pointer type. Then we'll use store to set the value at the address to the current value of the variable being referenced.

class Compiler {

  ...

  compileExpression(exp, destination, context) {

    ...

    // Is a reference, push onto the stack and return its address
    if (exp.startsWith('&')) {
      const symbol = exp.substring(1);
      const tmp = context.scope.symbol();
      this.compileExpression(symbol, tmp, context);
      this.emit(1, `%${destination.value} = alloca ${tmp.type}, align 4`);
      destination.type = tmp.type + '*';
      this.emit(1, `store ${tmp.type} %${tmp.value}, ${destination.type} %${destination.value}, align 4`);
      return;
    }

    ...

  }

  ...

}

And now we're set to take the address of any code.

System calls

LLVM IR provides no high-level means for making system calls. The only way is to use inline assembly. This syntax is based on GCC inline assembly and is confusing, with few explained examples, and unhelpful error messages.

Thankfully the assembly code needed for a syscall is only one line, one word: the syscall assembly instruction. We use inline assembly variable-to-register mapping functionality to line up all the parameters for the syscall. Here is an example:

%result = call i64 asm sideeffect "syscall", "=r,{rax},{rdi},{rsi},{rdx}" (i64 %raxArg, i64 %rdiArg, i64 %rsiArg, i64 %rdxArg)

This says to execute the inline assembly string, "syscall". The sideeffect flag means that this assembly should always be run even if the result isn't used. =r means the inline assembly returns a value, and the rest of the string is the list of registers that arguments should be mapped to. Finally we call the function with all the LLVM variables we want to be mapped.

Eventually we should also use the inline assembly syntax to list registers that are modified so that LLVM can know to save them before and after.

Code

We'll add a mapping for syscall/sys_write and a helper function for generating syscall code using the example above as a template. We'll suport 64-bit Darwin and Linux kernels.

const SYSCALL_TABLE = {
  darwin: {
    sys_write: 0x2000004,
    sys_exit: 0x2000001,
  },
  linux: {
    sys_write: 1,
    sys_exit: 60,
  },
}[process.platform];

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
      '%': this.compileOp('urem'),
      '<': this.compileOp('icmp slt'),
      '=': this.compileOp('icmp eq'),
      'syscall/sys_write': this.compileSyscall(SYSCALL_TABLE.sys_write),
    };
  }

  ...

  compileSyscall(id) {
    return (args, destination, context) => {
      const argTmps = args.map((arg) => {
          const tmp = context.scope.symbol();
          this.compileExpression(arg, tmp, context);
          return tmp.type + ' %' + tmp.value;
      }).join(', ');
      const regs = ['rdi', 'rsi', 'rdx', 'r10', 'r8', 'r9'];
      const params = args.map((arg, i) => `{${regs[i]}}`).join(',');
      const idTmp = context.scope.symbol().value;
      this.emit(1, `%${idTmp} = add i64 ${id}, 0`)
      this.emit(1, `%${destination.value} = call ${destination.type} asm sideeffect "syscall", "=r,{rax},${params},~{dirflag},~{fpsr},~{flags}" (i64 %${idTmp}, ${argTmps})`);
    }
  }
}

`>`, `/`

Finally, we have a few new operations to add support for. But they'll be pretty simple using the compileOp helper function.

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
      '/': this.compileOp('udiv'),
      '%': this.compileOp('urem'),
      '<': this.compileOp('icmp slt'),
      '>': this.compileOp('icmp sgt'),
      '=': this.compileOp('icmp eq'),
      'syscall/sys_write': this.compileSyscall(SYSCALL_TABLE.sys_write),
    };
  }

  ...

}

print

We're ready to give our print function a shot.

$ cat test.lisp
(def print-char (c)
     ; First argument is stdout
     ; Second argument is a pointer to a char array (of length one)
     ; Third argument is the length of the char array
     (syscall/sys_write 1 &c 1))

(def print (n)
     (if (> n 9)
         (print (/ n 10)))

     ; 48 is the ASCII code for '0'
     (print-char (+ 48 (% n 10))))

(def main ()
     (print 1234)
     0)
$ node ulisp.js test.lisp
$ ./build/a.out
1234

Looks good! In the next post we'll talk about tail call elimination.

It's been a slow month for the blog. But new post on compiler basics is up! Printing integers to stdout and making syscalls in LLVM (all without arrays). This was a pre-req for playing with tail-call elimination (post coming soon) https://t.co/fDtblUZRI8
— Phil Eaton (@phil_eaton) June 23, 2019

Food deliveries, Bayes and Computational Statistics

Bytepawn - Marton Trencseni — Sat, 22 Jun 2019 01:00:00 GMT

I was grabbing a burger at Shake Shack, Mall of the Emirates in Dubai, when I noticed this notebook on the counter. The staff is using it to track food deliveries and each service (Carriage, Talabat, UberEats, Deliveroo) has its own column with the order numbers. Let's assume this is the only page for the day, and ask ourselves: given this data, what is the probability that UberEats is the most popular food delivery service?.

Add Netlify Identity Authentication to any React App in 5 minutes with React Context, Hooks and Suspense

swyx's site RSS Feed — Fri, 21 Jun 2019 14:33:26 GMT

Adding authentication is a pain point for many React beginners. We’ve made it ridiculously easy to add Netlify Identity onto any React app, including create-react-app, Gatsby,…

A really cheap chatbot in Python

Ben E. C. Boyter — Wed, 19 Jun 2019 03:00:00 GMT

Let me start by stating that I think all chat-bots are rubbish and that a better solution is a decent search implementation. Type in some terms and get some useful results. Much better than typing to a machine.

A while ago I was working at a client site where there was an internal discussion group about building a chat-bot. Many meetings had passed, discussing using neural networks, building flows, should we use IBM Watson. Lots of talk, but not much delivery.

How to customise Jetty embedded in Spark Java framework

Subclassed — Tue, 18 Jun 2019 14:43:52 GMT

EDIT: I uploaded a full working example, ready to be customized, on github. If you just want a working solution, go there. Otherwise, keep reading for the explanation.

When it comes to Java microframeworks for API development, I've been using Spark for some time in a couple of different projects. For most casual needs, it works out of the box; however, there are circumstances where the "easy" options for configuring the embedded Jetty instance, are simply not enough. In that case, you have to take control and basically replace the instance with one that you completely control.

Due to how Jetty works, there are multiple moving parts: the server takes care of things like thread pools, the socket is responsible for SSL and other low-level protocol stuff, and the handler is where you do high-level middleware (changing headers etc). This means that you might need several factories to take care of all these. The good news is that Spark is granular enough that you can (more or less) limit yourself to what you need. The main pattern is: there will be a factory for each element, with a create() method returning an interface; you implement the interface and swap out the default factory with your own. However, these factories at the moment (Spark 2.8) are somewhat nested, so depending on where you need to work, you might have to replace a bunch of classes before you hit the point you're interested in.

The main entry point is EmbeddedServers.add(identifier, serverFactory). This is what you'll call from the main() method you use for all the post() and get() configuration directives. It is important that EmbeddedServers should appear right at the top, before any other configuration directive, otherwise Spark will use its default factory. It should look more or less like this:

EmbeddedServers.add(
     EmbeddedServers.Identifiers.JETTY, 
     new MyWonderfulServerFactory());

For the identifier, we re-use the default one because otherwise Spark will take over again. Nothing to do there.

The server factory is where real work begins. Your class (or lambda) needs to implement the spark.embeddedserver.EmbeddedServerFactory interface, which has one method:

public EmbeddedServer create(
            Routes routeMatcher, 
            StaticFilesConfiguration staticFilesConfiguration, 
            ExceptionMapper exceptionMapper, 
            boolean hasMultipleHandler)

As a starter, you can copy the content of spark.embeddedserver.jetty.EmbeddedJettyFactory as-is. You don't need constructors (unless you want them). Strictly speaking you don't need withThreadPool() and withHttpOnly() methods either, but I suggest you keep them anyway; just change the return type to match MyWonderfulServerFactory.

Create() does three things:

Initializing the route matcher, which you probably don't want to touch;
Initializing the Jetty handler for that matcher, which you may want to configure for things like header manipulation and other middleware;
Creating the embedded server, which is likely what you are after.

I will assume we want to tweak n.3. The main place where to pay attention is this line at the end of create():

return (new EmbeddedJettyServer(this.serverFactory, handler)
        ).withThreadPool(this.threadPool);

This will return the default configuration, and we don't want that. So we change it to something like:

return (new MyWonderfulEmbeddedServer(handler)
        ).withThreadPool(this.threadPool);

Now we need a MyWonderfulEmbeddedServer class, which should implement spark.embeddedserver.EmbeddedServer. Again, as a starting point, you can copy spark.embeddedserver.jetty.EmbeddedJettyServer, and change the return types to match. I also suggest you get rid of the factory parameter in constructor and the related field, which adds a bit of unnecessary complexity. That factory is actually used only in one place:

if (this.threadPool == null) {
    this.server = this.serverFactory.create(
       maxThreads, minThreads, threadIdleTimeoutMillis);
} else {
    this.server = this.serverFactory.create(this.threadPool);
}

Which you can replace with the following (straight from spark.embeddedserver.jetty.JettyServer):

if (this.threadPool == null) {
    if (maxThreads > 0) {
       int min = minThreads > 0 ? minThreads : 8;
       int idleTimeout = threadIdleTimeoutMillis > 0 ? threadIdleTimeoutMillis : '\uea60';
       server = new Server(new QueuedThreadPool(maxThreads, min, idleTimeout));
    } else {
       server = new Server();
    }
} else {
    this.server = threadPool != null ? new Server(threadPool) : new Server();
}

It's a bunch of stuff related to the amount of threads and timeouts. If that's what you were trying to configure, btw, this is where you can do it. To be honest, I believe Spark developers intended that the "proper" way, to do that particular customisation, would be to keep the factory parameter as it is, implement your own alternative to the badly-named JettyServer class -- it should be something like JettyThreadConfigFactory, really -- which implements the similarly badly-named JettyServerFactory, and pass it to the constructor in MyWonderfulServerFactory.create().

In my case, though, I was after an SSL customisation, and for that I had to swap out yet another factory, spark.embeddedserver.jetty.SocketConnectorFactory, mentioned in the second half of this block:

ServerConnector connector;
if (sslStores == null) {
    connector = SocketConnectorFactory.createSocketConnector(
         this.server, host, port);
} else {
    connector = SocketConnectorFactory.createSecureSocketConnector(
         this.server, host, port, sslStores);
}

In this case there is no interface, you can just extend the existing class with what you need. I wanted to enforce a particular set of SSL ciphers and protocols, so I overrode createSecureSocketConnector() adding the following bits:

sslContextFactory.setExcludeProtocols("SSLv3", "SSLv2", "TLSv1.2");
// first we clear existing exclusions
sslContextFactory.setExcludeCipherSuites(new String[]{});
// then we re-add what we need
sslContextFactory.setIncludeCipherSuites(bigArrayOfCipherNames);

And that's it. Now you know how to instantiate your own Jetty instance for Spark. It's a bit convoluted, and hopefully a "spark 3" will give us a better architecture to work with, one day. In the meantime, this is how you can do it.

How to dump macOS security-rights database

Subclassed — Tue, 18 Jun 2019 14:43:38 GMT

macOS handles some security items in a custom database, which may or may not be SQLite. The official way to interact with such database, outside of Objective-C, is the /usr/bin/security utility, with the parameter authorizationdb and specifying the required operation on the class of rights, e.g. security authorizationdb read system.login.console

Unfortunately, there is no way (that I could find) to simply list all classes. I was trying to uninstall something that might have had references in that db, but I wasn't sure about the class it might be registered under, so I wanted to dump them all.

Luckily I found what looks looks like a comprehensive list of rights. After a quick scraping job with Python, I had a list that I could use like this:

cat osxrights.txt | xargs -I % sh -c 'sudo security authorizationdb read %' | grep -B 10 myAnnoyingItemToRemove

... and the bundle wasn't anywhere, so I could just chuck it.

(Note: had the item been found, I would have had to dump the whole security class to a .plist file with security authorizationdb read the.class > my.plist , edited the file to remove it, then write it back to the db with security authorizationdb write the.class < my.plist )

How good does the Monaro look

Ben E. C. Boyter — Tue, 11 Jun 2019 03:00:00 GMT

About 12 years ago my old man was working as one of those hire a husband/jim’s maintenance deals. This was in the lower Blue Mountains. During my university break I didn’t have much on and would often come with him as something to do. We were attempting to repair a deck of some flea hole house near Glenbrook. The place would have been improved only by a match and petrol. The owner was around and mentioned that the place was to be rented out and real estate had just sent some people on the way to inspect.

A non-gamers attempt at Sea of Thieves

Ben E. C. Boyter — Mon, 10 Jun 2019 03:00:00 GMT

A long weekend just recently passed in NSW Australia. With a larger than normal amount of time on my hands I decided to try out the game Sea of Thieves. Ignoring the fact that the NBN rollout here has ruined my ADSL2+ which has gone from a 18-19 Mbps connection to 3 Mbps and that it took 24 hours to download, what follows is probably one of the more memorable gaming experiences I have ever had.

Creating an index of Apple Watch/MacOS/iOS features per-country

Tom Forbes — Tue, 04 Jun 2019 23:52:41 GMT

tl;dr: check out https://applewatchfeatures.com/ , https://iosfeatures.com/ and https://macosfeatures.com/ I recently brought an Apple Watch. It’s a pretty fantastic product! One thing that quite annoyed me while evaluating if I should buy it is that features can vary quite drastically per country, ...

About

infrequent oscillations — Mon, 03 Jun 2019 19:00:00 GMT

Social networks Fediverse: @zimpenfish@social.rjp.is Fediverse: @zimpenfish@octodon.social Goodreads Twitter Spotify last.fm Jabber Also reachable via Telegram, Signal, WhatsApp, LINE, WeChat, Viber, Threema, and probably some others I’ve forgotten. Creative outlets Instagram Soundcloud YouTube Flickr (largely just an Instagram backup now) Tumblr Sometimes my music escapes into the wild and appears on real services. Spotify Deezer Tidal Buy my stuff Bandcamp Spoonflower Etsy Redbubble Society6 Displate Coding shenanigans Bitbucket Github Exercise shenanigans Strava Garmin Connect

The Collatz conjecture

Bytepawn - Marton Trencseni — Sun, 02 Jun 2019 01:00:00 GMT

The Collatz conjecture is a conjecture in mathematics that concerns a sequence defined as follows: start with any positive integer n. Then each term is obtained from the previous term as follows: if the previous term is even, the next term is one half the previous term. If the previous term is odd, the next term is 3 times the previous term plus 1. The conjecture is that no matter what value of n, the sequence will always reach 1.

Cleaning up Google Purchases

Nemo's Home — Sat, 01 Jun 2019 03:00:00 GMT

The Google Purchase History feature has been doing rounds in the news recently. In case you missed it, go to https://myaccount.google.com/purchases right now and make sure you are logged in with your personal gmail account to see what all Google thinks you’ve bought.

For me it lists purchases going as far back as 2013, which include:

All of my Amazon Purchases (including Kindle and Audible)
Flipkart/Sneapdeal purchases
Gifts I’ve bought for others on various platforms
All my iCloud purchases
Purchases on Steam
BigBasket purchases
Google Playstore purchases as well, of course
And much, much more.

For each of the purchases, it remembers the price, the taxes, as well as the delivery address used.

While this isn’t shocking in the least, I was surprised, because as a Infosec professional, I’ve disabled all of google’s invasive tracking features:

All my Activity Controls are paused.
Google Location history is disabled for my account.
I have Shared endorsements turned off.
Ad personalization is turned off.
I used to run with the Protect my Choices extension till a while back to avoid targeted advertising.

Regardless, the Google purchases page had hundreds of results, going back half a decade. Google currently does not offer a way to delete collected purchases directly, or to pause this collection in any way. The only way is to find the emails that Google scanned, and delete them.

I ended up deleting everything from the following email addresses:

auto-confirm@amazon.com
auto-confirm@amazon.in
cs@flipkart.com
digital-no-reply@amazon.com
do_not_reply@audible.com
do_not_reply@gog.comorders@services.target.com
ebay@ebay.in
googleplay-noreply@google.com
help@stickermule.com
mail@info.fabfurnish.com
no-reply@flipkart.com
no-reply@paytm.com
no_reply@email.apple.com
noreply@flipkart.com
noreply@pizzahut.co.in
noreply@snapdeals.co.in
noreply@steampowered.com
notification@wish.com
order-update@amazon.in
orders@services.target.com
payments-messages@amazon.in
return@amazon.in
ship-confirm@amazon.com
ship-confirm@amazon.in
shipment-tracking@amazon.com
shipment-tracking@amazon.in
updates@myntra.com

Note that deleting the email doesn’t seem to be sufficient either, you need to clear your Trash, and then wait for a while (almost 2 days for me) before the system refreshes. After 3 days of just deleting mails, I finally got this screen:

Google seems to be picking up all kinds of emails, including:

Invoices
Shipment Confirmations / Updates
Return confirmations
Payment Confirmations
Order Cancellations
Payment Failures (gasp!)

Warning about Deletions

I’ve already switched away from Amazon/Flipkart emails from my Gmail. But deleting invoices from your inbox isn’t always the best idea. Most websites will let you re-download invoices (Amazon/Flipkart do), but take care not to delete any necessary emails that you might need for warranty claims or any other purpose later.

MNIST pixel attacks with Pytorch

Bytepawn - Marton Trencseni — Sat, 01 Jun 2019 01:00:00 GMT

It’s easy to build a CNN that does well on MNIST digit classification. How easy is it to break it, to distort the images and cause the model to misclassify?

News sites in a post-advertisement world

Ben Overmyer's Site — Wed, 29 May 2019 03:00:00 GMT

This post is a response to “News Websites are Dumpster Fires”, which rails against the current business model of most news websites.

The problem that faces news websites is not as simple as just a business model change.

The purpose of a news website is to provide accurate, current information on topics relevant to the health and progress of society. This is an expensive endeavor. Journalism isn’t easy - indeed, sometimes it can be dangerous, depending on the topic covered. It requires training, equipment, and no small amount of social savvy.

A news site can’t be funded by the government without the appearance of state bias. The ad-driven model currently employed by the majority of news sites is unsustainable. How, then, can these sites be supported in a way that doesn’t compromise their core mission?

This is a complicated question, and not one I have a solution for. I suspect that the future lies in content producers being funded by the public in some way. The following article may be of interest in this regard: Why isn’t there a news organization truly funded by the public

Writing an x86 emulator from scratch in JavaScript: 1. a stack and register machine

Notes on software development — Tue, 21 May 2019 03:00:00 GMT

Better yet, take a look at this post walking through emulating x86 ELF binaries in Go:
Emulating linux/AMD64 userland: interpreting an ELF binary

Next up in emulator basics: <! forgive me, for I have sinned >
2. system calls

In this post we'll create a small virtual machine in JavaScript and use it to run a simple C program compiled with GCC for an x86_64 (or AMD64) CPU running Linux.

All source code is available on Github.

Virtual machine data storage

Our virtual machine will have two means of storing data: registers and an integer stack. Each register can store a 64-bit integer. The stack is an array of 8-bit (or 1 byte) integers.

We'll make the following registers available for modification and use by the program(mer):

RDI, RSI, RSP, RBP, RAX, RBX, RCX, RDX, R8, R9, R10, R11, R12, R13, R14, R15

The RSP register is used by the virtual machine for tracking the location of the last entry in the stack. It will be modified by the virtual machine when it encounters the POP, PUSH, CALL and RET instructions we'll support. We'll get into the specifics shortly.

And we'll make the following registers available for use (but not modification) by the program(mer):

RIP, CS, DS, FS, SS, ES, GS, CF, ZF, PF, AF, SF, TF, IF, DF, OF

Each of these has a special meaning but we'll focus on RIP. The RIP register contains the address of the instruction currently being interpreted by our virtual machine. After every instruction the virtual machine will increment the value in this register -- except for a few special instructions like CALL and RET.

Memory addresses

It will become useful to provide direct access to memory with a special syntax. We'll focus just on 64-bit addresses that will look like this:

  MOV QWORD PTR [RBP - 8], 12

This asks for the value 12 to be written into the memory address at RBP - 8 bytes. The QWORD PTR part clarifies that we want to write 8 bytes worth of the value. Since 12 is less than 8 bytes, the rest will be filled with zeros.

  ADD RAX, QWORD PTR [RBP - 8]

This asks for eight bytes starting from the memory address RBP - 8 to be added to the value in RAX and stored back in RAX.

Virtual machine instruction set

In our virtual machine we'll define support for the following instructions:

MOV $REGISTER, $REGISTER or $MEMORY ADDRESS or $LITERAL NUMBER
- This instruction copies the second value into the first.
ADD $REGISTER, $REGISTER or $MEMORY ADDRESS
- This instruction adds the second value into the first and stores the result into the first.
PUSH $REGISTER
- This instruction will decrement the RSP register by 8 bytes and store the value at the bottom of the stack.
POP $REGISTER
- This instruction will increment the RSP register by 8 bytes, remove the last element in the stack (at the bottom), and store it into the register.
CALL $LABEL
- This instruction will push the value in the RIP register (plus one) onto the stack and set the RIP register to the line of code of the label. More on this later.
RET
- This instruction will remove the value at the bottom of the stack and store it in the RIP register.

Now we have more than enough instructions to write some interesting programs for the virtual machine.

Virtual machine semantics

We'll make one last assumption before explaining further. In our programs, there must be a main label which must contain a RET instruction. Once we hit the terminal RET, we will exit the virtual machine and set the exit code to the value stored in the RAX register.

Let's look at a simple program:

main: ; the required main label
  MOV RAX, 1 ; store 1 in RAX
  MOV RDI, 2 ; store 2 in RDI
  ADD RAX, RDI ; store the result of adding RAX and RDI in RAX
  RET ; give control back to the virtual machine

When we run this program, first we initialize a stack (we'll give it 1000 elements) and set the RSP register to 1000 (the top of the stack). Then we look for the main label and set the RIP register to 1, the line number after the label appears (0). Then until the RIP register is 1000 again, we interpret the instruction at the line number stored in the RIP register. Once the RIP register hits 1000, we exit the program setting RAX as the exit code.

One more example

Now let's look at one more program:

plus:
  ADD RDI, RSI
  MOV RAX, RDI
  RET

main:
  MOV RDI, 1
  MOV RSI, 2
  CALL plus
  RET

Our virtual machine will start at the line after the main label. Then it will store 1 into RDI and 2 into RSI. Then it will jump to the second line in the program to add RDI and RSI and store the result in RDI. Then it will copy RDI into RAX and return control to the final line. This last RET will in turn return control to the virtual machine. Then the program will exit with exit code 3.

Parsing

Now that we've finished up describing our virtual machine language and semantics, we need to parse the instructions into a format we can easily interpret.

To do this we'll iterate over the program skip any lines that start with a dot. These are virtual machine directives that are important for us to ignore for now. We'll also remove any characters including and following a semi-colon or hash-tag, until the end of the line. These are comments.

We'll store a dictionary of label names to line numbers (the line number of the label plus one) and without the colon.

And we'll store the instructions as an array of objects composed of an operation and optional operands.

Code

function parse(program) {
  const labels = {};
  const instructions = [];

  const lines = program.split('\n');

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace
    // TODO: handle each line
  }

  return { labels, instructions };
}

First let's handle the directives we want to ignore:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.')) {
      continue;
    }
  }

And then comments:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';') || line.startsWith('#')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes('#')) {
      line = line.split('#')[0];
    }

    if (!line) {
      continue;
    }
  }

And then labels:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';') || line.startsWith('#')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes('#')) {
      line = line.split('#')[0];
    }

    if (!line) {
      continue;
    }

    if (line.includes(':')) {
      const label = line.split(':')[0];
      labels[label] = instructions.length;
      continue;
    }
  }

And finally instruction parsing plus the rest:

function parse(program) {
  const labels = {};
  const instructions = [];

  const lines = program.split('\n');

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes(':')) {
      const label = line.split(':')[0];
      labels[label] = instructions.length;
      continue;
    }

    const operation = line.split(/\s/)[0].toLowerCase();
    const operands = line.substring(operation.length).split(',').map(t => t.trim());
    instructions.push({
      operation,
      operands,
    });
  }

  return { labels, instructions };
}

Hurray! A brittle parser.

Interpreting

We've already described the semantics a few times. So let's get started with the foundation and initialization.

We'll use BigInts because JavaScript integers are 53-bits wide. This isn't incredibly important in our basic programs but it will quickly became painful without.

And we'll make process memory available as an array of 8-bit integers. In order to make this easy to use, we'll also provide helper function for writing to and reading from memory.

const fs = require('fs');

const REGISTERS = [
  'RDI', 'RSI', 'RSP', 'RBP', 'RAX', 'RBX', 'RCX', 'RDX', 'RIP', 'R8',
  'R9', 'R10', 'R11', 'R12', 'R13', 'R14', 'R15', 'CS', 'DS', 'FS',
  'SS', 'ES', 'GS', 'CF', 'ZF', 'PF', 'AF', 'SF', 'TF', 'IF', 'DF', 'OF',
];

function writeMemoryBytes(process, address, value, size) {
  for (let i = 0n; i < size; i++) {
    value >>= i * 8n;
    process.memory[address + i] = value & 0xFFn;
  }
}

function readMemoryBytes(process, address, size) {
  let value = 0n;
  for (let i = 0n; i < size; i++) {
    value |= (process.memory[address + i] || 0n) << (i * 8n);
  }
  return value;
}

function interpret(process) {
  // TODO: interpret
}

function main(file) {
  const memory = new Array(10000);
  const code = fs.readFileSync(file).toString();
  const { instructions, labels } = parse(code);

  const registers = REGISTERS.reduce((rs, r) => ({ ...rs, [r]: 0n }), {});
  registers.RIP = BigInt(labels.main === undefined ? labels._main : labels.main);
  registers.RSP = BigInt(memory.length - 8);

  const process = {
    registers,
    memory,
    instructions,
    labels,
  };

  writeMemoryBytes(process, registers.RSP, registers.RSP, 8);

  interpret(process);
  return Number(registers.RAX);
}

process.exit(main(process.argv[2]));

We'll accept _main as an entry point as well as main to support our macOS users. If you know why our macOS users use _main I'd love to know.

To interpret, we grab the instruction pointed to in RIP and switch on the operation.

function interpret(process) {
  do {
    const instruction = process.instructions[process.registers.RIP];
    switch (instruction.operation.toLowerCase()) {
      case 'mov':
        break;
      case 'add':
        break;
      case 'call':
        break;
      case 'ret':
        break;
      case 'push':
        break;
      case 'pop':
        break;
    }
  } while (process.registers.RIP != BigInt(readMemoryBytes(process, memory.length - 8, 8)));
}

Interpreting MOV

Example:

  MOV RAX, 1
  MOV RAX, RDI
  MOV QWORD PTR [RBP - 8], 8

This instruction will store a value into a register or address and increment RIP. If the left-hand side is a memory address we will write to memory.

      case 'mov': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const rhs = interpretValue(process, instruction.operands[1]);
        if (REGISTERS.includes(lhs)) {
          process.registers[lhs] = rhs;
        } else {
          writeMemoryBytes(process, lhs.address, rhs, lhs.size);
        }
        process.registers.RIP++;
        break;
      }

We're delegating to a helper function to handle registers vs. memory addresses vs. literals appropriately. Without memory addresses it's a simple function:

function interpretValue(process, value, { lhs } = { lhs: false }) {
  if (REGISTERS.includes(value)) {
    if (lhs) {
      return value;
    } else {
      return process.registers[value];
    }
  }

  return BigInt.asIntN(64, value);
}

We need to do some hacking to support memory addresses:

function interpretValue(process, value, { lhs } = { lhs: false }) {
  if (REGISTERS.includes(value)) {
    if (lhs) {
      return value;
    } else {
      return process.registers[value];
    }
  }

  if (value.startsWith('QWORD PTR [')) {
    const offsetString = value.substring('QWORD PTR ['.length, value.length - 1).trim();
    if (offsetString.includes('-')) {
      const [l, r] = offsetString.split('-').map(l => interpretValue(process, l.trim()));
      const address = l - r;
      const bytes = 8; // qword is 8 bytes
      if (lhs) {
        return { address, size: bytes };
      } else {
        return readMemoryBytes(process, address, bytes);
      }
    }

    throw new Error('Unsupported offset calculation: ' + value);
  }

  return BigInt.asIntN(64, value);
}

Interpreting ADD

Example:

  ADD RAX, RDI

This instruction will combine both registers and store the result in the first, then increment the RIP register.

      case 'add': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const rhs = interpretValue(process, instruction.operands[1]);
        process.registers[lhs] += rhs;
        process.registers.RIP++;
        break;
      }

Interpreting CALL

Example:

  CALL plus

This instruction store RIP (plus one, to continue after the call instruction) on the stack and sets RIP to the location specified by the label.

      case 'call': {
        process.registers.RSP -= 8n;
        writeMemoryBytes(process, process.registers.RSP, process.registers.RIP + 1n, 8);
        const label = instruction.operands[0];
        process.registers.RIP = process.labels[label];
        break;
      }

Interpreting RET

Example:

RET

This instruction removes the last element from the stack and stores it in the RIP register.

      case 'ret': {
        const value = readMemoryBytes(process, process.registers.RSP, 8);
        process.registers.RSP += 8n;
        process.registers.RIP = value;
        break;
      }

Interpreting PUSH

Example:

  PUSH RAX

This instruction stores the value in the register on the stack and increments RIP.

      case 'push': {
        const value = interpretValue(process, instruction.operands[0]);
        process.registers.RSP -= 8n;
        writeMemoryBytes(process, process.registers.RSP, value, 8);
        process.registers.RIP++;
        break;
      }

Interpreting POP

Example:

  POP RAX

This instruction removes the last element from the stack and stores it into the register specified. Then it increments RIP.

      case 'pop': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const value = readMemoryBytes(process, process.registers.RSP, 8);
        process.registers.RSP += 8n;
        process.registers[lhs] = value;
        process.registers.RIP++;
        break;
      }

All together

$ cat test1.asm
main: ; the required main label
  MOV RAX, 1 ; store 1 in RAX
  MOV RDI, 2 ; store 2 in RDI
  ADD RAX, RDI ; store the result of adding RAX and RDI in RAX
  RET ; give control back to the virtual machine
$ node emulator.js test1.asm
$ echo $?
3

And finally, let's see what we can do with a simple C program:

$ cat plus.c
long main() {
  long a = 5;
  long b = 6;
  return a + b;
}
$ gcc -S -masm=intel -o plus.s plus.c
$ node emulator.js plus.s
$ echo $?
11

And we've got the start of a working x86_64/AMD64 emulator.

Next steps

We aren't setting flags appropriately to support conditionals, so that's low-hanging fruit. Additionally, syscalls open up a new world (that we'll end up needing since exit codes are limited to 8-bits of information). Additionally, our parsing is brittle. Dealing with ELF files may be a better direction to go and also enables more. We'll explore these aspects and others in follow-up posts.

Human interest

I originally intended to build a GameBoy emulator because the hardware is simple and uniform. But I found it easiest to start hacking together an AMD64 emulator because AMD64 is well-documented and gcc is easy enough to use. I'm still interested though unless/until I figure out how to emulate a graphics card for AMD64.

It's tricky! But not that tricky. I built a graphical debugger around this emulator to help out with the logic and off-by-one errors. But ultimately it's been surprising to me how easy it is to get started -- especially when I'm not concerned about absolute correctness (yet).

Here's my first post on a series on emulator basics. It's baby's first stack and register virtual machine and it turns out it runs x86 code. https://t.co/WiWmGedawt #linux #assembly https://t.co/xjiMkhgpdN
— Phil Eaton (@phil_eaton) May 24, 2019

Improving a session cookie

infrequent oscillations — Mon, 20 May 2019 16:22:25 GMT

In which we improve a session cookie with signing and sanity

Migrating DNSCrypt Server to Docker

Nemo's Home — Sat, 18 May 2019 03:00:00 GMT

I’ve been running a personal DNSCrypt server in Bangalore for the last 2 years. When I set it up, it was just a compiled version of dnscrypt-wrapper, which was the bare minimum setup I could do.

Since then, I’ve upgraded it to a distribution supported version, but recent changes in dnscrypt key rotation, I’ve been wanting to setup something automated as well.

The easiest way was to switch to the official DNSCrypt Docker image, which does both key generation and certificate rotation. Since my public key was already present in the DNSCrypt Server lists, I was not too keen to regenerate a new key.

The primary challenge was ensuring that the docker container picks up my existing keys without trying to generate new ones from scratch. It was basically 2 steps:

Match the directory structure that the container expects.
Invoke the container directly into start mode while passing existing keys.

Directory Structure

I copied my keys (public.key, secret.key) to /etc/dnscrypt-keys and ran the following:

echo 2.dnscrypt-cert.captnemo.in > provider_name
touch provider_info.txt # I couldn't figure out how to output the same info, so kept it blank
hexdump -ve '1/1 "%.2x"' < public.key > public.key.txt

Then I ensured that the file permissions are matching what the container expects:

chmod 640 secret.key
chmod 644 public.key
chown root:1002 public.key secret.key
chmod 644 provider_name

This is how the final permissions looked for the directory (/etc/dnscrypt-keys)

-rw-r-----   1 root 1002    64 May 18 07:15 secret.key
-rw-r--r--   1 root 1002    32 May 18 07:15 public.key
-rw-r--r--   1 root root    28 May 18 07:19 provider_name
-rw-r--r--   1 root root     0 May 18 07:23 provider_info.txt
-rw-r--r--   1 root root    64 May 18 07:25 public.key.txt
drwxr-xr-x   2 root root  4096 May 18 07:26 .

Running the Container

Then, I directly ran dnscrypt-wrapper container:

docker run --detatched --restart=unless-stopped --volume /etc/dnscrypt-keys:/opt/dnscrypt-wrapper/etc/keys --publish 10.47.0.5:4434:443/tcp --publish 10.47.0.5:4434:443/udp jedisct1/dnscrypt-server start

I pass a host path mount instead of creating a Docker Volume, since they can get deleted in regular docker prune.

Here, 10.47.0.5 is the “Anchor IP”, which Digital Ocean internally maps to my Floating IP.

The container comes up, generates new short-term keys and goes live:

Starting DNSCrypt service for provider:
2.dnscrypt-cert.captnemo.in
Starting pre-service scripts in /etc/runit_init.d
setup in directory /opt/unbound/etc/unbound
generating unbound_server.key
Generating RSA private key, 3072 bit long modulus (2 primes)
.......++++
...................++++
e is 65537 (0x010001)
generating unbound_control.key
Generating RSA private key, 3072 bit long modulus (2 primes)
.........................++++
........................................++++
e is 65537 (0x010001)
create unbound_server.pem (self signed certificate)
create unbound_control.pem (signed client certificate)
Signature ok
subject=CN = unbound-control
Getting CA Private Key
Setup success. Certificates created. Enable in unbound.conf file to use
ok: run: unbound: (pid 28) 300s
ok: run: dnscrypt-wrapper: (pid 31) 300s
ok: run: unbound: (pid 28) 600s
ok: run: dnscrypt-wrapper: (pid 31) 600s

Once the server was up, I verified connectivity with dnscrypt-proxy and it worked perfectly.

Future Scope

Right now, I have a single container that does 2 things:

Certificate Rotation via a service that checks it every 30 minutes.
DNSCrypt Service, which is accessible over the internet.

For (1) to work, it needs access to the Private Keys that are used to sign the temporary certificates that last 24 hours. Since both things are managed within the same container, the container ends up with both network and long-term keys access. This means, any RCE on the service can result in the long-term keys being compromised.

A simple fix for this would be to separate out the Certificate Rotation part into a separate “mode” on the Docker Image, which can be called independently. This would allow someone to run certificate rotation on a second container using a scheduler, but with far more limitations (such as no network access). A common file-mount between both the containers can take care of sharing the temporary keys between the containers, and a simple unix socket on the shared-file-mount can be used to signal a certificate rotation (this triggers the dnscrypt service restart, so it picks the new cert).

How To Name Things

swyx's site RSS Feed — Thu, 16 May 2019 03:00:00 GMT

There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. - Leon Bambrick

Tail call elimination

Notes on software development — Tue, 14 May 2019 03:00:00 GMT

In this post we'll explore what tail calls are, why they are useful, and how they can be eliminated in an interpreter, a compiler targeting C++, and a compiler targeting LLVM IR.

Tail calls

A tail call is a function call made at the end of a block that returns the value of the call (some languages do not force this return requirement). Here are a few examples.

function tailCallEx1() {
  // Loops forever but is a tail call.
  return tailCallEx1();
}

function tailCallEx2(x) {
  if (x) {
    return tailCallEx2(x - 1);
  }

  return 1;
}

function tailCallEx3(x) {
  return x && tailCallEx(x - 1) || 1;
}

function tailCallEx4(x) {
  switch (x) {
    case 0:
      return 1;
    default:
      return tailCallEx4(x - 1);
  }
}

And here are some examples of non-tail calls.

function nonTailCallEx1(x) {
  if (x) {
    // Not a tail call because the call is not the value returned.
    return 1 + nonTailCallEx1(x - 1);
  }

  return 0;
}

function nonTailCallEx2(x) {
  if (x) {
    const r = nonTailCallEx2(x - 1);
    // Not a tail call because the value is not *immediately* returned.
    console.log(r);
    return r;
  }

  return 1;
}

Why is this important?

Some languages can rewrite a recursive tail call as a jump/branch/goto instead of a function call. This allows:

Potential performance gain if function calls have large overhead
No stack overflows due to no nested function call stacks

Implementation 1: Interpreter

Given a tail call recursive fibonacci:

function fibonacci(a, b, n) {
  if (n === 0) {
    return a;
  }

  if (n === 1) {
    return b;
  }

  return fibonacci(b, a + b, n - 1);
}

Here is how we could transform (by hand) this without a tail call.

function fibonacci(a, b, n) {
  while (true) {
    if (n === 0) {
      return a;
    }

    if (n === 1) {
      return b;
    }

    const a1 = b;
    const b1 = a + b;
    const n1 = n - 1;
    a = a1;
    b = b1;
    n = n1;
  }
}

If this was written in a language with labels and goto we could simplify the code slightly by doing that. But it is the same effect as a loop.

Since we're in an interpreter (that isn't JIT compiling), we cannot pick between these two and must merge them. So we put all function bodies in a loop and break if it isn't a tail call. Otherwise we line up the paremeters and let the loop take us back.

Here is an example of this strategy used in a Scheme interpreter written in D.

// Define a new function with name `name` and add it to the context.
Value namedLambda(Value arguments, Context ctx, string name) {
  auto funArguments = car(arguments);
  auto funBody = cdr(arguments);

  Value defined(Value parameters, void** rest) {
    Context newCtx = ctx.dup;

    // Copy the runtime calling context to the new context.
    Context runtimeCtx = cast(Context)(*rest);
    auto runtimeCallingContext = runtimeCtx.callingContext;
    newCtx.callingContext = runtimeCallingContext.dup;

    Value result;
    // Loop forever, will break immediately if not a tail call
    bool tailCalling = false;
    while (true) {
      if (valueIsList(funArguments)) {
        auto keyTmp = valueToList(funArguments);
        auto valueTmp = valueToList(parameters);
        while (true) {
          auto key = valueToSymbol(keyTmp[0]);
          auto value = valueTmp[0];

          newCtx.set(key, value);

          // TODO: handle arg count mismatch
          if (valueIsList(keyTmp[1])) {
            keyTmp = valueToList(keyTmp[1]);
            valueTmp = valueToList(valueTmp[1]);
          } else {
            break;
          }
        }
      } else if (valueIsSymbol(funArguments)) {
        auto key = valueToSymbol(funArguments);
        newCtx.set(key, car(parameters));
      } else if (!valueIsNil(funArguments)) {
        error("Expected symbol or list in lambda formals", funArguments);
      }

      if (!tailCalling) {
        newCtx.callingContext.push(Tuple!(string, Delegate)(name, &defined));
      }

      result = eval(withBegin(funBody), cast(void**)[newCtx]);

      // In a tail call, let the loop carry us back.
      if (newCtx.doTailCall == &defined) {
        tailCalling = true;
        parameters = result;
        newCtx.doTailCall = null;
      } else {
        break; // Not in a tail call, we're done a regular function call.
      }
    }

    return result;
  }

  return makeFunctionValue(name, &defined, false);
}

We can not eliminate mutually recursive tail calls with this method. We could use continuation-passing style but that would not have addressed the concern: not making a function call.

Implementation 2: Compiling to C++

The strategy here is the same as in the interpreter except for that since tail call recursive functions are known at compile time, we can generate non-generalized code in function bodies.

Here is how a JavaScript compiler transforms the above fibonacci implementation into C++:

void tco_fib(const FunctionCallbackInfo<Value> &args) {
  Isolate *isolate = args.GetIsolate();
  double tco_n = toNumber(args[0]);
  double tco_a = toNumber(args[1]);
  double tco_b = toNumber(args[2]);

tail_recurse_1:

    ;

  bool sym_if_test_58 = (tco_n == 0);
  if (sym_if_test_58) {
    args.GetReturnValue().Set(Number::New(isolate, tco_a));
    return;
  }

  bool sym_if_test_70 = (tco_n == 1);
  if (sym_if_test_70) {
    args.GetReturnValue().Set(Number::New(isolate, tco_b));
    return;
  }

  Local<Value> sym_arg_83 = Number::New(isolate, (tco_n - 1));
  Local<Value> sym_arg_92 = Number::New(isolate, (tco_a + tco_b));
  tco_n = toNumber(sym_arg_83);
  tco_a = tco_b;
  tco_b = toNumber(sym_arg_92);
  goto tail_recurse_1;
}

This is implemented by checking every function call. If the function call is in tail call position, we generate code for jumping to the beginning of the function. Otherwise, we generate a call as usual.

Here is how the tail call check and code-generation is done in the compiler:

function compileCall(
  context: Context,
  destination: Local,
  ce: ts.CallExpression,
) {
  let tcoLabel;
  let tcoParameters;
  if (ce.expression.kind === ts.SyntaxKind.Identifier) {
    const id = identifier(ce.expression as ts.Identifier);
    const safe = context.locals.get(mangle(context.moduleName, id));

    if (safe && context.tco[safe.getCode()]) {
      const safeName = safe.getCode();
      tcoLabel = context.tco[safeName].label;
      tcoParameters = context.tco[safeName].parameters;
    }
  }

  ...

  if (ce.expression.kind === ts.SyntaxKind.Identifier) {
    const id = identifier(ce.expression as ts.Identifier);
    const mangled = mangle(context.moduleName, id);
    const safe = context.locals.get(mangled);

    if (safe) {
      if (tcoLabel) {
        args.forEach((arg, i) => {
          compileParameter(
            context,
            tcoParameters[i],
            i,
            i === args.length - 1,
            arg,
          );
        });

        context.emitStatement(`goto ${tcoLabel}`);
        context.emit('', 0);
        destination.tce = true;
        return;
      }
    }
  }

  ...

  // Otherwise generate regular function call

This requires you to have been building up the state throughout the AST to know whether or not any particular call is in tail position.

Implementation 3: Compiling to LLVM IR

LLVM IR is the most boring because all you do is mark any tail call as being a tail call. Then so long as the call meets some requirements, the key one being that the result of the call must be returned immediately, LLVM will generate a jump instead of a call for you.

Given the following lisp-y implementation of the same tail call recursive fibonacci function (compiler here):

(def fib (a b n)
     (if (= n 0)
         a
       (fib b (+ a b) (- n 1))))

We generate the following LLVM IR:

define i64 @fib(i64 %a, i64 %b, i64 %n) {
  %ifresult13 = alloca i64, align 4
  %sym14 = add i64 %n, 0
  %sym15 = add i64 0, 0
  %sym12 = icmp eq i64 %sym14, %sym15
  br i1 %sym12, label %iftrue16, label %iffalse17
iftrue16:
  %sym18 = add i64 %a, 0
  store i64 %sym18, i64* %ifresult13, align 4
  br label %ifend19
iffalse17:
  %sym21 = add i64 %b, 0
  %sym23 = add i64 %a, 0
  %sym24 = add i64 %b, 0
  %sym22 = add i64 %sym23, %sym24
  %sym26 = add i64 %n, 0
  %sym27 = add i64 1, 0
  %sym25 = sub i64 %sym26, %sym27
  ; NOTE the `tail` before `call` here
  %sym20 = tail call i64 @fib(i64 %sym21, i64 %sym22, i64 %sym25)
  ret i64 %sym20
  store i64 %sym20, i64* %ifresult13, align 4
  br label %ifend19
ifend19:
  %sym11 = load i64, i64* %ifresult13, align 4
  ret i64 %sym11
}

The only difference between supporting tail call elimination in is whether the call instruction is preceeded by a tail directive. That makes the implementation very simple:

const isTailCall = module.exports.TAIL_CALL_ENABLED &&
                   context.tailCallTree.includes(validFunction.value);
const maybeTail = isTailCall ? 'tail ' : '';
this.emit(1, `%${destination.value} = ${maybeTail}call ${validFunction.type} @${validFunction.value}(${safeArgs})`);
if (isTailCall) {
  this.emit(1, `ret ${destination.type} %${destination.value}`);
}

Generated assembly

The resulting generated code (run through llc) for that call will be:

...
    add rax, rsi
    dec rdx
    mov rdi, rsi
    mov rsi, rax
    jmp _fib               ## TAILCALL
...

And if tail call elimination is disabled:

...
    add rax, rsi
    dec rdx
    mov rdi, rsi
    mov rsi, rax
    call    _fib
...

Summary

The last bit I haven't covered is how you track whether or not a call is in tail position. That is difficult to cover in a blog post because it's a matter of you propagating/not propagating at each syntax node type. But generally speaking, if the syntax node is not in tail position (e.g. not the last expression in a block), you drop the tail state you've built up. When you make a function call, you add the function name to the tail state.

But I will be covering this in detail in the LLVM case in the next post in my compiler basics series.

Put together a survey and summary of tail call elimination, the effect and implementation, in an interpreter, a compiler targeting C++, and a compiler targeting LLVM IR. https://t.co/pXiLoXjw2u
— Phil Eaton (@phil_eaton) May 15, 2019

Solving CIFAR-10 with Pytorch and SKL

Bytepawn - Marton Trencseni — Tue, 14 May 2019 01:00:00 GMT

CIFAR-10 is a classic image recognition problem, consisting of 60,000 32x32 pixel RGB images (50,000 for training and 10,000 for testing) in 10 categories: plane, car, bird, cat, deer, dog, frog, horse, ship, truck. Convolutional Neural Networks (CNN) do really well on CIFAR-10, achieving 99%+ accuracy. The Pytorch distribution includes an example CNN for solving CIFAR-10, at 45% accuracy. I will use that and merge it with a Tensorflow example implementation to achieve 75%. We use torchvision to avoid downloading and data wrangling the datasets. Like in the MNIST example, I use Scikit-Learn to calculate goodness metrics and plots.

Football update, 2018-19 season

infrequent oscillations — Mon, 13 May 2019 14:00:18 GMT

In which we import numbers and update information

Updates to Iron Arachne: The World API and more

Ben Overmyer's Site — Sun, 12 May 2019 03:00:00 GMT

It’s been awhile since I blogged about Iron Arachne. Much has changed since then.

The various disparate APIs have been collapsed into a single “World API.” This makes it much easier to change, which means I can iterate faster and release more frequently. Also, the front-facing website is no longer a static JavaScript-powered site, but rather a full Laravel application. This lets me do a few clever things which I’ll get into later.

The culture generator has added appearance generation. Skin colors, hair colors, face shapes, and so forth add to the depth of the output.

The region generator has dramatically improved economic traits. The trade goods are derived from the skill of the local artisans, the local agriculture, and the local available raw materials now. They also take into account the culture of the town.

Deities created by the pantheon generator now adhere to the culture the religion is from. Also, there are subtle improvements to how deities’ domains are generated. No longer will multiple gods have the same domain.

The heraldry generator has had a few tweaks, too, though no new fields or charges have been added. Because the app was rewritten in Laravel, I now make use of caching via Redis to deliver consistent images and blazons. For example, here’s one I quite like:

Per saltire azure and argent, a lion’s head erased Or

Some of the groundwork has been laid for organizing regions into countries. This is a step into a more difficult part of the world generator, since now I need to take into account location relative to other locations. Inevitably, I’ll need to create a map generator. But that’s a story for another day.

Asymmetrical Football

infrequent oscillations — Thu, 09 May 2019 06:45:51 GMT

In which an overlap is missing.

Processing Large Files – Java, Go and 'hitting the wall'

Ben E. C. Boyter — Wed, 08 May 2019 03:00:00 GMT

It started with this chain of blog posts,

Using Java to Read Really, Really Large Files -> Processing Large Files in Java -> Processing Large Files – Java, Go and ‘hitting the wall’

I thought after reading them I would add to the chain.

While I don’t like solving arbitrary company made “programming tests” when doing interviews https://boyter.org/2016/09/companies-recruiting/ whenever I see a post comparing Go to Java where Java is trounced in performance gets my attention. Unless its a very short lived command line application where JVM startup is a problem or some super optimized Go solution in my experience so far Java should generally be slightly faster. The computer language benchmarks tend to show the same result so its not a flight of fancy for me.

Writing a lisp compiler from scratch in JavaScript: 4. LLVM conditionals and compiling fibonacci

Notes on software development — Sat, 04 May 2019 03:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
Next in compiler basics:
5. LLVM system calls
6. an x86 upgrade

In this post we'll extend the compiler's LLVM backend to support compiling conditionals such that we can support an implementation of the fibonacci algorithm.

Specifically we're aiming for the following:

$ cat tests/fib.lisp
(def fib (n)
     (if (< n 2)
         n
       (+ (fib (- n 1)) (fib (- n 2)))))

(def main ()
     (fib 8))
$ node src/ulisp.js tests/fib.lisp
$ ./build/prog
$ echo $?
21

To do this we'll have to add the <, - and if built-ins.

All source code is available on Github.

Subtraction

This is the easiest to add since we already support addition. They are both arithmetic operations that produce an integer. We simply add a mapping of - to the LLVM instruction sub so our LLVM backend constructor (src/backends/llvm.js) looks like this:

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),

...

Less than

The < builtin is a logical operation. These are handled differently from arithmetic operations in LLVM IR. A logical operation looks like this:

%3 = icmp slt i32 %1, %2

This says that we're doing an integer comparison, icmp, (with signed less than, slt) on the i32 integers in variables %1 and %2.

We can shim this into our existing compileOp helper like so:

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '<': this.compileOp('icmp slt'),

...

Conditionals

The last part we need to add is support for conditional execution of code at runtime. Assembly-like languages handle this with "jumps" and "labels". Jumping causes execution to continue at the address being jumped to (instead of just the line following the jump instruction). Labels give you a way of naming an address instead of having to calculate it yourself. Our code will look vaguely like this:

  %test = icmp slt i32 %n, %1
  br i1 %test, label %iftrue, label %iffalse
iftrue:
  ; do true stuff
iffalse:
  ; do false stuff

  ; do next stuff

The br instruction can jump (or branch) conditionally or unconditionally. This snippet demonstrates a conditional jump.

But there are a few things wrong with this pseudo-code. First off if the condition is true, execution will just continue on into the false section once finished. Second, LLVM IR actually requires all labels to end with a branch instruction. So we'll add a new label after the true and false section called ifresult and jump to it unconditionally after both.

  %test = icmp slt i32 %n, %1
  br i1 %test, label %iftrue, label %iffalse
iftrue:
  ; do true stuff
  br label %ifresult
iffalse:
  ; do false stuff
  br label %ifresult
ifresult:
  ; do next stuff

Scope

One last thing we'll need to do before implementing the code generation for this is to update our Scope class to accept symbol prefixes so we can pass our labels through Scope to make sure they are unique but still have useful names.

...

class Scope {

  ...

  symbol(prefix = 'sym') {
    const nth = Object.keys(this.locals).length + 1;
    return this.register(prefix + nth);
  }

...

compileIf

Now we can add a primitive function mapping if to a new compileIf helper and implement the helper.

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '<': this.compileOp('icmp slt'),
      'if': this.compileIf.bind(this),

...

  compileIf([test, thenBlock, elseBlock], destination, scope) {
    const testVariable = scope.symbol();

    // Compile expression and branch
    this.compileExpression(test, testVariable, scope);
    const trueLabel = scope.symbol('iftrue');
    const falseLabel = scope.symbol('iffalse');
    this.emit(1, `br i1 %${testVariable}, label %${trueLabel}, label %${falseLabel}`);

    // Compile true section
    this.emit(0, trueLabel + ':');
    this.compileExpression(thenBlock, destination, scope);
    const endLabel = scope.symbol('ifend');
    this.emit(1, 'br label %' + endLabel);
    this.emit(0, falseLabel + ':');

    // Compile false section
    this.compileExpression(elseBlock, destination, scope);
    this.emit(1, 'br label %' + endLabel);

    // Compile cleanup
    this.emit(0, endLabel + ':');
  }
...

Note that this code generation sends the destination variable into both the true and false sections. Let's try it out.


$ node src/ulisp.js tests/fib.lisp
llc: error: llc: build/prog.ll:19:3: error: multiple definition of local value named 'sym5'
  %sym5 = add i32 %sym15, %sym16
  ^
child_process.js:665
    throw err;
    ^

Error: Command failed: llc -o build/prog.s build/prog.ll
llc: error: llc: build/prog.ll:19:3: error: multiple definition of local value named 'sym5'
  %sym5 = add i32 %sym15, %sym16

That's annoying. An unfortunate aspect of LLVM's required
single-static assignment form is that you cannot reuse variable names
within a function even if it is not possible for the variable to be
actually reused.
To work around this we need to allocate memory on the stack, store the
result in each true/false section in this location, and read from this
location afterward to store it in the destination variable.
Stack memory instructions
LLVM IR gives us alloca to allocate memory on the stack,
store to store memory at a stack address, and
load to read the value at a stack address into a
variable. Here's a simple example:
%myvar = add i32 42, 0
%stackaddress = alloca i32, align 4
store i32 %myvar, i32* %stackaddress, align 4
%newvar = load i32, i32* %stackaddress, align 4

Such that newvar is now 42.
compileIf again
Applying this back to our compileIf helper gives us:
...

  compileIf([test, thenBlock, elseBlock], destination, scope) {
    const testVariable = scope.symbol();
    const result = scope.symbol('ifresult');
    // Space for result
    this.emit(1, `%${result} = alloca i32, align 4`);

    // Compile expression and branch
    this.compileExpression(test, testVariable, scope);
    const trueLabel = scope.symbol('iftrue');
    const falseLabel = scope.symbol('iffalse');
    this.emit(1, `br i1 %${testVariable}, label %${trueLabel}, label %${falseLabel}`);

    // Compile true section
    this.emit(0, trueLabel + ':');
    const tmp1 = scope.symbol();
    this.compileExpression(thenBlock, tmp1, scope);
    this.emit(1, `store i32 %${tmp1}, i32* %${result}, align 4`);
    const endLabel = scope.symbol('ifend');
    this.emit(1, 'br label %' + endLabel);
    this.emit(0, falseLabel + ':');

    // Compile false section
    const tmp2 = scope.symbol();
    this.compileExpression(elseBlock, tmp2, scope);
    this.emit(1, `store i32 %${tmp2}, i32* %${result}, align 4`);
    this.emit(1, 'br label %' + endLabel);

    // Compile cleanup
    this.emit(0, endLabel + ':');
    this.emit(1, `%${destination} = load i32, i32* %${result}, align 4`);
  }

...

Trying it out
We run our compiler one more time:
$ node src/ulisp.js tests/fib.lisp
$ ./build/prog
$ echo $?
21

And get what we expect!
Next up

Tail call optimization
Lists and dynamic memory
Strings?
Foreign function calls?
Self-hosting?

Latest post in the compiler basics series: using LLVM conditionals in compiling a fibonacci program https://t.co/A72yEDQ8sd
— Phil Eaton (@phil_eaton) May 5, 2019



Solving MNIST with Pytorch and SKL
Bytepawn - Marton Trencseni — Thu, 02 May 2019 01:00:00 GMT
MNIST is a classic image recognition problem, specifically digit recognition. It contains 70,000 28x28 pixel grayscale images of hand-written, labeled images, 60,000 for training and 10,000 for testing. Convolutional Neural Networks (CNN) do really well on MNIST, achieving 99%+ accuracy. The Pytorch distribution includes a 4-layer CNN for solving MNIST. Here I will unpack and go through this example. We use torchvision to avoid downloading and data wrangling the datasets. Finally, instead of calculating performance metrics of the model by hand, I will extract results in a format so we can use SciKit-Learn's rich library of metrics. 

 


Come over for dinner
CRAIG KERSTIENS — Wed, 01 May 2019 23:55:56 GMT
When I first moved to the Bay area I was fresh out of grad school. I was frequently heading out to dinner or to happy hour after work with colleagues. I was young and single, so why not of course. As time passed, marriage, kids, etc. the ability to go out for a quick drink or dinner was competing with various priorities. Dinner and drinks with co-workers was always a great time. It wasn’t just about hanging out, it built rapport and trust which I found made me a more effective teammate and product manager. It was about 8 years ago that I started to implement a variation of heading out for dinner and drinks.
I started inviting people over for dinner.
I still do this regularly. Roughly once a week we end up hosting someone for dinner. Sometimes it is a single person, sometimes it is a group of people. Sometimes it is co-workers, sometimes former colleagues, often friends that don’t work in tech. Growing up in the south it was common to have people over, I’d said we did that just as much as going out to dinner with folks. You’d get an invite to go to someone elses place and you’d show up with a bottle of wine or flowers in hand. Initially when I asked people in the Bay area over for dinner I’d get weird looks. Over? Like to your house?
The reaction from folks at the end of the night was very often… that was really fun. Thanks for the invite, I can’t remember the last time I just sat down at someones place, had a good meal, and conversation.
Once I found early success with this I started implementing it pretty methodically. When remote workers were in town I’d make sure to place them at thet top of the list to come if the scheduling worked. Same when friends visit from out of town. I’d also try to regularly rotate through my teams and those that report to me. At one point when I had 22 engineers that I was leading product for I had to do a bit of juggling and stagger things a bit, groups of 4 folks or so at a time and each would be over about once every 6 months. I made life a bit easier on myself by grouping vegans together, and vegetarians, and meat eaters. When you’re cooking in the Bay area you probably don’t have a 500 sq. foot kitchen, so cooking 2-3 different meals adds complexity. Thus simplifying.
All of this was for selfish reasons. I could excuse myself for 15 minutes to give the kids a bath, to read them a story, to tuck them in. At the same time I was able to continue building a rapport with co-workers that allowed us both to work together better.


Responsibility and ownership
Notes on software development — Tue, 30 Apr 2019 03:00:00 GMT
Responsibility is only possible by granting ownership and setting
expectations. If you don't turn over ownership, don't expect folks to
take responsibility. When you grant ownership and set expectations,
you'll be astounded what folks will accomplish without you.
I am astounded.


Reinforcement Learning: Game Theory
swyx's site RSS Feed — Mon, 29 Apr 2019 06:59:47 GMT
RL with multiple actors


The VIM Experience
Omar Abid - Personal Blog — Sat, 27 Apr 2019 03:00:00 GMT
7 years journey with VIM


Talking on the phone: better communication
CRAIG KERSTIENS — Wed, 24 Apr 2019 23:55:56 GMT
I interact with a lot of people in a given week, a few in person and far more on video and conference calls. I don’t claim to be a perfect person to talk to on the phone, but over the past several years I’ve noticed how painful some conference call experiences can be. As more and more work is conducted virtually and not face to face an ability to do communicate well on conference/voice calls is tied to what success you can deliver. It isn’t about having a fancy phone or high bandwidth video call–though that can at times be useful


Let’s start with local/remote teams


This one is a huge problem as companies go from 100% local to partially remote. When you start a meeting take note of how many folks are local vs. remote. If you have the majority of attendees on a video or conference call you’re likely fine. If however the majority is local you need to make a concerted effort to pause and ask for input from the phone. This is doubley true on large conference calls… in those cases it can be best to explicitly call out remote folks to chime in.


Talk to the phone


When you’re in person folks can tend to wander, look away, work on a whiteboard. All of these things can be very collaborative when in person but are absolutely terrible on the phone. When you’re not speaking into where the microphone is you will immediately come across choppy. The subtle take away is you’re not a professional. That doesn’t mean you’re doing it intentionally, but when interacting via voice you have to be extra intentional to come across how you intend.


Strong, short words


This applies when in person, but is especially true on the phone when you can’t read someone else. If you don’t have verbal cues to feed off from the other person then you need to listen to audible ones. If you’re talking for 5 minutes non-stop then you’ve learned nothing about how the other person is reacting. At times you may need to talk that long. I like to talk, but after 5 minutes talking non-stop on the phone I have to make sure I stop and say I’ve been talking too long. One it is only fair, and two it is a converation and you need to ensure that it’s flowing both ways.


Gone with the wind


Your environment and surroundings are absolutely terrible to a productive conversation. If you’re walking and there is any sense of wind, if the room is echo-ey, fix it. I’ve seen this both with people taking calls in transit, but also with conference rooms that weren’t well designed. For in transit invest in a great headset if you’ll be taking calls. Test several, read some reviews. If it is a conference actively ask others on calls with you. To just assume it’s fine will lead to people silently dismissing you’re not serious because calls with others are smoother.


Overcommunicate


Speed up, slow down, raise your voice, lower your voice. On the phone you only have one form of expression: your voice. Use it. For some this takes practice. Whenever I’m talking to press or analysts I make sure I’m standing. I make sure to not have coffee 3-4 hrs before because otherwise I’m overly animated. Make sure to take pauses. An explicit pause can give others a chance to chime in and ask questions. When you hear someone on the phone asking a question and you already started talking because of latency finish, then apologize, and ask for them to go ahead.


Be intentional


No matter who you are or what you do you’re on the phone plenty. Whether a 1:1 call or a conference call. Most of use don’t like these calls and feel drained or frustrated by them. I don’t claim that most of my calls feel like a friday night our at a baseball game, but by being explicit in how we communicate I do feel that you can make calls not feel like busy work. Instead they can be a productive way to communicate if you make sure you’re communicating clearly and explicitly, but also listening and engaging equally with the other side of the line.


Learning Gears
swyx's site RSS Feed — Fri, 19 Apr 2019 03:00:00 GMT
Translations welcome! (Português)


Using email as an effective tool
CRAIG KERSTIENS — Tue, 16 Apr 2019 23:55:56 GMT
I send way too many emails in a day. My inbox is very intermingled with my to do list and often represents some form of it. More relevant though is that email is a primary means of how I accomplish work. Being a PM I work cross functionally with other teams (from marketing, to engineering, to sales, to BD, to other product teams) and of course customers. Having to work so cross functionality I’ve found a lot of hacks I use to be able to better accomplish your goals with email, here is a collection of some of those.
Let me be clear, this is not another post about inbox 0, how I swapped to slack. This instead is how I use email to more effectively communicate and get people to engage. In other words it is about making emails more useful, not just getting through them faster. And onto those tips.


Don’t leave a document in a document


Often times I’ve found folks will collaborate in a document during a meeting or take notes there. After the meeting folks will email the document around, but few seldom actually open. Reasons may be they’re not logged in on their phone or it may be they just don’t care that much, I’m not really sure. What I do know is that by taking the notes and action items from the doc and including them in the email you will get more people paying attention to them. If you really want to get good at this when there is significant revisions of a work in progress document, re-circulate the updated version via email. Again not just a link to it, but the document itself.


Mail merge isn’t just for marketing


Years ago I sent a company wide request for feedback (to about 120 people). I got less than 3 responses. I ran an experiment the next time I needed the same thing using a mail merge so my same email seemed personal and was from me to them instead of some large alias. I got a response rate of over 45%. Use this wisely… not every email you send needs a response. A broad update can absolutely be to team@, but when you need to get actual feedback and people don’t seem to chime in this approach works great.


Conversation begets conversation


Often times when there is an email to team@ folks just simply click archive. A great tip for managers and leadership is to chime in on these emails right away. It can help promote some positive discussion and cause folks to actually take notice of updates that might otherwise end up archived. Be careful of the inverse of this though, replying to all mass emails can create a culture where everyone thinks it’s their job to +1 and thumbs up things. There is a balance to this one between team productivity and team morale.


Text is boring


Email is not like reading a novel (except when my colleague Daniel Farina writes them), if I wanted to read a novel I’d actually go read a novel. For email I want the clear concise points broken out. You should think about how easy it is to read on a phone. If a single paragraph consumes my entire screen you’ve lost your readers attention and that content goes else where. A number of things can help for readability:

Short paragraphs and usage of line breaks
Bullets are your friend (aren’t you enjoying this list already?)
Numbers are also great as they call attention to themselves

Bold/italics can also be useful, but assume many people use text only clients so you need to be careful. Also really don’t over-use these


Add extra tooling


Gmail and Outlook are both continuing to improve over time. With the demise of Inbox Gmail itself is seeing some more of those features. Reminders about old un-responded to emails are great. But you can do much more by adding in a third party tool. Boomerang and yesware are two popular ones. These can help with things like returning an email to your inbox in a couple weeks so it can be out of sight for now but come back later. You can also schedule emails to be sent later at a time that may be more ideal for someone to read it, instead of sending it at midnight what about 9am so it appears more top of their inbox?


Do you know about muting?


Muting is the single greatest feature within gmail. Email is a great place for an archive and history, it is a great place to clearly communicate things, it can be a horrible place when it comes to generating a bunch of noise.
Personally I subscribe to lots of lists. I have filters for many of those, but many I actually want to see. That doesn’t mean I care about every response to every thread on those lists. When engineering is discussing whether to use CircleCI vs. Travis I see that the discussion is happening but don’t care too much on the outcome. Mute will silence that entire thread and move it to archived for me. It will only re-appear if I am explicitly added to the To:
The fact that it persists in my archive is key for me. If I want to go back later and search to see the outcome I can, but I don’t have to create a new rule, or archive each reply to the thread.


These are just a few


These are only a few tips, but ones I’ve found extremely useful on two sides. The first is being more control of my inbox so it doesn’t control me. A workflow that works for me which includes mute, filtering lists, scheduling emails for later helps with that. The second is to me email isn’t something I have to do, it is a tool to work more effectively. Being intentional on how I construct messages, when I send them, and what is clearly communicated is key for my job.
I’m curious, what are some of your favorite email tips @craigkerstiens.


Developer Empowerment - Move fast, don't break things (for long), be productive
Ben E. C. Boyter — Tue, 16 Apr 2019 03:00:00 GMT
Recently my job/career path has had me looking “under the hood” at clients development practices. The idea being to see where their strengths and weaknesses are and then make actionable suggestions.

Generally every client wants the same outcome. They want to know how to improve software quality and deliver features faster.

When doing this I focus on the development side, but honestly a lot of this applies to DevOp’s as well. In most situations I realized very quickly that I would be terrified to make any changes in the environments I am working with. No safety net where the penalty for mistakes is very high. I also see the same issues over and over again. When asked to articulate what the issues I saw at one client I came back with the words “You need to focus on developer empowerment”.


SVM with Pytorch
Bytepawn - Marton Trencseni — Tue, 16 Apr 2019 01:00:00 GMT
I use the standard Iris dataset for supervised learning with a Support Vector Machine model using Pytorch's autograd.




Stripping Audible DRM
Nemo's Home — Sun, 14 Apr 2019 03:00:00 GMT
Self-Guide for stripping the Audible DRM, in similar vein as my Kindle Self-Guide.


  Download the aax file from Audible website.
  Run the inAudible-NG Rainbrow crack table against the AAX file.


Easiest way is via docker/podman:

cd ~/Music/Audiobooks
podman run -v $(pwd):/data ryanfb/inaudible@sha256:b66738d235be1007797e3a0a0ead115fa227e81e2ab5b7befb97d43f7712fac5
for i in "*.m4a"; do fix-audible-m4a "$i";done


The cool part about this is that the entire activation is done offline, and runs a Rainbow Table attack against the Audible DRM. To make the process faster in the future, you can save your “activation bytes” (8 hex characters) and directly use them with ffmpeg to decode instead:

ffmpeg -loglevel panic -y -activation_bytes ${AUDIBLE_ACTIVATION_BYTES} -i "$aax_file" -c:a copy -vn "$m4a_file"

A small percentage of Audible AAX files have a incorrect bit set in the “Audio Object Type Specific Config” in the ESDS atom in an M4A file, which leads to them not playing in
Firefox/Android and some other players. To fix this, I have a fix-audible-m4a script called above.

References


  https://github.com/ryanfb/docker_inaudible_rainbowcrack
  https://github.com/inAudible-NG/tables
  https://rentry.co/n4ost



Interpreting TypeScript
Notes on software development — Sun, 14 Apr 2019 03:00:00 GMT
In addition to providing a static type system and compiler for a
superset of JavaScript, TypeScript makes much of its functionality
available programmatically. In this post we'll use the TypeScript
compiler API to build an interpreter. We'll build off of a TypeScript
wiki
article
and cover a few areas that were confusing to me as I built out
a separate project.
The end result we're building will look like this:
$ cat test.ts # A program we can interpret
print(1 + 5);
$ tsc interpreter.ts # Build the source code for the interpreter
$ node interpreter.js test.ts # Run the interpreter against test program
6

All code is available on Github.
Setup
To begin with, we need Node.js and some dependencies:
$ yarn add typescript @types/node

Then we can begin the first stage of an interpreter: parsing the code.
Parsing
Parsing a fixed set of files is simple enough. We pass a list of files
to createProgram along with compiler options. But, as a user, we
don't want to keep track of all files used by a program
(i.e. everything we import). The most ideal situation is to pass a
single-file entrypoint (something like a main.js) and have our
interpreter figure out all the imports and handle them
recursively. More on this later, for now we'll just parse the
single-file entrypoint.
import * as ts from 'typescript';

const TS_COMPILER_OPTIONS = {
  allowNonTsExtensions: true,
};

function parse(fileName: string): ts.Program {
  return ts.createProgram([fileName], TS_COMPILER_OPTIONS);
}

function interpret(program: ts.Program) { // TODO }

function main(entrypoint: string) {
  const program = parse(entrypoint);
  interpret(program);
}

main(process.argv[2]);

interpret and ts.Program
A program contains all source files as well as any implicitly needed
TypeScript definition files (for us it will just be the TypeScript
definitions for the Node.js standard library).

  The program also gives us access to a type checker that we can use
  to query the type of any node in the program tree. We'll get into
  this in another post.
Our interpret program will iterate over all the source files, ignoring
the TypeScript definition files, and call interpretNode on all the
elements of the source file.
function interpretNode(node: ts.Node) { // TODO }

function interpret(program: ts.Program) {
  return program.getSourceFiles().map((source) => {
    const { fileName } = source;
    if (fileName.endsWith('.d.ts')) {
      return;
    }

    const results = [];
    ts.forEachChild(source, (node) => {
      results.push(interpretNode(node));
    });
    return results;
  });
}

interpretNode and ts.Node
A Node is a wrapper for most elements of what we consider a program to
be, such as a binary expression (2 + 3), a literal
expression (2), a function call expression
(a(c)), and so forth. When exploring a parser, it takes
time to become familiar with the particular way that a parser breaks
out a program into a tree of nodes.
As a concrete example, the following program:
print(a);

Will be built into ts.Node tree along these lines:
Node: ExpressionStatement: print(a);
  Node: CallExpression: print, a
    Node: Identifier: print
    Node: Identifier: a
Node: EndOfFileToken

And another example:
1 + 3;

Will be built into a ts.Node tree along these lines:
Node: Expression: 1 + 3
  Node: BinaryExpression: 1, 3, +
    Node: NumericLiteral: 1
    Node: NumericLiteral: 3
    Node: PlusToken
Node: EndOfFileToken

But how would one come to know this?
Exploring the ts.Node tree
The easiest thing to do is throw an error on every Node type we don't
yet know about and fill in support for each program we throw at the
interpreter.
For example:
function interpretNode(node: ts.Node) {
  switch (node.kind) {
    default:
      throw new Error('Unsupported node type: ' + ts.SyntaxKind[node.kind]);
  }
}

Now let's run our interpreter against an input file,
test.ts, that combines these two to make a
semi-interesting program:
$ cat test.ts
print(1 + 2);
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: ExpressionStatement
...

And we see an outer wrapper, an ExpressionStatement. To proceed we
look up the definition of an ExpressionStatement in TypeScript source
code,
src/compiler/types.ts
to be specific. This file will become our best friend. Hit ctrl-f and
look for "interface ExpressionStatement ". We see that it has only one
child, expression, so we call interpretNode
on this recursively:
function interpretNode(node: ts.Node) {
  switch (node.kind) {
    case ts.SyntaxKind.ExpressionStatement: {
      const es = node as ts.ExpressionStatement;
      return interpretNode(es.expression);
    }
    default:
      throw new Error('Unsupported node type: ' + ts.SyntaxKind[node.kind]);
  }
}

Thankfully TypeScript will be very quick to call us out if we
misunderstand this structure.

  It's pretty weird to me that the ts.Node tree is organized such that
  I must cast at each ts.Node but that's what they do even in the
  TypeScript source so I don't think I'm misunderstanding.
Now we recompile and run the interpreter against the program to
discover the next ts.Node type.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: CallExpression
...

Cool! Back to
src/compiler/types.ts.
Call expressions are complex enough that we'll break out handling them
into a separate function.
interpretCall and ts.CallExpression
From our reading of types.ts we need to handle the expression that
evaluates to a function, and we need to handle its parameters. We'll
just call interpretNode on each of these to get their
real value. And finally we'll call the function with the arguments.
function interpretCall(ce: ts.CallExpression) {
  const fn = interpretNode(ce.expression);
  const args = ce.arguments.map(interpretNode);
  return fn(...args);
}

function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.CallExpression: {
      const ce = node as ts.CallExpression;
      return interpretCall(ce);
    }
    ...
  }
}


  Please ignore the fact that we are not correctly setting
  this here.
Recompile and let's see what's next!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: Identifier
...

And back to types.ts.
ts.Identifier
In order to support identifiers in general we'd need to have a context
we could use to look up the value of an identifier. But we don't have
a context like this right now so we'll add builtin support for a
print function so we can get some output!
function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.Identifier: {
      const id = (node as ts.Identifier).escapedText as string;
      if (id === 'print') {
        return function (...args) { console.log(...args); };
      }

      throw new Error('Unsupported identifier: ' + id);
    }
    ...
  }
}

Recompile and let's see what's next!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: BinaryExpression
...

And we're finally into the parameters.
interpretBinaryExpression and ts.BinaryExpression
Looking into types.ts for this Node type suggests we may want to break
this out into its own function as well; there are a ton of operator
types. Within the interpretBinaryExpression helper we'll
interpret each operand and then switch on the operator type. We'll
throw an error on operators we don't know about -- all of them at
first:
function interpretBinaryExpression(be: ts.BinaryExpression) {
  const left = interpretNode(be.left);
  const right = interpretNode(be.right);
  switch (be.operatorToken.kind) {
    default:
      throw new Error('Unsupported operator: ' + ts.SyntaxKind[be.operatorToken.kind]);
  }
}

function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.BinaryExpression: {
      const be = node as ts.BinaryExpression;
      return interpretBinaryExpression(be);
    }
    ...
  }
}

We know the drill.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: FirstLiteralToken
...

At this point we're actually failing first on an unknown node type
rather than an operator. This is because we interpret the operands
(which are numeric literals) before we look up the operator. Time to
revisit types.ts!
ts.FirstLiteralToken, ts.NumericLiteral
Looking at types.ts shows us that FirstLiteralToken is a
synonym for NumericLiteral. The latter name is more
obvious, so let's add that to our supported Node list:
function interpretNode() {
  switch (node.kind) {
    ...
      case ts.SyntaxKind.NumericLiteral: {
      const nl = node as ts.NumericLiteral;
      return Number(nl.text);
    }
    ...
  }
}

And we keep going!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported operator: PlusToken
...

And we're into unknown operator territory!
interpretBinaryExpression and ts.PlusToken
A simple extension to our existing
interpretBinaryExpression, we return the sum of the left
and right values:
function interpretBinaryExpression(be: ts.BinaryExpression) {
  const left = interpretNode(be.left);
  const right = interpretNode(be.right);
  switch (be.operatorToken.kind) {
    case ts.SyntaxKind.PlusToken:
      return left + right;
    default:
      throw new Error('Unsupported operator: ' + ts.SyntaxKind[be.operatorToken.kind]);
  }
}

And we give it another shot.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: EndOfFileToken
...

ts.SyntaxKind.EndOfFileToken
Our final Node type before a working program, we simply do nothing:
function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.EndOfFileToken:
      break;
    ...
  }
}

One more time:
$ tsc interpreter.ts
$ node interpreter.js test.ts
3

A working program! And if we jiggle the test?
$ cat test.ts
print(1 + 5);
$ node interpreter.js test.ts
6

We're well on our way to interpreting TypeScript, and have gained some
familiarity with the TypeScript Compiler API.
All code is available on Github.
Quick intro to the TypeScript Compiler API by writing an interpreter https://t.co/QKz3XtOuP9
— Phil Eaton (@phil_eaton) April 14, 2019
 


Writing a web server from scratch: 1. HTTP and sockets
Notes on software development — Sat, 06 Apr 2019 03:00:00 GMT
Say we have some HTML:
<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

And say we'd like to be able to render this page in a web browser. If
the server is hosted locally we may want to enter
localhost:9000/hello-world.html in the address bar, hit
enter, make a request (done by the browser), receive a response (sent
by some server), and render the result (done by the browser).
Here is a minimal, often incomplete, and unsafe Node.js program (about
100 LoC) that would serve this (code available on
Github):
const fs = require('fs');
const net = require('net');

const CRLF = '\r\n';

const HELLO_WORLD = `<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>`;
const NOT_FOUND = `<html>
  <body>
    <h1>Not found</h1>
  </body>
</html>`;

class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
    this.request = {
      statusLine: null,
      headers: {},
      body: null,
    };
  }

  parse(buffer) {
    const lines = buffer.toString().split(CRLF);

    // Parse/store status line if necessary
    if (!this.request.statusLine) {
      const [method, path, protocol] = lines.shift().split(' ');
      this.request.statusLine = { method, path, protocol };
    }

    // Parse/store headers if the body hasn't begun
    if (!this.request.body) {
      for (let line = lines.shift(); lines.length; line = lines.shift()) {
        // Reached the end of headers, double CRLF
        if (line === '') {
          this.request.body = '';
          break;
        }

        const [key, value] = line.split(':');

        const safeKey = key.toLowerCase();
        if (!this.request.headers[safeKey]) {
          this.request.headers[safeKey] = [];
        }

        this.request.headers[safeKey].push(value.trimStart());
      }
    }

    this.request.body += lines.join(CRLF);
  }

  requestComplete() {
    if (!this.request.statusLine || !Object.keys(this.request.headers).length || this.request.body === null) {
      return false;
    }

    const [contentLength] = this.request.headers['content-length'] || [];
    if (this.request.statusLine.method !== 'GET' && this.request.body.length !== contentLength) {
      return false;
    }

    return true;
  }

  sendResponse() {
    const response = { status: 200, statusMessage: 'OK', body: '' };

    if (this.request.statusLine.path === '/hello-world.html') {
      response.body = HELLO_WORLD;
    } else {
      response.status = 404;
      response.statusMessage = 'NOT FOUND';
      response.body = NOT_FOUND;
    }

    const serialized = 'HTTP/1.1 ${response.status} ${response.statusMessage}' + CRLF +
                       'Content-Length: ' + response.body.length + CRLF + CRLF +
                       response.body;
    this.connection.write(serialized);
  }

  handle(buffer) {
    this.parse(buffer);

    if (!this.requestComplete()) {
      return;
    }

    this.sendResponse();

    // Other-wise the connection may attempt to be re-used, we don't support this.
    this.connection.end();
  }
}

function handleConnection(connection) {
  const handler = new HTTPRequestHandler(connection);
  connection.on('data', (buffer) => handler.handle(buffer));
}

const server = net.createServer(handleConnection);

server.listen('9000');

So what's going on?
The protocol
HTTP (version 1.1, specifically) is a convention for connecting over
TCP/IP and sending plain-text messages between two processes. HTTP
messages are broken into two categories: requests (the sender of a
request is called a "client") and responses (the sender of a response
is called a "server").
HTTP is important because it is the default protocol of web
browsers. When we type in localhost:9000/hello-world.html
and hit enter, the browser will open an TCP/IP connection to the
location localhost on the port 9000 and send
an HTTP request. If/when it receives the HTTP response from the server
it will try to render the response.
An HTTP request
A bare minimum HTTP/1.1 request (defined
here) based
on the request for localhost:9000/hello-world.html is the
following:
GET /hello-world.html HTTP/1.1\r\nHost: localhost:9000\r\n\r\n


  The spec explicitly requires the \r\n combo to
  represent a newline instead of simply \n.
If we printed out this request it would look like this:
GET /hello-world.html HTTP/1.1
Host: localhost:9000

Components of an HTTP request
An HTTP/1.1 request is made up of a few parts:

[Mandatory]: The status line (the first line) followed by a CRLF (the \r\n combo)
[Mandatory]: HTTP headers separated by a CRLF and followed by an additional CRLF
[Optional]: The request body

The status line consists of the request method (e.g. GET, POST, PUT,
etc.), the path for the request, and the protocol -- all separated by
a space.
An HTTP header is a key-value pair separated by a colon. Spaces
following the colon are ignored. The key is case insensitive. Only
the Host header appears to be mandatory. Since these
headers are sent by the client they are intended for the server's use.
The request body is text and is only relevant for requests of certain
methods (e.g. POST but not GET).
An HTTP response
A bare minimum HTTP/1.1 response (defined
here) based
on the file we wanted to send back is the following:
HTTP/1.1 200 OK\r\n\r\n<html>\n  <body>\n    <h1>Hello world!</h1>\n  </body>\n</html>

If we printed out this response it would look like this:
HTTP/1.1 200 OK

<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

Components of an HTTP response
An HTTP/1.1 response is made up of a few parts:

[Mandatory]: The status line (the first line) followed by a CRLF
[Optional]: HTTP headers separated by a CRLF and followed by an additional CRLF
[Optional]: The request body

The status line consists of the protocol, the status code, and the
status message -- all separated by a space.
HTTP response headers are the same as HTTP request headers although in
a response they are directives from the server to the client. There
are many standard
headers that
are used for such things as setting cache rules, setting cookies,
settings response type (e.g. HTML vs CSS vs PNG so the browser knows
how to handle the response).
The response body is similar to the HTTP request body.
Sockets
Most operating systems have a built-in means of connecting over TCP/IP
(and sending and receiving messages) called "sockets". Sockets allow
us to treat TCP/IP connections like files in memory. Most programming
languages have a built-in socket library. Node.js provides a
high-level interface for listening on a port and handling new
connections.
function handleConnection(connection) {
  connection.on('data', (buffer) => doSomething???);
}

const server = net.createServer(handleConnection);

server.listen('9000');

Once the program is listening, clients can open TCP/IP connections to
the address (localhost) and port (9000) and
our program takes over from there. Each connection is handled
separately and receives "data" events. Each data event includes new
bytes available for us to handle.
Let's encapsulate the state of each connection in HTTPRequestHandler
class. Its function will be to parse data as it becomes available and
respond to the request when the request is done.
class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
  }

  parse(buffer) {}

  requestComplete() {}

  sendResponse() {}

  handle(buffer) {
    this.parse(buffer);

    if (!this.requestComplete()) {
      return;
    }

    this.sendResponse();

    // Other-wise the connection may attempt to be re-used, we don't support this.
    this.connection.end();
  }
}

function handleConnection(connection) {
  const handler = new HTTPRequestHandler(connection);
  connection.on('data', (buffer) => handler.handle(buffer));
}

...

There are three functions we need to implement
now: parse(buffer), requestComplete(),
and sendResponse.
parse(buffer)
This function will be responsible for progressively pulling out data
from the buffer. If the status line has not been received, it will try
to grab the status line. If the body has not yet started, it will
accumulate headers. Then it will continue accumulating the body until
we close the connection (this will happen implicitly when
requestComplete() returns true).
class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
    this.request = {
      statusLine: null,
      headers: {},
      body: null,
    };
  }

  parse(buffer) {
    const lines = buffer.toString().split(CRLF);

    // Parse/store status line if necessary
    if (!this.request.statusLine) {
      const [method, path, protocol] = lines.shift().split(' ');
      this.request.statusLine = { method, path, protocol };
    }

    // Parse/store headers if the body hasn't begun
    if (this.request.body === null) {
      for (let line = lines.shift(); lines.length; line = lines.shift()) {
        // Reached the end of headers, double CRLF
        if (line === '') {
          this.request.body = '';
          break;
        }

        const [key, value] = line.split(':');

        const safeKey = key.toLowerCase();
        if (!this.request.headers[safeKey]) {
          this.request.headers[safeKey] = [];
        }

        this.request.headers[safeKey].push(value.trimStart());
      }
    }

    this.request.body += lines.join(CRLF);
  }

...

}

requestComplete()
This function will look at the internal request state and return false
if the status line has not been received, no headers have been
received (although this is stricter than the HTTP/1.1 standard
requires), or if the body length is not equal to the value of the
Content-Length header.
class HTTPRequestHandler {

...

  requestComplete() {
    if (!this.request.statusLine || !Object.keys(this.request.headers).length || this.request.body === null) {
      return false;
    }

    const [contentLength] = this.request.headers['content-length'] || [];
    if (this.request.statusLine.method !== 'GET' && this.request.body.length !== contentLength) {
      return false;
    }

    return true;
  }

...

}

sendResponse()
Finally we'll hard-code two responses (one for the valid request for
/hello-world.html and a catch-all 404 response for every other
request). These responses need to be serialized according the HTTP
response format described above and written to the connection.
class HTTPRequestHandler {

...

  sendResponse() {
    const response = { status: 200, statusMessage: 'OK', body: '' };

    if (this.request.statusLine.path === '/hello-world.html') {
      response.body = HELLO_WORLD;
    } else {
      response.status = 404;
      response.statusMessage = 'NOT FOUND';
      response.body = NOT_FOUND;
    }

    const serialized = 'HTTP/1.1 ${response.status} ${response.statusMessage}' + CRLF +
                       'Content-Length: ' + response.body.length + CRLF + CRLF +
                       response.body;
    this.connection.write(serialized);
  }  

...

}

Run it
Now that we've got all the pieces we can finally run the initial program:
$ node uweb.js &
$ open localhost:9000/hello-world.html

And we see the page! Try any other path and we receive a 404.
Review and next steps
We covered the basics of HTTP/1.1: a very simple, plain-text protocol
oriented around requests and responses over a TCP/IP connection. We
realize we need to know little about anything but parsing and
formatting text on top of the TCP/IP blackbox called sockets. We
created a simple application that returns different responses based on
the request. But we're a far shot from a more general library, a web
framework. Future posts will explore this transition as well as
performance and more features.
Code is available on Github.
First post in a new series on web server basics starting with HTTP and sockets (using JavaScript/Node.js). https://t.co/uBiNfOBJeZ
— Phil Eaton (@phil_eaton) April 7, 2019
 


Media Clipping using ffmpeg and Cache Eviction 2 Random for Disk Caching at Scale
Ben E. C. Boyter — Thu, 04 Apr 2019 03:00:00 GMT
A short time ago one of the more interesting blog posts (to me anyway) about cache eviction popped up on Hacker News which prompted me to post the following comment.

Love reading this. It has always been one of those interesting things I kept in the back of my mind in my day to day.


I was very excited when I actually got to implement it on a real world project.


I was writing a scale out job which used ffmpeg to make clips of video files. To speed it up I kept the downloaded files (which could be 150 GB in size) as a local cache. Quite often a clip is made of the same file. When the disk was full (there was a separate disk for download and clip output) selected two of the downloaded files randomly and deleted the older one. Loop till there was enough disk space, or no files.


Bash : so long, and thanks for all the fish !
Linux Engineer's random thoughts — Sun, 31 Mar 2019 01:00:00 GMT
I recently took the plunge and decided to move to fish, a friendly modern shell.
Background
I'm heavily invested in bash. I wrote countless scripts, for my personal use, and for various jobs or projects, so I didn't think it would be an easy thing to move away from it …


OKRs aren't going to fix your communication issues
CRAIG KERSTIENS — Sat, 30 Mar 2019 22:55:56 GMT
Talking with a startup a few days ago they asked for my opinions on OKRs. I have slightly mixed opinions on them overall and started to disclose some of those. Though in sharing some of this I had a few immediate realizations that might be broadly applicable. The crux of his question was, at what stage should we put them in place. I’ve seen a few companies try to put in some form of OKR, and most were met with pretty mixed results. The reason is that OKRs need to change something about your behavior otherwise why put them in place… either change something about the goals you would otherwise have or the methods at which you went about achieving them.
Stepping back a bit, my first question and a very focusing question on almost any situation to ask is “What problem are we trying to solve?” In our conversation he actually paused a bit. As he paused a bit longer it was clear that question had not been fully asked or answered.
The first and most common case I see with startups trying to put in place OKRs, v2moms, management by objectives is that the team is not aligned and focused on the same goals. But my follow-on question is consistently, have you communicated what you decided you goals were.
Startups tend to go through some distinct growing phases. The early stages all the founders are in a room together building out the product. When you get the first few engineers you expand out a little, but still in a single co-working conference room easily. Eventually you need a real office. At the real office stage you start to have an all hands where, this is probably gathered around a large lunch table at first. At all hands no one takes meeting minutes and sends out a recap, instead people take some notes and you assume everyone was present.
But, at about 20 people you have at least one person that misses the weekly team meeting and misses something key. In a 1:1 you catch it that it was talked about as a priority… but they weren’t there. This very subtle change I’ve seen linger all the way up to a 70/80 person org. I’ve observed management meetings where someone missed and a key member was entirely mis-aligned on what the goals were for months following.
That was a long detour, but the point is that explicit formal communication is a big change for early stage companies. Distributed teams tend to do this better than in person teams, but it is also not guaranteed.
OKRs present a heavy-weight answer to the problem. OKRs tend to require hours and maybe even days to determine what are the right goals and metrics. Even if they are quick, does the process of OKRs change how you structure your team and work significantly for the next few weeks/months. If not, could you much more easily get away with… wait for it… emailing out what the company says priorities are. Email out the meeting notes from your all hands meeting. Email out (gasp) a recap of what you discussed and are thinking about as a management team. Sure every manager could go and have a 1:1 and recap the points for 30 minutes with each of their employees. Or you could use this thing that we’ve had for a little while… email.


Writing a simple JSON path parser
Notes on software development — Wed, 27 Mar 2019 02:00:00 GMT
Let's say we want to implement a simple list filtering language so
we can enter a.b = 12 and return only results in a
list where the a column is an object that contains a
field b that is set to the value 12. What would a
filter(jsonPath, equals, listOfObjects) function look
like?
If we only needed to support object lookup, we might
implement filter by splitting the jsonPath
on periods and look at each object in the listOfObjects
for matching values. It might look something like this:
function filter(jsonPath, equals, listOfObjects) {
  const parts = jsonPath.split('.');

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

require('assert').deepEqual(
  filter('foo.bar', 12, [{ foo: { bar: 12 } }, { foo: null }]),
  [{ foo: { bar: 12 } }],
);

That doesn't work too badly. We haven't handled edge cases like a
jsonPath of foo..bar or
bar.. But those would not be difficult to handle:
function filter(jsonPath, equals, listOfObjects) {
  if (jsonPath.charAt(0) === '.') {
    throw new Error('JSON path cannot begin with a dot, in: ' + jsonPath);
  } else if (jsonPath.charAt(jsonPath.length - 1) === '.') {
    throw new Error('JSON path cannot end with a dot, in: ' + jsonPath);
  }

  const parts = jsonPath.split('.');
  if (parts.reduce((hasEmptyPart, part) => hasEmptyPart || part.length === 0, false)) {
    throw new Error('JSON path cannot contain an empty section, in: ' + jsonPath);
  }

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

And we now handle the most obvious invalid path cases.
Arrays?
If we want to support array path syntax, things get harder. For
example:
require('assert').deepEqual(
  filter('foo.bar[0].biz', 14, [{ foo: { bar: [ { biz: 14 }, { biz: 19 } ] } }, { foo: { bar: null } }]),
  [{ foo: { bar: [ { biz: 14 }, { biz: 19 } ] } }],
);

We could try to stick with the hammer that is
String.prototype.split and write some really messy
code. :) Or we could switch to an approach that gives us more
control. Let's do that.
We'll build a very simple lexer that will iterate over each character
accumulating characters into individual tokens that represent the
pieces of the path. Let's start by supporting the original
jsonPath syntax and error-handling.
function getJsonPathParts(path) {
  const parts = [];
  let currentToken = '';

  for (let i = 0; i < path.length; i++) {
    const c = path[i];
    switch (c) {
      case '.': {
        if (!currentToken) {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }
        parts.push(currentToken);
        currentToken = '';
        break;
      }
      default: {
        currentToken += c;
        break;
      }
    }
  }

  if (!currentToken) {
    throw new Error('JSON path cannot end with dot, in: ' + path);
  }

  parts.push(currentToken);
  return parts;
}

function filter(jsonPath, equals, listOfObjects) {
  const parts = getJsonPathParts(jsonPath);

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

Not too bad!
Arrays?
Right. Let's build on getJsonPathParts to support array
syntax. Along with that we're going to impose some restrictions. The
object path parts must be only alphanumeric characters plus dashes and
underscores. The array index must only be numeric characters. Anything
else should throw an error.
function getJsonPathParts(path) {
  const parts = [];
  let currentToken = '';
  let inArray = false;

  for (let i = 0; i < path.length; i++) {
    const c = path[i];
    switch (c) {
      case '.': {
        if (currentToken === '') {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }

        parts.push(currentToken);
        currentToken = '';
        break;
      }
      case '[': {
        if (inArray) {
          throw new Error('JSON path contains unexpected left bracket, in: ' + path);
        }

        if (currentToken === '') {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }

        parts.push(currentToken);
        currentToken = '';
        inArray = true;
        break;
      }
      case ']': {
        if (!inArray) {
          throw new Error('JSON path contains unexpected right bracket, in: ' + path);
        }

        if (currentToken === '') {
          throw new Error('JSON path array index must not be empty, in: ' + path);
        }

        // Array indices are recorded as numbers, not strings.
        currentToken = parseInt(currentToken, 10);
        inArray = false;
        break;
      }
      default: {
        const code = c.charCodeAt(0);

        if (inArray) {
          if (code >= '0'.charCodeAt(0) && code <= '9'.charCodeAt(0)) {
            currentToken += c;
            continue;
          }

          throw new Error('JSON path array index must be numeric, in: ' + path);
        }

        if ((code >= 'A'.charCodeAt(0) && code <= 'z'.charCodeAt(0)) ||
            (code >= '0'.charCodeAt(0) && code <= '9'.charCodeAt(0)) ||
            ['-', '_'].includes(c)) {
          currentToken += c;
          continue;
        }

        throw new Error('JSON path part must contain only alphanumeric characters, in: ' + path);
      }
    }
  }

  if (currentToken === '') {
    throw new Error('JSON path cannot end with dot, in: ' + path);
  }

  parts.push(currentToken);
  return parts;
}

require('assert').deepEqual(getJsonPathParts('foo.bar[0].biz'), ['foo', 'bar', 0, 'biz']);

Now we've got a simple JSON path parser with decent error handling! Of
course we wouldn't want to use this little library in production until
we had some serious test coverage. But writing tests and calling out
my mistakes will be left here as an exercise for the reader. :)
New (short) post on parsing JSON paths in JavaScript https://t.co/mIjOMugA7C
— Phil Eaton (@phil_eaton) March 28, 2019
 


Kindle Hacks, A Self-guide
Nemo's Home — Tue, 26 Mar 2019 02:00:00 GMT
I run a non-standard Kindle configuration:


  Jailbroken (because I want to own the device, not rent it)
  Runs KOReader (because I want to read EPUBs and PDFs with reflow.)
  DRM Stripping (because I want to own the book, not rent it)


Since I don’t do any of these often enough to automate it, this is a self guide to help me follow these steps the next time I have to do any of this. No guarantees of this being helpful to anyone else but me.

Jailbreak

The lifehacker guide on how to jailbreak your kindle is a good starting point [archived]. The mobileread forums have the definitive guides. Also see this FAQ on the mobileread wiki.

(Most of these only cover modern paperwhite kindles)

Maintaining the Jailbreak

Sometimes, Kindle firmware updates will stop the Jailbreak. Search for your firmware on mobileread forums. See this link for the 5.8 series.

Copy the .bin file to your kindle root directory and trigger a manual firmware update. That should reboot and re-affirm the jailbreak. To trigger a manual firmware update, go to the Kindle Menu and click “Update”. If it is greyed out, check if the file was copied correctly, and try rebooting.

Applications

Once you have a jailbreak, the rest is mostly installing packages via MRPI. I keep a ready directory of packages I can copy as-is to my Kindle. The current listing is at https://paste.ubuntu.com/p/CXS5hYZdqc/ with most of it just being koreader.

koreader is a FOSS document viewer for E Ink devices that supports Kindle, Kobo, PocketBook, Ubuntu Touch and Android devices.

The primary 2 packages are:


  Update_KUALBooklet_v2.7_install.bin
  update_kpvbooklet_0.6.6_install.bin


Run ;log mrpi via search after copying them to re-install them if needed.

koreader

Download the latest release from GitHub.

You should download the kindle5-linux-gnueabi package for modern Paperwhites. Unzip it to the copy directory mentioned above.

Aside: koreader has a linux appimage version for desktops, which I package for AUR.

DRM Related Stuff

DRM is inherently bad for users. If I switch my Ebook reader from Kindle (which are great as of today) to
a Kobo tomorrow, I want my content to stay with me.

There are much better websites that explain the issues with DRM, so go visit: fckdrm.com, DefectiveByDesign.org, or EFF/drm.

The primary tool for stripping DRM from Kindle books is apprenticeharper’s DeDRM Repo which works as a Calibre Plugin. If you are running calibre with Python 3 (such as via the calibre-python3 package on Arch Linux) - you should install the DeDRM plugin from the python3 fork. Compress the DeDRM_plugin directory into a flat-zip file and use that in Calibre.

Getting the Key

My current key is saved in pass:

pass show Keys/Kindle.k4i |jq

Save it in a file, which you can import to Calibre.

If you don’t have the key or if the above isn’t valid, see this comment on r/ebooks [archived].

Importing the Key


  At the bottom-left of the plugin’s customization dialog, you will see a button labeled “Import Existing Keyfiles”. Use this button to import existing ‘.k4i’ key files. Key files might come from being exported from this plugin, or may have been generated using the kindlekey.pyw script running under Wine on Linux systems.


I once did some trickery on the kindlekey.pyw application to get it working on my system, but I didn’t take notes. If I ever do this again - AUTOMATE THIS.

Getting a copy of the encrypted book

There are multiple sources for you to try.


  Amazon website’s My Content page is the easiest. It doesn’t work for books with special typesetting - quite rare. Prefer this over everything else.
  Download via the Kindle for PC application (See next section).
  Get the KFX file from your Kindle device.
  Copy the KFX/AZW file from the Android/iOS application.


Kindle for PC

Stripping DRM for any medium is always a cat-and-mouse game. Amazon keeps changing the DRM format in every Kindle firmware update, which is why the recommended method is to use a known/older version of the Kindle for Mac/PC Application as your source.

Note: The 1.24.3 release does not work on Linux. If you’re on Linux, you must instead download the 1.17.0 release instead (sha256=14e0f0053f1276c0c7c446892dc170344f707fbfe99b6951762c120144163200).


  Install Kindle for PC. It does work on Wine. Make sure you download 1.24.3 (51068). I trust filehippo for this. The sha256sum for the installer is c7a1a93763d102bca0fed9c16799789ae18c3322b1b3bdfbe8c00422c32f83d7.
  Install then launch it, and download the book.
  Go to ~/Documents/My Kindle Content
  Find book by Last Modified Date.
  Run calibredb add book.azw. If all goes well, the book should show up in your library, and you should be able to convert it.




Reference Files

I have a backup of my current Kindle files at http://ge.tt/75zk4Dv2 in case you need any of the files mentioned above. Checksums for the files are below, since ge.tt doesn’t believe in HTTPS:

e3b05193ed9d0b482f01dfb550eba67f3b113b5165aae5632379cf35fec2f59d  copy.tar.gz
14e0f0053f1276c0c7c446892dc170344f707fbfe99b6951762c120144163200  KindleForPC-installer-1.17.44170.exe
c7a1a93763d102bca0fed9c16799789ae18c3322b1b3bdfbe8c00422c32f83d7  KindleForPC-installer-1.24.51068.exe
50bb0e5d9c03bcb79b17c1b7063cefd2c947a9d1c4392814e6ec05225296472a  kual-helper-0.5.N.zip
39352b4b68993680f06d5ecc57ce7ec4c271b6b5f2386ea998027420c45f2acd  KUAL-KDK-1.0.azw2
ceb207ee4c8d3674f308ff91432aeabf213b203571e270f70b8ae218df6ded7d  KUAL-KDK-2.0.azw2
fce02f0e104e846f1e4cc0e029500c5a722614d63a47035d78ea4cf59f67a448  kual-mrinstaller-1.6.N.zip
4a6de1fafe47ec0e3bfb529edead401c92e66b00697d507abe945679b3b7bc65  KUAL-v2.7.zip
253d0b00b31d62ef9dadb7ca88b98e2718cb35246816b3c50dd63c0a7ef28a52  Update_jailbreak_hotfix_1.14_5.8.10_install.bin
cc63ba1b454d1f32492c835f108ee04aaa80e6e7a95f12b7216c2c015daa2fbc  Update_jailbreak_hotfix_1.14_nomax_install.bin



Unsupervised Learning: Feature Transformation
swyx's site RSS Feed — Thu, 21 Mar 2019 09:21:12 GMT
Presenting the same information a different way... helps! Plus, one algorithm that does better than Principal Components Analysis!


Unsupervised Learning: Feature Selection
swyx's site RSS Feed — Thu, 21 Mar 2019 09:15:55 GMT
Breaking the Curse of Dimensionality!!


Unsupervised Learning: Clustering
swyx's site RSS Feed — Thu, 21 Mar 2019 09:12:22 GMT
Single Linkage, K-Means, Soft Clustering, and Kleinberg Impossibility


Write My Code For Me
swyx's site RSS Feed — Thu, 21 Mar 2019 02:00:00 GMT
My thoughts to self while weighing adding netlify functions:create to Netlify Dev. Originally a gist


Placeholder Horse Races
infrequent oscillations — Wed, 20 Mar 2019 08:15:09 GMT
More fake horse races as filler content.


Tips for your first tech conference
CRAIG KERSTIENS — Mon, 18 Mar 2019 22:55:56 GMT
I make it to a lot of conferences these days. I often see colleagues, former colleagues, and friends at these conferences. Sometimes it is friends I haven’t seen in a few years, sometimes I just saw the same person in a different country the week before. Conferences now are much easier for me, in fact it is a bit hard to recall what the experience was like when I first started attending, but I’m at least going to try to give some input so others can have a smoother first experience.


Most people know no one


There is a very strong chance you’re attending your first conference by yourself or just with a single colleague you know. Being suddenly surrounded by thousands of people you don’t know can be intimidating. The reality though is most folks there are just like you and know very few other people. Most people don’t go to multiple conferences in a year, they go to 1. There are often a lot of opportunities to get to know folks. Some of the best times are during breaks between the talk track, and whenever there is food/beverages. If there is breakfast served at the conference as much as you’d like to sleep in wake up and go have breakfast and chat with someone. At lunch if there is a half full table, go ask if a seat is taken and join the table vs. sitting at a completely open one. During the breaks don’t rush to get to the talk 10 minutes before it begins, stay and mingle with folks.
If you’re really feeling shy wonder the booth floor. Those folks are paid to be there to talk to you. Let them tell you about their product or service, it’ll at least be a start to some flowing conversation.


Speakers can be your friends too


If you’re new often the speakers can seem extra intimidating to get to know. Don’t let that be the case. If one is talking about a topic you’re excited about hang around after the talk and discuss it with them. Speakers aren’t necessarily any more technically advanced than you are they just happened to get a talk accepted.


Study the agenda


The default is to look at the list of talks being presented and pick out which ones you want to attend. But pay closer attention at so many conferences there are a bunch of other great activities to participate in. PyCon is a model example of this. In addition to all the talks there is:

PyLadies Auction - Which is a great night of dinner, drinks, and an auction of fun Python focused or other items which benefits PyLadies.
[5k run] - You wake up early in the morning and run (it should be noted that 5:30 am does not sound fun to me, but to each his own)
Evening dinners - If you’re worried about having dinner plans then this is a great option at a fun venue already coordinated.

If you’re not at something as structured as PyCon then there are a few places you can find what is happening. These days conferences often have a slack channel. Join it, find out when people get into town. Find out if people like to hike or visit breweries, join in with others. There is also twitter. Twitter things don’t get planned far in advance, but often you can find last minute folks that are coordinating things.


Stay at the conference hotel


If there is an official conference hotel or hotels, stay there. This opinion may be a little more controversial, but stay with me. First you’ll find folks hanging around the hotel bar after or just sitting around mingling. It’s a just one more way to not feel isolated. Second, it often helps the conference. Conferences can take on big risks with minimum rooms filled at a hotel. They do this to get good rates, but it also presents risk. Staying at the hotel both keeps you around others that are there for the conference, but also helps the conference itself.


You don’t have to attend all the talks


Talks are super helpful. You can learn a lot in a compressed period of time and then often get where to go next to learn further. Great ones teach you something but are also entertaining presentations as well. But, you don’t have to attend all the talks. Many conferences record the talks and make them available later. Personally I make a note of ones I wanted to see, as well as ones that others said were especially great. Then I go to youtube and watch them at 1.5 or 2x speed to get through more of them in less time.


Conclusion


Going to a conference may absolutely feel intimidating if it is your first. There will be a little of that first day of school jitters. What do you wear, who will you talk to, what if no one sits with you at lunch. Just know that for most people there it really is like the first day of school for them too. Even though it may be intimidating hopefully the above helps you jump in with a little less apprehension.


Regarding My Blogs
Ben Overmyer's Site — Mon, 18 Mar 2019 02:00:00 GMT
I’ve been frustrated lately with the writing side of Hugo. It’s a great tool for publishing and content transformation. It plays well with CI/CD tools. My static site host, Netlify, makes hosting sites built with Hugo a breeze. However, when it comes to being able to write whenever the mood strikes, Hugo is horrible. Yes, technically, I can just use GitHub’s web IDE to write a new post. I don’t have to use the git+vim workflow that I often do when writing Markdown. But it feels very… clunky. Inserting images is a slow and manual process. Layout requires messing with the site theme and its CSS, making it unwieldy for individual posts. Hence, all my posts on this blog look the same.
I decided to return to the GUI blog engine world. With that decision behind me, I started looking at the options. Having been out of that particular game for awhile, I hoped that some new tool had come along that I could check out. In fact, there was - a blog engine called Write Freely. So it entered the running alongside the usual suspects (Blogger, Ghost, and WordPress). Blogger hasn’t changed. It has some new themes since I last looked at it seriously ten years ago, but the interface has not aged well. Ghost charges a ridiculous amount of money for a hosted blog, and the self-hosted version still hasn’t acquired some of the quality-of-life features that WordPress has.
Write Freely is interesting, especially since it’s in the federation space, but it’s not yet mature enough for my tastes. This, despite the fact that I contributed code to it. So that leaves WordPress. And rather than deal with hosting it myself, I decided to pay for hosting on wordpress.com. It’s not ideal (the Personal version doesn’t allow installing plugins), but it works well enough and gives me the GUI features I missed. At time of this writing, I’m still trying to figure out how to migrate all of my Hugo posts to WordPress. No tool exists for that purpose. I’ll probably have to write something myself. Update = I did end up writing a tool to handle this. I’ll write a blog post about it later.


How to remove all bars in fullscreen mode in Firefox on MacOS/OSX  ("kiosk mode")
Subclassed — Thu, 14 Mar 2019 14:32:28 GMT
Procedure
If you want to remove all toolbars from Firefox in fullscreen mode on Mac, this is the procedure:

Type about:config: in the location bar to open advanced settings. (If it is the first time you do it, you might have to click "confirm" or something like that in the scary screen that comes up first.)
search for full-screen-api.allow-trusted-requests-only and double-click the line that is found, so the value changes to false
Bring up the Bookmark Toolbar if you don't have it (menu View -> Toolbars -> Bookmark Toolbar)
Right-click on empty space in the Bookmark Toolbar, and select New Bookmark...
In the Name field, type something like "Enter Fullscreen" (it doesn't really matter, just don't leave it blank)
In the Location field, enter the following line:
javascript:document.getElementsByTagName('html')[0].mozRequestFullScreen();void(0)

Click Add to save the bookmark.


Now, when you click on that bookmark, Firefox will go "real fullscreen", hiding all toolbars. To get them back, you will have to press ESC.

Now, there is a bit of a snag. That preference in about:config: is set to true by default for a reason: it is a bit unsafe to allow any website to use fullscreen-enabling javascript. For this reason I would suggest to only do this for limited periods, then switch it back once you're done.

Long-winded explanation

FF has a preference, browser.fullscreen.autohide, that in theory should govern the behaviour of toolbars when going fullscreen. It is set to true by default, meaning toolbars should disappear. However, for some reason this property has (almost) always failed to work on MacOS/OSX builds of Firefox.
There used to be a few extensions that corrected that behaviour, but they've all gone away when Firefox switched to WebExtension APIs, which basically sandboxed extensions more rigidly and blocked them from changing the browser interface as deeply as they could before. This was the price for the dramatic performance improvement seen in Firefox 57+.
This workaround hence relies on the new Fullscreen WebAPI, which is supposed to be a new, standard way to enable fullscreen via Javascript on any browser (well, any browser that implements it - just a few at the moment). However, because of security concerns, this API is locked down by default; so, in order to use it, we first have to relax checks as described above. Once that is done, we can launch a bit of JS from a bookmark that triggers "real" fullscreen.
Credit to this AskDifferent post where I first found the trick.


My Personal Complaints about Programming in Go
Ben E. C. Boyter — Thu, 14 Mar 2019 02:00:00 GMT
Go as a language is fairly decent. However because questions about why I have issues with tends to come up often enough on the company slack programming channel (see what I did there?) I figured I would write them down and put it here so I can point people at a link when they ask what my complaints are.

For the record I have been using Go heavily for the last year or so, writing command line applications, scc, lc and API’s. These include large scale API’s for clients to a syntax highlighter that will be used in https://searchcode.com/ sometime soon.


Give me back my monolith
CRAIG KERSTIENS — Wed, 13 Mar 2019 22:55:56 GMT
It feels like we’re starting to pass the peak of the hype cycle of microservices. It’s no longer multiple times a week we now see a blog post of “How I migrated my monolith to 150 services”. Now I often hear a bit more of the counter: “I don’t hate my monolith, I just care that things stay performant”. We’ve actually seen some migrations from micro-services back to a monolith. When you go from one large application to multiple smaller services there are a number of new things you have to tackle, here is a rundown of all the things that were simple that you now get to re-visit:


Setup went from intro chem to quantum mechanics


Setting up a basic database and my application with a background process was a pretty defined process. I’d have the readme on Github, and often in an hour or maybe a few I’d be up and running when I started on a new project. Onboarding a new engineering, at least for an initial environment would be done in the first day. As we ventured into micro-services onboarding time skyrocketed. Yes, we have docker and orchestration such as K8s these days to help, but the time from start to up and running a K8s cluster just to onboard a new engineer is orders of magnitude larger than we saw a few years ago. For many junior engineers this is a burden that really is unnecessary complexity.


So long for understanding our systems


Lets stay on the junior engineer perspective for just a moment. Back when we had monolithic apps if you had an error you had a clear stacktrace to see where it originated from and could jump right in and debug. Now we have a service that talks to another service, that queues something on a message bus, that another service processes, and then we have an error. We have to piece together all of these pieces to eventually learn that service a was on version 11 and service q was expecting vesion 12 already. This in contrast to my standard consolidated log, and lets not forget my interactive terminal/debugger for when I wanted to go step by step through the process. Debugging and understanding is now inherintly more complicated.


If we can’t debug them, maybe we can test them


Continuous integration and continuous development is now starting to become common place. Most new apps I see now days automatically build and run their tests with a new PR and require tests to pass and review before check-in. These are great processes to have in place and have been a big shift for a lot of companies. But now to really test my service I have to bring up a complete working version of my application. Remember back to onboarding that new engineer with their 150 service K8s cluster? Well now we get to teach our CI system how to bring up all those systems to actually test that things are working. That is probably a bit too much effort so we’re just going to test each piece in isolation, I’m sure our specs were good enough that APIs are clean and service failure is isolated and won’t impact others.


All the trade-offs are for a good reason. Right?


There are a lot of reasons to migrate to micro-services. I’ve heard cases for more agility, for scaling your teams, for performance, to give you a more resilient service. The reality we’ve invested decades into development practices and tooling around monoliths that are still maturing. In my day to day I work with a lot of folks from all different stacks. Usually we’re talking about scaling because they’re running into limits of a single node Postgres database. Most of our conversation focuses on scaling the database.
But in all the conversations I’m fascinated to learn about their architecture. Where are they in their journey to micro-services. It has been an interesting trend to see more and more reactions “We’re happy with our monolithic app.” The road to micro-services may work fine for lots, and the benefits may outweigh the bumpy road to get there, but personally give me my monolithic app and a beach somewhere and I’ll be happy.


Using Que instead of Sidekiq
Pete Keen — Wed, 13 Mar 2019 02:00:00 GMT
A project I've had on the back burner for quite awhile is my own little marketing automation tool.
Not that existing tools like Drip or ConvertKit aren't adequate, of course.
They do the job and do it well.
I enjoy owning my own infrastructure, however, and after Drip changed direction and raised prices I found myself without a home for my mailing list.
I thought, why not now?

One vital component of any broadcast email system is fanout, where you merge the message you want to send with the list of people that should receive it.
The easiest way to fanout is to just loop over the list of recipients and enqueue a job for each:
Contact.not_opted_out.each do |contact|
  BroadcastMessageDeliver.perform_async(contact.id, the_message.id)
end
This is simple and works great. However, it's not super efficient. We can do better.
If we're using Sidekiq we can use push_bulk:
Contact.not_opted_out.find_in_batches do |batch|
  Sidekiq::Client.push_bulk(
    class: 'BroadcastMessageDeliver', 
    args: batch.map { |c| [c.id, the_message.id] }
  )
end
The find_in_batches call is a built-in ActiveRecord method that will give you all of the records in the scope in batches, which is just an array of ActiveRecord objects.
Sidekiq::Client.push_bulk eliminates the vast majority of Redis round trips that the simple version does because it pushes the whole batch in one Redis call.
We can still do better, though. Instead of using Sidekiq we can use Que.
Que is a background processing system like Sidekiq that keeps jobs in a PostgreSQL table instead of in a Redis list.
It uses PostgreSQL's native listen/notify system to make job starts basically instantaneous, rather than polling like what DelayedJob does.
Using the database as the queue has a number of advantages over systems that use two data stores. In particular, ACID guarantees and atomic backups are important to me because I'm running this all myself. The fewer moving parts the better.
The other thing you can do is insert directly into the que_jobs table:
ActiveRecord::Base.connection.execute(%Q{
  INSERT INTO que_jobs (job_class, args)
  SELECT
    'BroadcastMessageDeliver' as job_class,
    jsonb_build_array(#{the_message.id}, x.id) as args
  FROM
    (#{Contact.not_opted_out.select(:id).to_sql}) x
})
The que_jobs table is just a database table, which means you can insert into it however you want.
For example, Que::Job.enqueue just creates a record and saves it, it doesn't use any ActiveRecord hooks at all.
We can eliminate almost every round trip and application-level loop by letting the database do all the work.
Benchmarks (local Redis and local PostgreSQL, 5000 records):

Sidekiq loop: 1.9 seconds
Sidekiq batches: 0.3 seconds
Que direct insert: 0.7 seconds

Wait... that's... slower?
I'm as surprised as you are, but there turns out to be a pretty good reason.
Que performs a bunch of check constraints on the incoming data to make sure it's coherent and ready to run. Here's all the things it checks:
Check constraints:
    "error_length"
    "job_class_length"
    "queue_length"
    "valid_args"
    "valid_data"

valid_data in particular does a handful of expensive-ish operations on the incoming json data.
So I guess the lesson here is to always validate your assumptions.
I assumed that eliminating round trips would make things faster but because of other constraints and validations it's actually slower.
Still, it's considerably faster than the simple version (which is still no slouch, let's be honest), my marketing system gets all those in-database queue benefits, and I find it aesthetically pleasing. I think I'll keep it.


Why I love building developer products
CRAIG KERSTIENS — Tue, 12 Mar 2019 22:55:56 GMT
For much of my career I’ve been focused on building out developer or data focused products with the customer in some form or fashion being a developer on the other end. I fully realize now that I’m destined to spend the rest of my career in that space, either that or trying my hand at wine making. There are a few things that I personally find rewarding about the space that I’ve shared with a number of people individually lately and thought I would share more broadly.
First, software really is eating the world. Often developers and entrepeurs ask about what they could build that would be a good business. The reality is about anything that has not been modernized to as a service and improved with software could be. We have far less developers in the world than we need to execute on all the ways we could improve products and life. To me what is interesting is the last part of that last sentence. It is not that the market for developers is huge, which I do believe it is. It more that when we automate with systems we can get amazing economies of scale. I know folks that reminisce and talk about how we’re more stressed being always connected and such and that in the old days people got out and worked the fields and enjoyed the sun. They also absolutely physically exhausted their bodies in the process. The ability to make life better at scale is an interesting one and often done through systems we develop.
The second reason is the challenge and the reward. Developers are notoriously tough critics. If it feels/smells like marketing then they have an allergic reaction, and that is probably because much of it is done poorly. If they experience really positive/great marketing they latch on more than the average person. Most developers are by nature a bit skeptical… for some reason this resonates with me. But, I’ve found they are the largest/biggest supporters once they’re excited about something. Personally I’d rather folks more critical and then have a few diehard fans than a whole bunch of people I can try to some site with ads and convert them, but then never actually connect with my product/brand.
The third and final reason is, I can understand it. Having worked at small startups, side projects, and at large enterprise I can understand the challenges as each different level. Building a product that I can directly relate to and connect with is some how easier to stay motivated. I’m sure instagram has a lot of interesting technical challenges, or that scaling an ad network is extremely challenging. But building a product that I could see myself directly benefiting from makes everyone’s input on the team valuable and key. This changes the entire dynamic from product team to engineering to marketing.
Now days I dont get much chance to ship code to production, usually that is what I threaten when things are running behind, but at heart I’m still very much an engineer. Building products that make developers lives better and more productive may not change the world, but hopefully it better enables other developers to.


Hacker News Embeddings with PyTorch
Bytepawn - Marton Trencseni — Tue, 12 Mar 2019 01:00:00 GMT
A PyTorch model is trained on public Hacker News data, embedding posts and comments into a high-dimensional vector space, using the mean squared error (MSE) of dot products as the loss function. The resulting model is reasonably good at finding similar posts and recommending posts for users.




Getting Closure on React Hooks
swyx's site RSS Feed — Mon, 11 Mar 2019 02:00:00 GMT
Learn React Hooks by building a clone of React - in 30 minutes!


The Statistical Impossibility of Dropped Points in the Mind of a Football Manager
infrequent oscillations — Sun, 10 Mar 2019 17:52:32 GMT
Who avoided dropping *any* points after 30 games?


Writing a lisp compiler from scratch in JavaScript: 3. LLVM
Notes on software development — Sun, 10 Mar 2019 02:00:00 GMT

  Previously in compiler basics:
  <! forgive me, for I have sinned >
  

  1. lisp to assembly
  

  2. user-defined functions and variables
  

  

  Next in compiler basics:
  

  4. LLVM conditionals and compiling fibonacci
  

  5. LLVM system calls
  

  6. an x86 upgrade
In this post we'll extend the
compiler to emit LLVM
IR as an option instead of x86
assembly.
All source code is available on Github.
LLVM IR is a portable, human-readable, typed, assembly-like syntax
that LLVM can apply optimizations
on before generating assembly for the target architecture. Many
language implementors choose to compile to LLVM IR specifically to
avoid needing to implement sophisticated optimizations.
But the biggest reason I'm adding an LLVM backend is so that I can
punt on implementing register
allocation. This
is the technique that allows you to generically use as many registers
as possible before storing local variables on the stack. While
register allocation algorithms are not that difficult, I got
bored/lazy trying to implement this for ulisp. And LLVM IR provides
"infinite" locals that get mapped as needed to registers and the stack
-- implementing register allocation.
LLVM IR basics
In LLVM IR, all local variables must be prefixed
with %. All global variables (including function names)
must be prefixed with @. LLVM IR must be in
single-static
assignment
(SSA) form, which means that no variable is assigned
twice. Additionally, literals cannot be assigned to variables
directly. So we'll work around that by adding 0 to the
literal. Furthermore, we'll take advantage of
the add, sub, and mul
operations built into LLVM IR.
; x = 4
%x = add i32 4, 0

The type that the operation is operating on must be specified after
the operation name. In this case we are specifying
that add is operating on and returning 32-bit integers.
While this might seem very inefficient, we'll see in the end that
LLVM easily optimizes this away.
Function definition
Functions are defined at the top-level and are much simpler than x86
assembly since the details of calling conventions are handled by LLVM.
; (def plus (a b) (+ a b))
define i32 @plus (i32 a, i32 b) {
  %res = add i32 a, b
  ret i32 %res
}

In ulisp, all functions will return a result (and the only supported
type for now are 32-bit integers). So we annotate the definition with
this return type (i32 in define
i32). Finally, we return inside the function with
the ret instruction that must also specify the type
(again i32).
Generating LLVM IR
We are going to generate LLVM IR as text. But any large project will
benefit from generating LLVM IR via
API.
Supporting multiple backends
The goal is to be able to switch at compile-time between generating
x86 assembly or generating LLVM IR. So we'll need to reorganize ulisp
a little bit.
We'll edit src/ulisp.js to accept a second argument to
specify the backend (and from now on we'll default to LLVM).
const cp = require('child_process');
const fs = require('fs');

const { parse } = require('./parser');
const backends = require('./backend');

function main(args) {
  const input = fs.readFileSync(args[2]).toString();

  let backend;
  switch (args[3]) {
    case 'llvm':
    case undefined:
      backend = backends.llvm;
      break;
    case 'x86':
      backend = backends.x86;
      break;
    default:
      console.log('Unsupported backend ' + args[3]);
  }

  const [ast] = parse(input);
  const program = backend.compile(ast);

  try {
    fs.mkdirSync('build');
  } catch (e) {}
  backend.build('build', program);
}

main(process.argv);

The LLVM backend
We'll add src/backend/llvm.js and expose
compile and build functions.
compile(ast)
This will work the same as it did for the x86 backend, creating a new
Compiler helper object, creating a scope manager (which
we'll get into in more detail shortly), and generating code from the
AST wrapped in a begin.
module.exports.compile = function(ast) {
  const c = new Compiler();
  const scope = new Scope();
  c.compileBegin(ast, scope.symbol(), scope);
  return c.getOutput();
};

build(buildDir, output)
The job of build will be to clean up the build directory,
write any output as needed to the directory, and compile the written
output. Since we're dealing with LLVM IR, we first call
llc on the IR file to
get an assembly file. Then we can call gcc on the
assembly to get a binary output.
const cp = require('child_process');
const fs = require('fs');

...

module.exports.build = function(buildDir, program) {
  const prog = 'prog';
  fs.writeFileSync(buildDir + `/${prog}.ll`, program);
  cp.execSync(`llc -o ${buildDir}/${prog}.s ${buildDir}/${prog}.ll`);
  cp.execSync(`gcc -o ${buildDir}/${prog} ${buildDir}/${prog}.s`);
};

Taking advantage of locals
Before we get too far into the specifics of LLVM IR code generation,
let's build out the infrastructure to take advantage of "infinite"
locals. In particular, we want a local-manager (Scope)
with four functions:

register(local: name): for tracking user variables and mapping to safe names
symbol(): for tracking internal temporary variables
get(local: name): for returning the safe name of a user variable
copy(): for duplicating the local-tracker when we enter a new scope

It is important to track and map user variables into safe names so we
don't accidentally conflict between variable names used by the user
and names used by the compiler itself.
register(local)
When we register, we'll want to replace any unsafe characters that
Lisp allows but LLVM likely won't. For now, we'll just replace any
dashes in the name (since dashes are fine in variables in Lisp) with
underscores. Then we'll add a number to the end of the local name
until we have a safe name that doesn't exist already. Finally we
return that safe name after storing the mapping.
class Scope {
  constructor() {
    this.locals = {};
  }

  register(local) {
    let copy = local.replace('-', '_');
    let n = 1;
    while (this.locals[copy]) {
      copy = local + n++;
    }

    this.locals[local] = copy;
    return copy;
  }
}

symbol()
This is a simple function that will return one new unused safe name
that we can store things in.
class Scope {
  ...

  symbol() {
    const nth = Object.keys(this.locals).length + 1;
    return this.register('sym' + nth);
  }

  ...
}

We start off by making up a name based on the prefix sym
and a suffix of the current key length and pass that into the
register function to make sure we get a safe name.
get(local)
This function is a very simple lookup to return the safe name for a
user variable. It is up to the caller of this function to handle if
the user variable does not exist in scope (and perhaps throw a
compiler error back to the programmer).
class Scope {
  ...

  get(local) {
    return this.locals[local];
  }

  ...
}

copy()
Finally, we want to expose a copy function so we can duplicate the
local storage before entering a new scope. (A variable inside a
function should not exist in scope outside the function.)
class Scope {
  ...

  copy() {
    const c = new Scope();
    c.locals = { ...this.locals };
    return c;
  }

  ...
}

Back to codegen!
As mentioned in module.exports.compile, we're going to
use a Compiler that exposes a number of compiler helpers:

emit(depth, code): an internal helper for outputting indented lines of code
compileBegin(ast, destination, scope): compiles a begin block
compileExpression(ast, destination, scope): compiles variable references, literals, and passes on function calls
compileCall(functionName, ast, destination, scope): compiles a function call
compileDefine([functionName, parameters, ...body], destination, scope): compiles a function definition
compileOp(op): helper function for generating code for primitive operations like add
getOutput(): returns the code generated by the compiler

emit(depth, code)
Like we had in the x86 backend, this will indent the code two spaces
depth times and write it to the buffer we track generated
code.
class Compiler {
  constructor() {
    this.outBuffer = [];
  }

  emit(depth, code) {
    const indent = new Array(depth + 1).join('  ');
    this.outBuffer.push(indent + code);
  }
}

compileBegin(ast, destination, scope)
Our first compiler function actually does no code generation
itself. We'll call compileExpression on each item within
the begin block. And we'll pass the destination to the
last expression in the list so that the value of a begin block is set
to the value of its last expression. All other expressions will
receive a temporary variable to store results.
class Compiler {
  ...

  compileBegin(body, destination, scope) {
    body.forEach((expression, i) =>
      this.compileExpression(
        expression,
        i === body.length - 1 ? destination : scope.symbol(),
        scope,
      ),
    );
  }

  ...
}

Example:
(begin 1 2) ; returns 2

compileExpression(ast, destination, scope)
This is the most generic compile function. If the ast is a list
(representing a function call), it will pass compilation off to
compileCall. Otherwise the only non-function call parts
of the language are variable references and numeric literals.
class Compiler {
  ...

  compileExpression(exp, destination, scope) {
    // Is a nested function call, compile it
    if (Array.isArray(exp)) {
      this.compileCall(exp[0], exp.slice(1), destination, scope);
      return;
    }

    // If numeric literal, store to destination register by adding 0.
    if (Number.isInteger(exp)) {
      this.emit(1, `%${destination} = add i32 ${exp}, 0`);
      return;
    }

    // If is local, store to destination register similarly.
    const res = scope.get(exp);
    if (res) {
      this.emit(1, `%${destination} = add i32 %${res}, 0`);
    } else {
      throw new Error(
        'Attempt to reference undefined variable or unsupported literal: ' +
          exp,
      );
  }

  ...
}

Example:
1
...
a
...
(+ 1 a)

compileCall(functionName, arguments, destination, scope)
Most function calls will automatically compile arguments before
calling the function. However, certain control-flow primitives don't
do this (e.g. def, if, etc.). Macros in Lisp
allow you to add new control-flow primitives (even if you don't use it
to modify control-flow). But we will ignore user-defined primitives
for now.
We'll keep a list of control-flow primitives and pass off compilation
to them if the function name matches a primitive. Otherwise, we'll
look up the function name in scope (to find its safe name), compile
the arguments, and call the function with the results of the
arguments.
class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
    };
  }

  ...

  compileCall(fun, args, destination, scope) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, destination, scope);
      return;
    }

    const validFunction = scope.get(fun);
    if (validFunction) {
      const safeArgs = args
        .map((a) => {
          const res = scope.symbol();
          this.compileExpression(a, res, scope);
          return 'i32 %' + res;
        })
        .join(', ');
      this.emit(1, `%${destination} = call i32 @${validFunction}(${safeArgs})`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }
  }

  ...
}

Yay LLVM for simplifying calls!
Example:
(foo 1)
...
(+ 1 2)

compileDefine([functionName, parameters, ...body], destination, scope)
This is the last undefined compile function we've used. The call
signature may look funny but we write less code if we keep the
primitive signatures the same. In any case, JavaScript's destructuring
makes it pretty enough.
Aside from code generation, we also need to add the function itself to
scope so we can look it up later in use. Additionally we need to
create a copy of the current scope for the body of the function. And
we'll add the parameter names themselves to the child scope.
class Compiler {
  ...

  compileDefine([name, params, ...body], destination, scope) {
    // Add this function to outer scope
    const safeName = scope.register(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    const safeParams = params.map((param) =>
      // Store parameter mapped to associated local
      childScope.register(param),
    );

    this.emit(
      0,
      `define i32 @${safeName}(${safeParams
        .map((p) => `i32 %${p}`)
        .join(', ')}) {`,
    );

    // Pass childScope in for reference when body is compiled.
    const ret = childScope.symbol();
    this.compileExpression(body[0], ret, childScope);

    this.emit(1, `ret i32 %${ret}`);
    this.emit(0, '}\n');
  }

  ...
}

Example:
(def plus (a b) (+ a b))

compileOp(op)
The last function mentioned above will help us expose some useful
primitives.  This function will take a string builtin operation and
return a function that can be used to generate code when the operation
is called.
class Compiler {
  ...

  compileOp(op) {
    return ([a, b], destination, scope) => {
      const arg1 = scope.symbol();
      const arg2 = scope.symbol();
      this.compileExpression(a, arg1, scope);
      this.compileExpression(b, arg2, scope);
      this.emit(1, `%${destination} = ${op} i32 %${arg1}, %${arg2}`);
    };
  }

  ...
}

This allows us to add some builtin ops as primitives (even though they
aren't control-flow modifying).
class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
    };
  }

  ...
}

Example:
(+ 1 2)

Hello world!
Putting it all together, we'll compile this Lisp program:
(def plus-two (a b)
     (+ a (+ b 2)))

(def main ()
     (plus-two 3 (plus-two 1 1)))

To get 9.
$ node src/ulisp.js tests/function_definition.lisp
$ ./build/prog
$ echo $?
9

Generated code
The generated LLVM can be found in ./build/prog.ll:
define i32 @plus_two(i32 %a, i32 %b) {
  %sym7 = add i32 %a, 0
  %sym9 = add i32 %b, 0
  %sym10 = add i32 2, 0
  %sym8 = add i32 %sym9, %sym10
  %sym6 = add i32 %sym7, %sym8
  ret i32 %sym6
}

define i32 @main() {
  %sym6 = add i32 3, 0
  %sym8 = add i32 1, 0
  %sym9 = add i32 1, 0
  %sym7 = call i32 @plus_two(i32 %sym8, i32 %sym9)
  %sym5 = call i32 @plus_two(i32 %sym6, i32 %sym7)
  ret i32 %sym5
}

You can see all these unnecessary add, ... 0
instructions. But let's look at the x86 assembly that LLVM generates
in build/prog.s:
...

_plus_two:                              ## @plus_two
        .cfi_startproc
## %bb.0:
                                        ## kill: def $esi killed $esi def $rsi
                                        ## kill: def $edi killed $edi def $rdi
        leal    2(%rdi,%rsi), %eax
        retq
        .cfi_endproc
                                        ## -- End function
...

And we see that LLVM easily optimized the inefficiencies away. :)
Next up

Compiling conditionals
Tail call optimization
Lists and dynamic memory
Strings?
Foreign function calls?
Self-hosting?

Adding an LLVM backend to ulisp (small Lisp compiler in JavaScript) https://t.co/VIddKW1r3N
— Phil Eaton (@phil_eaton) March 10, 2019
 


Trot Trot Machine Trotting
infrequent oscillations — Thu, 07 Mar 2019 13:08:25 GMT
In which we enter the CHALLENGE STAKES CUP PACE TROT


Getting back into MMORPGs
Ben Overmyer's Site — Wed, 06 Mar 2019 02:00:00 GMT
As a result of my one-year boycott of Activision-Blizzard, I found myself without any of my usual games to play. I decided to take the opportunity to return to a genre I haven’t played much of in the past few years - MMORPGs.
Aside from a month where I played World of Warcraft following the new expansion, and about three weeks playing Final Fantasy XIV after that, I haven’t played MMOs much in the last three years. With Pantheon: Rise of the Fallen still a long way off, Crowfall still in alpha, and Camelot Unchained in… whatever state it’s in, I decided to have a look at games I’ve played in the past for something to occupy my attention.
I settled on four options - Elder Scrolls Online, Black Desert Online, Project Gorgon, and Guild Wars 2. ArenaNet’s layoffs killed my enthusiasm for Guild Wars 2, so that dropped off the list. Black Desert Online had some strange bugs with keyboard controls (like erasing all my key binds) that left a sour taste in my mouth. Some friends of mine decided to try playing ESO together, so I decided to join them. But, I also have renewed interest in Project Gorgon because of its influx of Pantheon fans, so I’ve decided to alternate between those two.
I never got very far in either game. In Elder Scrolls Online, my highest character was level eight. My character in Project Gorgon still hasn’t left the starter town. Of the two, ESO seems to be holding my attention better. The story is pulling me along. That got me to thinking… what exactly do I want out of an MMORPG?
These days twitch shooters usually are my go-to. But with most people obsessed with battle arena games (which I hate) and Overwatch and Destiny 2 in my boycott, there isn’t much to play that interests me. So I’ve decided to take a different angle. What if I actually relax with video games, instead of get a drip-feed of adrenaline? So that’s what I’m trying to do with ESO and Project Gorgon: relax. Sit back and enjoy a slower pace.
So far it seems to be working.
I may write in the future about my adventures as a wood elf nightblade in Elder Scrolls Online. I’m taking to the character’s playstyle pretty well. For now, I’ll finish with this thought: when’s the last time you relaxed with a video game?


How can I help? East coast vs. West coast mentalities
CRAIG KERSTIENS — Sun, 03 Mar 2019 22:55:56 GMT
Often times when I’m traveling on the east coast, whether it is NYC area or back home down south I try to spend some time to catch up with various people. In catching up we’ll spend some time talking about what we’re both up to, thoughts on tech or in general, and at the end I typically ask “Is there anything in particular I can help with?” More often than not the answer to this question isn’t super substantial, which is fine. But what is surprising is the stark contrast on reactions to this question and how it differs from west coast vs. east coast.
On the east coast when I ask “How can I help?” I get a look of:

What’s the catch?
Okay, what is it you want me to do for you?
There is no way you actually intend to help…

Meanwhile on the west coast the mentality is generally quite different. I often have people showing their own willingness to help. It may be someone you know personally that is offering help, but have also seen where someone gets asked for input or help, refers someone else and now you have two people that have known each other for under 5 minutes and one willing to dispense helpful advice or assistance.
The how I can help mentality of the west coast seems to follow that offering assistance is not a zero sum game, and by extending some effort now it hopefully comes back around when they need it themselves.


rxe: literate and composable regular expressions
Bytepawn - Marton Trencseni — Sat, 02 Mar 2019 01:00:00 GMT
rxe is a thin wrapper around Python's re module. The various rxe functions are wrappers around corresponding re patterns. For example, rxe.digit().one_or_more('a').whitespace() corresponds to \da+\s. Because rxe uses parentheses but wants to avoid unnamed groups, the internal (equivalent) representation is actually \d(?:a)+\s. This pattern can always be retrieved with get_pattern().




Mouldeth Thy Lord
infrequent oscillations — Fri, 01 Mar 2019 23:18:27 GMT
Moulding a tribute to Our Lord Helix for Helix Day


AOT-compilation of Javascript with V8
Notes on software development — Tue, 26 Feb 2019 02:00:00 GMT
tldr; I'm working on a AOT-compiled Javascript implementation called
jsc.
Many dynamically typed programming languages have implementations that
compile to native binaries:

Python: Cython
Common Lisp: SBCL
Scheme: Chicken Scheme

The benefits of compiling dynamically typed languages are similar to
those of compiling statically typed languages:

Simplified deployment via a single binary
Simplified foreign-function interfaces
e.g. Embedded C/C++ strings


Predictable performance compared to JIT compiling interpreters
Performance gains compared to non-JIT compiling interpreters

I (re)discovered a common technique for compiling dynamic languages
while developing BSDScheme,
an interpreter and compiler for Scheme. In this technique, you use
core parts of the runtime code as a library that is imported and
referenced by compiled code.
You save time building object-memory representations, memory
management, operations, interacting with existing libraries, etc. when
an interpreter already exists. The runtime as a library (plus existing
parser frontends) allows you to focus solely on code generation of
control flow.
The first pass
I wrote the initial version of jsc
in Rust using Dave Herman's
esprit parser (supports a subset
of ES6 that includes all of ES5).
The interesting parts of the runtime are taken care of by V8, e.g.:

V8::String - a Javascript string object
V8::String::NewFromUtf8(isolate, "hello world!") - C++ string to Javascript string object


V8::Number - a Javascript number object
V8::Number::New(isolate, 10) - C++ double to Javascript number object


Heap allocations
Calling convention

And so on.
An example
This first version of jsc could take the following Javascript:
function fib(n, a, b) {
    if (n == 0) {
        return a;
    }

    if (n == 1) {
        return b;
    }

    return fib(n - 1, b, a + b);
}

function main() {
  console.log(fib(50, 0, 1));
}

And produce the following C++:
#include <string>
#include <iostream>

#include <node.h>

using v8::Array;
using v8::Boolean;
using v8::Context;
using v8::Exception;
using v8::Function;
using v8::FunctionTemplate;
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Null;
using v8::Number;
using v8::Object;
using v8::String;
using v8::False;
using v8::True;
using v8::Value;

void fib_0(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
  Local<Value> n_1 = args[0];
  Local<Value> a_2 = args[1];
  Local<Value> b_3 = args[2];
tail_recurse_4:

  Local<Context> ctx_5 = isolate->GetCurrentContext();
  Local<Object> global_6 = ctx_5->Global();
  Local<Function> Boolean_7 = Local<Function>::Cast(global_6->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_8(n_1);
  std::string string_tmp_9(*utf8value_tmp_8);
  String::Utf8Value utf8value_tmp_10(Number::New(isolate, 0));
  std::string string_tmp_11(*utf8value_tmp_10);
  Local<Value> argv_12[] = { (n_1->IsBoolean() || Number::New(isolate, 0)->IsBoolean()) ? Boolean::New(isolate, n_1->ToBoolean(isolate)->Value() == Number::New(isolate, 0)->ToBoolean(isolate)->Value()) : ((n_1->IsNumber() || Number::New(isolate, 0)->IsNumber()) ? Boolean::New(isolate, n_1->ToNumber(isolate)->Value() == Number::New(isolate, 0)->ToNumber(isolate)->Value()) : ((n_1->IsString() || Number::New(isolate, 0)->IsString()) ? Boolean::New(isolate, string_tmp_9 == string_tmp_11) : (False(isolate)))) };
  Local<Value> result_13 = Boolean_7->Call(Null(isolate), 1, argv_12);
  if (result_13->ToBoolean()->Value()) {
    // return a;
    args.GetReturnValue().Set(a_2);
    return;


  }

  Local<Context> ctx_14 = isolate->GetCurrentContext();
  Local<Object> global_15 = ctx_14->Global();
  Local<Function> Boolean_16 = Local<Function>::Cast(global_15->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_17(n_1);
  std::string string_tmp_18(*utf8value_tmp_17);
  String::Utf8Value utf8value_tmp_19(Number::New(isolate, 1));
  std::string string_tmp_20(*utf8value_tmp_19);
  Local<Value> argv_21[] = { (n_1->IsBoolean() || Number::New(isolate, 1)->IsBoolean()) ? Boolean::New(isolate, n_1->ToBoolean(isolate)->Value() == Number::New(isolate, 1)->ToBoolean(isolate)->Value()) : ((n_1->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? Boolean::New(isolate, n_1->ToNumber(isolate)->Value() == Number::New(isolate, 1)->ToNumber(isolate)->Value()) : ((n_1->IsString() || Number::New(isolate, 1)->IsString()) ? Boolean::New(isolate, string_tmp_18 == string_tmp_20) : (False(isolate)))) };
  Local<Value> result_22 = Boolean_16->Call(Null(isolate), 1, argv_21);
  if (result_22->ToBoolean()->Value()) {
    // return b;
    args.GetReturnValue().Set(b_3);
    return;


  }

  // return fib(n - 1, b, a + b);
  Local<Value> arg_23 = (n_1->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? (Number::New(isolate, n_1->ToNumber(isolate)->Value() - Number::New(isolate, 1)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<Value> arg_24 = b_3;
  Local<Value> arg_25 = (a_2->IsString() || b_3->IsString()) ? Local<Value>::Cast(String::Concat(a_2->ToString(), b_3->ToString())) : Local<Value>::Cast((a_2->IsNumber() || b_3->IsNumber()) ? (Number::New(isolate, a_2->ToNumber(isolate)->Value() + b_3->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate)));
  Local<FunctionTemplate> ftpl_27 = FunctionTemplate::New(isolate, fib_0);
  Local<Function> fn_26 = ftpl_27->GetFunction();
  fn_26->SetName(String::NewFromUtf8(isolate, "fib_0"));
  n_1 = arg_23;
  a_2 = arg_24;
  b_3 = arg_25;
  goto tail_recurse_4;

}

void jsc_main(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
tail_recurse_5:

  // console.log(fib(50, 0, 1))
  Local<Value> dot_parent_7 = isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console"));
  Local<String> property_8 = String::NewFromUtf8(isolate, "log");
  while (dot_parent_7->IsObject() && !dot_parent_7.As<Object>()->HasOwnProperty(isolate->GetCurrentContext(), property_8).ToChecked()) {
    dot_parent_7 = dot_parent_7.As<Object>()->GetPrototype();
  }
  Local<Value> dot_result_6 = dot_parent_7.As<Object>()->Get(isolate->GetCurrentContext(), property_8).ToLocalChecked();
  Local<Value> arg_9 = Number::New(isolate, 50);
  Local<Value> arg_10 = Number::New(isolate, 0);
  Local<Value> arg_11 = Number::New(isolate, 1);
  Local<FunctionTemplate> ftpl_13 = FunctionTemplate::New(isolate, fib_0);
  Local<Function> fn_12 = ftpl_13->GetFunction();
  fn_12->SetName(String::NewFromUtf8(isolate, "fib_0"));
  Local<Value> argv_14[] = { arg_9, arg_10, arg_11 };
  Local<Value> result_15 = fn_12->Call(Null(isolate), 3, argv_14);
  Local<Value> arg_16 = result_15;
  Local<Function> fn_17 = Local<Function>::Cast(dot_result_6);
  Local<Value> argv_18[] = { arg_16 };
  Local<Value> result_19 = fn_17->Call(dot_parent_7, 1, argv_18);
  result_19;

}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

This output gets compiled (by jsc) as a Node
addon using
node-gyp.
The compiled addon is loaded by a single-line Javascript file generated by jsc:
$ rm -rf build
$ jsc fib.js
$ cat build/fib.js
require("build/Release/fib.node").jsc_main()
$ node build/fib.js
12586269025

Analysis
The code was a mess of bad formatting, unnecessary locals, inefficient
basic operations (e.g. huge, often unnecessary Boolean conversions),
and so on. The unnecessary locals was partially a by-product of
single-pass code generation. And the unnecessary conversions was
partly due to ignoring types (even types of literals that you don't
need Typescript/Flow to provide).
After I got this proof-of-concept working for basic examples, I wanted
to rewrite it around destination-driven code
generation,
a technique by Kent Dybvig used in V8's baseline compiler. And after a
few weeks not getting far in a refactor in Rust, I rewrote the
compiler in Typescript.
The second pass
Written in Typescript and using the Typescript compiler
API,
this second iteration was built to do destination-driven code
generation and leaf type propagation. Destination-driven code
generation allows a single-pass code generator to reduce redundant
reassignments. And leaf type propagation allows simple, obvious
optimizations such as just calling V8::Boolean::IsTrue()
on a statically-known boolean rather than calling
V8::Value::Equals().
Example
Given the same fibonacci Javascript program from before, this
iteration produces the following C++:
#include "lib.cc"

void tco_fib(const FunctionCallbackInfo<Value>& _args) {
  Isolate* isolate = _args.GetIsolate();
  std::vector<Local<Value>> args(_args.Length());;
  for (int i = 0; i < _args.Length(); i++) args[i] = _args[i];
tail_recurse_0:

  Local<Number> sym_rhs_4 = Number::New(isolate, 0);
  Local<Boolean> sym_anon_2 = args[0]->StrictEquals(sym_rhs_4) ? True(isolate) : False(isolate);
  if (sym_anon_2->IsTrue()) {
    _args.GetReturnValue().Set(args[1]);
    return;
  }

  Local<Number> sym_rhs_11 = Number::New(isolate, 1);
  Local<Boolean> sym_anon_9 = args[0]->StrictEquals(sym_rhs_11) ? True(isolate) : False(isolate);
  if (sym_anon_9->IsTrue()) {
    _args.GetReturnValue().Set(args[2]);
    return;
  }

  Local<Number> sym_rhs_19 = Number::New(isolate, 1);
  Local<Value> sym_arg_17 = genericMinus(isolate, args[0], sym_rhs_19);
  Local<Value> sym_arg_21 = genericPlus(isolate, args[1], args[2]);
  args[0] = sym_arg_17;
  args[1] = args[2];
  args[2] = sym_arg_21;
  goto tail_recurse_0;

  return;
}

void jsc_main(const FunctionCallbackInfo<Value>& _args) {
  Isolate* isolate = _args.GetIsolate();
  std::vector<Local<Value>> args(_args.Length());;
  for (int i = 0; i < _args.Length(); i++) args[i] = _args[i];
tail_recurse_1:

  Local<Number> sym_arg_29 = Number::New(isolate, 100);
  Local<Number> sym_arg_30 = Number::New(isolate, 0);
  Local<Number> sym_arg_31 = Number::New(isolate, 1);
  Local<Value> sym_args_32[] = { sym_arg_29, sym_arg_30, sym_arg_31 };
  Local<Function> sym_fn_33 = FunctionTemplate::New(isolate, tco_fib)->GetFunction();
  sym_fn_33->SetName(String::NewFromUtf8(isolate, "tco_fib"));
  Local<Value> sym_arg_28 = sym_fn_33->Call(sym_fn_33, 3, sym_args_32);

  Local<Value> sym_args_34[] = { sym_arg_28 };
  Local<Value> sym_parent_37 = isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console"));
  Local<Value> sym_anon_36 = sym_parent_37.As<Object>()->Get(String::NewFromUtf8(isolate, "log"));
  Local<Function> sym_fn_35 = Local<Function>::Cast(sym_anon_36);
  Local<Value> sym_anon_27 = sym_fn_35->Call(sym_fn_35, 1, sym_args_34);

  return;
}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

Analysis
Common code (genericPlus, genericMinus) and
all imports have been pulled into lib.cc for clarity. And
the entire result is run through
clang-format if it is
present on the system.
The benefit of leaf type propagation can be seen everywhere a local is
declared that is not Local and specifically in if
tests on statically known booleans:
...
Local<Boolean> sym_anon_2 = args[0]->StrictEquals(sym_rhs_4) ? True(isolate) : False(isolate);
  if (sym_anon_2->IsTrue()) {
...

It's obvious to a human that there is another optimization you could
do here by not wrapping this check in a V8::Boolean at
all. The only types tracked in destinations are V8 types, not yet C++
types. But not needing to passing this through a bool
toBoolean(Value v) wrapper is still an improvement.
In general, unboxing has not really been explore. But the ultimate
goal is to use Typescript types to produce function- or block-level
unboxed versions -- perhaps using a toggle in code to specify safety à
la Common Lisp.
Next steps
I broke tests and regressed on syntax support in the Typescript port,
so that's the first step. The second step is enough syntax to support
more interesting benchmarks than the fibonacci example (which has
comparative performance to Node.js/V8 but isn't saying much).
After that:

Unboxed expressions
Unboxed blocks
Foreign-function interface
Self-hosting
Node-API compatible runtime without Node

Companion blog post to my talk on an AOT-compiled Javascript implementation built on Typescript https://t.co/0aHVJ9UzYh
— Phil Eaton (@phil_eaton) February 26, 2019
 


Learn Enough tmux to be Productive in 5 Minutes
Ben E. C. Boyter — Sun, 24 Feb 2019 02:00:00 GMT
tmux the terminal multiplexer is one of those brilliant tools I use all day. I am not a tmux expert, nor do I have any need to be. What follows is the bare minimum you need to know to feel productive in tmux, written for some work colleagues.
The following assumes you have not configured or tweaked tmux in any way opting for the default settings.
From the command line to start or connect to tmux


Dealing with dead disks in a btrfs RAID1 array
Nemo's Home — Sun, 24 Feb 2019 02:00:00 GMT
tl;dr: Check your disk usage v/s RAID capacity to ensure that you can remove a disk before trying. If you can connect a new disk without removing the old one, run a btrfs replace - it is much faster.



My homeserver setup has a 4 disk setup:


  128GB Samsung EVO 850 SSD as the primary disk (root volume)
  A 3 Disk btrfs RAID1 Array that I use for almost everything else.


The 3 disks were:


  A WD-3.5inch-3TB that I shelled from a WD-MyBook. This was the oldest disk in the array
  2xSeagate 2.5-inch-3TB external disks that I shelled from Seagate Expansion disks.


The WD disk had been giving rising errors recently, and I was noticing hangs on the system as well:


  My Steam saves would take time, and hang the game.
  Kodi would ocassionaly hang just switching between screens as it would load images from disk.
  gitea, which writes a lot to disk would get similar issues.


I asked a question on r/archlinux and confirmed that it indeed a dead disk.

Ordered a new Seagate Barracuda 3TB the next day, but my peculiar setup caused me a lot of pain before I could remove the dead disk. The primary issue was with the limited number of SATA connectors I had (just 4). The original setup had /dev/sdb,/dev/sdc,/dev/sdd as the three RAID disks with /dev/sdb being the dying WD.

This is what all I tried:


  Removing /dev/sdb and adding a new disk the array (/dev/sde). Unfortunately, to add a disk to the array, you have to mount it first, and the setup just refused to mount in degraded mode. (It didn’t give a visibly error, so I didn’t know why)
  I tried to keep the old disk attached over USB on a friend’s suggestion, but that didn’t work either. This was likely a cable issue, and I didn’t investigate this further.
  Booting with the original three disks but replacing the dying disk with the new one post boot. Didn’t work as I kept getting read/write errors to sdb even after it was disconnected.


In short:


  the system refused to mount the raid array with a missing disk (and I didn’t want to risk a boot with the array unavailable)
  I couldn’t do a live replace because I had a limited number of SATA connectors.


What worked:

Running a btrfs device delete and leting it run overnight. It gave an error after quite a long time that finally helped me figure out the problem:

btrfs device delete /dev/sdb1 /mnt/xwing
ERROR: error removing device '/dev/sdb1': No space left on device

btrfs fi df /mnt/xwing
Data, RAID1: total=2.98TiB, used=2.98TiB
System, RAID1: total=32.00MiB, used=544.00KiB
Metadata, RAID1: total=5.49GiB, used=4.81GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


The RAID array was 2.7TBx3 disks and I was storing roughly 2.98TB of data. To switch to a RAID1 setup with just 2 disks, I needed to delete some data. I ended up clearing out a few steam games (bye bye Witcher 3) and ran another btrfs device delete to resolve the issue.

If you are faced with a situation where you have to remove a device, but can’t do a live replace, here’s what you need:


  Check that your disk removal does not impact any data storage. Your n-1 disk array should have enough capacity to store everything.
  Run a btrfs device delete
  Reboot
  Re-attach new disk, and then run a btrfs device add


As a retro, I posted a summary with the issues I faced on the btrfs mailing list



If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:


  Part 1, Hardware
  Part 2, Terraform/Docker
  Part 3, Learnings
  Part 4, Migrating from Google (and more)
  Part 5, Home Server Networking
  Part 6, btrfs RAID device replacement


If you have any comments, reach out to me


Unsupervised Learning: Information Theory Recap
swyx's site RSS Feed — Sat, 23 Feb 2019 15:45:41 GMT
A small detour to catch up on the basics of Information Theory we'll need


Unsupervised Learning: Randomized Optimization
swyx's site RSS Feed — Sat, 23 Feb 2019 14:32:43 GMT
Hill Climbing, Simulated Annealing, Genetic Algorithms, oh my!


Unsupervised Learning: Randomized Optimization
swyx's site RSS Feed — Sat, 23 Feb 2019 14:32:43 GMT
Hill Climbing, Simulated Annealing, Genetic Algorithms, oh my!


Supervised Learning: Bayesian Inference
swyx's site RSS Feed — Sat, 23 Feb 2019 11:02:44 GMT
Or, the unreasonable effectiveness of dumb rules


Post Bootcamp Mindmap
swyx's site RSS Feed — Thu, 21 Feb 2019 02:00:00 GMT
What are all the things you can do to improve as a developer after graduating from college or bootcamp?


Fixing the Exim 'certificate name mismatch' error
infrequent oscillations — Wed, 20 Feb 2019 15:23:50 GMT
Spoiler: it was manualroute to bare IPs


Goal Differences
infrequent oscillations — Tue, 19 Feb 2019 11:22:37 GMT
In which we are disappointed by goal differences.


Automatic Insurance
Pete Keen — Tue, 19 Feb 2019 02:00:00 GMT
The Setup

$1.5 million of term life insurance on me
A smaller amount of term life insurance on my wife
A private disability insurance policy on me that pays $5,000/mo if I can't program computers anymore, for up to 2 years
Employer sponsored short and long term disability insurance
Employer sponsored health insurance
Auto, home, and umbrella insurance


Why
My future income is currently my family's largest asset.
Term life protects my family if I die before the kids go to college.
Disability protects them if I'm incapacitated but don't die.
Health protects our assets from medical bankruptcy.
Auto, home and umbrella protect our assets from car crashes, fires, and slip and fall incidents.
We own term life insurance because it's the simplest, cheapest possible product.
If you die they pay. If you die before 2 years have passed they'll dig into your application and make sure you didn't lie or omit anything, but that's the only wrinkle.
Never buy whole, universal, or variable life insurance. It combines insurance with investing on very bad (for the customer) terms.
I have both employer sponsored and private policies because my health history makes it difficult and expensive to get the amount of insurance that we need.
The combination, while a little more complicated, helps me sleep easier.


Supervised Learning: Bayesian Learning
swyx's site RSS Feed — Sat, 16 Feb 2019 04:37:05 GMT
Lets update our priors! Oh wait we have none.


Supervised Learning: VC Dimensions
swyx's site RSS Feed — Fri, 15 Feb 2019 10:14:24 GMT
It's not venture capital, that's for sure


Supervised Learning: Computational Learning Theory
swyx's site RSS Feed — Fri, 15 Feb 2019 09:03:05 GMT
What's the big O of machine learning? Lets put some formal theory around HOW we learn!


SQL: One of the most valuable skills
CRAIG KERSTIENS — Tue, 12 Feb 2019 22:55:56 GMT
I’ve learned a lot of skills over the course of my career, but no technical skill more useful than SQL. SQL stands out to me as the most valuable skill for a few reasons:

It is valuable across different roles and disciplines
Learning it once doesn’t really require re-learning
You seem like a superhero. You seem extra powerful when you know it because of the amount of people that aren’t fluent

Let me drill into each of these a bit further.


SQL a tool you can use everywhere


Regardless of what role you are in SQL will find a way to make your life easier. Today as a product manager it’s key for me to look at data, analyze how effective we’re being on the product front, and shape the product roadmap. If we just shipped a new feature, the data on whether someone has viewed that feature is likely somewhere sitting in a relational database. If I’m working on tracking key business metrics such as month over month growth, that is likely somewhere sitting in a relational database. At the other end of almost anything we do there is likely a system of record that speaks SQL. Knowing how to access it most natively saves me a significant amount of effort without having to go ask someone else the numbers.
But even before becoming a product manager I would use SQL to inform me about what was happening within systems. As an engineer it could often allow me to pull information I wanted faster than if I were to script it in say Ruby or Python. When things got slow in my webapp having an understanding of the SQL that was executed and ways to optimize it was indespensible. Yes, this was going a little beyond just a basic understanding of SQL… but adding an index to a query instead of rolling my own homegrown caching well that was well worth the extra time learning.


SQL is permanent


I recall roughly 20 years ago creating my first webpage. It was magical, and then I introduced some Javascript to make it even more impressive prompting users to click Yes/No or give me some input. Then about 10 years later jQuery came along and while it was a little more verbose at times and something new to learn it made things prettier overall so I committed to re-learning the jQuery approach to JS. Then it just picked up pace with Angular -> React/Ember, and now I have an entire pipeline to introduce basic Javascript into my website and the reality is I’m still trying to accomplish the same thing I was 20 years ago by having someone click Yes/No.
SQL in contrast doesn’t really change. Caveat: It has changed–there is modern sql, but I’d still argue less dramatically than other language landscapes. Yes we get a new standard every few years and occasionally something new comes along like support for window functions or CTEs, but the basics of SQL are pretty permanent. Learning SQL once will allow you to re-use it heavily across your career span without having to re-learn. Don’t get me wrong I love learning new things, but I’d rather learn something truly new than just yet another way to accomplish the same task.


SQL: Seem better than you are


SQL is an underlearned skill, the majority of application developers just skip over it. Because so few actually know SQL well you can seem more elite than you actually are. In past companies with hundreds of engineers I’d get a question several times a week from junior to principal engineers of: “hey can you help me figure out how to write a query for this?” Because you’re skilled at something so few others are you can help them out which always makes life a little easier when you have a question for them.
So if you’re not already proficient what are you waiting for, do you want to seem like a SQL badass yet?
If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.


More Music
infrequent oscillations — Tue, 12 Feb 2019 20:00:47 GMT
Some more music and a lunatic from Bolton


PyTorch Basics: Solving the Ax=b matrix equation with gradient descent
Bytepawn - Marton Trencseni — Fri, 08 Feb 2019 01:00:00 GMT
I will show how to solve the standard A x = b matrix equation with PyTorch. This is a good toy problem to show some guts of the framework without involving neural networks.




Setting up a new personal blog
Ben Overmyer's Site — Wed, 06 Feb 2019 02:00:00 GMT
Having a blog built using a static site generator was nice from a maintenance perspective, but from a writing perspective, it’s a bit of an inconvenience.
So now I’ve created a new blog on this address using WordPress. I already use WordPress for the Silver Gryphon Games website and I maintain the Geek Partnership Society website (which also uses WordPress), so I might as well add a third one to the mix.
I will probably migrate all of my existing blog posts to this site. First though I’ll need to write a utility to migrate from Hugo to WordPress. That kind of tool doesn’t exist yet.


Goodbye XenSever - Hello XCP-ng
smcleod.net — Tue, 05 Feb 2019 15:00:00 GMT
In 2018 I set out to replace our XenSever 7.2 based virtualisation after Citrix essentially screwed over free / open source users.
This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs.

High Level Selection Considerations#
I evaluated a number of options, with the primary candidates below.
Key criteria (at a high level) I was evaluating:

Ease of moving from our existing XenServer 7.2 based hypervisor clusters.
Security (architecture, hardening, monitoring, logging).
Cost (both licensing if any and self-support / management costs).
VM Performance (Storage IOPs and throughput, Network latency and throughput, Processing latency, steal from over-provisioned workloads).
Management UI/UX and performance (for BAU activities).
Management / Cluster SPOFs, fail-over and redundancy.
Installation and upgrade process.
Update and security patching frequency.
Networking design and complexity.
Community (size, engagement, acceptance of suggestions / MRs).
Reliable live VM migrations.
Ease of management for a small team (Part of a low TCO).
Risk of vendor and technology lock-in.
Risk of survival (will it still be well maintained over the next 1-3 years).
Stability and reliability above practically all else.

XenServer 7.6 (w/ Paid License)#
For#

Potentially priority support from Citrix for issues.
Easy upgrade from 7.2.
Fresh installs and upgrades are simple, painless and easy to pxeboot, licensing can be a pain after install however.

Against#

Slow moving development.
Incredibly expensive, it would cost us something like $140,800 AUD per year (shelf price) for our 32 hosts (each with 2 sockets).
Features heavily restricted by licensing model.
Mostly older-fashioned enterprise users.
No web management interface (although you can use Xen Orchestra from the folks behind XCP-ng).
Diminishing community since XenServer licensing changes in 7.3.
Poor storage performance compared to KVM based solutions.
Many outdated packages, while kind of based on CentOS 7, there are a great deal of packages from older releases or completely custom rebuilt.
Uses old technologies like EXT3 and doesn’t support SSD/Flash TRIM/DISCARD functions.
SELinux not enforcing or supported.
OpenvSwitch still uses a lot of Dom0 CPU (same as XenServer).
Dom0 often ends up limiting VM storage operations (tapdisk maxing out Dom0 CPU).
Applying updates can be painful or at least slow, requiring binary ISO files to be downloaded from Citrix.
Uses Jira for bug tracking (I just can’t stand the thing, it’s painfully laggy, give my GitLab or Github over it any day).

XCP-ng#
XCP-ng is a relatively recent fork from XenServer after it was open sourced, tracking upstream but clearly prioritising modernisation and community.


SQLite Parameter Queries
infrequent oscillations — Mon, 04 Feb 2019 21:15:00 GMT
Since sqlite doesn’t support stored procedures, it’s tricky to parameterise a query...


Supervised Learning: Support Vector Machines
swyx's site RSS Feed — Fri, 01 Feb 2019 08:03:32 GMT
Transforming dimensions, nearest neighbors, and boosting, all in service of "drawing the best line".


More Football Facts And Figures
infrequent oscillations — Wed, 30 Jan 2019 22:53:52 GMT
In which we amuse ourselves with football facts.


Using mkcert to Set Up Local SSL on macOS's Built in Apache
Donat Studios — Wed, 30 Jan 2019 02:16:47 GMT
A little under a year ago I wrote a walkthrough: Set up a Self Signed Certificate on macOS's Built in Apache.
Recently a new tool showed up called mkcert has come and made life way easier.
It's a wonderful tool that instead of building Self-Signed Certificates, sets your local machine up with a certificate signing authority and creates valid certificates. 
You can read more about how it works at the link above. Needless to say, the process is much less painful than creating self signed certificates.
Installing mkcert
The easiest way to install mkcert is via Homebrew.
Beyond mkcert we should also install nss which helps mkcert set up Firefox to respect its certificates.
$ brew install mkcert nss
Once that's complete, we need mkcert to set itself up as a signing authority on our local machine.
$ mkcert -install
Creating the Certificates
Once we have mkcert set up, the next step is to generate the certificates for the intended local domain(s). 
You may include multiple domains or wildcard subdomains. You can find more information about mkcert's capabilities here.
$ mkdir /tmp/crt && cd /tmp/crt
$ mkcert local.example.com
You should get an output something like as follows
Using the local CA at "/Users/user/Library/Application Support/mkcert" ✨

Created a new certificate valid for the following names ????
 - "local.example.com"

The certificate is at "./local.example.com.pem" and the key at "./local.example.com-key.pem" ✅
Now that we have our certificate, we can move it into the apache /etc directory:
$ sudo mkdir /etc/apache2/ssl
$ sudo mv *.pem /etc/apache2/ssl
Apache Configuration
Next, all we need to do is configure Apache. 
Follow the Apache Configuration section's instructions, stopping and returning here before the "Chrome / Safari Configuration" section on my previous post.
When you reach the portion about configuring your VirtualHosts, the only difference is the filenames will look more akin to this pattern: 
  SSLEngine on
  SSLCertificateFile "/private/etc/apache2/ssl/local.example.com.pem"
  SSLCertificateKeyFile "/private/etc/apache2/ssl/local.example.com-key.pem"
Unlike the prior walkthrough which restarted Apache with a reboot, we instead now only need to restart Apache.
$ sudo apachectl restart
And we're done! Unlike setting up a self signed certificate, we do not need to add anything to our keychain or browsers. We are ready to go, your certificate should work as expected.


Iron Arachne Updates for January
Ben Overmyer's Site — Wed, 30 Jan 2019 02:00:00 GMT
It’s been a dry spell for Iron Arachne for the past couple months. The holidays and work on Silver Gryphon Games meant that I had no time for coding outside of work.
Coming up in February I have a two-week trip to Thailand. I won’t be coding during that, either!
Yet, I have been spooling up some projects for Iron Arachne. Here are some of the things I’ll be doing this half of the year:

Fixing a few bugs with the heraldry generator
Writing a language description generator
Expanding the Æther setting generator (under the Silver Gryphon Games banner)

I plan on doing work to make publishing and discovering these generators easier. That means releasing several of them as Docker images on Docker Hub, among other things.
The generator community is a talented group. More links to authors’ tools are going to appear on the Iron Arachne website.
I’m thinking about what kinds of generators I should be writing for Iron Arachne. Many of the ideas I’ve been running with so far have been things that’ve done many times before by other people. This includes name generators, character generators, and so forth. I’d like to cover new territory now that I have more experience with writing these tools.


Supervised Learning: Ensemble Learning and AdaBoost
swyx's site RSS Feed — Sun, 27 Jan 2019 23:35:27 GMT
Better together - how bootstrapping samples of data can work better than the entire dataset, and how to boost it even further, and faster


Linear Typewriters, Part 1
infrequent oscillations — Sun, 27 Jan 2019 11:24:08 GMT
Adventures in optimising a linear typewriter, part 1


Supervised Learning: Instance-based Learning and K-Nearest Neighbors
swyx's site RSS Feed — Sun, 27 Jan 2019 04:38:52 GMT
Regression isn't the only way. What if we were far, far... lazier about it?


Books I abandoned
Krishna's blog — Sun, 27 Jan 2019 02:00:00 GMT
Discover the books this reader abandoned and why, from a dictator's story to a Nobel Prize winner's account of Soviet concentration camps.


Automating a Call Center with Machine Learning
Bytepawn - Marton Trencseni — Sun, 27 Jan 2019 01:00:00 GMT
Over a period of 6 months, we rolled out a Machine Learning model to predict a customer’s delivery (latitude, longitude). During the recent holiday peak, this ML model handled most of Fetchr’s order scheduling.




Supervised Learning: Neural Networks
swyx's site RSS Feed — Tue, 22 Jan 2019 08:03:54 GMT
That one time we tried to emulate our brains with computer chips


Supervised Learning: Regression
swyx's site RSS Feed — Tue, 22 Jan 2019 06:37:24 GMT
Drawing lines among dots and more!


Supervised Learning: Classification Learning & Decision Trees
swyx's site RSS Feed — Tue, 22 Jan 2019 05:51:50 GMT
The simplest form of Classification algorithm


Machine Learning: An Overview
swyx's site RSS Feed — Tue, 22 Jan 2019 02:38:21 GMT
The three major branches of ML


My Spin on the Unix Philosophy
Ben Overmyer's Site — Tue, 22 Jan 2019 02:00:00 GMT
The Unix philosophy guides the development of both Unix and its derivatives. It’s a good philosophical core to judge new projects by. I’m increasingly using the Unix philosophy in my own works. This is particularly true of new versions of Iron Arachne command line tools. There are multiple versions of the Unix philosophy bouncing around on the web. Some are longer or more stringent than others. What follows is my own interpretation of it.
Rule One: Tight Focus
Each program should do one thing well. Corollary: Each program should do only one thing.
Rule Two: Predictable Flexibility
Expect each program’s output to be the input for some unknown other program. Design for flexibility. Corollary: Also design first for command-line interfaces, and avoid interactive interfaces. Assume machines will use your program more often than humans.
Rule Three: Rapid Iteration
Release early and often. Corollary: It’s not only OK but desirable to have minimal change between iterations.
Rule Four: Automated Development
Write tools to assist development instead of seeking out human help. Corollary: It’s fine to ask for help when you’ve tried the above to no success.


Use Go to Restore Glacier to S3
Ben E. C. Boyter — Tue, 22 Jan 2019 02:00:00 GMT
I had a great deal of difficulty searching for an answer to this I have included it here to hopefully save someone else the effort.
How to make a S3 request to retrieve items that have transitioned into AWS Glacier from an S3 bucket using the Go AWS SDK API.
package main

import (
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/awserr"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
		Region: aws.String("ap-southeast-2"),
	})

	svc := s3.New(sess)
	input := &s3.RestoreObjectInput{
		Bucket: aws.String("BucketName"),
		Key:    aws.String("FileKey"),
		RestoreRequest: &s3.RestoreRequest{
			Days: aws.Int64(1),
			GlacierJobParameters: &s3.GlacierJobParameters{
				Tier: aws.String("Expedited"), // Valid values: Expedited | Standard | Bulk
			},
		},
	}

	result, err := svc.RestoreObject(input)
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case s3.ErrCodeObjectAlreadyInActiveTierError:
				fmt.Println(s3.ErrCodeObjectAlreadyInActiveTierError, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return
	}

	fmt.Println(result)
}


Windows
Notes on software development — Sun, 20 Jan 2019 02:00:00 GMT
It has been six years since I last used Windows for any remotely
serious software development. I've used Ubuntu, Arch, or FreeBSD
since. But eventually I spent so much time working around common
workplace tasks that I decided to put Windows 10 Pro on my work
laptop.
Windows Subsystem for Linux
Introduced in 2016, this technology allows Windows to run unmodified
Linux binaries. The core feat being syscall
translation.
It works nearly flawlessly. This means I can do all my Go, Node,
PostgreSQL development on Windows without a virtual machine using
bash, tmux, git, emacs, etc.
I've seen a few minor exceptions over the course of regular software
development in WSL:

ss/netstat does not work
vagrant does not work

More generally, Linux programs are heavily file-oriented. And Windows
I/O is not designed well for
that.
In the worst cases (installing/adding Node packages) it can take
minutes to do operations that would take Linux seconds.
Vagrant
Vagrant-Windows interoperability is abysmal.
As noted above, you cannot manage Hyper-V from vagrant within WSL. So
you're stuck using Powershell. Even then, managing synced files from
vagrant is a nightmare. The default sync method requires you to sign
in using your Windows Live username and password on every
reboot. But Node package installation attempts some file operations
that are not supported over the default synced, network filesystem.
When I switched to rsync vagrant wouldn't reliable sync when the
virtual machine went down and came back up.
After hours of trying to get some files synced with vagrant I gave up.
Hyper-V
Hyper-V's GUI is much more complex/feature-complete than VirtualBox.
It even provides a Ubuntu-quick-install that I used to jump right in.
I don't recommend using this though because it gives you no option but
an 11GB hard disk. I didn't realize this until I went through an hour
or two of post-install customization only to run out of space. Too
lazy to boot into a live CD to grow the root filesystem I reinstalled
with a more suitable 64GB drive and went through the hour-long
post-install customization process again.
Networking in Hyper-V is more complex/feature-complete than VirtualBox
as well. To access a Hyper-V machine you must create a new virtual
network interface manually and associate it. Static IP address appear
to be controlled at the host networking level (e.g. Control Panel)
instead of within the Hyper-V interface. This highlights how these
virtual interfaces are first-class, but overcomplicates the process of
getting started.
Ultimately I gave up on a static IP address and decided to reboot less
frequently.
Performance-wise Hyper-V machines are exactly as expected: excellent.
Misc
Docker support on Windows needs work. It took me a while to understand
how Docker interacts with the WSL filesystem and what I needed to do
to allow Docker to mount. The complexity is similar on macOS when you
want to mount privileged directories like /var, but the experience is
worse on Windows.
Apparently Windows does have tiling window managers, but I have not
tried one out yet.
Powershell, a language with real types, is pretty compelling. But I
have not spent enough time with it to be efficient. And since WSL is
mostly good enough I don't really plan to.
Windows doesn't allow you to delete any files that are "in use". This
is kinda cool except for that the errors you get when trying to delete
files that are in use are useless. They are even more useless when you
get the plain "could not delete directory" when you try to delete a
directory with some file inside it that is in use. I had to start
deleting files within by hand until I found the one I realized was in
use.
Conclusion
If you have never run Linux or FreeBSD, don't use this post as an
excuse not to. You should run Linux or FreeBSD for the experience. But
if you've reached diminishing returns in your Linux/FreeBSD use,
Windows as a development environment has come a long way. It may be
the best platform available for software development, the profession.
Some notes on my experience having replaced Arch Linux with Windows on my work laptop https://t.co/8asxZmspwR
— Phil Eaton (@phil_eaton) January 20, 2019
 


Writing a lisp compiler from scratch in JavaScript: 2. user-defined functions and variables
Notes on software development — Sun, 20 Jan 2019 02:00:00 GMT

  Previously in compiler basics:
  <! forgive me, for I have sinned >
  

  1. lisp to assembly
  

  

  Next in compiler basics:
  

  3. LLVM
  

  4. LLVM conditionals and compiling fibonacci
  

  5. LLVM system calls
  

  6. an x86 upgrade
In this post we'll extend the compiler to support defining functions
and variables. Additionally, we'll require the program's entrypoint to
be within a main function.
The resulting code can be found
here.
Function definition
The simplest function definition we need to support is for our main
function. This will look like this:
$ cat basic.lisp
(def main ()
     (+ 1 2))

Where compiling and running it should produce a return code of 3:
$ node ulisp.js basic.lisp
$ ./build/a.out
$ echo $?
3

Parsing function definitions
The entire language is defined in S-expressions and we already parse
S-expressions.
$ node
> const { parse } = require('./parser');
> JSON.stringify(parse('(def main () (+ 1 2))'));
'[[["def","main",[],["+",1,2]]],""]'

So we're done!
Code generation
There are two tricky parts to code generation once function
definitions are introduced:

Functions definitions are not expressions (in assembly)
Function calling conventions for the callee
Variable scope

Function definitions
A function definition looks like a function call. So we'll need to
keep a list of "primitive" functions that handle what looks like
function calls differently.
function compile_define() {
  // TODO
}

const primitive_functions = {
  def: compile_define,
};

Then in our compile_call function we need to see if the function
being "called" is in this list. If so, we allow the associated
callback to handle compilation.
function compile_call(fun, args, destination) {
  if (primitive_functions[fun]) {
    primitive_functions[fun](args, destination);
    return;
  }

  // Save param registers
  args.map((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile registers and store as params
  args.map((arg, i) => compile_expression(arg, PARAM_REGISTERS[i], scope));

  emit(1, `CALL ${BUILTIN_FUNCTIONS[fun] || scope[fun]}`);

  // Restore param registers
  args.map((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

Now we can begin thinking about compile_define. It takes args
which will be a list of three elements containing the function's:

name
parameters
and body


  It does not use destination because we're treating function
  definitions as statements for now and not as expressions. If we were
  treating it as an expression, we might store the address of the
  function in the destination register.

  We keep destination around to keep the primitive function signatures
  consistent.
Based on how we called functions before and how we defined the
hard-coded add function, we know what a function definition in
assembly generally looks like. And we know the arguments to the
function when called will be in RDI, RSI, and RDX.
function compile_define([name, parameters, body]) {
  // Function name becomes a label we can CALL
  emit(0, `${name}:`);

  // Something to do with RDI, RSI, RDX and the parameters variable?

  // We renamed compile_argument to compile_expression to be more general
  compile_expression(body[0], 'RAX');

  // Maybe some cleanup to do with RDI, RSI, RDX?

  emit(1, 'RET\n');
}

Not a bad first sketch. But how do we match up RDI, RSI, RDX and
the user-defined parameters variable names? For example in the
following:
(def plus-two (a)
     (+ a 2))

It's clear to us that a must match up to RDI. In order to do this
we need to track all variables in a scope dictionary mapping the
variable name to the register where it's stored.
Additionally, keeping track of scope can help us fail quickly in the
following scenario:
(def plus-two (a)
     (+ b 2))

The variable b is used but never defined. It has not been added to
the scope dictionary. So our compiler can fail quickly saying there is
an undefined variable being referenced.
Taking this a step further, what if we want to catch the following
too:
(def plus-two (a)
     (plus a 2))

We're trying to call plus but it has not been defined. We should be
able to fail quickly here too. But that means we're need to track the
scope of function names in addition to variables. We'll choose to
track function names and variable names in the same scope dictionary.

  This is the distinction between a lisp-1 and a lisp-2. We are a
  lisp-1 like Scheme because we have a single scope. Common Lisp is a
  lisp-2 because it stores function name scope separately from
  variable name scope.
Implementing scope
We need to revise every compile function to accept a scope dictionary
(specifically: compile, compile_expression, compile_call, and
compile_define). If a variable is referenced, we need to look up
it's location in the scope dictionary. If a variable is defined
(e.g. a function name or a function parameter) we need to add a
mapping to the scope dictionary.
Modifying compile_expression is easiest:
function compile_expression(arg, destination, scope) {
  // Is a nested function call, compile it
  if (Array.isArray(arg)) {
    compile_call(arg[0], arg.slice(1), destination, scope);
    return;
  }

  if (scope[arg] || Number.isInteger(arg)) {
    emit(1, `MOV ${destination}, ${scope[arg] || arg}`);
  } else {
    throw new Error('Attempt to reference undefined variable or unsupported literal: ' + arg);
  }
}

Next we modify compile_call:
function compile_call(fun, args, destination, scope) {
  if (primitive_functions[fun]) {
    primitive_functions[fun](args, destination, scope);
    return;
  }

  // Save param registers
  args.map((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile registers and store as params
  args.map((arg, i) => compile_expression(arg, PARAM_REGISTERS[i], scope));

  const validFunction = BUILTIN_FUNCTIONS[fun] || scope[fun];
  if (validFunction) {
    emit(1, `CALL ${validFunction}`);
  } else {
    throw new Error('Attempt to call undefined function: ' + fun);
  }

  // Restore param registers
  args.map((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

And then compile_define where we modify scope for the first time:
function compile_define([name, params, ...body], destination, scope) {
  // Add this function to outer scope
  scope[name] = name.replace('-', '_');

  // Copy outer scope so parameter mappings aren't exposed in outer scope.
  const childScope = { ...scope };

  emit(0, `${scope[name]}:`);

  params.forEach(function (param, i) {
    const register = PARAM_REGISTERS[i];
    // Store parameter mapped to associated register
    childScope[param] = register;
  });

  // Pass childScope in for reference when body is compiled.
  compile_expression(body[0], 'RAX', childScope);

  emit(1, 'RET\n');
}

And finally we need to modify the entrypoint compile:
module.exports.compile = function (ast) {
  emit_prefix();
  // Pass in new, empty scope mapping
  compile_call(ast[0], ast.slice(1), 'RAX', {});
  emit_postfix();
}

And scope-wise we're pretty good!
Function calling convention: callee
We currently have a problem that we're using parameters registers to
store local variables that messes up with how we are storing
parameters for function calls within the function itself.
Ideally we could store function local variables (including the
parameters when we get them) separately from how we store function
call parameters within the function.
Thankfully according to the calling convention we've followed, we're
given a set of registers that are callee-preserved. Of them we'll use
RBX, RBP, and R12 in that order. This allows us to mess with so
long as we store them and restore them within the function.
Applying the same storing/restoring strategy to local variables as we
did for parameters, we get:
const LOCAL_REGISTERS = [
  'RBX',
  'RBP',
  'R12',
];

function compile_define([name, params, ...body], destination, scope) {
  // Add this function to outer scope
  scope[name] = name.replace('-', '_');

  // Copy outer scope so parameter mappings aren't exposed in outer scope.
  const childScope = { ...scope };

  emit(0, `${scope[name]}:`);

  params.forEach(function (param, i) {
    const register = PARAM_REGISTERS[i];
    const local = LOCAL_REGISTERS[i];
    emit(1, `PUSH ${local}`);
    emit(1, `MOV ${local}, ${register}`);
    // Store parameter mapped to associated local
    childScope[param] = local;
  });

  // Pass childScope in for reference when body is compiled.
  compile_expression(body[0], 'RAX', childScope);

  params.forEach(function (param, i) {
    // Backwards first
    const local = LOCAL_REGISTERS[params.length - i - 1];
    emit(1, `POP ${local}`);
  });

  emit(1, 'RET\n');
}

And we're set.
Cleanup
We've still got a few messes going on:

emit_prefix wraps out entire body in _main, we're requiring our own main now
emitting to stdout instead of to a file
multiple function definitions is treated as nonsense

Starting first, we rewrite emit_prefix and emit_postfix so that
our _main just calls main.
function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  emit(0, 'plus:');
  emit(1, 'ADD RDI, RSI');
  emit(1, 'MOV RAX, RDI');
  emit(1, 'RET\n');
}

function emit_postfix() {
  emit(0, '_main:');
  emit(1, 'CALL main');
  emit(1, 'MOV RDI, RAX'); // Set exit arg
  emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`);
  emit(1, 'SYSCALL');
}

Next to deal with writing to a file instead of stdout, we need our
emit function to write to a buffer. We'll let ulisp.js write that
buffer to a file. Because we're incredibly lazy, we'll do this all
globally.
let OUT = '';

function emit(depth, args) {
  const indent = new Array(depth + 1).join('  ');
  OUT += `${indent}${args}\n`;
}

...

module.exports.compile = function (ast) {
  OUT = '';

  emit_prefix();
  compile_call(ast[0], ast.slice(1), 'RAX', {});
  emit_postfix();

  return OUT;
}

And modify ulisp.js:
const cp = require('child_process');
const fs = require('fs');

const { parse } = require('./parser');
const { compile } = require('./compiler');

function main (args) {
  const input = fs.readFileSync(args[2]).toString();
  const [ast] = parse(input);
  const program = compile(ast);

  try {
    fs.mkdirSync('build');
  } catch (e) {}
  fs.writeFileSync('build/prog.s', program);
  cp.execSync('gcc -mstackrealign -masm=intel -o build/a.out build/prog.s');
}

main(process.argv);

And we're finally ready to run a simple program.
A program!
$ cat test.lisp
(def main () (+ 1 2))
$ node ulisp.js test.lisp
$ ./build/a.out
$ echo $?
3

Hurray! Now let's try defining and calling a second function
to validate parameter behavior.
$ cat test.lisp
(def plus-two (a)
     (+ a 2))

(def main ()
     (plus-two 3))
$ node ulisp.js test.lisp
$ ./build/a.out
./compiler.js:106
    throw new Error('Attempt to call undefined function: ' + fun);
    ^

Error: Attempt to call undefined function: p2

...

We start getting some really weird errors. And the reason is because
our compiler doesn't know how to deal with sibling S-expressions.
So we'll introduce a new primitive function called begin that calls
all it's sibling functions and returns the value of the last
call. Then we'll wrap the program in an implicit begin so we don't
need to.
function compile_begin(body, destination, scope) {
  body.forEach((expression) => compile_expression(expression, 'RAX', scope));
  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

const primitive_functions = {
  def: compile_define,
  begin: compile_begin,
};

...

module.exports.compile = function (ast) {
  OUT = '';

  emit_prefix();
  compile_call('begin', ast, 'RAX', {});
  emit_postfix();

  return OUT;
}

And we try our test program again. :)
$ cat test.lisp
(def plus-two (a)
     (+ a 2))

(def main ()
     (plus-two 3))
$ node ulisp.js test.lisp
$ ./build/a.out
$ echo $?
5

And that's all there is to it! Stay tuned for the next post on
conditionals and tail-call optimization.
Part two on compiler basics using JavaScript: user-defined functions and variables https://t.co/XOam67HO8h
— Phil Eaton (@phil_eaton) January 20, 2019
 


STAR Apps - A New Generation of Front-End Tooling for Development Workflows
swyx's site RSS Feed — Fri, 18 Jan 2019 02:00:00 GMT
Published on CSS Tricks as STAR Apps: A New Generation of Front-End Tooling for Development Workflows


Cool Facts about Cuttlefish
swyx's site RSS Feed — Tue, 15 Jan 2019 02:00:00 GMT
This was just a short rant I prepared for Netlify's Allhands where we were asked to give a short lightning talk on any topic we wished. I picked cuttles. I actually saw one in…


A technologist's prayer book
jodavaho.io — Mon, 14 Jan 2019 02:00:00 GMT
I recently found a list of old “prayers”, probably inspired by the “litanies”
that are everpresent in the Warhammer 40k universe. I’ll keep a growing list
here. These are free for your use, under the creative commons license.
Chapter one

In the beginning of any focused work, and to set intention for your building.


We dedicate these hours to the advancement of understanding. We thank
humanity for this opportunity. May our children find use of our work.


On the complexity of life and the search for perfection


The universe is clockwork driven by entropy. We thank the omni-algorithms for imparting upon us turing completeness such that we emulate free will. May our meta-algorithms drive us on to perfection.


Blessed be the complexity of life’s outcomes. May we honor the meta-algorithms of others as they honor the meta-algorithms in us.


On the use of life


May we have breadth over years, and depth across days.

License
CC BY-SA 4.0


Sloc Cloc and Code a Performance Update
Ben E. C. Boyter — Wed, 09 Jan 2019 02:00:00 GMT
Update 2019-03-13
This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16
Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28
Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19
Sloc Cloc and Code a Performance Update 2019-01-09
Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab

I thought I had finished with my code counter Sloc Cloc and Code (AKA scc) https://github.com/boyter/scc/ for a while. The what I had hoped to be final blog post about it https://boyter.org/posts/sloc-cloc-code-performance/ however did mention that the building of the language features in it were a cause of slowdown,


Setting up this blog
Krishna's blog — Wed, 09 Jan 2019 02:00:00 GMT
This article provides a step-by-step guide on setting up a static blog using Hugo, Git, Nginx, and Let's Encrypt on a Linux server.


5 things that happened in Data Science in 2018
Bytepawn - Marton Trencseni — Wed, 09 Jan 2019 01:00:00 GMT
2018 was a hot year for Data Science and AI. Here we picked out 5 highlights, which in our opinion shaped the field in the past year.




Trimming the fat from a Golang binary
Ben E. C. Boyter — Sun, 06 Jan 2019 02:00:00 GMT
I have recently started work again on searchcode server with the continued plan to upgrade searchcode to use the same code-base.
One big issue with this which I recently tweeted about was that Java has a decent well documented fast library for just about everything, with the exception of a good syntax highlighter like Pygments http://pygments.org/ or Chroma https://github.com/alecthomas/chroma
It is however possible to use Pygments in Java using Jython http://pygments.org/docs/java/ which was very appealing to me as Pygments is the current highlighter used by searchcode.com


Books I read in 2018
Krishna's blog — Sun, 06 Jan 2019 02:00:00 GMT
A quick round-up of some of the best books read in 2018, ranging from fantasy to spy novels to maths and politics.


Skin in the Game
Krishna's blog — Sun, 06 Jan 2019 02:00:00 GMT
A critical review of the book Skin in the Game by Nassim Nicholas Taleb.


What I Learned And Did In 2018
infrequent oscillations — Sat, 05 Jan 2019 15:13:35 GMT
A brief list of things I learnt or created in 2018. If I missed something, please tell me! Thanks!


Building c-base @ 35C3 with Flowhub
Henri Bergius — Sat, 05 Jan 2019 02:00:00 GMT
The 35th Chaos Communication Congress is now over, and it is time to write about how we built the software side of the c-base assembly there.

c-base at 35C3

The Chaos Communication Congress is a major fixture of the European security and free software scene, with thousands of attendees. As always, the “mother of all hackerspaces” had a big presence there, with a custom booth that we spend nearly two weeks constructing.



This year’s theme was “Refreshing Memories”, and accordingly we brought various elements of the history of the c-base space station to the event. On hardware side we had things like a scale model the c-base antenna, as well as vintage arcade machines and various artifacts from over the years.

With software, we utilized the existing IoT infrastructure at c-base to control lights, sound, and drive videos and other information to a set of information displays. All of course powered by Flowhub.

This was a quite full-stack development effort, involving microcontroller firmware programming, server-side NoFlo and MsgFlo development, and front-end infoscreen web design. We also did quite a bit of devopsing with Travis CI, Docker, and docker-compose.

Local MsgFlo setup

The first step in bringing c-base’s IoT setup was to prepare a “portable” version of the environment. An MQTT broker, MsgFlo, some components, and a graph with any on-premise c-base hardware or service dependencies removed. As this was for a CCC event, we decided to call it c3-flo (in comparison to the c-flo that we run at c-base).

We already have a quite nice setup where our various systems get built and tested on Travis, and uploaded to Docker Hub’s cbase namespace. Some repositories weren’t yet integrated, and so the first step was to Dockerize them.

To make the local setup simple to manage, we decided to go with a single docker-compose environment that would start all systems needed. This would be easy to run on any x86 machine, and provide us with a quite comprehensive set of features from the IoT parts to NASA’s Open MCT dashboard.

Of course we kept adding to the system throughout 35C3, but in the end the graph looked like the following:



WiFi setup

To make our setup more portable, we decided to bring a local instance of the “c-base botnet” WiFi used to Congress. This way all of our IoT devices could work at 35C3 with the exact same firmware and networking setup as they do at c-base.

Normally Congress doesn’t recommend running your own access point. But if needed, there are guidelines available on how to do it properly if needed. As it happens, out of this year’s 558 unofficial access points, the c-base one was the only one conforming to the guidelines (commentary around the 25 minute mark).



Info displays

Like any station, c-base has a set of info screens showing various announcements, timelines, and statistics. These are built with Raspberry Pi 3s running Chrome in Kiosk Mode, with a single-page webapp that connects to our MsgFlo infrastructure over WebSockets with msgflo-browser.

Each screen has a customized rotation of different pages to show, and we can send URLs to announce events like members arriving to c-base or a space launch livestream via MQTT.



For 35C3 we built a new set of pages tailed for the Congress experience:


  Tweaked version of the normal c-base events view showing current and upcoming talks
  Video player to rotate various videos from the history of c-base
  Photo slideshow with a nice set of pictures from c-base
  Countdown screen for some event (c-base crash, teardown of the assembly at the end of Congress)


Crashing c-base

Highlight of the whole assembly was a re-enactment of the c-base crash from billions of years ago. Triggered by a dropped bottle of space soda, this was an experience incorporating video, lights, and audio that we ran several times every day of the conference.



The c-base crash animation was managed by a NoFlo graph integrated to the our MsgFlo setup with the standard noflo-runtime-msgflo tool. With this we could trigger the “crash” with a MQTT message (sent by a physical button), and run a timed sequence of actions on lights, a sound system, and our info screens.



Timeline manager

There were some new components that we had to build for this purpose. The most important was a Timeline component that was upstreamed as part of the noflo-tween animation library.



With this you can define a multi-tracked timeline as JSON or YAML, with actions triggered on each track on their appropriate second. With MsgFlo this meant we could send timed commands to different devices and create a coordinated experience.

For example, our animation started by showing a short video on all info screens. When the bottle fell in the video, we triggered the appropriate soundtrack, and switched the lights through various animation modes. After the video ended, we switched to a “countdown to crash” screen, and turned all lights to a red alert mode.

After the crash happened, everything went dark for a few seconds, before the c-base assembly was returned into its normal state.

Controlling McLighting

All LED strips we used at 35C3 were run using the McLighting firmware. By default it allows switching between different light modes with a simple WebSocket API.

For our requirements, we wanted the capability to send new commands to the lights with minimal latency, and to be able to restore the lights to whatever mode they had before the crash started in the end.



The component is available in noflo-mclighting. The only thing you need is running the NoFlo graph in the same network as the LED strips, and to send the WebSocket addresses of your LED strips to the component. After that you can control them with normal NoFlo packets.

Finally

The whole setup took a couple of days to get right, especially regarding timings and tweaking the light modes. But, it was great! You can see a video of it below:



And if you’re interested in experimenting this stuff, check out the “portable c-base IoT setup” at https://github.com/c-base/c3-flo.


Vignettes Redux Redux
infrequent oscillations — Fri, 04 Jan 2019 13:40:43 GMT
In which we reduce films down to their essence.


Football Statistics
infrequent oscillations — Thu, 03 Jan 2019 06:31:40 GMT
In which David Wagner is simulataneously right and wrong.


Annual Planning
Beetle Space — Tue, 01 Jan 2019 10:00:00 GMT
New Year’s Resolutions. It’s that time of year again. Below I’ve
accumulated some resources that give interesting tips on how to think
about them, along with my own thoughts.
In this series, I will use the words goals and resolutions interchangeably.

Links
First, a dump of the …

Hacker News Personal Blogs 2019 | Top 100 Blogs

36th Chaos Communication Congress

Tamago

In Defense of Hammers

Personal goaling

The modern web is becoming an unusable, user-hostile wasteland

Put All Your Pelican Posts in One Org File

Distributing malware with Google images, service workers and vegan Twitter bots

Pytorch in 2019

BHAGs

Learn to drive the Pan-American Highway

Writing a lisp compiler from scratch in JavaScript: 6. an x86 upgrade

Stack-based languages

Boilerplate

Scope

compileExpression

compileCall

prepareArithmeticWrappers

prepareGeneral

prepareRax

prepareArithmeticWrappers

compileBegin

compileDefine

Our first program

prepareLogicalWrappers

prepareSyscallWrappers

compileIf

Fibonacci

Nov 2019 Recap

The Great British Baking Show

Learning to use emacs

Warren Buffett style fundamental metrics of long-term company performance

Confusion and disengagement in meetings

Calibration curves for delivery prediction with Scikit-Learn

Using simulated self-play to solve all OpenAI Gym classic control problems with Pytorch

An interview on what makes Postgres unique (extensions)

Applying policy gradient to OpenAI Gym classic control problems with Pytorch

Swyx Faves: Software Engineering Daily

Narrow Waists

The Perfect Restaurant

The Three F's of Open Source Development

Fix it, Fork it, Fuck off

Fix it

Oct 2019 Recap

Rust 2020: A better ecosystem

Linux Security Summit Europe 2019

Exploiting race conditions using the scheduler — Jann Horn

Sloc Cloc and Code - Can a crusty Go program outperform a well written Rust Project?

ELCE, OS Summit and KVM Forum 2019 notes

Interesting Upcoming pgDays

Machine Learning at Fetchr

Private Conversations are Private

Ensembles vs Committees

Solving the CartPole Reinforcement Learning problem with Pytorch

Getting started with security keys

Updates (Jan 2020)

Jun 2020

Jan 2020

Dec 2019

Nov 2019

Security best practices

A few basics for improving your online security right now

Getting more advanced

More tips for the increasingly paranoid

Switch your carrier to Google Fi

If you want another safety net

Enable Google Advanced Protection

Consider Advanced Protection for your Google accounts

Further reading

What are security keys?

The small hardware devices to enhance your online security.

Security key history

FIDO, U2F, CTAP, WebAuthn.. where did it come from?

What is passwordless?

What is this passwordless thing all about?

Security keys vs. auth apps

Are security keys really any better than TOTP code generator apps for 2FA?

TOTP risks

Why authenticator apps aren't perfect

Security key benefits