Hacker News Personal Blogs 2019 | Top 100 Blogs

36th Chaos Communication Congress

Linux Engineer's random thoughts — Sat, 28 Dec 2019 14:00:00 GMT

I’m at CCC for the first time this year ! Here my notes for a few of the talks.

Tamago

In an ideal world, one could pick the programming language of their choice and always generate the optimal machine code. But we live in the real world, and it isn't …

In Defense of Hammers

swyx's site RSS Feed — Thu, 26 Dec 2019 02:00:00 GMT

Let's say you needed a multipurpose tool. Which of these would you pick?

Personal goaling

Bytepawn - Marton Trencseni — Sun, 22 Dec 2019 01:00:00 GMT

The meta-goal of goaling is to stretch yourself to achieve more, and to feel good about what you’ve achieved. Whatever happened this year, it’s always possible to achieve a lot more and feel better about yourself next year. To hijack a Feynman quote, there is plenty of room at the top.

Put All Your Pelican Posts in One Org File

Beetle Space — Thu, 19 Dec 2019 10:00:00 GMT

Pelican is a static site generator. Out of the box it supports two formats: Restructured Text and Markdown. There exist plugins to support the Org format. For a while, I would write my posts in org mode, with one file per post. But I longed for a way to have …

Distributing malware with Google images, service workers and vegan Twitter bots

Tom Forbes — Sun, 15 Dec 2019 12:19:01 GMT

What started with a simple image search for “Skippers hat” quickly turned into investigating a clever malware distribution network utilizing service workers, which is somehow linked to a group of hardcore vegan twitter bots. The hat at the entrance of the rabbit hole was innocent enough: However, wh...

Pytorch in 2019

Bytepawn - Marton Trencseni — Thu, 12 Dec 2019 01:00:00 GMT

2019 was another big year for Pytorch, one of the most popular Deep Learning libraries out there. Pytorch has become the de facto deep learning library used for research thanks to it’s dynamic graph model which allows fast model experimentation. It’s also become production ready, with support for mobile and infrastructure tooling such as Tensorboard.

BHAGs

swyx's site RSS Feed — Wed, 11 Dec 2019 02:00:00 GMT

Recently, I asked tweeple:

Learn to drive the Pan-American Highway

The Road Chose Me — Mon, 09 Dec 2019 18:00:33 GMT

As many of you know, long ago – in fact before I had even heard the word “Overlanding” – I set out to drive my little Jeep Wrangler from the top of Alaska to...

Writing a lisp compiler from scratch in JavaScript: 6. an x86 upgrade

Notes on software development — Sun, 08 Dec 2019 02:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
5. LLVM system calls

This post upgrades the ulisp x86 backend from using a limited set of registers (with no spilling support) to solely using the stack to pass values between expressions.

This is a slightly longer post since we've got a lot of catchup to do to get to feature parity with the LLVM backend. Namely:

"Infinite" locals, parameters
Function definitions
Variable references
Arithmetic and logical operations
If
Syscalls

We'll tackle the first four points first and finish up with the last two. This way we can support the same fibonacci program that prints integers to stdout that we support in the LLVM backend.

As always the code is available on Github.

But first a digression into how this is suddenly easy for us to do with x86 and one-pass (sorta) code generation.

Stack-based languages

Stack-based languages have the extremely convenient attribute that values are (by default) stored on the stack, which allows a code generator targeting a stack-based language the option to omit handling register allocation. And as it happens, x86 has enough support to make it easy to treat as a stack machine.

As we build out the code generator for x86 as a stack machine we need to keep two commitments in mind:

Every expression must pop all arguments/operands
Every expression must store one result back on the stack

In the future, we may replace the second commitment. But for now it is more than enough.

Boilerplate

We'll start with the existing x86 backend code and strip all the implementation code:

const cp = require('child_process');
const fs = require('fs');
const os = require('os');

let GLOBAL_COUNTER = 0;

const SYSCALL_MAP = {
  darwin: {
    exit: '0x2000001',
    write: '0x2000004',
  },
  linux: {
    exit: 60,
    write: 1,
  },
}[os.platform()];

class Scope {}

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      if: this.compileIf.bind(this),
      ...this.prepareArithmeticWrappers(),
      ...this.prepareLogicalWrappers(),
      ...this.prepareSyscallWrappers(),
    };
  }

  prepareArithmeticWrappers() {}

  prepareLogicalWrappers() {}

  prepareSyscallWrappers() {}

  emit(depth, args) {
    if (depth === undefined || args === undefined) {
      throw new Error('Invalid call to emit');
    }

    const indent = new Array(depth + 1).join('  ');
    this.outBuffer.push(indent + args);
  }

  compileExpression(arg, scope, depth) {}

  compileIf([test, then, els], scope, depth) {}

  compileBegin(body, scope, depth, topLevel = false) {}

  compileDefine([name, params, ...body], scope, depth) {}

  compileCall(fun, args, scope, depth) {}

  emitPrefix() {
    this.emit(1, '.global _main\n');

    this.emit(1, '.text\n');
  }

  emitPostfix() {
    this.emit(0, '_main:');
    this.emit(1, 'CALL main');
    this.emit(1, 'MOV RDI, RAX'); // Set exit arg
    this.emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`);
    this.emit(1, 'SYSCALL');
  }

  getOutput() {
    const output = this.outBuffer.join('\n');

    // Leave at most one empty line
    return output.replace(/\n\n\n+/g, '\n\n');
  }
}

module.exports.compile = function(ast) {
  const c = new Compiler();
  c.emitPrefix();
  const s = new Scope();
  c.compileBegin(ast, s, 1, true);
  c.emitPostfix();
  return c.getOutput();
};

module.exports.build = function(buildDir, program) {
  const prog = 'prog';
  fs.writeFileSync(`${buildDir}/${prog}.s`, program);
  cp.execSync(
    `gcc -mstackrealign -masm=intel -o ${buildDir}/${prog} ${buildDir}/${prog}.s`,
  );
};

The prefix and postfix stays mostly the same as the original implementation. But we'll assume a couple of new helpers to get us in parity with the LLVM backend:

compileDefine
compileBegin
compileIf
compileCall
prepareArithmeticWrappers
prepareLogicalWrappers
prepareSyscallWrappers

The prepareArithmeticWrappers helper will define wrappers for arithmetic instructions. The prepareLogicalWrappers helper will define wrappers for logical instructions. And the prepareSyscallWrappers helper will define a wrapper for syscalls and generate builtins based on the SYSCALL_MAP entries.

Scope

Similar to our LLVM backend's Context and Scope helpers we'll define our own for the x86 backend. Since we're placing all locals on the stack, the two most important things Scope will do for us are:

Map identifiers to escaped strings
Store and increment the location of the local on the stack

Here's what it will look like:

class Scope {
  constructor() {
    this.localOffset = 1;
    this.map = {};
  }

  assign(name) {
    const safe = name.replace('-', '_');
    this.map[safe] = this.localOffset++;
    return safe;
  }

  symbol() {
    return this.localOffset++;
  }

  lookup(name) {
    const safe = name.replace('-', '_');
    if (this.map[safe]) {
      return { name: safe, offset: this.map[safe] };
    }

    return null;
  }

  copy() {
    const s = new Scope();
    // In the future we may need to store s.scopeOffset = this.scopeOffset + 1
    // so we can read outer-scoped values at runtime.
    s.map = { ...this.map };
    return s;
  }
}

compileExpression

An expression will be one of:

A function call (possibly a builtin like def or +)
A literal value (e.g. 29)
A reference (e.g. &c)
An identifier (e.g. my-var)

We'll handle compiling an expression in that order. If the AST argument passed to compileExpression is an array, we will call compileCall and return.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }
  }

If the AST is a number, we will push the number onto the stack and return.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }
  }

If the AST is a string that starts with & we will look up the location of the identifier after the &, push its location onto the stack and return.

We count on the Scope storing its offset from the "frame pointer", which we will later set up to be stored in RBP.

Locals will be stored after the frame pointer and parameters will be stored before it. So we'll need to add or subtract from the frame pointer depending on if we need a positive or negative offset from it.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }

    if (arg.startsWith('&')) {
      const { offset } = scope.lookup(arg.substring(1));
      // Copy the frame pointer so we can return an offset from it
      this.emit(depth, `MOV RAX, RBP`);
      const operation = offset < 0 ? 'ADD' : 'SUB';
      this.emit(depth, `${operation} RAX, ${Math.abs(offset * 8)} # ${arg}`);
      this.emit(depth, `PUSH RAX`);
      return;
    }
  }

Finally, we'll look up the identifier and copy the value (in its offset on the stack) to the top of the stack.

  compileExpression(arg, scope, depth) {
    // Is a nested function call, compile it
    if (Array.isArray(arg)) {
      this.compileCall(arg[0], arg.slice(1), scope, depth);
      return;
    }

    if (Number.isInteger(arg)) {
      this.emit(depth, `PUSH ${arg}`);
      return;
    }

    if (arg.startsWith('&')) {
      const { offset } = scope.lookup(arg.substring(1));
      // Copy the frame pointer so we can return an offset from it
      this.emit(depth, `MOV RAX, RBP`);
      const operation = offset < 0 ? 'ADD' : 'SUB';
      this.emit(depth, `${operation} RAX, ${Math.abs(offset * 8)} # ${arg}`);
      this.emit(depth, `PUSH RAX`);
      return;
    }

    // Variable lookup
    const { offset } = scope.lookup(arg);
    if (offset) {
      const operation = offset < 0 ? '+' : '-';
      this.emit(
        depth,
        `PUSH [RBP ${operation} ${Math.abs(offset * 8)}] # ${arg}`,
      );
    } else {
      throw new Error(
        'Attempt to reference undefined variable or unsupported literal: ' +
          arg,
      );
    }
  }

And that's it for handling expression! Let's add compileCall support now that we referenced it.

compileCall

A call will first check if the call is a builtin. If so, it will immediately pass control to the builtin.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }
  }

Otherwise it will compile every argument to the call (which will leave all the resulting values on the stack.)

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));
  }

Then we will check that function is defined and call it.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));

    const fn = scope.lookup(fun);
    if (fn) {
      this.emit(depth, `CALL ${fn.name}`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }
  }

Then we'll reset the stack pointer (to maintain our commitment) based on the number of arguments and push RAX (where the return result of the function call will be stored) onto the stack. We'll make two minor optimizations for when there is exactly zero or one argument to the function.

  compileCall(fun, args, scope, depth) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, scope, depth);
      return;
    }

    // Compile registers and store on the stack
    args.map((arg, i) => this.compileExpression(arg, scope, depth));

    const fn = scope.lookup(fun);
    if (fn) {
      this.emit(depth, `CALL ${fn.name}`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }

    if (args.length > 1) {
      // Drop the args
      this.emit(depth, `ADD RSP, ${args.length * 8}`);
    }

    if (args.length === 1) {
      this.emit(depth, `MOV [RSP], RAX\n`);
    } else {
      this.emit(depth, 'PUSH RAX\n');
    }
  }

When there is only one argument, we can just set the top value on the stack to be the return result of the call rather than resetting the stack pointer just to push onto it.

And that's it for compileCall! Now that we've got a feel for expressions and function calls, let's add some simple arithmetic operations.

prepareArithmeticWrappers

There are two major kind of arithmetic instructions we'll wrap for now:

"General" instructions that operate on two arguments, putting the return result in the first argument
"RAX" instructions that operate on RAX and the first argument, putting the return result in RAX and possibly RDX

prepareGeneral

This helper will compile its two arguments and pop the second argument into RAX. This is because x86 instructions typically require one argument to be a register if one argument is allowed to be a memory address.

We'll use the stack address as the first argument so 1) that non-commutative operations are correct and 2) the result is stored right back onto the stack in the right location.

    const prepareGeneral = (instruction) => (arg, scope, depth) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);
      this.emit(depth, `POP RAX`);

      // Compile operation
      this.emit(depth, `${instruction.toUpperCase()} [RSP], RAX`);

      this.emit(depth, `# End ${instruction.toUpperCase()}`);
    };

prepareRax

This helper will similarly compile its two arguments and pop the second argument into RAX. But the RAX-implicit instructions require the argument to be stored in a register so we'll use the XCHG instruction to swap RAX with the value on the top of the stack (the first argument).

    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

This may seem roundabout but remember that we must pop all arguments to the instruction to maintain our commitment.

Next we'll zero out the RDX register if the operation is DIV, perform the operation, and store the result on the top of the stack.

    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

      // Reset RDX for DIV
      if (instruction.toUpperCase() === 'DIV') {
        this.emit(depth, `XOR RDX, RDX`);
      }

      // Compiler operation
      this.emit(depth, `${instruction.toUpperCase()} QWORD PTR [RSP]`);

      // Swap the top of the stack
      this.emit(depth, `MOV [RSP], ${outRegister}`);
    };

We parameterize the out register because the % wrapper will call DIV but need RDX rather than RAX after the operation.

prepareArithmeticWrappers

Putting everything together we get:

  prepareArithmeticWrappers() {
    // General operatations
    const prepareGeneral = (instruction) => (arg, scope, depth) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);
      this.emit(depth, `POP RAX`);

      // Compile operation
      this.emit(depth, `${instruction.toUpperCase()} [RSP], RAX`);

      this.emit(depth, `# End ${instruction.toUpperCase()}`);
    };

    // Operations that use RAX implicitly
    const prepareRax = (instruction, outRegister = 'RAX') => (
      arg,
      scope,
      depth,
    ) => {
      depth++;
      this.emit(depth, `# ${instruction.toUpperCase()}`);

      // Compile first argument, store in RAX
      this.compileExpression(arg[0], scope, depth);

      // Compile second argument
      this.compileExpression(arg[1], scope, depth);

      // POP second argument and swap with first
      this.emit(depth, `POP RAX`);
      this.emit(depth, `XCHG [RSP], RAX`);

      // Reset RDX for DIV
      if (instruction.toUpperCase() === 'DIV') {
        this.emit(depth, `XOR RDX, RDX`);
      }

      // Compiler operation
      this.emit(depth, `${instruction.toUpperCase()} QWORD PTR [RSP]`);

      // Swap the top of the stack
      this.emit(depth, `MOV [RSP], ${outRegister}`);
    };

    return {
      '+': prepareGeneral('add'),
      '-': prepareGeneral('sub'),
      '&': prepareGeneral('and'),
      '|': prepareGeneral('or'),
      '=': prepareGeneral('mov'),
      '*': prepareRax('mul'),
      '/': prepareRax('div'),
      '%': prepareRax('div', 'RDX'),
    };
  }

Next we'll tackle compileBegin and compileDefine.

compileBegin

A begin form is an expression made up of a series of expressions where all expression values are thrown away and the last expression value is the result of the begin form.

To compile this form we will compile each expression passed in and pop from the stack to throw its value away. If the expression is the last in the list we will not pop since it is the result of the begin form.

We will add one exception to this popping logic: if the begin is called from the top-level we will omit the popping.

  compileBegin(body, scope, depth, topLevel = false) {
    body.forEach((expression, i) => {
      this.compileExpression(expression, scope, depth);
      if (!topLevel && i < body.length - 1) {
        this.emit(depth, `POP RAX # Ignore non-final expression`);
      }
    });
  }

That's it for compileBegin!

compileDefine

The prelude for a function definition will add its name to scope, push the current frame pointer (RBP) onto the stack and store the current stack pointer (RSP) as the new frame pointer (RBP).

Remember that we use the frame pointer as a point of reference when setting and getting local and parameter values. It works out entirely by convention.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);
  }

Next we copy the parameters into local scope at their negative (from the frame pointer) location. In the future we may decide to actually copy in the parameter values into the local stack but for now there's no benefit.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });
  }

Next we'll compile the body of the function within a begin block.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });

    // Pass childScope in for reference when body is compiled.
    this.compileBegin(body, childScope, depth);
  }

Then in the postlude we'll pop the stack (for the return result of the begin form), save it in RAX, pop the previous frame pointer back to restore the calling frame, and return.

  compileDefine([name, params, ...body], scope, depth) {
    // Add this function to outer scope
    const safe = scope.assign(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    this.emit(0, `${safe}:`);
    this.emit(depth, `PUSH RBP`);
    this.emit(depth, `MOV RBP, RSP\n`);

    // Copy params into local scope
    params.forEach((param, i) => {
      childScope.map[param] = -1 * (params.length - i - 1 + 2);
    });

    // Pass childScope in for reference when body is compiled.
    this.compileBegin(body, childScope, depth);

    // Save the return value
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `POP RBP\n`);

    this.emit(depth, 'RET\n');
  }

And now we're ready to compile a simple program!

Our first program

Here's a simple one we can support:

$ cat tests/meaning-of-life.lisp
(def main ()
  (+ 8 (* 2 17)))

We'll compile this program without the ulisp kernel (which contains a lisp library we cannot currently compile):

$ node src/ulisp.js tests/meaning-of-life.lisp --no-kernel --backend x86
$ ./build/prog
$ echo $?
42

Not bad. Let's finish up with support for prepareLogicalWrappers, prepareSyscallWrappers, and compileIf.

prepareLogicalWrappers

Storing logical results as values is a bit of pain. Most of the internet wants you to use branching. And a better compiler may optimize an idiom like (if (> 5 2) ...) into a single branch.

But we're going to resort to an instruction I just learned about called CMOV. This allows us to conditionally assign a value based on flags, similar to how you can conditionally branch.

Otherwise we'll follow a pattern similar to our arithmetic wrappers. At the end of the procedure we will have a 0 or a 1 on the top of the stack.

  prepareLogicalWrappers() {
    const prepareComparison = (operator) => {
      return {
        [operator]: (arg, scope, depth) => {
          depth++;
          this.emit(depth, `# ${operator}`);

          // Compile first argument, store in RAX
          this.compileExpression(arg[0], scope, depth);

          // Compile second argument
          this.compileExpression(arg[1], scope, depth);
          this.emit(depth, `POP RAX`);

          // Compile operation
          this.emit(depth, `CMP [RSP], RAX`);

          // Reset RAX to serve as CMOV* dest, MOV to keep flags (vs. XOR)
          this.emit(depth, `MOV RAX, 0`);

          // Conditional set [RSP]
          const operation = {
            '>': 'CMOVA',
            '>=': 'CMOVAE',
            '<': 'CMOVB',
            '<=': 'CMOVBE',
            '==': 'CMOVE',
            '!=': 'CMOVNE',
          }[operator];
          // CMOV* requires the source to be memory or register
          this.emit(depth, `MOV DWORD PTR [RSP], 1`);
          // CMOV* requires the dest to be a register
          this.emit(depth, `${operation} RAX, [RSP]`);
          this.emit(depth, `MOV [RSP], RAX`);
          this.emit(depth, `# End ${operator}`);
        },
      };
    };

    return {
      ...prepareComparison('>'),
      ...prepareComparison('>='),
      ...prepareComparison('<'),
      ...prepareComparison('<='),
      ...prepareComparison('=='),
      ...prepareComparison('!='),
    }
  }

prepareSyscallWrappers

This helper is similar to compileCall except for that it needs to follow the SYS V ABI and use the SYSCALL instruction rather than CALL.

  prepareSyscallWrappers() {
    const registers = ['RDI', 'RSI', 'RDX', 'R10', 'R8', 'R9'];

    const wrappers = {};
    Object.keys(SYSCALL_MAP).forEach((key, obj) => {
      wrappers[`syscall/${key}`] = (args, scope, depth) => {
        if (args.length > registers.length) {
          throw new Error(`Too many arguments to syscall/${key}`);
        }

        // Compile first
        args.forEach((arg) => this.compileExpression(arg, scope, depth));

        // Then pop to avoid possible register contention
        args.forEach((_, i) =>
          this.emit(depth, `POP ${registers[args.length - i - 1]}`),
        );

        this.emit(depth, `MOV RAX, ${SYSCALL_MAP[key]}`);
        this.emit(depth, 'SYSCALL');
        this.emit(depth, `PUSH RAX`);
      };
    });

    return wrappers;
  }

And we're set! Last up is compileIf.

compileIf

This is standard code generation but gets a little tricky due to our stack commitments. Testing must pop the test value off the stack. And then/else blocks must push a value onto the stack (even if there is no else block).

Here is an example we'd like to support:

(if (foo)
  (do-bar))

We compile the test and branch:

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);
  }

Then we compile the then block and jump to after the else block afterward.

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);

    // Compile then section
    this.emit(depth, `# If then`);
    this.compileExpression(then, scope, depth);
    this.emit(depth, `JMP .after_${branch}\n`);
  }

Finally we compile the else block if it exists, and otherwise we push a zero onto the stack (possibly to represent null).

  compileIf([test, then, els], scope, depth) {
    this.emit(depth, '# If');
    // Compile test
    this.compileExpression(test, scope, depth);
    const branch = `else_branch` + GLOBAL_COUNTER++;
    // Must pop/use up argument in test
    this.emit(0, '');
    this.emit(depth, `POP RAX`);
    this.emit(depth, `TEST RAX, RAX`);
    this.emit(depth, `JZ .${branch}\n`);

    // Compile then section
    this.emit(depth, `# If then`);
    this.compileExpression(then, scope, depth);
    this.emit(depth, `JMP .after_${branch}\n`);

    // Compile else section
    this.emit(depth, `# If else`);
    this.emit(0, `.${branch}:`);
    if (els) {
      this.compileExpression(els, scope, depth);
    } else {
      this.emit(1, 'PUSH 0 # Null else branch');
    }
    this.emit(0, `.after_${branch}:`);
    this.emit(depth, '# End if');
  }

And we're ready for an interesting program! Let's print (to stdout) the result of fib(20).

Fibonacci

$ cat ./tests/fib.lisp
(def fib (n)
     (if (< n 2)
       n
       (+ (fib (- n 1)) (fib (- n 2)))))

(def main ()
     (print (fib 20)))

And check out the kernel:

$ cat ./lib/kernel.lisp
(def print-char (c)
     (syscall/write 1 &c 1))

(def print (n)
     (if (> n 9)
       (print (/ n 10)))

     (print-char (+ 48 (% n 10))))

Compile and run it:

$ node src/ulisp.js tests/fib.lisp --backend x86
$ ./build/prog
6765

And we're in business!

Latest post in the compiler basics series: an x86 upgrade. We've got basic syscall support, "infinite" locals and parameters, and if/else. More than enough to handle printing integers to stdout and recursive fibonacci. https://t.co/B3OV0vEX1V
— Phil Eaton (@phil_eaton) December 8, 2019

Nov 2019 Recap

swyx's site RSS Feed — Tue, 03 Dec 2019 02:00:00 GMT

Hello mailing list friends!

The Great British Baking Show

Tilting at Windmills — Mon, 02 Dec 2019 02:00:00 GMT

Just what is it about The Great British Baking Show? (or The Great British Bake-Off as it's called in its homeland.) In case you haven't…

Learning to use emacs

Ben Overmyer's Site — Mon, 02 Dec 2019 02:00:00 GMT

I've been a vim user for somewhere around 10 years. While I'm comfortable enough in it, my knowledge and practice of it are still relatively amateurish. For a long time now I have heard others sing the praises of emacs, and since I'm trying to expand my horizons, it sounded like a good idea to try and pick it up.

I'm writing this blog post in emacs. Where possible, emacs is the first editor that I reach for, even though I know I could accomplish my goals faster and with less pain if I stuck to what I know. It's an interesting experience.

I like that emacs is more natural-feeling than vim. The key incantations to do things are relatively intuitive. However, there is a lot of memorization I'm going to need to do in order to be effective with it.

This month will be my trial period. By the end of December, I'll have figured out whether I want to continue using emacs, or switch back to vim.

Warren Buffett style fundamental metrics of long-term company performance

Bytepawn - Marton Trencseni — Mon, 02 Dec 2019 01:00:00 GMT

I look at some fundamental charts of Apple, Activision Blizzard and Intel.

Confusion and disengagement in meetings

Notes on software development — Sat, 30 Nov 2019 02:00:00 GMT

The quickest way to cut through confusion or disagreement among otherwise amiable and honest folks is to ask questions.

Ask early so you don't waste time. But it's not enough to just ask clarifying questions because the answers won't always be clear.

Sounds like Human Interaction 101, and maybe it is. These techniques show up more when discussing outcomes and very rarely when discussing assumptions.

Meetings are called to discuss outcomes, not assumptions. But assumptions almost always need to be called into question too.

If you have clarity personally but you observe confusion and disengagement, ask questions and summarize. Someone must be aware of the group and be willing to sound dumb.

If you aren't aware of confusion or disengagement, start paying attention. Addressing doesn't need to be hard and is personally meaningful.

Address confusion and disengagement in meetings by asking questions and summarizing, whether you're confused or not. Question outcomes _and_ assumptions. https://t.co/2OPifEBSq5
— Phil Eaton (@phil_eaton) December 1, 2019

Calibration curves for delivery prediction with Scikit-Learn

Bytepawn - Marton Trencseni — Thu, 21 Nov 2019 01:00:00 GMT

I show calibration curves for four different binary classification Scikit-Learn models we built for delivery prediction at Fetchr, trained using real-world data: LogisticRegression, DecisionTree, RandomForest and GradientBoosting.

Using simulated self-play to solve all OpenAI Gym classic control problems with Pytorch

Bytepawn - Marton Trencseni — Thu, 14 Nov 2019 01:00:00 GMT

I use simulated self-play by ranking episodes by summed reward. Game outcomes are divided in two by cutting at the median, winners are assigned +1 rewards, losers are assigned -1 rewards, like in games like Go and Chess. Unlike naive policy gradient descent used in previous posts, this version solves all OpenAI classic control problems, albeit slowly.

An interview on what makes Postgres unique (extensions)

CRAIG KERSTIENS — Wed, 13 Nov 2019 22:55:56 GMT

I’ve been at dinners before with developers that admitted developers, themselves included, can be a bit opinionated. In one case one said for example, “I love Postgres, but I have no idea why.” They were sitting at the wrong table to use Postgres as an example… But it is quite often that I am asked Why Postgres.

In fact a little over a year ago good friend Dimitri Fontaine asked if he could interview me for a book he’s working on for Postgres. I’ve long said their is a shortage of good books about Postgres and he’s done a great job with his in providing a guide targetted at developers, not just DBAs, that want to become better with their database. What follows is the excerpt of the interview from the book. And if you’re interested in picking up a copy he was friendly enough to share a discount code you can find below.

Intro

Craig heads up the Cloud team @citusdata now running product for Azure Postgres since being acquired by Microsoft. Citus extends Postgres to be a horizontally scalable distributed database. If you have a database, especially Postgres, that needs to scale beyond a single node (typically at 100GB and up) Craig is always happy to chat and see if Citus can help.

Previously Craig has spent a number of years @heroku, a platform-as-a-service, which takes much of the overhead out of IT and lets developers focus on building features and adding value. The bulk of Craig’s time at Heroku was spent running product and marketing for Heroku Data.

In your opinion, how important are extensions for the PostgreSQL open source project and ecosystem?

To me the extension APIs and growing ecosystem of extensions are the biggest advancement to Postgres in probably the last 10 years. Extensions have allowed Postgres to extend beyond a traditional relational database to much more of a data platform. Whether it’s the initial NoSQL datatypes (if we exclude XML that is) in hstore, to the rich feature set in geospatial with GIS, or approximation algorithms such as HyperLogLog or TopN you have extensions that now by themselves take Postgres into a new frontier.

Extensions allow the core to move at a slower pace, which makes sense. Each new feature in core means it has to be thoroughly tested and safe. That’s not to say that extensions don’t, but extensions that can exist outside core, then become part of the contrib provide a great on ramp for things to move much faster.

What are your favorite PostgreSQL extensions, and why?

My favorite three extensions are:

pg_stat_statements

Citus

HyperLogLog

pg_stat_statements is easily the most powerful extension for an application developer without having to understand deep database internals to get insights to optimize their database. For many application developers the database is a black box, but pg_stat_statements is a great foundation for AI for your database that I only expect to be improved upon in time.

Citus: I’m of course biased because I work there, but I followed Citus and pg_shard for 3 years prior to joining. Citus turns Postgres into a horizontally scalable database. Under the covers it’s sharded, but application developers don’t have to think or know about that complexity. With Citus Postgres is equipped to tackle larger workloads than ever before as previously Postgres was constrained to a single box or overly complicated architectures.

HyperLogLog: I have a confession to make. In part I just love saying it, but it also makes you seem uber-intelligent when you read about the algorithm itself. “K minimum value, bit observable patterns, stochastic averaging, harmonic averaging.” I mean who doesn’t want to use something with all those things in it? In simpler terms, it’s close enough approximate uniques that are compose-able with a really small footprint on storage. If you’re building something like a web analytics tool HyperLogLog is an obvious go to.

How do you typically find any extension you might need? Well, how do you know you might need a PostgreSQL extension in the first place?

pgxn.org and github are my two go-tos. Though Google also tends to work pretty well. And of course I stay up to date on new ones via PostgresWeekly.com.

Though in reality I often don’t always realize I need one. I search for the problem I’m trying to solve and discover it. I would likely never search for HyperLogLog, but a search for Postgres approximate count or approximate distincts would yield it pretty quickly.

Is there any downside you could think of when your application code base now relies on some PostgreSQL extension to run? I could think of extension’s availability in cloud and SaaS offerings, for instance.

It really depends. There are extensions that are much more bleeding edge, and ones that are more mature. Many of the major cloud providers support a range of extensions, but they won’t support any extension. If they do support it there isn’t a big downside to leveraging it. If they don’t you need to weigh the cost of running and managing Postgres yourself vs. how much value that particular extension would provide. As with all things managed vs. not, there is a trade-off there and you need to decide which one is right for you.

Though if something is supported and easy to leverage wherever you run, by all means, go for it.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Applying policy gradient to OpenAI Gym classic control problems with Pytorch

Bytepawn - Marton Trencseni — Tue, 12 Nov 2019 01:00:00 GMT

I try to generalize the policy gradient algorithm as introduced earlier to solve all the OpenAI classic control problems. It works for CartPole and Acrobot, but not for Pendulum and MountainCar environments.

Swyx Faves: Software Engineering Daily

swyx's site RSS Feed — Mon, 11 Nov 2019 02:00:00 GMT

> "How to become a 10x engineer: listen to @softwaredaily everyday." - Amjad Masad

Narrow Waists

swyx's site RSS Feed — Sun, 10 Nov 2019 02:00:00 GMT

When talking about "Narrow Waists" I should clarify that I'm only referring to the term from the somewhat obscure "Internet Architecture" model of the different technology layers (my notes here):

The Perfect Restaurant

swyx's site RSS Feed — Tue, 05 Nov 2019 02:00:00 GMT

This is a rant of uninformed thoughts and theories about running a restaurant.

The Three F's of Open Source Development

Ben E. C. Boyter — Mon, 04 Nov 2019 02:00:00 GMT

I write and publish open source code. I quite often get bug fixes, patches, requests and questions regarding the things I have published. Most of the feedback is respectful and helpful. However occasionally there are other requests. This page is where I plan to redirect them as a final warning before I block and ignore them.

The three F’s to open source development.

Fix it, Fork it, Fuck off

Fix it

If you find a bug, notice missing, incorrect or misleading documentation, or want an enhancement, your first option is to make a request to fix it. Raise the issue in the bug/request tracker of choice. Heck even a tweet at the owner might be good enough. However that is not your only option. You can choose submit a patch after you fix it yourself, or if you lack the skills, pay someone else to do so. You could also offer to pay the maintainer to resolve this for you. If there is enough value for you to raise the issue, perhaps there is enough value to pay to resolve it?

Oct 2019 Recap

swyx's site RSS Feed — Sat, 02 Nov 2019 02:00:00 GMT

Hello mailing list friends!

Linux Security Summit Europe 2019

Linux Engineer's random thoughts — Thu, 31 Oct 2019 17:00:00 GMT

Following my ELCE notes, you'll find below my notes for some of the talks the Linux Security Summit Europe 2019.

Exploiting race conditions using the scheduler — Jann Horn

The first bug Jann talked about is a race between mremap and fallocate. mremap allows moving a memory mapping from an address …

Sloc Cloc and Code - Can a crusty Go program outperform a well written Rust Project?

Ben E. C. Boyter — Thu, 31 Oct 2019 02:00:00 GMT

For those that wish to follow along with the presentation use the below link,

https://boyter.org/static/gophercon-syd-presentation/

You can press S to see the speaker notes as you go along.

ELCE, OS Summit and KVM Forum 2019 notes

Linux Engineer's random thoughts — Thu, 31 Oct 2019 01:00:00 GMT

After a two years hiatus, I couldn't miss this year's ELCE happening in Lyon, the first in France since 10 years ago in Grenoble, which coincidentally was one of my first conferences. OS Summit and KVM forum events are co-located and with the same ticket, which is nice, and why …

Interesting Upcoming pgDays

CRAIG KERSTIENS — Tue, 29 Oct 2019 22:55:56 GMT

I’ve been to a lot of conferences over the years. PgConf EU, PostgresOpen, too many pgDays to count, and even more none Postgres conferences (OSCON, Strangeloop, Railsconf, PyCon, LessConf, and many more). I’ve always found Postgres conferences one of the best places to get training and learn about what’s new with Postgres (in addition to Dimitri’s recent book, more on that below). They’re my regular stop to catch up on all the new features of a release before it comes out, and often there is a talk highlighting what is new with a simple easy to understand summary once released.

I just got back from PGConf EU a little over a week ago and it was a great time. I’m sure we’ll see some rundowns of it start appearing on Postgres planet. But, as far as I’m concerned PGConf EU is in the past (unless your counting next year which is in Berlin-in which case I’ll see you there). For me it’s time to look to the future and there are a number of upcoming pgDays I’m looking forward to.

The first two I want to highlight are separate events, but you’ll notice they’re scheduled nicely for you to easily attend both. With a day in between for travel you’ll find that many speakers and attendees depart one and head straight to the other. It makes for an easy opportunity to visit two cities, see two different communities, and yet not have to spend too much time traveling. The first is Nordic pgDay in Helsinki. It’s coming up on March 24. The second is pgDay Paris on March 26. Both of these are great single track conferences. If you’re in Europe of fancy a trip to Europe I recommend giving them a look, and even better the CFPs are open so consider submitting.

Another pgDay I have to mention is right in my backyard. pgDay SF I’m particularly excited for a few reasons:

San Francisco is very much a central tech hub, which means a great chance of learning from folks at many many interesting tech companies in attendance
Just like Nordic and Paris this is a single track conference, which I’m a personal fan of because you can have continuity between talks and shared conversation with other attendees
The venue! If you’re not from the bay area you may not be aware, but Swedish American Hall is well known music venue within SF. It’s had many famous artists over the years and now pgDay SF joins the ranks.

This isn’t an exhaustive list of course, just a few on my personal list that I hope to make it to. If you’re there and see me make sure to say hi!

Machine Learning at Fetchr

Bytepawn - Marton Trencseni — Tue, 29 Oct 2019 01:00:00 GMT

Opportunities for automating, optimizing and enabling processes with ML at a delivery company such as Fetchr are plentiful. We put three families of ML models into production. These 3 areas are: Scheduling, Notifications and Operational choice.

Private Conversations are Private

swyx's site RSS Feed — Sun, 27 Oct 2019 03:00:00 GMT

I recently made a mistake. I make many, but this involved someone important to me and to people I know, so it stands out among the general cacophony of my many other failures. I wanted to own up to what I did, explain how I handled it, and in general take ownership of the problem.

Ensembles vs Committees

swyx's site RSS Feed — Sat, 26 Oct 2019 03:00:00 GMT

A useful dichotomy for team dynamics.

Solving the CartPole Reinforcement Learning problem with Pytorch

Bytepawn - Marton Trencseni — Tue, 22 Oct 2019 01:00:00 GMT

The CartPole problem is the Hello World of Reinforcement Learning, originally described in 1985 by Sutton et al. The environment is a pole balanced on a cart. CartPole is one of the environments in OpenAI Gym, so we don't have to code up the physics. Here I walk through a simple solution using Pytorch.

Donating $10k to FreeCodeCamp

swyx's site RSS Feed — Sun, 20 Oct 2019 03:00:00 GMT

I benefited from FreeCodeCamp and would like to give back

What are you working on ?

Linux Engineer's random thoughts — Sun, 20 Oct 2019 01:00:00 GMT

This was previously published on LinkedIn

Until very recently I couldn’t really answer this question. But it has shipped, so now I can tell you: I've been working on Edge Computing.

To be more precise, this is the first compute service available to ISP subscribers directly on their ISP …

Django Docker Box is now an official Django project!

Tom Forbes — Fri, 18 Oct 2019 00:40:54 GMT

When I was working on adding queryset.explain() to Django I got annoyed by how complex it was to set up a local Django environment with multiple databases and versions. The traditional way of handling this was to use django-box which utilizes Vagrant to spin up a VM and install different types of da...

Testing my dotfiles with Github actions

Tom Forbes — Sat, 12 Oct 2019 21:01:55 GMT

I went through a phase that involved reinstalling MacOS way, way too often. At a certain point I got fed up with installing things over and over again and decided to version my dotfiles. And so https://github.com/orf/dotfiles was born. This reolved around a bootstrap.sh script that clones your dotfi...

Elasticsearch mappings - How to add Facets and Aggregations

Ben E. C. Boyter — Sat, 12 Oct 2019 03:00:00 GMT

Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub

In elasticsearch mappings define type and values in documents. You use them to specify that fields within your document should be treated as numbers, dates, geo-locations and whatever other types elasticsearch supports. You can also define the stemming algorithm used and other useful index fields.

Interpreting Go

Notes on software development — Sat, 12 Oct 2019 03:00:00 GMT

After spending some time at work on tooling for keeping documentation in sync with Go struct definitions I had enough exposure to Go's built-in parsing package that next steps were clear: write an interpreter. It's a great way to get more comfortable with a language's AST.

In this post we'll use the Go parser package to interpret the AST directly (as opposed to compiling to a bytecode VM) with enough to support a recursive implementation of the fibonacci algorithm:

package main

func fib(a int) int {
  if a == 1 {
    return 0
  }

  if a == 2 {
    return 1
  }

  return fib(a-1) + fib(a-2)
}

func main() {
  println(fib(15))
}

You'll note this isn't actually valid Go because we are using an undefined function println. We'll provide that for the runtime to make things easier on ourselves.

The fibonacci algorithm is my goto minimal program that forces us to deal with basic aspects of:

Function definitions
Function calls
Function arguments
Function return values
If/else
Assignment
Arithmetic and boolean operators

We'll do this in around 200 LoC. Project code is available on Github.

A follow-up post will cover support for an iterative fibonacci implementation with support for basic aspects of:

Local variables
Loops

First steps

I always start exploring an AST by practicing error-driven development. It's helpful to have the Go AST, parser, and token package docs handy as well.

We'll focus on single-file programs and start with parser.ParseFile. This function will return an *ast.File. This in turn contains a list of Decls. Unfortunately Go stops being helpful at this point because we have no clue what is going to implement this Decl interface. So we'll switch on the concrete type and error until we know what we need to know.

package main

import (
  "go/ast"
  "go/parser"
  "go/token"
  "io/ioutil"
  "log"
  "os"
  "reflect"
)

func interpret(f *ast.File) {
  for _, decl := range f.Decls {
    switch d := decl.(type) {
    default:
      log.Fatalf("Unknown decl type (%s): %+v", reflect.TypeOf(d), d)
    }
  }
}

func main() {
  fset := token.NewFileSet() // positions are relative to fset

  src, err := ioutil.ReadFile(os.Args[1])
  if err != nil {
    log.Fatalf("Unable to read file: %s", err.Error())
  }

  f, err := parser.ParseFile(fset, os.Args[1], src, 0)
  if err != nil {
    log.Fatalf("Unable to parse file: %s", err.Error())
  }

  interpret(f)
}

Build and run:

$ go build goi.go
$ ./goi fib.go
2019/10/12 09:43:48 Unknown decl type (*ast.FuncDecl): &{Doc:<nil> Recv:<nil> Name:fib Type:0xc000096320 Body:0xc00009a3c0}

Cool! This is the declaration of the fib function and its type is *ast.FuncDecl.

Interpreting ast.FuncDecl

A function declaration is going to need to add its name to a context map, mapped to a function reference for use in function calls. Since Go throws everything into the same context namespace this we can simply pass around a map of strings to values where a value can be any Go value. To facilitate this, we'll define a value struct to hold an integer to represent "kind" and an empty interface "value". When a value is referenced it will have to switch on the "kind" and then cast the "value".

Additionally, and unlike a value-oriented language like Scheme, we'll need to track a set of return values at all stages through interpretation so, when set, we can short circuit execution.

type kind uint

const (
  i64 kind = iota
  fn
  bl
)

type value struct {
  kind  kind
  value interface{}
}

type context map[string]value

func (c context) copy() context {
  cpy := context{}
  for key, value := range c {
    cpy[key] = value
  }

  return cpy
}

type ret struct {
  set bool
  vs  []value
}

func (r *ret) setValues(vs []value) {
  r.vs = vs
  r.set = true
}

func interpretFuncDecl(ctx context, r *ret, fd *ast.FuncDecl) {
  ctx[fd.Name.String()] = value{
    fn,
    func(ctx context, r *ret, args []value) {},
  }
}

func interpret(ctx context, f *ast.File) {
  for _, decl := range f.Decls {
    switch d := decl.(type) {
    case *ast.FuncDecl:
      interpretFuncDecl(ctx, nil, d)
    default:
      log.Fatalf("Unknown decl type (%s): %+v", reflect.TypeOf(d), d)
    }
  }
}

Now that we have the idea of return management and contexts set out, let's fill out the actual function declaration callback. Inside we'll need to copy the context so variables declared inside the function are not visible outside. Then we'll iterate over the parameters and map them in context to the associated argument. Finally we'll interpret the body.

func interpretBlockStmt(ctx context, r *ret, fd *ast.BlockStmt) {}

func interpretFuncDecl(ctx context, r *ret, fd *ast.FuncDecl) {
  ctx[fd.Name.String()] = value{
    fn,
    func(ctx context, r *ret, args []value) {
      childCtx := ctx.copy()
      for i, param := range fd.Type.Params.List {
        childCtx[param.Names[0].String()] = args[i]
      }

      interpretBlockStmt(childCtx, r, fd.Body)
    },
  }
}

And we'll add a call to the interpreted main to the end of the interpreter's main:

func main() {
  fset := token.NewFileSet() // positions are relative to fset

  src, err := ioutil.ReadFile(os.Args[1])
  if err != nil {
    log.Fatalf("Unable to read file: %s", err.Error())
  }

  f, err := parser.ParseFile(fset, os.Args[1], src, 0)
  if err != nil {
    log.Fatalf("Unable to parse file: %s", err.Error())
  }

  ctx := context{}
  interpret(ctx, f)
  var r ret
  ctx["main"].value.(func(context, *ret, []value))(ctx, &r, []value{})
}

Next step!

Interpreting ast.BlockStmt

For this AST node, we'll iterate over each statement and interpret it. If the return value has been set we'll execute the loop to short circuit execution.

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {}

func interpretBlockStmt(ctx context, r *ret, bs *ast.BlockStmt) {
  for _, stmt := range bs.List {
    interpretStmt(ctx, r, stmt)
    if r.set {
      return
    }
  }
}

Next step!

Interpreting ast.Stmt

Since ast.Stmt is another interface, we're back to error-driven development.

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  switch s := stmt.(type) {
  default:
    log.Fatalf("Unknown stmt type (%s): %+v", reflect.TypeOf(s), s)
  }
}

And the trigger:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:15:14 Unknown stmt type (*ast.ExprStmt): &{X:0xc0000a02c0}

Great! Checking the docs on ast.ExprStmt we'll just skip directly to a call to a new function interpretExpr:

func interpretExpr(ctx context, r *ret, expr ast.Expr) {}

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  switch s := stmt.(type) {
  case *ast.ExprStmt:
    interpretExpr(ctx, r, s.X)
  default:
    log.Fatalf("Unknown stmt type (%s): %+v", reflect.TypeOf(s), s)
  }
}

Moving on!

Interpreting ast.Expr

We've got another interface. Let's error!

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

And the trigger:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:19:16 Unknown expr type (*ast.CallExpr): &{Fun:println Lparen:146 Args:[0xc0000a2280] Ellipsis:0 Rparen:154}

Cool! For a call we'll evaluate the arguments, evaluate the function expression itself, cast the resulting value to a function, and call it.

func interpretCallExpr(ctx context, r *ret, ce *ast.CallExpr) {
  var fnr ret
  interpretExpr(ctx, &fnr, ce.Fun)
  fn := fnr.values[0]

  values := []value{}
  for _, arg := range ce.Args {
    var vr ret
    interpretExpr(ctx, &vr, arg)
    values = append(values, vr.values[0])
  }

  fn.value.(func(context, *ret, []value))(ctx, r, values)
}

All of this casting is unsafe because we aren't doing a type-checking stage. But we can ignore this because if a type-checking stage were introduced (which it need to be at some point), it would prevent bad casts from happening.

Handling more ast.Expr implementations

Let's give the interpreter a shot again:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:28:17 Unknown expr type (*ast.Ident): println

We'll need to add ast.Ident support to interpretCallExpr. This will be a simple lookup in context. We'll also add a setValue helper since the ret value is serving double-duty as a value passing mechanism and also a function's return value (solely where multiple value are a thing).

...


func (r *ret) setValue(v value) {
  r.values = []value{v}
  r.set = true
}

...

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.CallExpr:
    interpretCallExpr(ctx, r, e)
  case *ast.Ident:
    r.setValue(ctx[e.Name])
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

This is also a good time to add the println builtin to our top-level context.

func main() {
  ...

  ctx := context{}
  interpret(ctx, f)
  ctx["println"] = value{
    fn,
    func(ctx context, r *ret, args []value) {
      var values []interface{}
      for _, arg := range args {
        values = append(values, arg.value)
      }

      fmt.Println(values...)
    },
  }

  var r ret
  ctx["main"].value.(func(context, *ret, []value))(ctx, &r, []value{})
}

More ast.Expr implementations

Running the interpreter again we get:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:41:59 Unknown expr type (*ast.BasicLit): &{ValuePos:151 Kind:INT Value:15}

Easy enough: we'll switch on the "kind" and parse a string int to an int and wrap it in our value type.

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.CallExpr:
    interpretCallExpr(ctx, r, e)
  case *ast.Ident:
    r.setValue(ctx[e.Name])
  case *ast.BasicLit:
    switch e.Kind {
    case token.INT:
      i, _ := strconv.ParseInt(e.Value, 10, 64)
      r.setValue(value{i64, i})
    default:
      log.Fatalf("Unknown basiclit type: %+v", e)
    }
  default:
    log.Fatalf("Unknown expr type (%s): %+v", reflect.TypeOf(e), e)
  }
}

Now we run again:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:48:46 Unknown stmt type (*ast.IfStmt): &{If:38 Init:<nil> Cond:0xc0000ac150 Body:0xc0000ac1b0 Else:<nil>}

Cool, more control flow!

Interpreting ast.IfStmt

For ast.IfStmt we interpret the condition and, depending on the condition, interpret the body or the else node. In order to make empty else interpreting easier, we'll also add a nil short-circuit to interpretStmt.

func interpretIfStmt(ctx context, r *ret, is *ast.IfStmt) {
  interpretStmt(ctx, nil, is.Init)

  var cr ret
  interpretExpr(ctx, &cr, is.Cond)
  c := cr.valus[0]

  if c.value.(bool) {
    interpretBlockStmt(ctx, r, is.Body)
    return
  }

  interpretStmt(ctx, r, is.Else)
}


func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  if stmt == nil {
    return
  }

  switch s := stmt.(type) {
  case *ast.IfStmt:
    interpretIfStmt(ctx, r, s)

  ...

Let's try it out:

$ go build goi.go
$ ./goi fib.go
2019/10/12 10:56:28 Unknown expr type (*ast.BinaryExpr): &{X:a OpPos:43 Op:== Y:0xc00008a120}

Great!

Interpreting ast.BinaryExpr

An ast.BinaryExpr has an Op field that we'll switch on to decide what operations to do. We'll interpret the left side and then the right side and finally perform the operation and return the result. The three binary operations we use in this program are ==, + and -. We'll look these up in go/token docs to discover the associated constants.

func interpretBinaryExpr(ctx context, r *ret, bexpr *ast.BinaryExpr) {
  var xr, yr ret
  interpretExpr(ctx, &xr, bexpr.X)
  x := xr.values[0]
  interpretExpr(ctx, &yr, bexpr.Y)
  y := yr.values[0]

  switch bexpr.Op {
  case token.ADD:
    r.setValue(value{i64, x.value.(int64) + y.value.(int64)})
  case token.SUB:
    r.setValue(value{i64, x.value.(int64) - y.value.(int64)})
  case token.EQL:
    r.setValue(value{bl, x.value.(int64) == y.value.(int64)})
  default:
    log.Fatalf("Unknown binary expression type: %+v", bexpr)
  }
}

func interpretExpr(ctx context, r *ret, expr ast.Expr) {
  switch e := expr.(type) {
  case *ast.BinaryExpr:
    interpretBinaryExpr(ctx, r, e)

  ...

Let's try one more time!

$ go build goi.go
$ ./goi fib.go
2019/10/12 11:06:19 Unknown stmt type (*ast.ReturnStmt): &{Return:94 Results:[0xc000070540]}

Awesome, last step.

Interpreting ast.ReturnStmt

Based on the ast.ReturnStmt definition we'll have to interpret each expression and set all of them to the ret value.

func interpretReturnStmt(ctx context, r *ret, s *ast.ReturnStmt) {
  var values []value
  for _, expr := range s.Results {
    var r ret
    interpretExpr(ctx, &r, expr)
    values = append(values, r.values[0])
  }

  r.setValues(values)

  return
}

func interpretStmt(ctx context, r *ret, stmt ast.Stmt) {
  if stmt == nil {
    return
  }

  switch s := stmt.(type) {
  case *ast.ReturnStmt:
    interpretReturnStmt(ctx, r, s)

  ...

And let's try one last time:

$ go build goi.go
$ ./goi fib.go
377

Looking good. :) Let's try with another input:

$ cat fib.go
package main

func fib(a int) int {
  if a == 1 {
    return 0
  }

  if a == 2 {
    return 1
  }

  return fib(a-1) + fib(a-2)
}

func main() {
  println(fib(14))
}
$ ./goi fib.go
233

We've got the basics of an interpreter for Golang.

Here's a blog post on building a simple AST interpreter for Go to support running a recursive fibonacci implementation https://t.co/5Zz388d8ZN
— Phil Eaton (@phil_eaton) October 12, 2019

"No Code" Is A Lie

swyx's site RSS Feed — Fri, 11 Oct 2019 03:00:00 GMT

Originally published on the Webflow blog.

Guidance for conducting offsites

CRAIG KERSTIENS — Thu, 10 Oct 2019 23:55:56 GMT

Offsites an invaluable tool in getting a team aligned. I’ve been a part of organizations where offsites never happened, and then when they happened at a regular interval. Just because offsites happened it didn’t mean they had the same significant impact to alignment and ability to execute moving forward. What follows is a few key principles around conducting an impactful offsite.

Get out of the office

An offsite isn’t reserving a conference room for a full day and just sitting there meeting on special topics. A change of scenery is important, and nice scenery will impact your abiltiy to collaborate. One long time friend and colleague communicated to me about 10 years into his working career, that the thing he most values in a workplace is natural light. He remarked younger him wanted free food and drinks, now he finds him much happier and more productive with natural light above all other amenities. A similar environment for an offsite is very helpful.

The best option isn’t a hotel with a rented conference room, but rather an AirBnB with each person their own room, a large kitchen, and large living room. More on the why for each of those later.

From working with some teams that are often local and a portion of them remote, hosting the offsite within a 2 hr drive of your primary office can be good. You don’t waste the full day traveling, for remote team members they can also budget 1-2 days in the office to see other team members in addition to the time spent at the offsite.

Collaboration is key

Get rid of powerpoint presentations and someone presenting on a screen. That isn’t collaboration, that is a presentation. If you have a team of 8 and each person presents for an hour you haven’t collaborated for 8 hrs, you’ve been presented to for 7 and presented for 1. I personally don’t like team building exercises, but I’ll admit they can be useful here.

A presentation by the leadership at the offsite for a few minutes to frame things can be helpful. It’s key to not have a projector and folks staring at a screen. Instead having a few prepared notes and ability interactively discuss is great. Collaborating on a large notepad, or using sticky notes for ideas and brainstorming is great. One of my favorite things to do at an offsite is a gridding exercise.

2-3 days is ideal

You don’t need a full week offsite, but a 1 day offsite doesn’t really provide ideas to evolve and change over time. The ideal to me is 3 days where you can make some statements about ideas on day 1, and then drill deeper into those on day 2 or 3. Having the ability to do research on data and evaluate any plans you make is helpful. You also then can have good collaborative time, but also set aside time for contentious topics. If you cover a contentious topic and then jump right into collaboration without time to change your mindset you’re going to get a very different outcome. Breaking things up across days makes this way easier.

2-3 days also allows time for non work activities. Coming together as a team will let you work better in the future. Boardgames are a common favorite, video games are an occasional activity. Sometimes sitting by a fireplace and enjoying a nice glass of wine and just catching up.

Those moments can live on for a very long time. Many colleagues from years ago will recall how when playing “We didn’t play test this at all”, we brought back online an archived server intentionally to cause a page so an engineer would pick up his phone, only for one of us to play a card causing him to lose the game due to holding his phone. Those stories create a foundation and shared experience to build on in the future for how you work together

This takes me onto the next point…

Food

Too often offsites are a day of meetings in a conference room, followed by dinner out at a nice restaurant. The exact opposite works better. We’ve already talked about location. But food should be collaborative as well. Over 2-3 days you may do one dinner out, but even that isn’t the norm. Instead make a meal list ahead of time, and people spend time in the kitchen together. You’ll typically find you have 1-2 more cooks in the team than others, but then you have a lot of helpers.

It actually reminds me a bit of my wedding reception back home. I smoked about 80 lbs of meat for the reception. As we were taking the meat off, I walked back in to 4 generations of in-laws, the older ones teaching the younger ones how to pull pork.

Cooking together is another form of collaborating. You can easily learn about cultures and backgrounds. I didn’t realize melon with proscuitto was always supposed to be server with port if you’re in France, but now I can eat it no other way. Splitting up the duties so that 2-3 help cook a meal means you don’t spend all your time cooking, but you get people working together.

Don’t steal a weekend

I mentioned 2-3 days is ideal. Please don’t use one of those days as a weekend. As the leader or coordinator of such you may not have weekend obligations and happy to give up a day of it. But many on your team may have other commitements and such, an offsite is valuable, but asking people to give up their weekend to essentially do more work isn’t a great look.

Offsites for everyone

Offsites aren’t just for management. The folks in boots on the ground doing the work has as much need and value as the leadership team sitting there determining the next 5 year strategy. Allowing engineers that are building things, supporting customers, answering pages to brainstorm and collaborate on how to more effectively design the systems and move more quickly will pay huge dividends. You’ll find experiments, and then new tools and systems that allow you to ship things faster with higher quality. Collaborating with the product team is key as well, an engineering offsite in isolation leaves out the voice of the customer. A product offsite without engineering leaves you daydreaming of solutions without being grounded in the reality of what is possible.

How Elasticsearch Stores Documents

Ben E. C. Boyter — Thu, 10 Oct 2019 03:00:00 GMT

Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub

It is useful to understand is how elasticsearch stores documents. Documents that are indexed need to go into an index and have a type. Indexes can contain one or more types. This may sound limiting but you can search over all indexes or all types within an index or just one type within an index if you require.

API Design: Modifying Defaults

swyx's site RSS Feed — Thu, 10 Oct 2019 03:00:00 GMT

> This is a quick note on a API Design. I hope to make this an ongoing series.

My list of useful command line tools

Ben E. C. Boyter — Wed, 09 Oct 2019 03:00:00 GMT

A post that is constantly a work in progress.

ripgrep - Similar to grep, but a drop in replacement for large file scanning without unicode penalities. It’s also similar to Ag and Ack and optimised for searching code, ignoring .git directories and the like. https://github.com/burntsushi/ripgrep
tmux - Terminal multiplexer. I wrote a 5 min guide to this. Use it to have resumable sessions after closing ssh sessions and to have split pane windows.
bat - Replacement for cat but provides syntax highlighting and line numbers. Can work as a drop in replacement for cat as well as it detects redirects and removes the fancy output. https://github.com/sharkdp/bat
fd - Better than find. Recursively looks for whatever it is you need using regular expressions and like ripgrep ignores .git directories and the like. https://github.com/sharkdp/fd
fzf - Fuzzy finder. Hard to explain why this is so useful. Suggest checking it out https://github.com/junegunn/fzf
gron - Makes JSON files grepable by flattening the structure. Very useful when trying to find something in JSON quickly. https://github.com/TomNomNom/gron
hexyl - Hex view of files with highlighting and the like. Find BOM in files quickly! https://github.com/sharkdp/hexyl
hyperfine - A command line benchmark tool. Give it two commands hyperfine 'scc' 'tokei' to determine which one is faster. https://github.com/sharkdp/hyperfine
z - A shell script that enables quick jumping to your most recently used directories. For example I work with scc a lot and its directory is /c/Users/bboyter/Documents/Go/src/github.com/boyter/scc which is a pain to remember. Having changed to it a few times I can now type z scc and it takes me right there. A massive time saver. https://github.com/rupa/z/
zoxide - Similar to z above, but I prefer it over z these days https://github.com/ajeetdsouza/zoxide
stu - A small TUI application explorer for AWS S3 or any other S3 compatible storage. Very useful when used with localstack stu --endpoint-url http://localhost:4566 https://github.com/lusingander/stu

Ejectable Defaults

swyx's site RSS Feed — Sat, 05 Oct 2019 03:00:00 GMT

Borrowing heavily from Gatsby Themes to improve the Sapper developer experience, and a discussion of how Ejectable Defaults fits with the Zero Config movement

Sept 2019 Recap

swyx's site RSS Feed — Tue, 01 Oct 2019 03:00:00 GMT

Hello mailing list friends!

Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100

Ben E. C. Boyter — Mon, 30 Sep 2019 03:00:00 GMT

The command line tool I created Sloc Cloc and Code (scc) (which is now modified and maintained by many other excellent people) counts lines of code, comments and makes a complexity estimate for files inside a directory. The latter is something you need a good sample size to make good use of. The way it works is that it counts branch statements in code. However what does that actually mean? For example “This file has a complexity of 10” is not very useful without some context. To solve this issue I thought I would try to run scc at all the source code I could get my hands on. This would also allow me to see if there are any edge cases I didn’t consider in the tool itself. A brute force Q/A trial by fire.

Administering Kubernetes is hard

Notes on software development — Mon, 30 Sep 2019 03:00:00 GMT

Kubernetes is easy to use after some exposure; it's pretty convenient too. But it is super hard to set up.

eksctl is a good tool for folks who don't want to spend hours/days/weeks debugging VPC configuration in 1000s of lines of CloudFormation. None of the other tools seem to be that much easier to use (kops, kubeadm, etc.).

But even with EKS and eksctl you are constrained to Amazon Linux worker nodes. AMIs are practically impossible to discover.

I haven't spent much time with GKE.

And while eksctl operates on the right level for developers needing to administrate small/medium-sized systems, it... doesn't exist outside EKS.

It is unfortunate the only major container orchestration system is this complex to administer. The user-facing APIs are pretty solid and guide toward sustainable system design. It is really hard to see the value for most companies with medium-sized deployments tasked with administration. Among serious proprietary alternatives, sure, there's ECS and Google App Engine. But there's little advantage in existing Kubernetes user knowledge. The OSS alternatives don't have the adoption to seem like a good investment.

OpenStack's magnum or OpenShift seem like possible high-level providers for a generic environment. But neither are particularly known for stability.

In all, the ecosystem has gotten friendlier. There will probably be a time in the future (3-5 years from now?) when Kubernetes is fairly easy to administer.

I'd love to hear your thoughts and experiences administering Kubernetes.

I hope I hate this code one day

Tom Forbes — Sat, 28 Sep 2019 18:25:53 GMT

I remember the first program I built fully on my own: a music downloader. There was a site that would basically index random music files found on Google and present you with a way to search them. I guess they would periodically search google for things like intitle:index.of mp3 (which you can still ...

Friday Links 2019-09-27

infrequent oscillations — Fri, 27 Sep 2019 22:52:33 GMT

In which we copy an idea and post some links.

Git-Centric Workflow

swyx's site RSS Feed — Thu, 26 Sep 2019 03:00:00 GMT

Published on the Netlify Blog

Why I Won't Choose To Be A Manager

swyx's site RSS Feed — Thu, 26 Sep 2019 03:00:00 GMT

I recently decided that I will not choose to be a manager. This is how I arrived at that conclusion.

The World's Greatest Netlify Demo 2019

swyx's site RSS Feed — Tue, 24 Sep 2019 03:00:00 GMT

Note: this was published as the Complete Intro to Netlify on the Netlify Blog and the FreeCodeCamp Channel

Adverse Status Updates and Trust

swyx's site RSS Feed — Mon, 23 Sep 2019 03:00:00 GMT

I noticed an asymmetry between adverse status updates and trust that I figured it would be worth a quick comment on.

Europython 2019 Talk - Rewriting the Django autoreloader

Tom Forbes — Sun, 22 Sep 2019 22:38:42 GMT

Over two years ago I set out to try and improve Django’s autoreloader implementation. My PR was merged and the feature released in Django 2.2, and I thought I should at least mention it on my blog. The role of an autoreloader is conceptually pretty simple: When a developer changes a file your softwa...

Waterfall

swyx's site RSS Feed — Sun, 22 Sep 2019 03:00:00 GMT

The worlds of software, business, and music use the word "Waterfall" incredibly differently and they are completely ignorant of each other. I figured I would make a quick note to compare and contrast them!

The Checklist Manifesto – Atul Gawande

Dan Palmer — Sun, 22 Sep 2019 02:00:00 GMT

Not long after a recent one to one with my manager, discussing how we could improve our incident response process in engineering at Thread, I returned to my desk to find a copy of The Checklist Manifesto that he had kindly got for me.

This is less of a book review and more of some highlights that I wanted to pull out from the book. Going into it, I had already read about the effectiveness of checklists in preventing human error, particularly in commercial aviation and medicine, but the book still had some great points to make. I won’t mention on any of the evidence as the book goes into plenty of detail, I’m more interested in how checklists can be used and the effect they have had on various professions.

Redesigning my blog

Tom Forbes — Sat, 21 Sep 2019 23:28:03 GMT

I like Hugo a lot. It’s really fast and really simple. However it’s very much still evolving, and the theme I was using had not been updated in nearly a year. This meant that it broke with the new release of Hugo, and I was generally getting a bit sick of the design. Now I’m not a designer by any me...

Friday Links 20

infrequent oscillations — Fri, 20 Sep 2019 15:49:19 GMT

In which we copy an idea and post some links.

Your Site's Calling Card

swyx's site RSS Feed — Fri, 20 Sep 2019 03:00:00 GMT

> Note: this tutorial is now out of date - I don't pre-generate og:image cards anymore as it added too much time to my builds

A Better Way to Get Data

swyx's site RSS Feed — Sun, 15 Sep 2019 03:00:00 GMT

> Note: this was an unfinished draft, i published it anyway but it is incomplete

Why JavaScript Tooling Sucks

swyx's site RSS Feed — Sun, 15 Sep 2019 03:00:00 GMT

JavaScript Tooling is just too hard to use, and it's not your fault.

Bad Ways to Get Data

swyx's site RSS Feed — Sat, 14 Sep 2019 03:00:00 GMT

In working on my Sapper export library, I ran into a very peculiar problem: my pages were being generated a lot more slowly than expected:

Friday Links 2019-09-13

infrequent oscillations — Fri, 13 Sep 2019 17:07:35 GMT

In which we copy an idea and post some links.

Static Svelte: JavaScript Blogging with 93% less JavaScript

swyx's site RSS Feed — Fri, 13 Sep 2019 03:00:00 GMT

This blog now uses Svelte & Sapper as a static site generator, where it previously used React & Gatsby. This is achieved through Sapper's sapper export feature.

Using Next.js as a Static Site Generator for Netlify

swyx's site RSS Feed — Wed, 11 Sep 2019 03:00:00 GMT

How to deploy a Static Next.js Site to Netlify, and then upgrade it to use Dynamic Route Segments to become a full fledged Static Site Generator!

Building Authenticated Serverless Jamstack Apps with Gatsby and Netlify

swyx's site RSS Feed — Wed, 11 Sep 2019 03:00:00 GMT

Gatsby is more than a simple static site generator. It uses JavaScript to rehydrate Markup into a fully dynamic React app - which means you can use APIs to do all sorts of dynamic functionality!

GraphQL Variables and Persisted Queries

swyx's site RSS Feed — Mon, 09 Sep 2019 03:00:00 GMT

My mistakes and insights on GraphQL Variables and Persisted Queries.

r2wars 2019

Linux Engineer's random thoughts — Sun, 08 Sep 2019 01:00:00 GMT

You might have noticed I was at r2con 2019. This is my writeup for the r2wars 2019 challenge.

r2wars primer: a Core Wars-like game

In radare2, there's an intermediate language and VM called ESIL. It is used to emulate code, and supports many architectures. r2wars is built on top of …

Friday Links

infrequent oscillations — Fri, 06 Sep 2019 21:36:01 GMT

In which we copy an idea and post some links.

r2con 2019

Linux Engineer's random thoughts — Fri, 06 Sep 2019 01:00:00 GMT

I'm back in Barcelona for this year's edition of r2con. You can read my r2con 2018 report.

radare2 is debugging and reverse engineering toolkit. It's mostly used from the command line or through a programming interface (r2pipe) that is identical to the command line one.

Cutter

Cutter is the official …

Keyboard First

swyx's site RSS Feed — Thu, 05 Sep 2019 03:00:00 GMT

I enjoyed listening to Des and Paul's discussion of keyboard-first apps on their podcast today. They discussed the rise of the cmd+K UI paradigm in apps like:

Compilers Copout

swyx's site RSS Feed — Wed, 04 Sep 2019 03:00:00 GMT

Today I dropped out of the Compilers course I accidentally registered myself for. I'm not proud of it but there's too much I want to do and I know I'm just going to do a shit job at everything by spreading my self thin and also putting an intense gun to my head.

JAMstack for Indie Hackers

swyx's site RSS Feed — Mon, 02 Sep 2019 03:00:00 GMT

This article was published on Indie Hackers!

Book Poll 2019

swyx's site RSS Feed — Mon, 02 Sep 2019 03:00:00 GMT

I've been feeling like I've skewed too much on empty calories recently. So I ran this poll:

Unit testing C code with gtest

Notes on software development — Sat, 31 Aug 2019 03:00:00 GMT

This post covers building and testing a minimal, but still useful, C project. We'll use Google's gtest and CMake for testing C code. This will serve as a foundation for some upcoming posts/projects on programming Linux, userland networking and interpreters.

The first version of this post only included one module to test. The test/CMakeLists.txt would also only expose a single pass-fail status for all modules. The second version of this post extends the test/CMakeLists.txt to expose each test/*.cpp file as its own CMake test so that results are displayed by ctest per file. The second version also splits the original src/testy.c and include/testy/testy.h module into a widget and customer module to demonstrate the changes to the CMake configuration.

The "testy" sample project

In this project, we'll put source code in src/ and publicly exported symbols (functions, structs, etc.) in header files in include/testy/. There will be a main.c in the src/ directory. Tests are written in C++ (since gtest is a C++ testing framework) and are in the test/ directory.

Here's an overview of the source and test code.

This file has some library code that we should be able to test.

#include "testy/widget.h"

int private_ok_value = 2;

int widget_ok(int a, int b) {
  return a + b == private_ok_value;
}

This file handles exported symbols for widget code.

#ifndef _WIDGET_H_
#define _WIDGET_H_

int widget_ok(int, int);

#endif

src/customer.c

This file has some more library code that we should be able to test.

#include "testy/customer.h"

int customer_check(int a) {
  return a == 5;
}

include/testy/customer.h

This file handles exported symbols for customer code.

#ifndef _CUSTOMER_H_
#define _CUSTOMER_H_

int customer_check(int);

#endif

src/main.c

This is the entrypoint to a program built around libtesty.

#include "testy/customer.h"
#include "testy/widget.h"

int main() {
  if (widget_ok(1, 1)) {
    return customer_check(5);
  }

  return 0;
}

This is one of our test files. It registers test cases and uses gtest to make assertions. We need to wrap the testy/widget.h include in an extern "C" to stop C++ from name-mangling.

#include "gtest/gtest.h"

extern "C" {
#include "testy/widget.h"
}

TEST(widget, ok) {
  ASSERT_EQ(widget_ok(1, 1), 1);
}

TEST(testy, not_ok) {
  ASSERT_EQ(widget_ok(1, 2), 0);
}

You can see a good high-level overview of gtest testing utilities like ASSERT_EQ and TEST here.

test/customer.cpp

This is another one of our test files.

#include "gtest/gtest.h"

extern "C" {
#include "testy/customer.h"
}

TEST(customer, ok) {
  ASSERT_EQ(customer_check(5), 1);
}

TEST(testy, not_ok) {
  ASSERT_EQ(customer_check(0), 0);  
}

test/main.cpp

This is a standard entrypoint for the test runner.

#include "gtest/gtest.h"

int main(int argc, char **argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Building with CMake

CMake is a build tool that (among other things) produces a Makefile we can run to build our code. We will also use it for dependency management. But fundementally we use it because gtest requires it.

CMake options/rules are defined in a CMakeLists.txt file. We'll have one in the root directory, one in the test directory, and a template for one that will handle the gtest dependency.

A first draft of the top-level CMakeLists.txt might look like this:

cmake_minimum_required(VERSION 3.1)

project(testy)

##
### Source definitions ###
##

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources "${PROJECT_SOURCE_DIR}/src/*.c")

add_executable(testy ${sources})

Using include_directory will make sure we compile with the -I flag set up correctly for our include directory.

Using add_executable sets up the binary name to produce from the given sources. And we're using the file helper to get a glob match of C files rather than listing them all out verbatim in the add_executable call.

Building and running

CMake pollutes the current directory, and is fine running in a different directory, so we'll make a build/ directory so we don't pollute root. Then we'll build a Makefile with CMake, run Make, and run our program.

$ mkdir build
$ cd build
$ cmake ..
-- The C compiler identification is AppleClang 10.0.1.10010046
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build
$ make
[ 25%] Building C object CMakeFiles/testy.dir/src/customer.c.o
[ 50%] Building C object CMakeFiles/testy.dir/src/widget.c.o
[ 75%] Building C object CMakeFiles/testy.dir/src/main.c.o
[100%] Linking C executable testy
[100%] Built target testy
$ ./testy
$ echo $?
1

CMakeLists.txt.in

This template file handles downloading the gtest dependency from github.com pinned to a release. It will be copied into a subdirectory during the cmake .. step.

cmake_minimum_required(VERSION 3.1)

project(googletest-download NONE)

include(ExternalProject)
ExternalProject_Add(googletest
  GIT_REPOSITORY    https://github.com/google/googletest.git
  GIT_TAG           release-1.8.1
  SOURCE_DIR        "${CMAKE_BINARY_DIR}/googletest-src"
  BINARY_DIR        "${CMAKE_BINARY_DIR}/googletest-build"
  CONFIGURE_COMMAND ""
  BUILD_COMMAND     ""
  INSTALL_COMMAND   ""
  TEST_COMMAND      ""
)

Now we can tell CMake about it and how to build, within the top-level CMakeLists.txt file.

cmake_minimum_required(VERSION 3.1)

project(testy)

##
### Test definitions ###
##

configure_file(CMakeLists.txt.in
        googletest-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )
execute_process(COMMAND ${CMAKE_COMMAND} --build .
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )

add_subdirectory(${CMAKE_BINARY_DIR}/googletest-src
        ${CMAKE_BINARY_DIR}/googletest-build)

enable_testing()
add_subdirectory(test)

##
### Source definitions ###
##

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources
  "${PROJECT_SOURCE_DIR}/include/testy/*.h"
  "${PROJECT_SOURCE_DIR}/src/*.c")

add_executable(testy ${sources})

The add_subdirectory calls register a directory that contains its own CMakeLists.txt. It would fail now without a CMakeLists.txt file in the test/ directory.

test/CMakeLists.txt

This final file registers a unit_test executable compiling against the source and test code, and includes the project header files.

include_directories("${PROJECT_SOURCE_DIR}/include")

file(GLOB sources "${PROJECT_SOURCE_DIR}/src/*.c")
list(REMOVE_ITEM sources "${PROJECT_SOURCE_DIR}/src/main.c")

file(GLOB tests "${PROJECT_SOURCE_DIR}/test/*.cpp")
list(REMOVE_ITEM tests "${PROJECT_SOURCE_DIR}/test/main.cpp")

foreach(file ${tests})
  set(name)
  get_filename_component(name ${file} NAME_WE)
  add_executable("${name}_tests"
    ${sources}
    ${file}
    "${PROJECT_SOURCE_DIR}/test/main.cpp")
  target_link_libraries("${name}_tests" gtest_main)
  add_test(NAME ${name} COMMAND "${name}_tests")
endforeach()

We have to register a test for each file otherwise each file's tests won't show up by default (i.e. without a --verbose flag).

Building and running tests

Similar to building and running the source, we run CMake in a subdirectory but run make test or ctest after building all sources and tests with make.

$ cd build
$ cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build/googletest-download
Scanning dependencies of target googletest
[ 11%] Creating directories for 'googletest'
[ 22%] Performing download step (git clone) for 'googletest'
Cloning into 'googletest-src'...
Note: checking out 'release-1.8.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2fe3bd99 Merge pull request #1433 from dsacre/fix-clang-warnings
[ 33%] No patch step for 'googletest'
[ 44%] Performing update step for 'googletest'
[ 55%] No configure step for 'googletest'
[ 66%] No build step for 'googletest'
[ 77%] No install step for 'googletest'
[ 88%] No test step for 'googletest'
[100%] Completed 'googletest'
[100%] Built target googletest
-- Found PythonInterp: /usr/local/bin/python (found version "2.7.16")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/philipeaton/tmp/testy/build
$ make
[  4%] Building C object CMakeFiles/testy.dir/src/customer.c.o
[  9%] Building C object CMakeFiles/testy.dir/src/widget.c.o
[ 13%] Building C object CMakeFiles/testy.dir/src/main.c.o
[ 18%] Linking C executable testy
[ 18%] Built target testy
[ 22%] Building CXX object googletest-build/googlemock/gtest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 27%] Linking CXX static library libgtest.a
[ 27%] Built target gtest
[ 31%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 36%] Linking CXX static library libgmock.a
[ 36%] Built target gmock
[ 40%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 45%] Linking CXX static library libgmock_main.a
[ 45%] Built target gmock_main
[ 50%] Building CXX object googletest-build/googlemock/gtest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 54%] Linking CXX static library libgtest_main.a
[ 54%] Built target gtest_main
[ 59%] Building C object test/CMakeFiles/customer_tests.dir/__/src/customer.c.o
[ 63%] Building C object test/CMakeFiles/customer_tests.dir/__/src/widget.c.o
[ 68%] Building CXX object test/CMakeFiles/customer_tests.dir/customer.cpp.o
[ 72%] Building CXX object test/CMakeFiles/customer_tests.dir/main.cpp.o
[ 77%] Linking CXX executable customer_tests
[ 77%] Built target customer_tests
Scanning dependencies of target widget_tests
[ 81%] Building C object test/CMakeFiles/widget_tests.dir/__/src/customer.c.o
[ 86%] Building C object test/CMakeFiles/widget_tests.dir/__/src/widget.c.o
[ 90%] Building CXX object test/CMakeFiles/widget_tests.dir/widget.cpp.o
[ 95%] Building CXX object test/CMakeFiles/widget_tests.dir/main.cpp.o
[100%] Linking CXX executable widget_tests
[100%] Built target widget_tests

After running cmake and make, we're finally ready to run ctest.

$ ctest
Test project /Users/philipeaton/tmp/testy/build
    Start 1: customer
1/2 Test #1: customer ..........................   Passed    0.01 sec
    Start 2: widget
2/2 Test #2: widget ............................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec

Now we're in a good place with most of the challenges of unit testing C code (i.e. ignoring mocks) past us.

In preparation for a couple new articles on some C projects, here's a foundational post on building C code and writing/running unit tests with gtest and cmake https://t.co/aMVyr7LO73
— Phil Eaton (@phil_eaton) August 31, 2019

Netlify Year One - 360 Review

swyx's site RSS Feed — Sat, 31 Aug 2019 03:00:00 GMT

I thought I would write a quick coda to my Netlify Year One Recap. That post was entirely me looking back at myself. It's helpful to see what others said about me at my review.

(Incomplete) The introduction to TypeScript Generics you've been missing

swyx's site RSS Feed — Fri, 30 Aug 2019 03:00:00 GMT

> Note: this is an unfinished draft

The Case for the React Native Web Singularity

swyx's site RSS Feed — Fri, 30 Aug 2019 03:00:00 GMT

There is a possible "React Native Web Singularity", when it starts being a better standalone choice for developing for the mobile web than react-dom. If this speculation comes true, this would be gamechanging.

You Already Use Types

swyx's site RSS Feed — Thu, 29 Aug 2019 03:00:00 GMT

Published on Freecodecamp

Metrics Atlas

Bytepawn - Marton Trencseni — Thu, 29 Aug 2019 01:00:00 GMT

The idea is simple: write a document which helps new and existing people—both managers and individual contributors—get an objective, metrics-based picture of the business. This is helpful when new people join, when people start working in new segments of the business, and to understand other parts of the company.

Notes on 'Metaphysics and JavaScript'

swyx's site RSS Feed — Wed, 28 Aug 2019 03:00:00 GMT

Slides here

How to run Python and Portray on Netlify

swyx's site RSS Feed — Tue, 27 Aug 2019 03:00:00 GMT

How to deploy Python projects to Netlify

Playing Go with supervised learning in Pytorch

Bytepawn - Marton Trencseni — Sun, 25 Aug 2019 01:00:00 GMT

Using historic gameplay between strong Go players as training data, a CNN model is built to predict good Go moves on a standard 19x19 Go board.

Brave New Blog

Tilting at Windmills — Wed, 21 Aug 2019 03:00:00 GMT

Whew. It's been a while since I blogged. Astute readers (if there are any readers) might notice that the site looks a little bit different…

JAMstack or 'Pilha de Atolamento'? i18n in Gatsby + Netlify

swyx's site RSS Feed — Wed, 21 Aug 2019 03:00:00 GMT

Originally published on Scotch.io

Slowmad Q&A

swyx's site RSS Feed — Mon, 19 Aug 2019 03:00:00 GMT

The other thing about Netlify Year One that people are generally interested in is my experience of remote work. As I put it recently:

10,000 Loose Fans

swyx's site RSS Feed — Sun, 18 Aug 2019 03:00:00 GMT

I broke past 10,000 Twitter followers on Saturday. Obviously this is a completely arbitrary milestone and pretty minor in the hierarchy of needs. I'm not celebrating.

Why Naked Promises are Not Safe For Work

swyx's site RSS Feed — Wed, 14 Aug 2019 03:00:00 GMT

> Published on FreeCodeCamp's Developer News. A followup for this was published in Errors are not Exceptions

Downgrading my iPad 2 to iOS 8

Nemo's Home — Sun, 11 Aug 2019 03:00:00 GMT

Update: Apple is no longer signing iOS6 for the iPad 2, so this is no longer feasible.

I own a iPad 2 (GSM), which is rarely used these days because it is too slow with the latest iOS 9 upgrades. It is a 8 year old device, but I can’t just install Linux on it and make it usable, which is what I do with most other devices.

The next best thing was to downgrade the iOS version. The device is anyway un-supported at this point, so I might as well go there. Apple has restrictions on which iOS releases are installable at any point on any device, ~but thankfully they are still signing iOS6 for my device for some legal reasons~.

Steps:

Download the iOS firmware for your device from https://ipsw.me/#platform
Launch iTunes
Option+Click on the Restore button in iTunes
Select the file you just downloaded.
Disable iCloud on the device
Upgrade to iOS8

This is possible as long as Apple is signing the IPSW for your device. The case of iOS6 being signed seems to be true for:

iPhone 4S
iPad 2

Security Notes

Running an unsupported OS is not something I take lightly. Here’s a list of defensive measures I took to ensure that I’m not at risk while doing so:

Try to keep the device always in Airplane mode.
Keep sensitive data off the device. No photos/keychain sync for eg. Don’t enable Calendar/Contact/Media sync.
Enable Restrictions on the device:
- Restrict Safari to limited websites.
- Disable application installs.
- Disable iTunes store/iBooks store etc.
- Disable GPS/Bluetooth.
- Limit Background Refresh to very few trusted applications.
Limit number of applications (I only have Kybooks installed).
Disable Javascript on Safari.

If possible, I’d recommend using a separate Apple Account on the device.

Netlify Environment Variables

swyx's site RSS Feed — Fri, 09 Aug 2019 03:00:00 GMT

We usually use Environment Variables as (global) variables, but did you know they can literally configure your environments as well?

Netlify Year One

swyx's site RSS Feed — Fri, 09 Aug 2019 03:00:00 GMT

> Update: I have since left Netlify. Alex has published his own writeup on The Mo.

Camels Dressed As Unicorns

smcleod.net — Thu, 08 Aug 2019 11:10:00 GMT

Stop trying to hire with titles like ‘DevOps Engineer’ or ‘Cloud Engineer’

“DevOps is … not a job title”

“It’s more of a cultural practice, like innovation, and it makes just as little sense hiring “innovation engineers” and expecting your organisation to be innovative, without also creating the culture to foster innovation.” (Joel Shea when responding to this posted on LinkedIn)

In most cases it’s clear the organisation doesn’t truly know what they want or need and likely don’t understand the nuances of the aspects of engineering.

Looking For New Opportunities

smcleod.net — Tue, 06 Aug 2019 13:30:00 GMT

As stated in my announcement post - after 7~ years, I resigned from Infoxchange several months ago.

I planned on taking a holiday, relaxing, decompressing, dipping my toes in some off-the-shelf tech I haven’t spent a lot of time with and performing some personal growth through reading a number of books that have been sitting on my ’to read’ shelf.

This is exactly what I’ve done.

I’m now open to new opportunities

I’d rather find a workplace that I can add great value too and be happy to work with than rush into a new role quickly, obviously if something great turns up right away - I’ll pursue it.

Using Novela by Narative (updated)

swyx's site RSS Feed — Sun, 04 Aug 2019 03:00:00 GMT

With the growing community interest in Gatsby, we hope to create more resources that make it easier for anyone to grasp the power of this incredible tool.

Arabic name classification with Scikit-Learn and Pytorch

Bytepawn - Marton Trencseni — Fri, 02 Aug 2019 01:00:00 GMT

While working on arabic-vs-rest classification, I was curious how good out-of-the-box models perform with publicly available data, and then compare that with what we can achieve with internal data / features derived from millions of deliveries. We train Scikit-learn and Pytorch models for this classification task and achieve 90% prediction accuracy on publicly available data and out-of-the-box models, while internally 99% is achievable.

Fight Link Rot with Server- and Client-side Redirects (Netlify and Gatsby)

swyx's site RSS Feed — Mon, 22 Jul 2019 18:55:15 GMT

Why you need redirects on clientside and serverside and how to set that up with Gatsby and Netlify

Hiatus

infrequent oscillations — Sat, 20 Jul 2019 23:36:01 GMT

In which there is a temporary hiatus.

Writing an x86 emulator from scratch in JavaScript: 2. system calls

Notes on software development — Sat, 20 Jul 2019 03:00:00 GMT

Previously in emulator basics: <! forgive me, for I have sinned >
1. a stack and register machine

In this post we'll extend x86e to support the exit and write Linux system calls, or syscalls. A syscall is a function handled by the kernel that allows the process to interact with data outside of its memory. The SYSCALL instruction takes arguments in the same order that the regular CALL instruction does. But SYSCALL additionally requires the RAX register to contain the integer number of the syscall.

Historically, there have been a number of different ways to make syscalls. All methods perform variations on a software interrupt. Before AMD64, on x86 processors, there was the SYSENTER instruction. And before that there was only INT 80h to trigger the interrupt with the syscall handler (since interrupts can be used for more than just syscalls). The various instructions around interrupts have been added for efficiency as the processors and use by operating systems evolved.

Since this is a general need and AMD64 processors are among the most common today, you'll see similar code in every modern operating system such as FreeBSD, OpenBSD, NetBSD, macOS, and Linux. (I have no background in Windows.) The calling convention may differ (e.g. which arguments are in which registers) and the syscall numbers differ. Even within Linux both the calling convention and the syscall numbers differ between x86 (32-bit) and AMD64/x86_64 (64-bit) versions.

See this StackOverflow post for some more detail.

Code for this post in full is available as a Gist.

Exit

The exit syscall is how a child process communicates with the process that spawned it (its parent) when the child is finished running. Exit takes one argument, called the exit code or status code. It is an arbitrary signed 8-bit integer. If the high bit is set (i.e. the number is negative), this is interpreted to mean the process exited abnormally such as due to a segfault. Shells additionally interpret any non-zero exit code as a "failure". Otherwise, and ignoring these two common conventions, it can be used to mean anything the programmer wants.

The wait syscall is how the parent process can block until exit is called by the child and receive its exit code.

On AMD64 Linux the syscall number is 60. For example:

  MOV RDI, 0
  MOV RAX, 60
  SYSCALL

This calls exit with a status code of 0.

Write

The write syscall is how a process can send data to file descriptors, which are integers representing some file-like object. By default, a Linux process is given access to three file descriptors with consistent integer values: stdin is 0, stdout is 1, and stderr is 2. Write takes three arguments: the file descriptor integer to write to, a starting address to memory that is interpreted as a byte array, and the number of bytes to write to the file descriptor beginning at the start address.

On AMD64 Linux the syscall number is 1. For example:

  MOV RDI, 1   ; stdout
  MOV RSI, R12 ; address of string
  MOV RDX, 8   ; 8 bytes to write
  MOV RAX, 1   ; write
  SYSCALL

This writes 8 bytes to stdout starting from the string whose address is in R12.

Implementing syscalls

Our emulator is simplistic and is currently only implementing process emulation, not full CPU emulation. So the syscalls themselves will be handled in JavaScript. First we'll write out stubs for the two syscalls we are adding. And we'll provide a map from syscall id to the syscall.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {},
  60: function sys_exit(process) {},
};

We need to add an instruction handler to our instruction switch. In doing so we must convert the value in RAX from a BigInt to a regular Number so we can look it up in the syscall map.

      case 'syscall': {
        const idNumber = Number(process.registers.RAX);
        SYSCALLS_BY_ID[idNumber](process);
        process.registers.RIP++;
        break;
      }

Exit

Exit is really simple. It will be implemented by calling Node's global.process.exit(). Again we'll need to convert the register's BigInt value to a Number.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {},
  60: function sys_exit(process) {
    global.process.exit(Number(process.registers.RDI));
  },
};

Write

Write will be implemented by iterating over the process memory as bytes and by calling write() on the relevant file descriptor. We'll store a map of these on the process object and supply stdout, stderr, and stdin proxies on startup.

function main(file) {
  ...

  const process = {
    registers,
    memory,
    instructions,
    labels,
    fd: {
      // stdout
      1: global.process.stdout,
    }
  };

  ...
}

The base address is stored in RSI, the number of bytes to write are stored in RDX. And the file descriptor to write to is stored in RDI.

const SYSCALLS_BY_ID = {
  1: function sys_write(process) {
    const msg = BigInt(process.registers.RSI);
    const bytes = Number(process.registers.RDX);
    for (let i = 0; i < bytes; i++) {
      const byte = readMemoryBytes(process, msg + BigInt(i), 1);
      const char = String.fromCharCode(Number(byte));
      process.fd[Number(process.registers.RDI)].write(char);
    }
  },
  ...

All together

$ cat exit3.asm
main:
  MOV RDI, 1
  MOV RSI, 2
  ADD RDI, RSI

  MOV RAX, 60 ; exit
  SYSCALL
$ node emulator.js exit3.asm
$ echo $?
3

$ cat hello.asm
main:
  PUSH 10  ; \n
  PUSH 33  ; !
  PUSH 111 ; o
  PUSH 108 ; l
  PUSH 108 ; l
  PUSH 101 ; e
  PUSH 72  ; H

  MOV RDI, 1   ; stdout
  MOV RSI, RSP ; address of string
  MOV RDX, 56  ; 7 8-bit characters in the string but PUSH acts on 64-bit integers
  MOV RAX, 1   ; write
  SYSCALL

  MOV RDI, 0
  MOV RAX, 60
  SYSCALL
$ node emulator.js hello.asm
Hello!
$

Next steps

We still aren't setting flags appropriately to support conditionals, so that's low-hanging fruit. There are some other fun syscalls to implement that would also give us access to an emulated VGA card so we could render graphics. Syntactic support for string constants would also be convenient and more efficient.

Latest post in the emulator basics series up: implementing some syscalls starting with sys_exit and sys_write so we can print a nice hello message. https://t.co/NEfId0lnJx #javascript #x86
— Phil Eaton (@phil_eaton) July 20, 2019

Go Binary Sizes Have Been Pretty Stable

Donat Studios — Thu, 18 Jul 2019 03:55:21 GMT

Update October 2nd 2024: This was updated to include Go versions all the way up to 1.23

In 2013 I wrote an unintentionally inflammatory post: Go Binary Sizes Are Growing out of Control. I had imagined no one would read it, but it took off. I wrote it because I noticed my binaries getting larger as I upgraded versions of Go and found it curious.

Much has changed since then; biggest of all there have a handful of major releases of Go in that time.

I wanted to find out how much this had changed since my post went up. I used Docker to automate the process of compiling against major releases of Go. The exact code I used is on GitHub.

I hit a couple of hurdles. The oldest official version of Go on Docker Hub is 1.2 - which happens to be the version my previous post left off. Also, many of the programs from the previous comparison no longer compile in such old versions of Go.

Thus, consider this a standalone comparison rather than a direct continuation of the previous post.

I ended up going through most of my Go projects to find what would actually compile in ancient versions of Go.

The tools I compiled for the comparison are in order

hello world using fmt
hello world using println
Hookah - a tool for GitHub automation
imgavg - a tool for averaging a collection of images
sqlread - a tool for querying MySQL dumps without loading them into a database.

The Results

Without further ado, the raw data in megabytes.

binary	1.2	1.3	1.4	1.5	1.6	1.7	1.8	1.9	1.10	1.11	1.12	1.13	1.14	1.15	1.16	1.17	1.18	1.19	1.20	1.21	1.22	1.23
hello-nofmt	0.577	0.475	0.635	1.076	1.101	1.003	0.960	1.035	1.086	1.068	1.127	1.149	1.234	1.224	1.216	1.159	1.155	1.203	1.236	1.285	1.377	1.519
hello	2.240	1.824	1.941	2.367	2.288	1.634	1.552	1.860	2.012	1.907	1.997	2.009	2.068	2.035	1.937	1.772	1.764	1.814	1.852	1.806	1.894	2.129
hookah	8.352	6.737	6.868	7.684	8.739	6.391	6.553	6.805	7.311	7.282	8.234	8.289	8.315	7.191	6.907	6.848	7.014	7.260	7.477	7.620	7.892	8.464
imgavg	3.793	3.040	3.089	3.607	3.560	2.736	2.529	2.656	2.854	2.766	2.891	2.920	2.978	2.887	2.802	2.569	2.558	2.629	2.676	2.619	2.748	2.970
sqlread	4.730	3.648	3.739	4.242	4.192	3.170	3.043	3.091	3.284	3.144	3.302	3.316	3.373	3.265	3.149	2.989	2.968	3.057	3.110	3.171	3.291	3.538

Summary

The trendlines show the majority of the projects have a small downtick in size. The Go 1.5 and Go 1.6 releases caused a pretty universal uptick which 1.7 seems to have largely mitigated.

Hookah shows by far the most fluctuation in size by a pretty large margin. This is interesting because comparing dependencies the only builtin packages Hookah uses that sqlread does not are regexp and regexp/syntax. sqlread itself actually utilizes a pretty large number more. This could indicates the variation is deeper than the changes to the built in libraries, and potentially more to do with the compilation output and optimization.

Everything except for the simplest "Hello World" hello-nofmt has shown an overall decrease. hello-nofmt boils down to println("Hello, 世界") has shown an overall 658kb increase. This is likely the purest illustratration the growth in size of the runtime.

Conclusion

Go binaries are not "growing out of control". They're pretty stable and for the most part have shown a decrease.

I'm still hopeful future optimizations will help improve sizes. Given WASM becoming a major compilation target, size matters more than ever as it directly affect load times.

Tests or Types: Why Not Both?

swyx's site RSS Feed — Wed, 10 Jul 2019 03:00:00 GMT

Published on CSS Tricks as Types or Tests: Why Not Both?

Exploring prior beliefs with MCMC

Bytepawn - Marton Trencseni — Sat, 06 Jul 2019 01:00:00 GMT

I use PyMC3 to solve the food delivery toy problem and explore some alternative priors.

Design Issues of Sign in with Apple

Dan Palmer — Fri, 05 Jul 2019 02:00:00 GMT

Last month at their annual Worldwide Developers Conference (WWDC), most interesting announcements was Sign in with Apple. Built to compete with Facebook and Google’s single-sign-on (or social sign-on, SSO) offerings, Apple’s SSO will eschew control over the data and analytics that its competitors seek in favour of a privacy preserving design intended to advance Apple’s pro-privacy stance and ultimately to sell more devices by bringing more value to the Apple ecosystem.

Deploying a Create-React-App Website to Netlify with HTTPS

swyx's site RSS Feed — Wed, 03 Jul 2019 17:51:34 GMT

Tutorial on Deploying a Create-React-App Website to Netlify with HTTPS

How AWS and Go Transformed a Public Broadcaster

Ben E. C. Boyter — Wed, 03 Jul 2019 03:00:00 GMT

Today I presented at the Sydney Dev Warriors meet-up about a project I was working on from the start to end of 2018. In a nutshell it was a self serve digital archive for video, audio, images and metadata.

Since I am in the habit of collecting everything I do into one place I chose to do the presentation using reveal.js which means I can also include it here. How AWS and Go Transformed a Public Broadcasters Digital Archive.

Pass the SALT 2019 live report (part 3/3)

Linux Engineer's random thoughts — Wed, 03 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my part 1 and part 2 of my notes here.

Configurations, Do you prove yours ?

by Alexandre Brianceau (slides)

Alexandre went a bit more into details on the core concepts behind DevSecOps, and why continous configuration and observability …

Pass the SALT 2019 live report (part 2/3)

Linux Engineer's random thoughts — Tue, 02 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my part 1 of my notes and part 2 here.

Time-efficient assessment of open-source projects for Red Teamers

by Thomas Chauchefoin and Julien Szlamowicz (slides)

In their pentest team, Julien often does red team assessments with big scopes …

Pass the SALT 2019 live report (part 1/3)

Linux Engineer's random thoughts — Mon, 01 Jul 2019 01:00:00 GMT

This year I'm at the Pass the Salt 2019 conferences. You'll find my notes updated in real time here.

Kill MD5

by Ange Albertini (slides)

Ange starts by saying that he does not know much about crypto, and he had help from Marc Stevens, a hash-collision cryptographer. This talk should …

A/B tests: Moving Fast vs Being Sure

Bytepawn - Marton Trencseni — Mon, 01 Jul 2019 01:00:00 GMT

Most A/B testing tools default to α=0.05, meaning the expected false positive rate is 5%. In this post I explore the trade-offs between moving fast, ie. using higher α, versus being sure, ie. using lower α.

The Engineering Manager/Product Manager Marriage

CRAIG KERSTIENS — Sun, 30 Jun 2019 23:55:56 GMT

I’ve worked as a PM at a number of size companies for a few years now. At a startup and then as a part of a larger company once startups were acquired. I’ve been the first PM for a team as well as first for a company. I’ve written at times about product management, and today I’d like to drill into one aspect that doesn’t seem to get talked about enough and that is the pairing of product manager and engineering manager.

Mom vs. Dad

As parents my partner and I have learned very quickly that we need to have a consistent voice and unified view of things. I care that our son watches less power rangers otherwise he’s going to use his megazord powers on our TV and we’ll be watching a lot more of nothing. My partner cares that when we’re visiting family in the south they drink enough water so they don’t get dehydrated.

Meanwhile my kids are experts at leveraging us to get what they want. My daughter came to me last night asking if she could play on her iPad some. Not knowing if she had or if she’d already given an answer my safest question was have you asked your mom? In the absense of knowing my default isn’t a yes or no, it’s a “let me learn more.” and then potentially discuss it.

EM vs. PM

As a PM I want us to build a rich and powerful product, but there is a strong balance to doing too little vs. too much. It isn’t always a question of doing more, we need to make sure the product is well built. In order to do that we need to say no at times. Saying no, as well as yes, needs to come from a unified front, both engineering and product. If one side is agreeing without being aligned with the other half you’re going to wind up with a confused and frustrated team.

There are a number of ways engineering managers and product managers can stay aligned. The first starts with being explicit with each other, so if you’re struggling with one side communicating things the other don’t agree with… sit down and have a conversation about it. From an ongoing perspective you can get to a better aligned position to not have to discuss every question before you give an answer. My partner knows to make sure we don’t watch more than 2 episodes of power rangers and when we’re too amped up on it we shorten that.

Are you having regularly 1:1s with your counterpart?
Do you review emails out to the team ahead of time to capture feedback?
Do you have conversations where you’re not on the same page in an offline fashion as opposed to in team meetings?
Are you treating your EM/PM just like your partner at home? (I guess that does assume you treat your partner well, but that’s a whole other blog post)

Leaving Infoxchange

smcleod.net — Sat, 29 Jun 2019 12:00:00 GMT

After 7~ years, I have resigned from Infoxchange.

In 2012 I was hired to work with Infoxchange to “shake things up” (in the Systems Operations team) and stabilise the hosting environment.

As Team Lead of Systems Operations I was tasked with melding the team and then for us to work to provide a stable, robust, modern and scalable platform for Infoxchange developed application hosting and product delivery and in a short time, with a small budget - we did just that.

Betwixt Reason and Result

swyx's site RSS Feed — Sat, 29 Jun 2019 03:00:00 GMT

I'm writing to you from a cheap hostel in downtown Toronto (Hostelling International, highly recommend for solo travel, I have stayed in SF, LA, NY, PHL, TO, NZ and more I probably forget) where a cute anecdote just happened that illustrates an important principle.

Solve CORS once and for all with Netlify Dev

swyx's site RSS Feed — Fri, 28 Jun 2019 03:00:00 GMT

Published on Alligator.io

Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab

Ben E. C. Boyter — Thu, 27 Jun 2019 03:00:00 GMT

This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

A very brief update in the world of scc. I noticed that tokei.rs which used to offer badges seemed to be having issues. Since I had wanted an excuse to hook up a AWS Lambda behind an ALB I thought I would add scc as a ‘service’ with badges.

How I traded my first software project 15 years ago

Linux Engineer's random thoughts — Wed, 26 Jun 2019 01:00:00 GMT

2004

Gmail recently celebrated its 15 years anniversary, which led me down the path of how I got my first gmail invite. Back then, the private Gmail beta was all the rage. In the first few months, gmail invites were scarce. I got one by doing the most popular thing …

Writing a lisp compiler from scratch in JavaScript: 6. LLVM system calls

Notes on software development — Sat, 22 Jun 2019 03:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
Next in compiler basics:
5. an x86 upgrade

In this post we'll extend the ulisp compiler's LLVM backend to support printing integers to stdout.

Exit code limitations

Until now we've validated program state by setting the exit code to the result of the program computation. But the exit code is an eight bit integer. What if we want to validate a computation that produces a result larger than 255?

To do this we need a way to print integers. This is challenging because printing normally deals with byte arrays. libc's printf, for example, takes a byte array as its first argument.

The shortest path forward is to add support for system calls so we can print one character at a time. Here's a version of a print form that hacks around not having arrays to send each integer in a number to stdout.

(def print-char (c)
     ; First argument is stdout
     ; Second argument is a pointer to a char array (of length one)
     ; Third argument is the length of the char array
     (syscall/sys_write 1 &c 1))

(def print (n)
     (if (> n 9)
         (print (/ n 10)))

     ; 48 is the ASCII code for '0'
     (print-char (+ 48 (% n 10))))

In order to support this we need to add the syscall/sys_write, >, %, and / builtin forms. We'll also need to add support for taking the address of a variable.

All code is available on Github as is the particular commit related to this post.

References

The sys_write syscall requires us to pass the memory address of the byte array to write. We don't support arrays, but we can treat an individual variable as an array of length one by passing the variable's address.

If we were compiling to C we could just pass the address of a local variable. But LLVM doesn't allow us to take the address of variables directly. We need to push the variable onto the LLVM stack to get an address.

Under the hood LLVM will likely optimize this into a local variable reference instead of first pushing to the stack.

Since LLVM IR is typed, the value representing the address of a local variable will be a pointer type. We'll need to refer to all uses of this value as a pointer. So we'll need to modify ulisp to track local types rather than hard-coding i64 everywhere.

Scope

To begin we'll modify the Scope class to track types. We only need to do this on registration. Afterward, we'll have to find all uses of local variables to make sure they use the local's value and type fields appropriately.

class Scope {

  ...

  register(local) {
    let copy = local.replace('-', '_');
    let n = 1;
    while (this.locals[copy]) {
      copy = local + n++;
    }

    this.locals[local] = {
      value: copy,
      type: 'i64',
    };
    return this.locals[local];
  }

  ...

}

We won't go through every use of a Scope variable in this post, but you can find it in the related commit to ulisp.

Reference

The long-term approach for handling a reference syntactically is probably to rewrite &x to (& x) in the parser. The lazy approach we'll take for now is to handle a reference as a special kind of identifier in compileExpression.

We'll use the LLVM alloca instruction to create space on the stack. This will return a pointer and will turn the destination variable into a pointer type. Then we'll use store to set the value at the address to the current value of the variable being referenced.

class Compiler {

  ...

  compileExpression(exp, destination, context) {

    ...

    // Is a reference, push onto the stack and return its address
    if (exp.startsWith('&')) {
      const symbol = exp.substring(1);
      const tmp = context.scope.symbol();
      this.compileExpression(symbol, tmp, context);
      this.emit(1, `%${destination.value} = alloca ${tmp.type}, align 4`);
      destination.type = tmp.type + '*';
      this.emit(1, `store ${tmp.type} %${tmp.value}, ${destination.type} %${destination.value}, align 4`);
      return;
    }

    ...

  }

  ...

}

And now we're set to take the address of any code.

System calls

LLVM IR provides no high-level means for making system calls. The only way is to use inline assembly. This syntax is based on GCC inline assembly and is confusing, with few explained examples, and unhelpful error messages.

Thankfully the assembly code needed for a syscall is only one line, one word: the syscall assembly instruction. We use inline assembly variable-to-register mapping functionality to line up all the parameters for the syscall. Here is an example:

%result = call i64 asm sideeffect "syscall", "=r,{rax},{rdi},{rsi},{rdx}" (i64 %raxArg, i64 %rdiArg, i64 %rsiArg, i64 %rdxArg)

This says to execute the inline assembly string, "syscall". The sideeffect flag means that this assembly should always be run even if the result isn't used. =r means the inline assembly returns a value, and the rest of the string is the list of registers that arguments should be mapped to. Finally we call the function with all the LLVM variables we want to be mapped.

Eventually we should also use the inline assembly syntax to list registers that are modified so that LLVM can know to save them before and after.

Code

We'll add a mapping for syscall/sys_write and a helper function for generating syscall code using the example above as a template. We'll suport 64-bit Darwin and Linux kernels.

const SYSCALL_TABLE = {
  darwin: {
    sys_write: 0x2000004,
    sys_exit: 0x2000001,
  },
  linux: {
    sys_write: 1,
    sys_exit: 60,
  },
}[process.platform];

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
      '%': this.compileOp('urem'),
      '<': this.compileOp('icmp slt'),
      '=': this.compileOp('icmp eq'),
      'syscall/sys_write': this.compileSyscall(SYSCALL_TABLE.sys_write),
    };
  }

  ...

  compileSyscall(id) {
    return (args, destination, context) => {
      const argTmps = args.map((arg) => {
          const tmp = context.scope.symbol();
          this.compileExpression(arg, tmp, context);
          return tmp.type + ' %' + tmp.value;
      }).join(', ');
      const regs = ['rdi', 'rsi', 'rdx', 'r10', 'r8', 'r9'];
      const params = args.map((arg, i) => `{${regs[i]}}`).join(',');
      const idTmp = context.scope.symbol().value;
      this.emit(1, `%${idTmp} = add i64 ${id}, 0`)
      this.emit(1, `%${destination.value} = call ${destination.type} asm sideeffect "syscall", "=r,{rax},${params},~{dirflag},~{fpsr},~{flags}" (i64 %${idTmp}, ${argTmps})`);
    }
  }
}

`>`, `/`

Finally, we have a few new operations to add support for. But they'll be pretty simple using the compileOp helper function.

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
      '/': this.compileOp('udiv'),
      '%': this.compileOp('urem'),
      '<': this.compileOp('icmp slt'),
      '>': this.compileOp('icmp sgt'),
      '=': this.compileOp('icmp eq'),
      'syscall/sys_write': this.compileSyscall(SYSCALL_TABLE.sys_write),
    };
  }

  ...

}

print

We're ready to give our print function a shot.

$ cat test.lisp
(def print-char (c)
     ; First argument is stdout
     ; Second argument is a pointer to a char array (of length one)
     ; Third argument is the length of the char array
     (syscall/sys_write 1 &c 1))

(def print (n)
     (if (> n 9)
         (print (/ n 10)))

     ; 48 is the ASCII code for '0'
     (print-char (+ 48 (% n 10))))

(def main ()
     (print 1234)
     0)
$ node ulisp.js test.lisp
$ ./build/a.out
1234

Looks good! In the next post we'll talk about tail call elimination.

It's been a slow month for the blog. But new post on compiler basics is up! Printing integers to stdout and making syscalls in LLVM (all without arrays). This was a pre-req for playing with tail-call elimination (post coming soon) https://t.co/fDtblUZRI8
— Phil Eaton (@phil_eaton) June 23, 2019

Food deliveries, Bayes and Computational Statistics

Bytepawn - Marton Trencseni — Sat, 22 Jun 2019 01:00:00 GMT

I was grabbing a burger at Shake Shack, Mall of the Emirates in Dubai, when I noticed this notebook on the counter. The staff is using it to track food deliveries and each service (Carriage, Talabat, UberEats, Deliveroo) has its own column with the order numbers. Let's assume this is the only page for the day, and ask ourselves: given this data, what is the probability that UberEats is the most popular food delivery service?.

Add Netlify Identity Authentication to any React App in 5 minutes with React Context, Hooks and Suspense

swyx's site RSS Feed — Fri, 21 Jun 2019 14:33:26 GMT

Adding authentication is a pain point for many React beginners. We’ve made it ridiculously easy to add Netlify Identity onto any React app, including create-react-app, Gatsby, Next.js, or any other setup you may have, by wrapping it all into one simple React Hook! However, this article is more about effective design patterns for introducing authentication into React apps, and any reader should be able to write similar wrappers for their preferred provider.

A really cheap chatbot in Python

Ben E. C. Boyter — Wed, 19 Jun 2019 03:00:00 GMT

Let me start by stating that I think all chat-bots are rubbish and that a better solution is a decent search implementation. Type in some terms and get some useful results. Much better than typing to a machine.

A while ago I was working at a client site where there was an internal discussion group about building a chat-bot. Many meetings had passed, discussing using neural networks, building flows, should we use IBM Watson. Lots of talk, but not much delivery.

How to customise Jetty embedded in Spark Java framework

Subclassed — Tue, 18 Jun 2019 14:43:52 GMT

EDIT: I uploaded a full working example, ready to be customized, on github. If you just want a working solution, go there. Otherwise, keep reading for the explanation.

When it comes to Java microframeworks for API development, I've been using Spark for some time in a couple of different projects. For most casual needs, it works out of the box; however, there are circumstances where the "easy" options for configuring the embedded Jetty instance, are simply not enough. In that case, you have to take control and basically replace the instance with one that you completely control.

Due to how Jetty works, there are multiple moving parts: the server takes care of things like thread pools, the socket is responsible for SSL and other low-level protocol stuff, and the handler is where you do high-level middleware (changing headers etc). This means that you might need several factories to take care of all these. The good news is that Spark is granular enough that you can (more or less) limit yourself to what you need. The main pattern is: there will be a factory for each element, with a create() method returning an interface; you implement the interface and swap out the default factory with your own. However, these factories at the moment (Spark 2.8) are somewhat nested, so depending on where you need to work, you might have to replace a bunch of classes before you hit the point you're interested in.

The main entry point is EmbeddedServers.add(identifier, serverFactory). This is what you'll call from the main() method you use for all the post() and get() configuration directives. It is important that EmbeddedServers should appear right at the top, before any other configuration directive, otherwise Spark will use its default factory. It should look more or less like this:

EmbeddedServers.add(
     EmbeddedServers.Identifiers.JETTY, 
     new MyWonderfulServerFactory());

For the identifier, we re-use the default one because otherwise Spark will take over again. Nothing to do there.

The server factory is where real work begins. Your class (or lambda) needs to implement the spark.embeddedserver.EmbeddedServerFactory interface, which has one method:

public EmbeddedServer create(
            Routes routeMatcher, 
            StaticFilesConfiguration staticFilesConfiguration, 
            ExceptionMapper exceptionMapper, 
            boolean hasMultipleHandler)

As a starter, you can copy the content of spark.embeddedserver.jetty.EmbeddedJettyFactory as-is. You don't need constructors (unless you want them). Strictly speaking you don't need withThreadPool() and withHttpOnly() methods either, but I suggest you keep them anyway; just change the return type to match MyWonderfulServerFactory.

Create() does three things:

Initializing the route matcher, which you probably don't want to touch;
Initializing the Jetty handler for that matcher, which you may want to configure for things like header manipulation and other middleware;
Creating the embedded server, which is likely what you are after.

I will assume we want to tweak n.3. The main place where to pay attention is this line at the end of create():

return (new EmbeddedJettyServer(this.serverFactory, handler)
        ).withThreadPool(this.threadPool);

This will return the default configuration, and we don't want that. So we change it to something like:

return (new MyWonderfulEmbeddedServer(handler)
        ).withThreadPool(this.threadPool);

Now we need a MyWonderfulEmbeddedServer class, which should implement spark.embeddedserver.EmbeddedServer. Again, as a starting point, you can copy spark.embeddedserver.jetty.EmbeddedJettyServer, and change the return types to match. I also suggest you get rid of the factory parameter in constructor and the related field, which adds a bit of unnecessary complexity. That factory is actually used only in one place:

if (this.threadPool == null) {
    this.server = this.serverFactory.create(
       maxThreads, minThreads, threadIdleTimeoutMillis);
} else {
    this.server = this.serverFactory.create(this.threadPool);
}

Which you can replace with the following (straight from spark.embeddedserver.jetty.JettyServer):

if (this.threadPool == null) {
    if (maxThreads > 0) {
       int min = minThreads > 0 ? minThreads : 8;
       int idleTimeout = threadIdleTimeoutMillis > 0 ? threadIdleTimeoutMillis : '\uea60';
       server = new Server(new QueuedThreadPool(maxThreads, min, idleTimeout));
    } else {
       server = new Server();
    }
} else {
    this.server = threadPool != null ? new Server(threadPool) : new Server();
}

It's a bunch of stuff related to the amount of threads and timeouts. If that's what you were trying to configure, btw, this is where you can do it. To be honest, I believe Spark developers intended that the "proper" way, to do that particular customisation, would be to keep the factory parameter as it is, implement your own alternative to the badly-named JettyServer class -- it should be something like JettyThreadConfigFactory, really -- which implements the similarly badly-named JettyServerFactory, and pass it to the constructor in MyWonderfulServerFactory.create().

In my case, though, I was after an SSL customisation, and for that I had to swap out yet another factory, spark.embeddedserver.jetty.SocketConnectorFactory, mentioned in the second half of this block:

ServerConnector connector;
if (sslStores == null) {
    connector = SocketConnectorFactory.createSocketConnector(
         this.server, host, port);
} else {
    connector = SocketConnectorFactory.createSecureSocketConnector(
         this.server, host, port, sslStores);
}

In this case there is no interface, you can just extend the existing class with what you need. I wanted to enforce a particular set of SSL ciphers and protocols, so I overrode createSecureSocketConnector() adding the following bits:

sslContextFactory.setExcludeProtocols("SSLv3", "SSLv2", "TLSv1.2");
// first we clear existing exclusions
sslContextFactory.setExcludeCipherSuites(new String[]{});
// then we re-add what we need
sslContextFactory.setIncludeCipherSuites(bigArrayOfCipherNames);

And that's it. Now you know how to instantiate your own Jetty instance for Spark. It's a bit convoluted, and hopefully a "spark 3" will give us a better architecture to work with, one day. In the meantime, this is how you can do it.

How to dump macOS security-rights database

Subclassed — Tue, 18 Jun 2019 14:43:38 GMT

macOS handles some security items in a custom database, which may or may not be SQLite. The official way to interact with such database, outside of Objective-C, is the /usr/bin/security utility, with the parameter authorizationdb and specifying the required operation on the class of rights, e.g. security authorizationdb read system.login.console

Unfortunately, there is no way (that I could find) to simply list all classes. I was trying to uninstall something that might have had references in that db, but I wasn't sure about the class it might be registered under, so I wanted to dump them all.

Luckily I found what looks looks like a comprehensive list of rights. After a quick scraping job with Python, I had a list that I could use like this:

cat osxrights.txt | xargs -I % sh -c 'sudo security authorizationdb read %' | grep -B 10 myAnnoyingItemToRemove

... and the bundle wasn't anywhere, so I could just chuck it.

(Note: had the item been found, I would have had to dump the whole security class to a .plist file with security authorizationdb read the.class > my.plist , edited the file to remove it, then write it back to the db with security authorizationdb write the.class < my.plist )

How good does the Monaro look

Ben E. C. Boyter — Tue, 11 Jun 2019 03:00:00 GMT

About 12 years ago my old man was working as one of those hire a husband/jim’s maintenance deals. This was in the lower Blue Mountains. During my university break I didn’t have much on and would often come with him as something to do. We were attempting to repair a deck of some flea hole house near Glenbrook. The place would have been improved only by a match and petrol. The owner was around and mentioned that the place was to be rented out and real estate had just sent some people on the way to inspect.

A non-gamers attempt at Sea of Thieves

Ben E. C. Boyter — Mon, 10 Jun 2019 03:00:00 GMT

A long weekend just recently passed in NSW Australia. With a larger than normal amount of time on my hands I decided to try out the game Sea of Thieves. Ignoring the fact that the NBN rollout here has ruined my ADSL2+ which has gone from a 18-19 Mbps connection to 3 Mbps and that it took 24 hours to download, what follows is probably one of the more memorable gaming experiences I have ever had.

Creating an index of Apple Watch/MacOS/iOS features per-country

Tom Forbes — Tue, 04 Jun 2019 23:52:41 GMT

tl;dr: check out https://applewatchfeatures.com/ , https://iosfeatures.com/ and https://macosfeatures.com/ I recently brought an Apple Watch. It’s a pretty fantastic product! One thing that quite annoyed me while evaluating if I should buy it is that features can vary quite drastically per country, ...

About

infrequent oscillations — Mon, 03 Jun 2019 19:00:00 GMT

Social networks Fediverse: @zimpenfish@social.rjp.is Fediverse: @zimpenfish@octodon.social Goodreads Twitter Spotify last.fm Jabber Also reachable via Telegram, Signal, WhatsApp, LINE, WeChat, Viber, Threema, and probably some others I’ve forgotten. Creative outlets Instagram Soundcloud YouTube Flickr (largely just an Instagram backup now) Tumblr Sometimes my music escapes into the wild and appears on real services. Spotify Deezer Tidal Buy my stuff Bandcamp Spoonflower Etsy Redbubble Society6 Displate Coding shenanigans Bitbucket Github Exercise shenanigans Strava Garmin Connect

The Collatz conjecture

Bytepawn - Marton Trencseni — Sun, 02 Jun 2019 01:00:00 GMT

The Collatz conjecture is a conjecture in mathematics that concerns a sequence defined as follows: start with any positive integer n. Then each term is obtained from the previous term as follows: if the previous term is even, the next term is one half the previous term. If the previous term is odd, the next term is 3 times the previous term plus 1. The conjecture is that no matter what value of n, the sequence will always reach 1.

Cleaning up Google Purchases

Nemo's Home — Sat, 01 Jun 2019 03:00:00 GMT

The Google Purchase History feature has been doing rounds in the news recently. In case you missed it, go to https://myaccount.google.com/purchases right now and make sure you are logged in with your personal gmail account to see what all Google thinks you’ve bought.

For me it lists purchases going as far back as 2013, which include:

All of my Amazon Purchases (including Kindle and Audible)
Flipkart/Sneapdeal purchases
Gifts I’ve bought for others on various platforms
All my iCloud purchases
Purchases on Steam
BigBasket purchases
Google Playstore purchases as well, of course
And much, much more.

For each of the purchases, it remembers the price, the taxes, as well as the delivery address used.

While this isn’t shocking in the least, I was surprised, because as a Infosec professional, I’ve disabled all of google’s invasive tracking features:

All my Activity Controls are paused.
Google Location history is disabled for my account.
I have Shared endorsements turned off.
Ad personalization is turned off.
I used to run with the Protect my Choices extension till a while back to avoid targeted advertising.

Regardless, the Google purchases page had hundreds of results, going back half a decade. Google currently does not offer a way to delete collected purchases directly, or to pause this collection in any way. The only way is to find the emails that Google scanned, and delete them.

I ended up deleting everything from the following email addresses:

auto-confirm@amazon.com
auto-confirm@amazon.in
cs@flipkart.com
digital-no-reply@amazon.com
do_not_reply@audible.com
do_not_reply@gog.comorders@services.target.com
ebay@ebay.in
googleplay-noreply@google.com
help@stickermule.com
mail@info.fabfurnish.com
no-reply@flipkart.com
no-reply@paytm.com
no_reply@email.apple.com
noreply@flipkart.com
noreply@pizzahut.co.in
noreply@snapdeals.co.in
noreply@steampowered.com
notification@wish.com
order-update@amazon.in
orders@services.target.com
payments-messages@amazon.in
return@amazon.in
ship-confirm@amazon.com
ship-confirm@amazon.in
shipment-tracking@amazon.com
shipment-tracking@amazon.in
updates@myntra.com

Note that deleting the email doesn’t seem to be sufficient either, you need to clear your Trash, and then wait for a while (almost 2 days for me) before the system refreshes. After 3 days of just deleting mails, I finally got this screen:

Google seems to be picking up all kinds of emails, including:

Invoices
Shipment Confirmations / Updates
Return confirmations
Payment Confirmations
Order Cancellations
Payment Failures (gasp!)

Warning about Deletions

I’ve already switched away from Amazon/Flipkart emails from my Gmail. But deleting invoices from your inbox isn’t always the best idea. Most websites will let you re-download invoices (Amazon/Flipkart do), but take care not to delete any necessary emails that you might need for warranty claims or any other purpose later.

MNIST pixel attacks with Pytorch

Bytepawn - Marton Trencseni — Sat, 01 Jun 2019 01:00:00 GMT

It’s easy to build a CNN that does well on MNIST digit classification. How easy is it to break it, to distort the images and cause the model to misclassify?

News sites in a post-advertisement world

Ben Overmyer's Site — Wed, 29 May 2019 03:00:00 GMT

This post is a response to "News Websites are Dumpster Fires", which rails against the current business model of most news websites.

The problem that faces news websites is not as simple as just a business model change.

The purpose of a news website is to provide accurate, current information on topics relevant to the health and progress of society. This is an expensive endeavor. Journalism isn't easy - indeed, sometimes it can be dangerous, depending on the topic covered. It requires training, equipment, and no small amount of social savvy.

A news site can't be funded by the government without the appearance of state bias. The ad-driven model currently employed by the majority of news sites is unsustainable. How, then, can these sites be supported in a way that doesn't compromise their core mission?

This is a complicated question, and not one I have a solution for. I suspect that the future lies in content producers being funded by the public in some way. The following article may be of interest in this regard: Why isn't there a news organization truly funded by the public

Writing an x86 emulator from scratch in JavaScript: 1. a stack and register machine

Notes on software development — Tue, 21 May 2019 03:00:00 GMT

Better yet, take a look at this post walking through emulating x86 ELF binaries in Go:
Emulating linux/AMD64 userland: interpreting an ELF binary

Next up in emulator basics: <! forgive me, for I have sinned >
2. system calls

In this post we'll create a small virtual machine in JavaScript and use it to run a simple C program compiled with GCC for an x86_64 (or AMD64) CPU running Linux.

All source code is available on Github.

Virtual machine data storage

Our virtual machine will have two means of storing data: registers and an integer stack. Each register can store a 64-bit integer. The stack is an array of 8-bit (or 1 byte) integers.

We'll make the following registers available for modification and use by the program(mer):

RDI, RSI, RSP, RBP, RAX, RBX, RCX, RDX, R8, R9, R10, R11, R12, R13, R14, R15

The RSP register is used by the virtual machine for tracking the location of the last entry in the stack. It will be modified by the virtual machine when it encounters the POP, PUSH, CALL and RET instructions we'll support. We'll get into the specifics shortly.

And we'll make the following registers available for use (but not modification) by the program(mer):

RIP, CS, DS, FS, SS, ES, GS, CF, ZF, PF, AF, SF, TF, IF, DF, OF

Each of these has a special meaning but we'll focus on RIP. The RIP register contains the address of the instruction currently being interpreted by our virtual machine. After every instruction the virtual machine will increment the value in this register -- except for a few special instructions like CALL and RET.

Memory addresses

It will become useful to provide direct access to memory with a special syntax. We'll focus just on 64-bit addresses that will look like this:

  MOV QWORD PTR [RBP - 8], 12

This asks for the value 12 to be written into the memory address at RBP - 8 bytes. The QWORD PTR part clarifies that we want to write 8 bytes worth of the value. Since 12 is less than 8 bytes, the rest will be filled with zeros.

  ADD RAX, QWORD PTR [RBP - 8]

This asks for eight bytes starting from the memory address RBP - 8 to be added to the value in RAX and stored back in RAX.

Virtual machine instruction set

In our virtual machine we'll define support for the following instructions:

MOV $REGISTER, $REGISTER or $MEMORY ADDRESS or $LITERAL NUMBER
- This instruction copies the second value into the first.
ADD $REGISTER, $REGISTER or $MEMORY ADDRESS
- This instruction adds the second value into the first and stores the result into the first.
PUSH $REGISTER
- This instruction will decrement the RSP register by 8 bytes and store the value at the bottom of the stack.
POP $REGISTER
- This instruction will increment the RSP register by 8 bytes, remove the last element in the stack (at the bottom), and store it into the register.
CALL $LABEL
- This instruction will push the value in the RIP register (plus one) onto the stack and set the RIP register to the line of code of the label. More on this later.
RET
- This instruction will remove the value at the bottom of the stack and store it in the RIP register.

Now we have more than enough instructions to write some interesting programs for the virtual machine.

Virtual machine semantics

We'll make one last assumption before explaining further. In our programs, there must be a main label which must contain a RET instruction. Once we hit the terminal RET, we will exit the virtual machine and set the exit code to the value stored in the RAX register.

Let's look at a simple program:

main: ; the required main label
  MOV RAX, 1 ; store 1 in RAX
  MOV RDI, 2 ; store 2 in RDI
  ADD RAX, RDI ; store the result of adding RAX and RDI in RAX
  RET ; give control back to the virtual machine

When we run this program, first we initialize a stack (we'll give it 1000 elements) and set the RSP register to 1000 (the top of the stack). Then we look for the main label and set the RIP register to 1, the line number after the label appears (0). Then until the RIP register is 1000 again, we interpret the instruction at the line number stored in the RIP register. Once the RIP register hits 1000, we exit the program setting RAX as the exit code.

One more example

Now let's look at one more program:

plus:
  ADD RDI, RSI
  MOV RAX, RDI
  RET

main:
  MOV RDI, 1
  MOV RSI, 2
  CALL plus
  RET

Our virtual machine will start at the line after the main label. Then it will store 1 into RDI and 2 into RSI. Then it will jump to the second line in the program to add RDI and RSI and store the result in RDI. Then it will copy RDI into RAX and return control to the final line. This last RET will in turn return control to the virtual machine. Then the program will exit with exit code 3.

Parsing

Now that we've finished up describing our virtual machine language and semantics, we need to parse the instructions into a format we can easily interpret.

To do this we'll iterate over the program skip any lines that start with a dot. These are virtual machine directives that are important for us to ignore for now. We'll also remove any characters including and following a semi-colon or hash-tag, until the end of the line. These are comments.

We'll store a dictionary of label names to line numbers (the line number of the label plus one) and without the colon.

And we'll store the instructions as an array of objects composed of an operation and optional operands.

Code

function parse(program) {
  const labels = {};
  const instructions = [];

  const lines = program.split('\n');

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace
    // TODO: handle each line
  }

  return { labels, instructions };
}

First let's handle the directives we want to ignore:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.')) {
      continue;
    }
  }

And then comments:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';') || line.startsWith('#')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes('#')) {
      line = line.split('#')[0];
    }

    if (!line) {
      continue;
    }
  }

And then labels:

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';') || line.startsWith('#')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes('#')) {
      line = line.split('#')[0];
    }

    if (!line) {
      continue;
    }

    if (line.includes(':')) {
      const label = line.split(':')[0];
      labels[label] = instructions.length;
      continue;
    }
  }

And finally instruction parsing plus the rest:

function parse(program) {
  const labels = {};
  const instructions = [];

  const lines = program.split('\n');

  for (let i = 0; i < lines.length; i++) {
    let line = lines[i].trim(); // Remove any trailing, leading whitespace

    if (line.startsWith('.') || line.startsWith(';')) {
      continue;
    }

    if (line.includes(';')) {
      line = line.split(';')[0];
    }

    if (line.includes(':')) {
      const label = line.split(':')[0];
      labels[label] = instructions.length;
      continue;
    }

    const operation = line.split(/\s/)[0].toLowerCase();
    const operands = line.substring(operation.length).split(',').map(t => t.trim());
    instructions.push({
      operation,
      operands,
    });
  }

  return { labels, instructions };
}

Hurray! A brittle parser.

Interpreting

We've already described the semantics a few times. So let's get started with the foundation and initialization.

We'll use BigInts because JavaScript integers are 53-bits wide. This isn't incredibly important in our basic programs but it will quickly became painful without.

And we'll make process memory available as an array of 8-bit integers. In order to make this easy to use, we'll also provide helper function for writing to and reading from memory.

const fs = require('fs');

const REGISTERS = [
  'RDI', 'RSI', 'RSP', 'RBP', 'RAX', 'RBX', 'RCX', 'RDX', 'RIP', 'R8',
  'R9', 'R10', 'R11', 'R12', 'R13', 'R14', 'R15', 'CS', 'DS', 'FS',
  'SS', 'ES', 'GS', 'CF', 'ZF', 'PF', 'AF', 'SF', 'TF', 'IF', 'DF', 'OF',
];

function writeMemoryBytes(process, address, value, size) {
  for (let i = 0n; i < size; i++) {
    value >>= i * 8n;
    process.memory[address + i] = value & 0xFFn;
  }
}

function readMemoryBytes(process, address, size) {
  let value = 0n;
  for (let i = 0n; i < size; i++) {
    value |= (process.memory[address + i] || 0n) << (i * 8n);
  }
  return value;
}

function interpret(process) {
  // TODO: interpret
}

function main(file) {
  const memory = new Array(10000);
  const code = fs.readFileSync(file).toString();
  const { instructions, labels } = parse(code);

  const registers = REGISTERS.reduce((rs, r) => ({ ...rs, [r]: 0n }), {});
  registers.RIP = BigInt(labels.main === undefined ? labels._main : labels.main);
  registers.RSP = BigInt(memory.length - 8);

  const process = {
    registers,
    memory,
    instructions,
    labels,
  };

  writeMemoryBytes(process, registers.RSP, registers.RSP, 8);

  interpret(process);
  return Number(registers.RAX);
}

process.exit(main(process.argv[2]));

We'll accept _main as an entry point as well as main to support our macOS users. If you know why our macOS users use _main I'd love to know.

To interpret, we grab the instruction pointed to in RIP and switch on the operation.

function interpret(process) {
  do {
    const instruction = process.instructions[process.registers.RIP];
    switch (instruction.operation.toLowerCase()) {
      case 'mov':
        break;
      case 'add':
        break;
      case 'call':
        break;
      case 'ret':
        break;
      case 'push':
        break;
      case 'pop':
        break;
    }
  } while (process.registers.RIP != BigInt(readMemoryBytes(process, memory.length - 8, 8)));
}

Interpreting MOV

Example:

  MOV RAX, 1
  MOV RAX, RDI
  MOV QWORD PTR [RBP - 8], 8

This instruction will store a value into a register or address and increment RIP. If the left-hand side is a memory address we will write to memory.

      case 'mov': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const rhs = interpretValue(process, instruction.operands[1]);
        if (REGISTERS.includes(lhs)) {
          process.registers[lhs] = rhs;
        } else {
          writeMemoryBytes(process, lhs.address, rhs, lhs.size);
        }
        process.registers.RIP++;
        break;
      }

We're delegating to a helper function to handle registers vs. memory addresses vs. literals appropriately. Without memory addresses it's a simple function:

function interpretValue(process, value, { lhs } = { lhs: false }) {
  if (REGISTERS.includes(value)) {
    if (lhs) {
      return value;
    } else {
      return process.registers[value];
    }
  }

  return BigInt.asIntN(64, value);
}

We need to do some hacking to support memory addresses:

function interpretValue(process, value, { lhs } = { lhs: false }) {
  if (REGISTERS.includes(value)) {
    if (lhs) {
      return value;
    } else {
      return process.registers[value];
    }
  }

  if (value.startsWith('QWORD PTR [')) {
    const offsetString = value.substring('QWORD PTR ['.length, value.length - 1).trim();
    if (offsetString.includes('-')) {
      const [l, r] = offsetString.split('-').map(l => interpretValue(process, l.trim()));
      const address = l - r;
      const bytes = 8; // qword is 8 bytes
      if (lhs) {
        return { address, size: bytes };
      } else {
        return readMemoryBytes(process, address, bytes);
      }
    }

    throw new Error('Unsupported offset calculation: ' + value);
  }

  return BigInt.asIntN(64, value);
}

Interpreting ADD

Example:

  ADD RAX, RDI

This instruction will combine both registers and store the result in the first, then increment the RIP register.

      case 'add': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const rhs = interpretValue(process, instruction.operands[1]);
        process.registers[lhs] += rhs;
        process.registers.RIP++;
        break;
      }

Interpreting CALL

Example:

  CALL plus

This instruction store RIP (plus one, to continue after the call instruction) on the stack and sets RIP to the location specified by the label.

      case 'call': {
        process.registers.RSP -= 8n;
        writeMemoryBytes(process, process.registers.RSP, process.registers.RIP + 1n, 8);
        const label = instruction.operands[0];
        process.registers.RIP = process.labels[label];
        break;
      }

Interpreting RET

Example:

RET

This instruction removes the last element from the stack and stores it in the RIP register.

      case 'ret': {
        const value = readMemoryBytes(process, process.registers.RSP, 8);
        process.registers.RSP += 8n;
        process.registers.RIP = value;
        break;
      }

Interpreting PUSH

Example:

  PUSH RAX

This instruction stores the value in the register on the stack and increments RIP.

      case 'push': {
        const value = interpretValue(process, instruction.operands[0]);
        process.registers.RSP -= 8n;
        writeMemoryBytes(process, process.registers.RSP, value, 8);
        process.registers.RIP++;
        break;
      }

Interpreting POP

Example:

  POP RAX

This instruction removes the last element from the stack and stores it into the register specified. Then it increments RIP.

      case 'pop': {
        const lhs = interpretValue(process, instruction.operands[0], { lhs: true });
        const value = readMemoryBytes(process, process.registers.RSP, 8);
        process.registers.RSP += 8n;
        process.registers[lhs] = value;
        process.registers.RIP++;
        break;
      }

All together

$ cat test1.asm
main: ; the required main label
  MOV RAX, 1 ; store 1 in RAX
  MOV RDI, 2 ; store 2 in RDI
  ADD RAX, RDI ; store the result of adding RAX and RDI in RAX
  RET ; give control back to the virtual machine
$ node emulator.js test1.asm
$ echo $?
3

And finally, let's see what we can do with a simple C program:

$ cat plus.c
long main() {
  long a = 5;
  long b = 6;
  return a + b;
}
$ gcc -S -masm=intel -o plus.s plus.c
$ node emulator.js plus.s
$ echo $?
11

And we've got the start of a working x86_64/AMD64 emulator.

Next steps

We aren't setting flags appropriately to support conditionals, so that's low-hanging fruit. Additionally, syscalls open up a new world (that we'll end up needing since exit codes are limited to 8-bits of information). Additionally, our parsing is brittle. Dealing with ELF files may be a better direction to go and also enables more. We'll explore these aspects and others in follow-up posts.

Human interest

I originally intended to build a GameBoy emulator because the hardware is simple and uniform. But I found it easiest to start hacking together an AMD64 emulator because AMD64 is well-documented and gcc is easy enough to use. I'm still interested though unless/until I figure out how to emulate a graphics card for AMD64.

It's tricky! But not that tricky. I built a graphical debugger around this emulator to help out with the logic and off-by-one errors. But ultimately it's been surprising to me how easy it is to get started -- especially when I'm not concerned about absolute correctness (yet).

Here's my first post on a series on emulator basics. It's baby's first stack and register virtual machine and it turns out it runs x86 code. https://t.co/WiWmGedawt #linux #assembly https://t.co/xjiMkhgpdN
— Phil Eaton (@phil_eaton) May 24, 2019

Improving a session cookie

infrequent oscillations — Mon, 20 May 2019 16:22:25 GMT

In which we improve a session cookie with signing and sanity

Migrating DNSCrypt Server to Docker

Nemo's Home — Sat, 18 May 2019 03:00:00 GMT

I’ve been running a personal DNSCrypt server in Bangalore for the last 2 years. When I set it up, it was just a compiled version of dnscrypt-wrapper, which was the bare minimum setup I could do.

Since then, I’ve upgraded it to a distribution supported version, but recent changes in dnscrypt key rotation, I’ve been wanting to setup something automated as well.

The easiest way was to switch to the official DNSCrypt Docker image, which does both key generation and certificate rotation. Since my public key was already present in the DNSCrypt Server lists, I was not too keen to regenerate a new key.

The primary challenge was ensuring that the docker container picks up my existing keys without trying to generate new ones from scratch. It was basically 2 steps:

Match the directory structure that the container expects.
Invoke the container directly into start mode while passing existing keys.

Directory Structure

I copied my keys (public.key, secret.key) to /etc/dnscrypt-keys and ran the following:

echo 2.dnscrypt-cert.captnemo.in > provider_name
touch provider_info.txt # I couldn't figure out how to output the same info, so kept it blank
hexdump -ve '1/1 "%.2x"' < public.key > public.key.txt

Then I ensured that the file permissions are matching what the container expects:

chmod 640 secret.key
chmod 644 public.key
chown root:1002 public.key secret.key
chmod 644 provider_name

This is how the final permissions looked for the directory (/etc/dnscrypt-keys)

-rw-r-----   1 root 1002    64 May 18 07:15 secret.key
-rw-r--r--   1 root 1002    32 May 18 07:15 public.key
-rw-r--r--   1 root root    28 May 18 07:19 provider_name
-rw-r--r--   1 root root     0 May 18 07:23 provider_info.txt
-rw-r--r--   1 root root    64 May 18 07:25 public.key.txt
drwxr-xr-x   2 root root  4096 May 18 07:26 .

Running the Container

Then, I directly ran dnscrypt-wrapper container:

docker run --detatched --restart=unless-stopped --volume /etc/dnscrypt-keys:/opt/dnscrypt-wrapper/etc/keys --publish 10.47.0.5:4434:443/tcp --publish 10.47.0.5:4434:443/udp jedisct1/dnscrypt-server start

I pass a host path mount instead of creating a Docker Volume, since they can get deleted in regular docker prune.

Here, 10.47.0.5 is the “Anchor IP”, which Digital Ocean internally maps to my Floating IP.

The container comes up, generates new short-term keys and goes live:

Starting DNSCrypt service for provider:
2.dnscrypt-cert.captnemo.in
Starting pre-service scripts in /etc/runit_init.d
setup in directory /opt/unbound/etc/unbound
generating unbound_server.key
Generating RSA private key, 3072 bit long modulus (2 primes)
.......++++
...................++++
e is 65537 (0x010001)
generating unbound_control.key
Generating RSA private key, 3072 bit long modulus (2 primes)
.........................++++
........................................++++
e is 65537 (0x010001)
create unbound_server.pem (self signed certificate)
create unbound_control.pem (signed client certificate)
Signature ok
subject=CN = unbound-control
Getting CA Private Key
Setup success. Certificates created. Enable in unbound.conf file to use
ok: run: unbound: (pid 28) 300s
ok: run: dnscrypt-wrapper: (pid 31) 300s
ok: run: unbound: (pid 28) 600s
ok: run: dnscrypt-wrapper: (pid 31) 600s

Once the server was up, I verified connectivity with dnscrypt-proxy and it worked perfectly.

Future Scope

Right now, I have a single container that does 2 things:

Certificate Rotation via a service that checks it every 30 minutes.
DNSCrypt Service, which is accessible over the internet.

For (1) to work, it needs access to the Private Keys that are used to sign the temporary certificates that last 24 hours. Since both things are managed within the same container, the container ends up with both network and long-term keys access. This means, any RCE on the service can result in the long-term keys being compromised.

A simple fix for this would be to separate out the Certificate Rotation part into a separate “mode” on the Docker Image, which can be called independently. This would allow someone to run certificate rotation on a second container using a scheduler, but with far more limitations (such as no network access). A common file-mount between both the containers can take care of sharing the temporary keys between the containers, and a simple unix socket on the shared-file-mount can be used to signal a certificate rotation (this triggers the dnscrypt service restart, so it picks the new cert).

How To Name Things

swyx's site RSS Feed — Thu, 16 May 2019 03:00:00 GMT

> There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. - Leon Bambrick

Tail call elimination

Notes on software development — Tue, 14 May 2019 03:00:00 GMT

In this post we'll explore what tail calls are, why they are useful, and how they can be eliminated in an interpreter, a compiler targeting C++, and a compiler targeting LLVM IR.

Tail calls

A tail call is a function call made at the end of a block that returns the value of the call (some languages do not force this return requirement). Here are a few examples.

function tailCallEx1() {
  // Loops forever but is a tail call.
  return tailCallEx1();
}

function tailCallEx2(x) {
  if (x) {
    return tailCallEx2(x - 1);
  }

  return 1;
}

function tailCallEx3(x) {
  return x && tailCallEx(x - 1) || 1;
}

function tailCallEx4(x) {
  switch (x) {
    case 0:
      return 1;
    default:
      return tailCallEx4(x - 1);
  }
}

And here are some examples of non-tail calls.

function nonTailCallEx1(x) {
  if (x) {
    // Not a tail call because the call is not the value returned.
    return 1 + nonTailCallEx1(x - 1);
  }

  return 0;
}

function nonTailCallEx2(x) {
  if (x) {
    const r = nonTailCallEx2(x - 1);
    // Not a tail call because the value is not *immediately* returned.
    console.log(r);
    return r;
  }

  return 1;
}

Why is this important?

Some languages can rewrite a recursive tail call as a jump/branch/goto instead of a function call. This allows:

Potential performance gain if function calls have large overhead
No stack overflows due to no nested function call stacks

Implementation 1: Interpreter

Given a tail call recursive fibonacci:

function fibonacci(a, b, n) {
  if (n === 0) {
    return a;
  }

  if (n === 1) {
    return b;
  }

  return fibonacci(b, a + b, n - 1);
}

Here is how we could transform (by hand) this without a tail call.

function fibonacci(a, b, n) {
  while (true) {
    if (n === 0) {
      return a;
    }

    if (n === 1) {
      return b;
    }

    const a1 = b;
    const b1 = a + b;
    const n1 = n - 1;
    a = a1;
    b = b1;
    n = n1;
  }
}

If this was written in a language with labels and goto we could simplify the code slightly by doing that. But it is the same effect as a loop.

Since we're in an interpreter (that isn't JIT compiling), we cannot pick between these two and must merge them. So we put all function bodies in a loop and break if it isn't a tail call. Otherwise we line up the paremeters and let the loop take us back.

Here is an example of this strategy used in a Scheme interpreter written in D.

// Define a new function with name `name` and add it to the context.
Value namedLambda(Value arguments, Context ctx, string name) {
  auto funArguments = car(arguments);
  auto funBody = cdr(arguments);

  Value defined(Value parameters, void** rest) {
    Context newCtx = ctx.dup;

    // Copy the runtime calling context to the new context.
    Context runtimeCtx = cast(Context)(*rest);
    auto runtimeCallingContext = runtimeCtx.callingContext;
    newCtx.callingContext = runtimeCallingContext.dup;

    Value result;
    // Loop forever, will break immediately if not a tail call
    bool tailCalling = false;
    while (true) {
      if (valueIsList(funArguments)) {
        auto keyTmp = valueToList(funArguments);
        auto valueTmp = valueToList(parameters);
        while (true) {
          auto key = valueToSymbol(keyTmp[0]);
          auto value = valueTmp[0];

          newCtx.set(key, value);

          // TODO: handle arg count mismatch
          if (valueIsList(keyTmp[1])) {
            keyTmp = valueToList(keyTmp[1]);
            valueTmp = valueToList(valueTmp[1]);
          } else {
            break;
          }
        }
      } else if (valueIsSymbol(funArguments)) {
        auto key = valueToSymbol(funArguments);
        newCtx.set(key, car(parameters));
      } else if (!valueIsNil(funArguments)) {
        error("Expected symbol or list in lambda formals", funArguments);
      }

      if (!tailCalling) {
        newCtx.callingContext.push(Tuple!(string, Delegate)(name, &defined));
      }

      result = eval(withBegin(funBody), cast(void**)[newCtx]);

      // In a tail call, let the loop carry us back.
      if (newCtx.doTailCall == &defined) {
        tailCalling = true;
        parameters = result;
        newCtx.doTailCall = null;
      } else {
        break; // Not in a tail call, we're done a regular function call.
      }
    }

    return result;
  }

  return makeFunctionValue(name, &defined, false);
}

We can not eliminate mutually recursive tail calls with this method. We could use continuation-passing style but that would not have addressed the concern: not making a function call.

Implementation 2: Compiling to C++

The strategy here is the same as in the interpreter except for that since tail call recursive functions are known at compile time, we can generate non-generalized code in function bodies.

Here is how a JavaScript compiler transforms the above fibonacci implementation into C++:

void tco_fib(const FunctionCallbackInfo<Value> &args) {
  Isolate *isolate = args.GetIsolate();
  double tco_n = toNumber(args[0]);
  double tco_a = toNumber(args[1]);
  double tco_b = toNumber(args[2]);

tail_recurse_1:

    ;

  bool sym_if_test_58 = (tco_n == 0);
  if (sym_if_test_58) {
    args.GetReturnValue().Set(Number::New(isolate, tco_a));
    return;
  }

  bool sym_if_test_70 = (tco_n == 1);
  if (sym_if_test_70) {
    args.GetReturnValue().Set(Number::New(isolate, tco_b));
    return;
  }

  Local<Value> sym_arg_83 = Number::New(isolate, (tco_n - 1));
  Local<Value> sym_arg_92 = Number::New(isolate, (tco_a + tco_b));
  tco_n = toNumber(sym_arg_83);
  tco_a = tco_b;
  tco_b = toNumber(sym_arg_92);
  goto tail_recurse_1;
}

This is implemented by checking every function call. If the function call is in tail call position, we generate code for jumping to the beginning of the function. Otherwise, we generate a call as usual.

Here is how the tail call check and code-generation is done in the compiler:

function compileCall(
  context: Context,
  destination: Local,
  ce: ts.CallExpression,
) {
  let tcoLabel;
  let tcoParameters;
  if (ce.expression.kind === ts.SyntaxKind.Identifier) {
    const id = identifier(ce.expression as ts.Identifier);
    const safe = context.locals.get(mangle(context.moduleName, id));

    if (safe && context.tco[safe.getCode()]) {
      const safeName = safe.getCode();
      tcoLabel = context.tco[safeName].label;
      tcoParameters = context.tco[safeName].parameters;
    }
  }

  ...

  if (ce.expression.kind === ts.SyntaxKind.Identifier) {
    const id = identifier(ce.expression as ts.Identifier);
    const mangled = mangle(context.moduleName, id);
    const safe = context.locals.get(mangled);

    if (safe) {
      if (tcoLabel) {
        args.forEach((arg, i) => {
          compileParameter(
            context,
            tcoParameters[i],
            i,
            i === args.length - 1,
            arg,
          );
        });

        context.emitStatement(`goto ${tcoLabel}`);
        context.emit('', 0);
        destination.tce = true;
        return;
      }
    }
  }

  ...

  // Otherwise generate regular function call

This requires you to have been building up the state throughout the AST to know whether or not any particular call is in tail position.

Implementation 3: Compiling to LLVM IR

LLVM IR is the most boring because all you do is mark any tail call as being a tail call. Then so long as the call meets some requirements, the key one being that the result of the call must be returned immediately, LLVM will generate a jump instead of a call for you.

Given the following lisp-y implementation of the same tail call recursive fibonacci function (compiler here):

(def fib (a b n)
     (if (= n 0)
         a
       (fib b (+ a b) (- n 1))))

We generate the following LLVM IR:

define i64 @fib(i64 %a, i64 %b, i64 %n) {
  %ifresult13 = alloca i64, align 4
  %sym14 = add i64 %n, 0
  %sym15 = add i64 0, 0
  %sym12 = icmp eq i64 %sym14, %sym15
  br i1 %sym12, label %iftrue16, label %iffalse17
iftrue16:
  %sym18 = add i64 %a, 0
  store i64 %sym18, i64* %ifresult13, align 4
  br label %ifend19
iffalse17:
  %sym21 = add i64 %b, 0
  %sym23 = add i64 %a, 0
  %sym24 = add i64 %b, 0
  %sym22 = add i64 %sym23, %sym24
  %sym26 = add i64 %n, 0
  %sym27 = add i64 1, 0
  %sym25 = sub i64 %sym26, %sym27
  ; NOTE the `tail` before `call` here
  %sym20 = tail call i64 @fib(i64 %sym21, i64 %sym22, i64 %sym25)
  ret i64 %sym20
  store i64 %sym20, i64* %ifresult13, align 4
  br label %ifend19
ifend19:
  %sym11 = load i64, i64* %ifresult13, align 4
  ret i64 %sym11
}

The only difference between supporting tail call elimination in is whether the call instruction is preceeded by a tail directive. That makes the implementation very simple:

const isTailCall = module.exports.TAIL_CALL_ENABLED &&
                   context.tailCallTree.includes(validFunction.value);
const maybeTail = isTailCall ? 'tail ' : '';
this.emit(1, `%${destination.value} = ${maybeTail}call ${validFunction.type} @${validFunction.value}(${safeArgs})`);
if (isTailCall) {
  this.emit(1, `ret ${destination.type} %${destination.value}`);
}

Generated assembly

The resulting generated code (run through llc) for that call will be:

...
    add rax, rsi
    dec rdx
    mov rdi, rsi
    mov rsi, rax
    jmp _fib               ## TAILCALL
...

And if tail call elimination is disabled:

...
    add rax, rsi
    dec rdx
    mov rdi, rsi
    mov rsi, rax
    call    _fib
...

Summary

The last bit I haven't covered is how you track whether or not a call is in tail position. That is difficult to cover in a blog post because it's a matter of you propagating/not propagating at each syntax node type. But generally speaking, if the syntax node is not in tail position (e.g. not the last expression in a block), you drop the tail state you've built up. When you make a function call, you add the function name to the tail state.

But I will be covering this in detail in the LLVM case in the next post in my compiler basics series.

Put together a survey and summary of tail call elimination, the effect and implementation, in an interpreter, a compiler targeting C++, and a compiler targeting LLVM IR. https://t.co/pXiLoXjw2u
— Phil Eaton (@phil_eaton) May 15, 2019

Solving CIFAR-10 with Pytorch and SKL

Bytepawn - Marton Trencseni — Tue, 14 May 2019 01:00:00 GMT

CIFAR-10 is a classic image recognition problem, consisting of 60,000 32x32 pixel RGB images (50,000 for training and 10,000 for testing) in 10 categories: plane, car, bird, cat, deer, dog, frog, horse, ship, truck. Convolutional Neural Networks (CNN) do really well on CIFAR-10, achieving 99%+ accuracy. The Pytorch distribution includes an example CNN for solving CIFAR-10, at 45% accuracy. I will use that and merge it with a Tensorflow example implementation to achieve 75%. We use torchvision to avoid downloading and data wrangling the datasets. Like in the MNIST example, I use Scikit-Learn to calculate goodness metrics and plots.

Football update, 2018-19 season

infrequent oscillations — Mon, 13 May 2019 14:00:18 GMT

In which we import numbers and update information

Updates to Iron Arachne: The World API and more

Ben Overmyer's Site — Sun, 12 May 2019 03:00:00 GMT

It's been awhile since I blogged about Iron Arachne. Much has changed since then.

The various disparate APIs have been collapsed into a single "World API." This makes it much easier to change, which means I can iterate faster and release more frequently. Also, the front-facing website is no longer a static JavaScript-powered site, but rather a full Laravel application. This lets me do a few clever things which I'll get into later.

The culture generator has added appearance generation. Skin colors, hair colors, face shapes, and so forth add to the depth of the output.

The region generator has dramatically improved economic traits. The trade goods are derived from the skill of the local artisans, the local agriculture, and the local available raw materials now. They also take into account the culture of the town.

Deities created by the pantheon generator now adhere to the culture the religion is from. Also, there are subtle improvements to how deities' domains are generated. No longer will multiple gods have the same domain.

The heraldry generator has had a few tweaks, too, though no new fields or charges have been added. Because the app was rewritten in Laravel, I now make use of caching via Redis to deliver consistent images and blazons. For example, here's one I quite like:

Per saltire azure and argent, a lion's head erased Or

Some of the groundwork has been laid for organizing regions into countries. This is a step into a more difficult part of the world generator, since now I need to take into account location relative to other locations. Inevitably, I'll need to create a map generator. But that's a story for another day.

Asymmetrical Football

infrequent oscillations — Thu, 09 May 2019 06:45:51 GMT

In which an overlap is missing.

Processing Large Files – Java, Go and 'hitting the wall'

Ben E. C. Boyter — Wed, 08 May 2019 03:00:00 GMT

It started with this chain of blog posts,

Using Java to Read Really, Really Large Files -> Processing Large Files in Java -> Processing Large Files – Java, Go and ‘hitting the wall’

I thought after reading them I would add to the chain.

While I don’t like solving arbitrary company made “programming tests” when doing interviews https://boyter.org/2016/09/companies-recruiting/ whenever I see a post comparing Go to Java where Java is trounced in performance gets my attention. Unless its a very short lived command line application where JVM startup is a problem or some super optimized Go solution in my experience so far Java should generally be slightly faster. The computer language benchmarks tend to show the same result so its not a flight of fancy for me.

Writing a lisp compiler from scratch in JavaScript: 4. LLVM conditionals and compiling fibonacci

Notes on software development — Sat, 04 May 2019 03:00:00 GMT

Previously in compiler basics: <! forgive me, for I have sinned >
1. lisp to assembly
2. user-defined functions and variables
3. LLVM
Next in compiler basics:
5. LLVM system calls
6. an x86 upgrade

In this post we'll extend the compiler's LLVM backend to support compiling conditionals such that we can support an implementation of the fibonacci algorithm.

Specifically we're aiming for the following:

$ cat tests/fib.lisp
(def fib (n)
     (if (< n 2)
         n
       (+ (fib (- n 1)) (fib (- n 2)))))

(def main ()
     (fib 8))
$ node src/ulisp.js tests/fib.lisp
$ ./build/prog
$ echo $?
21

To do this we'll have to add the <, - and if built-ins.

All source code is available on Github.

Subtraction

This is the easiest to add since we already support addition. They are both arithmetic operations that produce an integer. We simply add a mapping of - to the LLVM instruction sub so our LLVM backend constructor (src/backends/llvm.js) looks like this:

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),

...

Less than

The < builtin is a logical operation. These are handled differently from arithmetic operations in LLVM IR. A logical operation looks like this:

%3 = icmp slt i32 %1, %2

This says that we're doing an integer comparison, icmp, (with signed less than, slt) on the i32 integers in variables %1 and %2.

We can shim this into our existing compileOp helper like so:

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      'if': this.compileIf.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '<': this.compileOp('icmp slt'),

...

Conditionals

The last part we need to add is support for conditional execution of code at runtime. Assembly-like languages handle this with "jumps" and "labels". Jumping causes execution to continue at the address being jumped to (instead of just the line following the jump instruction). Labels give you a way of naming an address instead of having to calculate it yourself. Our code will look vaguely like this:

  %test = icmp slt i32 %n, %1
  br i1 %test, label %iftrue, label %iffalse
iftrue:
  ; do true stuff
iffalse:
  ; do false stuff

  ; do next stuff

The br instruction can jump (or branch) conditionally or unconditionally. This snippet demonstrates a conditional jump.

But there are a few things wrong with this pseudo-code. First off if the condition is true, execution will just continue on into the false section once finished. Second, LLVM IR actually requires all labels to end with a branch instruction. So we'll add a new label after the true and false section called ifresult and jump to it unconditionally after both.

  %test = icmp slt i32 %n, %1
  br i1 %test, label %iftrue, label %iffalse
iftrue:
  ; do true stuff
  br label %ifresult
iffalse:
  ; do false stuff
  br label %ifresult
ifresult:
  ; do next stuff

Scope

One last thing we'll need to do before implementing the code generation for this is to update our Scope class to accept symbol prefixes so we can pass our labels through Scope to make sure they are unique but still have useful names.

...

class Scope {

  ...

  symbol(prefix = 'sym') {
    const nth = Object.keys(this.locals).length + 1;
    return this.register(prefix + nth);
  }

...

compileIf

Now we can add a primitive function mapping if to a new compileIf helper and implement the helper.

...

class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '<': this.compileOp('icmp slt'),
      'if': this.compileIf.bind(this),

...

  compileIf([test, thenBlock, elseBlock], destination, scope) {
    const testVariable = scope.symbol();

    // Compile expression and branch
    this.compileExpression(test, testVariable, scope);
    const trueLabel = scope.symbol('iftrue');
    const falseLabel = scope.symbol('iffalse');
    this.emit(1, `br i1 %${testVariable}, label %${trueLabel}, label %${falseLabel}`);

    // Compile true section
    this.emit(0, trueLabel + ':');
    this.compileExpression(thenBlock, destination, scope);
    const endLabel = scope.symbol('ifend');
    this.emit(1, 'br label %' + endLabel);
    this.emit(0, falseLabel + ':');

    // Compile false section
    this.compileExpression(elseBlock, destination, scope);
    this.emit(1, 'br label %' + endLabel);

    // Compile cleanup
    this.emit(0, endLabel + ':');
  }
...

Note that this code generation sends the destination variable into both the true and false sections. Let's try it out.


$ node src/ulisp.js tests/fib.lisp
llc: error: llc: build/prog.ll:19:3: error: multiple definition of local value named 'sym5'
  %sym5 = add i32 %sym15, %sym16
  ^
child_process.js:665
    throw err;
    ^

Error: Command failed: llc -o build/prog.s build/prog.ll
llc: error: llc: build/prog.ll:19:3: error: multiple definition of local value named 'sym5'
  %sym5 = add i32 %sym15, %sym16

That's annoying. An unfortunate aspect of LLVM's required
single-static assignment form is that you cannot reuse variable names
within a function even if it is not possible for the variable to be
actually reused.
To work around this we need to allocate memory on the stack, store the
result in each true/false section in this location, and read from this
location afterward to store it in the destination variable.
Stack memory instructions
LLVM IR gives us alloca to allocate memory on the stack,
store to store memory at a stack address, and
load to read the value at a stack address into a
variable. Here's a simple example:
%myvar = add i32 42, 0
%stackaddress = alloca i32, align 4
store i32 %myvar, i32* %stackaddress, align 4
%newvar = load i32, i32* %stackaddress, align 4

Such that newvar is now 42.
compileIf again
Applying this back to our compileIf helper gives us:
...

  compileIf([test, thenBlock, elseBlock], destination, scope) {
    const testVariable = scope.symbol();
    const result = scope.symbol('ifresult');
    // Space for result
    this.emit(1, `%${result} = alloca i32, align 4`);

    // Compile expression and branch
    this.compileExpression(test, testVariable, scope);
    const trueLabel = scope.symbol('iftrue');
    const falseLabel = scope.symbol('iffalse');
    this.emit(1, `br i1 %${testVariable}, label %${trueLabel}, label %${falseLabel}`);

    // Compile true section
    this.emit(0, trueLabel + ':');
    const tmp1 = scope.symbol();
    this.compileExpression(thenBlock, tmp1, scope);
    this.emit(1, `store i32 %${tmp1}, i32* %${result}, align 4`);
    const endLabel = scope.symbol('ifend');
    this.emit(1, 'br label %' + endLabel);
    this.emit(0, falseLabel + ':');

    // Compile false section
    const tmp2 = scope.symbol();
    this.compileExpression(elseBlock, tmp2, scope);
    this.emit(1, `store i32 %${tmp2}, i32* %${result}, align 4`);
    this.emit(1, 'br label %' + endLabel);

    // Compile cleanup
    this.emit(0, endLabel + ':');
    this.emit(1, `%${destination} = load i32, i32* %${result}, align 4`);
  }

...

Trying it out
We run our compiler one more time:
$ node src/ulisp.js tests/fib.lisp
$ ./build/prog
$ echo $?
21

And get what we expect!
Next up

Tail call optimization
Lists and dynamic memory
Strings?
Foreign function calls?
Self-hosting?

Latest post in the compiler basics series: using LLVM conditionals in compiling a fibonacci program https://t.co/A72yEDQ8sd
— Phil Eaton (@phil_eaton) May 5, 2019



Solving MNIST with Pytorch and SKL
Bytepawn - Marton Trencseni — Thu, 02 May 2019 01:00:00 GMT
MNIST is a classic image recognition problem, specifically digit recognition. It contains 70,000 28x28 pixel grayscale images of hand-written, labeled images, 60,000 for training and 10,000 for testing. Convolutional Neural Networks (CNN) do really well on MNIST, achieving 99%+ accuracy. The Pytorch distribution includes a 4-layer CNN for solving MNIST. Here I will unpack and go through this example. We use torchvision to avoid downloading and data wrangling the datasets. Finally, instead of calculating performance metrics of the model by hand, I will extract results in a format so we can use SciKit-Learn's rich library of metrics. 

 


Come over for dinner
CRAIG KERSTIENS — Wed, 01 May 2019 23:55:56 GMT
When I first moved to the Bay area I was fresh out of grad school. I was frequently heading out to dinner or to happy hour after work with colleagues. I was young and single, so why not of course. As time passed, marriage, kids, etc. the ability to go out for a quick drink or dinner was competing with various priorities. Dinner and drinks with co-workers was always a great time. It wasn’t just about hanging out, it built rapport and trust which I found made me a more effective teammate and product manager. It was about 8 years ago that I started to implement a variation of heading out for dinner and drinks.
I started inviting people over for dinner.
I still do this regularly. Roughly once a week we end up hosting someone for dinner. Sometimes it is a single person, sometimes it is a group of people. Sometimes it is co-workers, sometimes former colleagues, often friends that don’t work in tech. Growing up in the south it was common to have people over, I’d said we did that just as much as going out to dinner with folks. You’d get an invite to go to someone elses place and you’d show up with a bottle of wine or flowers in hand. Initially when I asked people in the Bay area over for dinner I’d get weird looks. Over? Like to your house?
The reaction from folks at the end of the night was very often… that was really fun. Thanks for the invite, I can’t remember the last time I just sat down at someones place, had a good meal, and conversation.
Once I found early success with this I started implementing it pretty methodically. When remote workers were in town I’d make sure to place them at thet top of the list to come if the scheduling worked. Same when friends visit from out of town. I’d also try to regularly rotate through my teams and those that report to me. At one point when I had 22 engineers that I was leading product for I had to do a bit of juggling and stagger things a bit, groups of 4 folks or so at a time and each would be over about once every 6 months. I made life a bit easier on myself by grouping vegans together, and vegetarians, and meat eaters. When you’re cooking in the Bay area you probably don’t have a 500 sq. foot kitchen, so cooking 2-3 different meals adds complexity. Thus simplifying.
All of this was for selfish reasons. I could excuse myself for 15 minutes to give the kids a bath, to read them a story, to tuck them in. At the same time I was able to continue building a rapport with co-workers that allowed us both to work together better.


Responsibility and ownership
Notes on software development — Tue, 30 Apr 2019 03:00:00 GMT
Responsibility is only possible by granting ownership and setting
expectations. If you don't turn over ownership, don't expect folks to
take responsibility. When you grant ownership and set expectations,
you'll be astounded what folks will accomplish without you.
I am astounded.


Reinforcement Learning: Game Theory
swyx's site RSS Feed — Mon, 29 Apr 2019 06:59:47 GMT
RL with multiple actors


Talking on the phone: better communication
CRAIG KERSTIENS — Wed, 24 Apr 2019 23:55:56 GMT
I interact with a lot of people in a given week, a few in person and far more on video and conference calls. I don’t claim to be a perfect person to talk to on the phone, but over the past several years I’ve noticed how painful some conference call experiences can be. As more and more work is conducted virtually and not face to face an ability to do communicate well on conference/voice calls is tied to what success you can deliver. It isn’t about having a fancy phone or high bandwidth video call–though that can at times be useful


Let’s start with local/remote teams


This one is a huge problem as companies go from 100% local to partially remote. When you start a meeting take note of how many folks are local vs. remote. If you have the majority of attendees on a video or conference call you’re likely fine. If however the majority is local you need to make a concerted effort to pause and ask for input from the phone. This is doubley true on large conference calls… in those cases it can be best to explicitly call out remote folks to chime in.


Talk to the phone


When you’re in person folks can tend to wander, look away, work on a whiteboard. All of these things can be very collaborative when in person but are absolutely terrible on the phone. When you’re not speaking into where the microphone is you will immediately come across choppy. The subtle take away is you’re not a professional. That doesn’t mean you’re doing it intentionally, but when interacting via voice you have to be extra intentional to come across how you intend.


Strong, short words


This applies when in person, but is especially true on the phone when you can’t read someone else. If you don’t have verbal cues to feed off from the other person then you need to listen to audible ones. If you’re talking for 5 minutes non-stop then you’ve learned nothing about how the other person is reacting. At times you may need to talk that long. I like to talk, but after 5 minutes talking non-stop on the phone I have to make sure I stop and say I’ve been talking too long. One it is only fair, and two it is a converation and you need to ensure that it’s flowing both ways.


Gone with the wind


Your environment and surroundings are absolutely terrible to a productive conversation. If you’re walking and there is any sense of wind, if the room is echo-ey, fix it. I’ve seen this both with people taking calls in transit, but also with conference rooms that weren’t well designed. For in transit invest in a great headset if you’ll be taking calls. Test several, read some reviews. If it is a conference actively ask others on calls with you. To just assume it’s fine will lead to people silently dismissing you’re not serious because calls with others are smoother.


Overcommunicate


Speed up, slow down, raise your voice, lower your voice. On the phone you only have one form of expression: your voice. Use it. For some this takes practice. Whenever I’m talking to press or analysts I make sure I’m standing. I make sure to not have coffee 3-4 hrs before because otherwise I’m overly animated. Make sure to take pauses. An explicit pause can give others a chance to chime in and ask questions. When you hear someone on the phone asking a question and you already started talking because of latency finish, then apologize, and ask for them to go ahead.


Be intentional


No matter who you are or what you do you’re on the phone plenty. Whether a 1:1 call or a conference call. Most of use don’t like these calls and feel drained or frustrated by them. I don’t claim that most of my calls feel like a friday night our at a baseball game, but by being explicit in how we communicate I do feel that you can make calls not feel like busy work. Instead they can be a productive way to communicate if you make sure you’re communicating clearly and explicitly, but also listening and engaging equally with the other side of the line.


Learning Gears
swyx's site RSS Feed — Fri, 19 Apr 2019 03:00:00 GMT
> Translations welcome! (Português)


Using email as an effective tool
CRAIG KERSTIENS — Tue, 16 Apr 2019 23:55:56 GMT
I send way too many emails in a day. My inbox is very intermingled with my to do list and often represents some form of it. More relevant though is that email is a primary means of how I accomplish work. Being a PM I work cross functionally with other teams (from marketing, to engineering, to sales, to BD, to other product teams) and of course customers. Having to work so cross functionality I’ve found a lot of hacks I use to be able to better accomplish your goals with email, here is a collection of some of those.
Let me be clear, this is not another post about inbox 0, how I swapped to slack. This instead is how I use email to more effectively communicate and get people to engage. In other words it is about making emails more useful, not just getting through them faster. And onto those tips.


Don’t leave a document in a document


Often times I’ve found folks will collaborate in a document during a meeting or take notes there. After the meeting folks will email the document around, but few seldom actually open. Reasons may be they’re not logged in on their phone or it may be they just don’t care that much, I’m not really sure. What I do know is that by taking the notes and action items from the doc and including them in the email you will get more people paying attention to them. If you really want to get good at this when there is significant revisions of a work in progress document, re-circulate the updated version via email. Again not just a link to it, but the document itself.


Mail merge isn’t just for marketing


Years ago I sent a company wide request for feedback (to about 120 people). I got less than 3 responses. I ran an experiment the next time I needed the same thing using a mail merge so my same email seemed personal and was from me to them instead of some large alias. I got a response rate of over 45%. Use this wisely… not every email you send needs a response. A broad update can absolutely be to team@, but when you need to get actual feedback and people don’t seem to chime in this approach works great.


Conversation begets conversation


Often times when there is an email to team@ folks just simply click archive. A great tip for managers and leadership is to chime in on these emails right away. It can help promote some positive discussion and cause folks to actually take notice of updates that might otherwise end up archived. Be careful of the inverse of this though, replying to all mass emails can create a culture where everyone thinks it’s their job to +1 and thumbs up things. There is a balance to this one between team productivity and team morale.


Text is boring


Email is not like reading a novel (except when my colleague Daniel Farina writes them), if I wanted to read a novel I’d actually go read a novel. For email I want the clear concise points broken out. You should think about how easy it is to read on a phone. If a single paragraph consumes my entire screen you’ve lost your readers attention and that content goes else where. A number of things can help for readability:

Short paragraphs and usage of line breaks
Bullets are your friend (aren’t you enjoying this list already?)
Numbers are also great as they call attention to themselves

Bold/italics can also be useful, but assume many people use text only clients so you need to be careful. Also really don’t over-use these


Add extra tooling


Gmail and Outlook are both continuing to improve over time. With the demise of Inbox Gmail itself is seeing some more of those features. Reminders about old un-responded to emails are great. But you can do much more by adding in a third party tool. Boomerang and yesware are two popular ones. These can help with things like returning an email to your inbox in a couple weeks so it can be out of sight for now but come back later. You can also schedule emails to be sent later at a time that may be more ideal for someone to read it, instead of sending it at midnight what about 9am so it appears more top of their inbox?


Do you know about muting?


Muting is the single greatest feature within gmail. Email is a great place for an archive and history, it is a great place to clearly communicate things, it can be a horrible place when it comes to generating a bunch of noise.
Personally I subscribe to lots of lists. I have filters for many of those, but many I actually want to see. That doesn’t mean I care about every response to every thread on those lists. When engineering is discussing whether to use CircleCI vs. Travis I see that the discussion is happening but don’t care too much on the outcome. Mute will silence that entire thread and move it to archived for me. It will only re-appear if I am explicitly added to the To:
The fact that it persists in my archive is key for me. If I want to go back later and search to see the outcome I can, but I don’t have to create a new rule, or archive each reply to the thread.


These are just a few


These are only a few tips, but ones I’ve found extremely useful on two sides. The first is being more control of my inbox so it doesn’t control me. A workflow that works for me which includes mute, filtering lists, scheduling emails for later helps with that. The second is to me email isn’t something I have to do, it is a tool to work more effectively. Being intentional on how I construct messages, when I send them, and what is clearly communicated is key for my job.
I’m curious, what are some of your favorite email tips @craigkerstiens.


Developer Empowerment - Move fast, don't break things (for long), be productive
Ben E. C. Boyter — Tue, 16 Apr 2019 03:00:00 GMT
Recently my job/career path has had me looking “under the hood” at clients development practices. The idea being to see where their strengths and weaknesses are and then make actionable suggestions.

Generally every client wants the same outcome. They want to know how to improve software quality and deliver features faster.

When doing this I focus on the development side, but honestly a lot of this applies to DevOp’s as well. In most situations I realized very quickly that I would be terrified to make any changes in the environments I am working with. No safety net where the penalty for mistakes is very high. I also see the same issues over and over again. When asked to articulate what the issues I saw at one client I came back with the words “You need to focus on developer empowerment”.


SVM with Pytorch
Bytepawn - Marton Trencseni — Tue, 16 Apr 2019 01:00:00 GMT
I use the standard Iris dataset for supervised learning with a Support Vector Machine model using Pytorch's autograd.




Stripping Audible DRM
Nemo's Home — Sun, 14 Apr 2019 03:00:00 GMT
Self-Guide for stripping the Audible DRM, in similar vein as my Kindle Self-Guide.


  Download the aax file from Audible website.
  Run the inAudible-NG Rainbrow crack table against the AAX file.


Easiest way is via docker/podman:

cd ~/Music/Audiobooks
podman run -v $(pwd):/data ryanfb/inaudible@sha256:b66738d235be1007797e3a0a0ead115fa227e81e2ab5b7befb97d43f7712fac5
for i in "*.m4a"; do fix-audible-m4a "$i";done


The cool part about this is that the entire activation is done offline, and runs a Rainbow Table attack against the Audible DRM. To make the process faster in the future, you can save your “activation bytes” (8 hex characters) and directly use them with ffmpeg to decode instead:

ffmpeg -loglevel panic -y -activation_bytes ${AUDIBLE_ACTIVATION_BYTES} -i "$aax_file" -c:a copy -vn "$m4a_file"

A small percentage of Audible AAX files have a incorrect bit set in the “Audio Object Type Specific Config” in the ESDS atom in an M4A file, which leads to them not playing in
Firefox/Android and some other players. To fix this, I have a fix-audible-m4a script called above.

References


  https://github.com/ryanfb/docker_inaudible_rainbowcrack
  https://github.com/inAudible-NG/tables
  https://rentry.co/n4ost



Interpreting TypeScript
Notes on software development — Sun, 14 Apr 2019 03:00:00 GMT
In addition to providing a static type system and compiler for a
superset of JavaScript, TypeScript makes much of its functionality
available programmatically. In this post we'll use the TypeScript
compiler API to build an interpreter. We'll build off of a TypeScript
wiki
article
and cover a few areas that were confusing to me as I built out
a separate project.
The end result we're building will look like this:
$ cat test.ts # A program we can interpret
print(1 + 5);
$ tsc interpreter.ts # Build the source code for the interpreter
$ node interpreter.js test.ts # Run the interpreter against test program
6

All code is available on Github.
Setup
To begin with, we need Node.js and some dependencies:
$ yarn add typescript @types/node

Then we can begin the first stage of an interpreter: parsing the code.
Parsing
Parsing a fixed set of files is simple enough. We pass a list of files
to createProgram along with compiler options. But, as a user, we
don't want to keep track of all files used by a program
(i.e. everything we import). The most ideal situation is to pass a
single-file entrypoint (something like a main.js) and have our
interpreter figure out all the imports and handle them
recursively. More on this later, for now we'll just parse the
single-file entrypoint.
import * as ts from 'typescript';

const TS_COMPILER_OPTIONS = {
  allowNonTsExtensions: true,
};

function parse(fileName: string): ts.Program {
  return ts.createProgram([fileName], TS_COMPILER_OPTIONS);
}

function interpret(program: ts.Program) { // TODO }

function main(entrypoint: string) {
  const program = parse(entrypoint);
  interpret(program);
}

main(process.argv[2]);

interpret and ts.Program
A program contains all source files as well as any implicitly needed
TypeScript definition files (for us it will just be the TypeScript
definitions for the Node.js standard library).

  The program also gives us access to a type checker that we can use
  to query the type of any node in the program tree. We'll get into
  this in another post.
Our interpret program will iterate over all the source files, ignoring
the TypeScript definition files, and call interpretNode on all the
elements of the source file.
function interpretNode(node: ts.Node) { // TODO }

function interpret(program: ts.Program) {
  return program.getSourceFiles().map((source) => {
    const { fileName } = source;
    if (fileName.endsWith('.d.ts')) {
      return;
    }

    const results = [];
    ts.forEachChild(source, (node) => {
      results.push(interpretNode(node));
    });
    return results;
  });
}

interpretNode and ts.Node
A Node is a wrapper for most elements of what we consider a program to
be, such as a binary expression (2 + 3), a literal
expression (2), a function call expression
(a(c)), and so forth. When exploring a parser, it takes
time to become familiar with the particular way that a parser breaks
out a program into a tree of nodes.
As a concrete example, the following program:
print(a);

Will be built into ts.Node tree along these lines:
Node: ExpressionStatement: print(a);
  Node: CallExpression: print, a
    Node: Identifier: print
    Node: Identifier: a
Node: EndOfFileToken

And another example:
1 + 3;

Will be built into a ts.Node tree along these lines:
Node: Expression: 1 + 3
  Node: BinaryExpression: 1, 3, +
    Node: NumericLiteral: 1
    Node: NumericLiteral: 3
    Node: PlusToken
Node: EndOfFileToken

But how would one come to know this?
Exploring the ts.Node tree
The easiest thing to do is throw an error on every Node type we don't
yet know about and fill in support for each program we throw at the
interpreter.
For example:
function interpretNode(node: ts.Node) {
  switch (node.kind) {
    default:
      throw new Error('Unsupported node type: ' + ts.SyntaxKind[node.kind]);
  }
}

Now let's run our interpreter against an input file,
test.ts, that combines these two to make a
semi-interesting program:
$ cat test.ts
print(1 + 2);
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: ExpressionStatement
...

And we see an outer wrapper, an ExpressionStatement. To proceed we
look up the definition of an ExpressionStatement in TypeScript source
code,
src/compiler/types.ts
to be specific. This file will become our best friend. Hit ctrl-f and
look for "interface ExpressionStatement ". We see that it has only one
child, expression, so we call interpretNode
on this recursively:
function interpretNode(node: ts.Node) {
  switch (node.kind) {
    case ts.SyntaxKind.ExpressionStatement: {
      const es = node as ts.ExpressionStatement;
      return interpretNode(es.expression);
    }
    default:
      throw new Error('Unsupported node type: ' + ts.SyntaxKind[node.kind]);
  }
}

Thankfully TypeScript will be very quick to call us out if we
misunderstand this structure.

  It's pretty weird to me that the ts.Node tree is organized such that
  I must cast at each ts.Node but that's what they do even in the
  TypeScript source so I don't think I'm misunderstanding.
Now we recompile and run the interpreter against the program to
discover the next ts.Node type.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: CallExpression
...

Cool! Back to
src/compiler/types.ts.
Call expressions are complex enough that we'll break out handling them
into a separate function.
interpretCall and ts.CallExpression
From our reading of types.ts we need to handle the expression that
evaluates to a function, and we need to handle its parameters. We'll
just call interpretNode on each of these to get their
real value. And finally we'll call the function with the arguments.
function interpretCall(ce: ts.CallExpression) {
  const fn = interpretNode(ce.expression);
  const args = ce.arguments.map(interpretNode);
  return fn(...args);
}

function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.CallExpression: {
      const ce = node as ts.CallExpression;
      return interpretCall(ce);
    }
    ...
  }
}


  Please ignore the fact that we are not correctly setting
  this here.
Recompile and let's see what's next!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: Identifier
...

And back to types.ts.
ts.Identifier
In order to support identifiers in general we'd need to have a context
we could use to look up the value of an identifier. But we don't have
a context like this right now so we'll add builtin support for a
print function so we can get some output!
function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.Identifier: {
      const id = (node as ts.Identifier).escapedText as string;
      if (id === 'print') {
        return function (...args) { console.log(...args); };
      }

      throw new Error('Unsupported identifier: ' + id);
    }
    ...
  }
}

Recompile and let's see what's next!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: BinaryExpression
...

And we're finally into the parameters.
interpretBinaryExpression and ts.BinaryExpression
Looking into types.ts for this Node type suggests we may want to break
this out into its own function as well; there are a ton of operator
types. Within the interpretBinaryExpression helper we'll
interpret each operand and then switch on the operator type. We'll
throw an error on operators we don't know about -- all of them at
first:
function interpretBinaryExpression(be: ts.BinaryExpression) {
  const left = interpretNode(be.left);
  const right = interpretNode(be.right);
  switch (be.operatorToken.kind) {
    default:
      throw new Error('Unsupported operator: ' + ts.SyntaxKind[be.operatorToken.kind]);
  }
}

function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.BinaryExpression: {
      const be = node as ts.BinaryExpression;
      return interpretBinaryExpression(be);
    }
    ...
  }
}

We know the drill.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: FirstLiteralToken
...

At this point we're actually failing first on an unknown node type
rather than an operator. This is because we interpret the operands
(which are numeric literals) before we look up the operator. Time to
revisit types.ts!
ts.FirstLiteralToken, ts.NumericLiteral
Looking at types.ts shows us that FirstLiteralToken is a
synonym for NumericLiteral. The latter name is more
obvious, so let's add that to our supported Node list:
function interpretNode() {
  switch (node.kind) {
    ...
      case ts.SyntaxKind.NumericLiteral: {
      const nl = node as ts.NumericLiteral;
      return Number(nl.text);
    }
    ...
  }
}

And we keep going!
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported operator: PlusToken
...

And we're into unknown operator territory!
interpretBinaryExpression and ts.PlusToken
A simple extension to our existing
interpretBinaryExpression, we return the sum of the left
and right values:
function interpretBinaryExpression(be: ts.BinaryExpression) {
  const left = interpretNode(be.left);
  const right = interpretNode(be.right);
  switch (be.operatorToken.kind) {
    case ts.SyntaxKind.PlusToken:
      return left + right;
    default:
      throw new Error('Unsupported operator: ' + ts.SyntaxKind[be.operatorToken.kind]);
  }
}

And we give it another shot.
$ tsc interpreter.ts
$ node interpreter.js test.ts
...
Error: Unsupported node type: EndOfFileToken
...

ts.SyntaxKind.EndOfFileToken
Our final Node type before a working program, we simply do nothing:
function interpretNode() {
  switch (node.kind) {
    ...
    case ts.SyntaxKind.EndOfFileToken:
      break;
    ...
  }
}

One more time:
$ tsc interpreter.ts
$ node interpreter.js test.ts
3

A working program! And if we jiggle the test?
$ cat test.ts
print(1 + 5);
$ node interpreter.js test.ts
6

We're well on our way to interpreting TypeScript, and have gained some
familiarity with the TypeScript Compiler API.
All code is available on Github.
Quick intro to the TypeScript Compiler API by writing an interpreter https://t.co/QKz3XtOuP9
— Phil Eaton (@phil_eaton) April 14, 2019
 


Writing a web server from scratch: 1. HTTP and sockets
Notes on software development — Sat, 06 Apr 2019 03:00:00 GMT
Say we have some HTML:
<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

And say we'd like to be able to render this page in a web browser. If
the server is hosted locally we may want to enter
localhost:9000/hello-world.html in the address bar, hit
enter, make a request (done by the browser), receive a response (sent
by some server), and render the result (done by the browser).
Here is a minimal, often incomplete, and unsafe Node.js program (about
100 LoC) that would serve this (code available on
Github):
const fs = require('fs');
const net = require('net');

const CRLF = '\r\n';

const HELLO_WORLD = `<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>`;
const NOT_FOUND = `<html>
  <body>
    <h1>Not found</h1>
  </body>
</html>`;

class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
    this.request = {
      statusLine: null,
      headers: {},
      body: null,
    };
  }

  parse(buffer) {
    const lines = buffer.toString().split(CRLF);

    // Parse/store status line if necessary
    if (!this.request.statusLine) {
      const [method, path, protocol] = lines.shift().split(' ');
      this.request.statusLine = { method, path, protocol };
    }

    // Parse/store headers if the body hasn't begun
    if (!this.request.body) {
      for (let line = lines.shift(); lines.length; line = lines.shift()) {
        // Reached the end of headers, double CRLF
        if (line === '') {
          this.request.body = '';
          break;
        }

        const [key, value] = line.split(':');

        const safeKey = key.toLowerCase();
        if (!this.request.headers[safeKey]) {
          this.request.headers[safeKey] = [];
        }

        this.request.headers[safeKey].push(value.trimStart());
      }
    }

    this.request.body += lines.join(CRLF);
  }

  requestComplete() {
    if (!this.request.statusLine || !Object.keys(this.request.headers).length || this.request.body === null) {
      return false;
    }

    const [contentLength] = this.request.headers['content-length'] || [];
    if (this.request.statusLine.method !== 'GET' && this.request.body.length !== contentLength) {
      return false;
    }

    return true;
  }

  sendResponse() {
    const response = { status: 200, statusMessage: 'OK', body: '' };

    if (this.request.statusLine.path === '/hello-world.html') {
      response.body = HELLO_WORLD;
    } else {
      response.status = 404;
      response.statusMessage = 'NOT FOUND';
      response.body = NOT_FOUND;
    }

    const serialized = 'HTTP/1.1 ${response.status} ${response.statusMessage}' + CRLF +
                       'Content-Length: ' + response.body.length + CRLF + CRLF +
                       response.body;
    this.connection.write(serialized);
  }

  handle(buffer) {
    this.parse(buffer);

    if (!this.requestComplete()) {
      return;
    }

    this.sendResponse();

    // Other-wise the connection may attempt to be re-used, we don't support this.
    this.connection.end();
  }
}

function handleConnection(connection) {
  const handler = new HTTPRequestHandler(connection);
  connection.on('data', (buffer) => handler.handle(buffer));
}

const server = net.createServer(handleConnection);

server.listen('9000');

So what's going on?
The protocol
HTTP (version 1.1, specifically) is a convention for connecting over
TCP/IP and sending plain-text messages between two processes. HTTP
messages are broken into two categories: requests (the sender of a
request is called a "client") and responses (the sender of a response
is called a "server").
HTTP is important because it is the default protocol of web
browsers. When we type in localhost:9000/hello-world.html
and hit enter, the browser will open an TCP/IP connection to the
location localhost on the port 9000 and send
an HTTP request. If/when it receives the HTTP response from the server
it will try to render the response.
An HTTP request
A bare minimum HTTP/1.1 request (defined
here) based
on the request for localhost:9000/hello-world.html is the
following:
GET /hello-world.html HTTP/1.1\r\nHost: localhost:9000\r\n\r\n


  The spec explicitly requires the \r\n combo to
  represent a newline instead of simply \n.
If we printed out this request it would look like this:
GET /hello-world.html HTTP/1.1
Host: localhost:9000

Components of an HTTP request
An HTTP/1.1 request is made up of a few parts:

[Mandatory]: The status line (the first line) followed by a CRLF (the \r\n combo)
[Mandatory]: HTTP headers separated by a CRLF and followed by an additional CRLF
[Optional]: The request body

The status line consists of the request method (e.g. GET, POST, PUT,
etc.), the path for the request, and the protocol -- all separated by
a space.
An HTTP header is a key-value pair separated by a colon. Spaces
following the colon are ignored. The key is case insensitive. Only
the Host header appears to be mandatory. Since these
headers are sent by the client they are intended for the server's use.
The request body is text and is only relevant for requests of certain
methods (e.g. POST but not GET).
An HTTP response
A bare minimum HTTP/1.1 response (defined
here) based
on the file we wanted to send back is the following:
HTTP/1.1 200 OK\r\n\r\n<html>\n  <body>\n    <h1>Hello world!</h1>\n  </body>\n</html>

If we printed out this response it would look like this:
HTTP/1.1 200 OK

<html>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

Components of an HTTP response
An HTTP/1.1 response is made up of a few parts:

[Mandatory]: The status line (the first line) followed by a CRLF
[Optional]: HTTP headers separated by a CRLF and followed by an additional CRLF
[Optional]: The request body

The status line consists of the protocol, the status code, and the
status message -- all separated by a space.
HTTP response headers are the same as HTTP request headers although in
a response they are directives from the server to the client. There
are many standard
headers that
are used for such things as setting cache rules, setting cookies,
settings response type (e.g. HTML vs CSS vs PNG so the browser knows
how to handle the response).
The response body is similar to the HTTP request body.
Sockets
Most operating systems have a built-in means of connecting over TCP/IP
(and sending and receiving messages) called "sockets". Sockets allow
us to treat TCP/IP connections like files in memory. Most programming
languages have a built-in socket library. Node.js provides a
high-level interface for listening on a port and handling new
connections.
function handleConnection(connection) {
  connection.on('data', (buffer) => doSomething???);
}

const server = net.createServer(handleConnection);

server.listen('9000');

Once the program is listening, clients can open TCP/IP connections to
the address (localhost) and port (9000) and
our program takes over from there. Each connection is handled
separately and receives "data" events. Each data event includes new
bytes available for us to handle.
Let's encapsulate the state of each connection in HTTPRequestHandler
class. Its function will be to parse data as it becomes available and
respond to the request when the request is done.
class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
  }

  parse(buffer) {}

  requestComplete() {}

  sendResponse() {}

  handle(buffer) {
    this.parse(buffer);

    if (!this.requestComplete()) {
      return;
    }

    this.sendResponse();

    // Other-wise the connection may attempt to be re-used, we don't support this.
    this.connection.end();
  }
}

function handleConnection(connection) {
  const handler = new HTTPRequestHandler(connection);
  connection.on('data', (buffer) => handler.handle(buffer));
}

...

There are three functions we need to implement
now: parse(buffer), requestComplete(),
and sendResponse.
parse(buffer)
This function will be responsible for progressively pulling out data
from the buffer. If the status line has not been received, it will try
to grab the status line. If the body has not yet started, it will
accumulate headers. Then it will continue accumulating the body until
we close the connection (this will happen implicitly when
requestComplete() returns true).
class HTTPRequestHandler {
  constructor(connection) {
    this.connection = connection;
    this.request = {
      statusLine: null,
      headers: {},
      body: null,
    };
  }

  parse(buffer) {
    const lines = buffer.toString().split(CRLF);

    // Parse/store status line if necessary
    if (!this.request.statusLine) {
      const [method, path, protocol] = lines.shift().split(' ');
      this.request.statusLine = { method, path, protocol };
    }

    // Parse/store headers if the body hasn't begun
    if (this.request.body === null) {
      for (let line = lines.shift(); lines.length; line = lines.shift()) {
        // Reached the end of headers, double CRLF
        if (line === '') {
          this.request.body = '';
          break;
        }

        const [key, value] = line.split(':');

        const safeKey = key.toLowerCase();
        if (!this.request.headers[safeKey]) {
          this.request.headers[safeKey] = [];
        }

        this.request.headers[safeKey].push(value.trimStart());
      }
    }

    this.request.body += lines.join(CRLF);
  }

...

}

requestComplete()
This function will look at the internal request state and return false
if the status line has not been received, no headers have been
received (although this is stricter than the HTTP/1.1 standard
requires), or if the body length is not equal to the value of the
Content-Length header.
class HTTPRequestHandler {

...

  requestComplete() {
    if (!this.request.statusLine || !Object.keys(this.request.headers).length || this.request.body === null) {
      return false;
    }

    const [contentLength] = this.request.headers['content-length'] || [];
    if (this.request.statusLine.method !== 'GET' && this.request.body.length !== contentLength) {
      return false;
    }

    return true;
  }

...

}

sendResponse()
Finally we'll hard-code two responses (one for the valid request for
/hello-world.html and a catch-all 404 response for every other
request). These responses need to be serialized according the HTTP
response format described above and written to the connection.
class HTTPRequestHandler {

...

  sendResponse() {
    const response = { status: 200, statusMessage: 'OK', body: '' };

    if (this.request.statusLine.path === '/hello-world.html') {
      response.body = HELLO_WORLD;
    } else {
      response.status = 404;
      response.statusMessage = 'NOT FOUND';
      response.body = NOT_FOUND;
    }

    const serialized = 'HTTP/1.1 ${response.status} ${response.statusMessage}' + CRLF +
                       'Content-Length: ' + response.body.length + CRLF + CRLF +
                       response.body;
    this.connection.write(serialized);
  }  

...

}

Run it
Now that we've got all the pieces we can finally run the initial program:
$ node uweb.js &
$ open localhost:9000/hello-world.html

And we see the page! Try any other path and we receive a 404.
Review and next steps
We covered the basics of HTTP/1.1: a very simple, plain-text protocol
oriented around requests and responses over a TCP/IP connection. We
realize we need to know little about anything but parsing and
formatting text on top of the TCP/IP blackbox called sockets. We
created a simple application that returns different responses based on
the request. But we're a far shot from a more general library, a web
framework. Future posts will explore this transition as well as
performance and more features.
Code is available on Github.
First post in a new series on web server basics starting with HTTP and sockets (using JavaScript/Node.js). https://t.co/uBiNfOBJeZ
— Phil Eaton (@phil_eaton) April 7, 2019
 


Media Clipping using ffmpeg and Cache Eviction 2 Random for Disk Caching at Scale
Ben E. C. Boyter — Thu, 04 Apr 2019 03:00:00 GMT
A short time ago one of the more interesting blog posts (to me anyway) about cache eviction popped up on Hacker News which prompted me to post the following comment.

Love reading this. It has always been one of those interesting things I kept in the back of my mind in my day to day.


I was very excited when I actually got to implement it on a real world project.


I was writing a scale out job which used ffmpeg to make clips of video files. To speed it up I kept the downloaded files (which could be 150 GB in size) as a local cache. Quite often a clip is made of the same file. When the disk was full (there was a separate disk for download and clip output) selected two of the downloaded files randomly and deleted the older one. Loop till there was enough disk space, or no files.


Bash : so long, and thanks for all the fish !
Linux Engineer's random thoughts — Sun, 31 Mar 2019 01:00:00 GMT
I recently took the plunge and decided to move to fish, a friendly modern shell.
Background
I'm heavily invested in bash. I wrote countless scripts, for my personal use, and for various jobs or projects, so I didn't think it would be an easy thing to move away from it …


OKRs aren't going to fix your communication issues
CRAIG KERSTIENS — Sat, 30 Mar 2019 22:55:56 GMT
Talking with a startup a few days ago they asked for my opinions on OKRs. I have slightly mixed opinions on them overall and started to disclose some of those. Though in sharing some of this I had a few immediate realizations that might be broadly applicable. The crux of his question was, at what stage should we put them in place. I’ve seen a few companies try to put in some form of OKR, and most were met with pretty mixed results. The reason is that OKRs need to change something about your behavior otherwise why put them in place… either change something about the goals you would otherwise have or the methods at which you went about achieving them.
Stepping back a bit, my first question and a very focusing question on almost any situation to ask is “What problem are we trying to solve?” In our conversation he actually paused a bit. As he paused a bit longer it was clear that question had not been fully asked or answered.
The first and most common case I see with startups trying to put in place OKRs, v2moms, management by objectives is that the team is not aligned and focused on the same goals. But my follow-on question is consistently, have you communicated what you decided you goals were.
Startups tend to go through some distinct growing phases. The early stages all the founders are in a room together building out the product. When you get the first few engineers you expand out a little, but still in a single co-working conference room easily. Eventually you need a real office. At the real office stage you start to have an all hands where, this is probably gathered around a large lunch table at first. At all hands no one takes meeting minutes and sends out a recap, instead people take some notes and you assume everyone was present.
But, at about 20 people you have at least one person that misses the weekly team meeting and misses something key. In a 1:1 you catch it that it was talked about as a priority… but they weren’t there. This very subtle change I’ve seen linger all the way up to a 70/80 person org. I’ve observed management meetings where someone missed and a key member was entirely mis-aligned on what the goals were for months following.
That was a long detour, but the point is that explicit formal communication is a big change for early stage companies. Distributed teams tend to do this better than in person teams, but it is also not guaranteed.
OKRs present a heavy-weight answer to the problem. OKRs tend to require hours and maybe even days to determine what are the right goals and metrics. Even if they are quick, does the process of OKRs change how you structure your team and work significantly for the next few weeks/months. If not, could you much more easily get away with… wait for it… emailing out what the company says priorities are. Email out the meeting notes from your all hands meeting. Email out (gasp) a recap of what you discussed and are thinking about as a management team. Sure every manager could go and have a 1:1 and recap the points for 30 minutes with each of their employees. Or you could use this thing that we’ve had for a little while… email.


Writing a simple JSON path parser
Notes on software development — Wed, 27 Mar 2019 02:00:00 GMT
Let's say we want to implement a simple list filtering language so
we can enter a.b = 12 and return only results in a
list where the a column is an object that contains a
field b that is set to the value 12. What would a
filter(jsonPath, equals, listOfObjects) function look
like?
If we only needed to support object lookup, we might
implement filter by splitting the jsonPath
on periods and look at each object in the listOfObjects
for matching values. It might look something like this:
function filter(jsonPath, equals, listOfObjects) {
  const parts = jsonPath.split('.');

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

require('assert').deepEqual(
  filter('foo.bar', 12, [{ foo: { bar: 12 } }, { foo: null }]),
  [{ foo: { bar: 12 } }],
);

That doesn't work too badly. We haven't handled edge cases like a
jsonPath of foo..bar or
bar.. But those would not be difficult to handle:
function filter(jsonPath, equals, listOfObjects) {
  if (jsonPath.charAt(0) === '.') {
    throw new Error('JSON path cannot begin with a dot, in: ' + jsonPath);
  } else if (jsonPath.charAt(jsonPath.length - 1) === '.') {
    throw new Error('JSON path cannot end with a dot, in: ' + jsonPath);
  }

  const parts = jsonPath.split('.');
  if (parts.reduce((hasEmptyPart, part) => hasEmptyPart || part.length === 0, false)) {
    throw new Error('JSON path cannot contain an empty section, in: ' + jsonPath);
  }

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

And we now handle the most obvious invalid path cases.
Arrays?
If we want to support array path syntax, things get harder. For
example:
require('assert').deepEqual(
  filter('foo.bar[0].biz', 14, [{ foo: { bar: [ { biz: 14 }, { biz: 19 } ] } }, { foo: { bar: null } }]),
  [{ foo: { bar: [ { biz: 14 }, { biz: 19 } ] } }],
);

We could try to stick with the hammer that is
String.prototype.split and write some really messy
code. :) Or we could switch to an approach that gives us more
control. Let's do that.
We'll build a very simple lexer that will iterate over each character
accumulating characters into individual tokens that represent the
pieces of the path. Let's start by supporting the original
jsonPath syntax and error-handling.
function getJsonPathParts(path) {
  const parts = [];
  let currentToken = '';

  for (let i = 0; i < path.length; i++) {
    const c = path[i];
    switch (c) {
      case '.': {
        if (!currentToken) {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }
        parts.push(currentToken);
        currentToken = '';
        break;
      }
      default: {
        currentToken += c;
        break;
      }
    }
  }

  if (!currentToken) {
    throw new Error('JSON path cannot end with dot, in: ' + path);
  }

  parts.push(currentToken);
  return parts;
}

function filter(jsonPath, equals, listOfObjects) {
  const parts = getJsonPathParts(jsonPath);

  function filterSingle(object) {
    let objectAtPath = object;
    let i = 0;
    for (let part = parts[i]; part && objectAtPath; part = parts[++i]) {
      objectAtPath = objectAtPath[part];
    }

    return i === parts.length && objectAtPath === equals;
  }

  return listOfObjects.filter(filterSingle);
}

Not too bad!
Arrays?
Right. Let's build on getJsonPathParts to support array
syntax. Along with that we're going to impose some restrictions. The
object path parts must be only alphanumeric characters plus dashes and
underscores. The array index must only be numeric characters. Anything
else should throw an error.
function getJsonPathParts(path) {
  const parts = [];
  let currentToken = '';
  let inArray = false;

  for (let i = 0; i < path.length; i++) {
    const c = path[i];
    switch (c) {
      case '.': {
        if (currentToken === '') {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }

        parts.push(currentToken);
        currentToken = '';
        break;
      }
      case '[': {
        if (inArray) {
          throw new Error('JSON path contains unexpected left bracket, in: ' + path);
        }

        if (currentToken === '') {
          throw new Error('JSON path cannot contain empty section, in: ' + path);
        }

        parts.push(currentToken);
        currentToken = '';
        inArray = true;
        break;
      }
      case ']': {
        if (!inArray) {
          throw new Error('JSON path contains unexpected right bracket, in: ' + path);
        }

        if (currentToken === '') {
          throw new Error('JSON path array index must not be empty, in: ' + path);
        }

        // Array indices are recorded as numbers, not strings.
        currentToken = parseInt(currentToken, 10);
        inArray = false;
        break;
      }
      default: {
        const code = c.charCodeAt(0);

        if (inArray) {
          if (code >= '0'.charCodeAt(0) && code <= '9'.charCodeAt(0)) {
            currentToken += c;
            continue;
          }

          throw new Error('JSON path array index must be numeric, in: ' + path);
        }

        if ((code >= 'A'.charCodeAt(0) && code <= 'z'.charCodeAt(0)) ||
            (code >= '0'.charCodeAt(0) && code <= '9'.charCodeAt(0)) ||
            ['-', '_'].includes(c)) {
          currentToken += c;
          continue;
        }

        throw new Error('JSON path part must contain only alphanumeric characters, in: ' + path);
      }
    }
  }

  if (currentToken === '') {
    throw new Error('JSON path cannot end with dot, in: ' + path);
  }

  parts.push(currentToken);
  return parts;
}

require('assert').deepEqual(getJsonPathParts('foo.bar[0].biz'), ['foo', 'bar', 0, 'biz']);

Now we've got a simple JSON path parser with decent error handling! Of
course we wouldn't want to use this little library in production until
we had some serious test coverage. But writing tests and calling out
my mistakes will be left here as an exercise for the reader. :)
New (short) post on parsing JSON paths in JavaScript https://t.co/mIjOMugA7C
— Phil Eaton (@phil_eaton) March 28, 2019
 


Kindle Hacks, A Self-guide
Nemo's Home — Tue, 26 Mar 2019 02:00:00 GMT
I run a non-standard Kindle configuration:


  Jailbroken (because I want to own the device, not rent it)
  Runs KOReader (because I want to read EPUBs and PDFs with reflow.)
  DRM Stripping (because I want to own the book, not rent it)


Since I don’t do any of these often enough to automate it, this is a self guide to help me follow these steps the next time I have to do any of this. No guarantees of this being helpful to anyone else but me.

Jailbreak

The lifehacker guide on how to jailbreak your kindle is a good starting point [archived]. The mobileread forums have the definitive guides. Also see this FAQ on the mobileread wiki.

(Most of these only cover modern paperwhite kindles)

Maintaining the Jailbreak

Sometimes, Kindle firmware updates will stop the Jailbreak. Search for your firmware on mobileread forums. See this link for the 5.8 series.

Copy the .bin file to your kindle root directory and trigger a manual firmware update. That should reboot and re-affirm the jailbreak. To trigger a manual firmware update, go to the Kindle Menu and click “Update”. If it is greyed out, check if the file was copied correctly, and try rebooting.

Applications

Once you have a jailbreak, the rest is mostly installing packages via MRPI. I keep a ready directory of packages I can copy as-is to my Kindle. The current listing is at https://paste.ubuntu.com/p/CXS5hYZdqc/ with most of it just being koreader.

koreader is a FOSS document viewer for E Ink devices that supports Kindle, Kobo, PocketBook, Ubuntu Touch and Android devices.

The primary 2 packages are:


  Update_KUALBooklet_v2.7_install.bin
  update_kpvbooklet_0.6.6_install.bin


Run ;log mrpi via search after copying them to re-install them if needed.

koreader

Download the latest release from GitHub.

You should download the kindle5-linux-gnueabi package for modern Paperwhites. Unzip it to the copy directory mentioned above.

Aside: koreader has a linux appimage version for desktops, which I package for AUR.

DRM Related Stuff

DRM is inherently bad for users. If I switch my Ebook reader from Kindle (which are great as of today) to
a Kobo tomorrow, I want my content to stay with me.

There are much better websites that explain the issues with DRM, so go visit: fckdrm.com, DefectiveByDesign.org, or EFF/drm.

The primary tool for stripping DRM from Kindle books is apprenticeharper’s DeDRM Repo which works as a Calibre Plugin. If you are running calibre with Python 3 (such as via the calibre-python3 package on Arch Linux) - you should install the DeDRM plugin from the python3 fork. Compress the DeDRM_plugin directory into a flat-zip file and use that in Calibre.

Getting the Key

My current key is saved in pass:

pass show Keys/Kindle.k4i |jq

Save it in a file, which you can import to Calibre.

If you don’t have the key or if the above isn’t valid, see this comment on r/ebooks [archived].

Importing the Key


  At the bottom-left of the plugin’s customization dialog, you will see a button labeled “Import Existing Keyfiles”. Use this button to import existing ‘.k4i’ key files. Key files might come from being exported from this plugin, or may have been generated using the kindlekey.pyw script running under Wine on Linux systems.


I once did some trickery on the kindlekey.pyw application to get it working on my system, but I didn’t take notes. If I ever do this again - AUTOMATE THIS.

Getting a copy of the encrypted book

There are multiple sources for you to try.


  Amazon website’s My Content page is the easiest. It doesn’t work for books with special typesetting - quite rare. Prefer this over everything else.
  Download via the Kindle for PC application (See next section).
  Get the KFX file from your Kindle device.
  Copy the KFX/AZW file from the Android/iOS application.


Kindle for PC

Stripping DRM for any medium is always a cat-and-mouse game. Amazon keeps changing the DRM format in every Kindle firmware update, which is why the recommended method is to use a known/older version of the Kindle for Mac/PC Application as your source.

Note: The 1.24.3 release does not work on Linux. If you’re on Linux, you must instead download the 1.17.0 release instead (sha256=14e0f0053f1276c0c7c446892dc170344f707fbfe99b6951762c120144163200).


  Install Kindle for PC. It does work on Wine. Make sure you download 1.24.3 (51068). I trust filehippo for this. The sha256sum for the installer is c7a1a93763d102bca0fed9c16799789ae18c3322b1b3bdfbe8c00422c32f83d7.
  Install then launch it, and download the book.
  Go to ~/Documents/My Kindle Content
  Find book by Last Modified Date.
  Run calibredb add book.azw. If all goes well, the book should show up in your library, and you should be able to convert it.




Reference Files

I have a backup of my current Kindle files at http://ge.tt/75zk4Dv2 in case you need any of the files mentioned above. Checksums for the files are below, since ge.tt doesn’t believe in HTTPS:

e3b05193ed9d0b482f01dfb550eba67f3b113b5165aae5632379cf35fec2f59d  copy.tar.gz
14e0f0053f1276c0c7c446892dc170344f707fbfe99b6951762c120144163200  KindleForPC-installer-1.17.44170.exe
c7a1a93763d102bca0fed9c16799789ae18c3322b1b3bdfbe8c00422c32f83d7  KindleForPC-installer-1.24.51068.exe
50bb0e5d9c03bcb79b17c1b7063cefd2c947a9d1c4392814e6ec05225296472a  kual-helper-0.5.N.zip
39352b4b68993680f06d5ecc57ce7ec4c271b6b5f2386ea998027420c45f2acd  KUAL-KDK-1.0.azw2
ceb207ee4c8d3674f308ff91432aeabf213b203571e270f70b8ae218df6ded7d  KUAL-KDK-2.0.azw2
fce02f0e104e846f1e4cc0e029500c5a722614d63a47035d78ea4cf59f67a448  kual-mrinstaller-1.6.N.zip
4a6de1fafe47ec0e3bfb529edead401c92e66b00697d507abe945679b3b7bc65  KUAL-v2.7.zip
253d0b00b31d62ef9dadb7ca88b98e2718cb35246816b3c50dd63c0a7ef28a52  Update_jailbreak_hotfix_1.14_5.8.10_install.bin
cc63ba1b454d1f32492c835f108ee04aaa80e6e7a95f12b7216c2c015daa2fbc  Update_jailbreak_hotfix_1.14_nomax_install.bin



Unsupervised Learning: Feature Transformation
swyx's site RSS Feed — Thu, 21 Mar 2019 09:21:12 GMT
Presenting the same information a different way... helps! Plus, one algorithm that does better than Principal Components Analysis!


Unsupervised Learning: Feature Selection
swyx's site RSS Feed — Thu, 21 Mar 2019 09:15:55 GMT
Breaking the Curse of Dimensionality!!


Unsupervised Learning: Clustering
swyx's site RSS Feed — Thu, 21 Mar 2019 09:12:22 GMT
Single Linkage, K-Means, Soft Clustering, and Kleinberg Impossibility


Write My Code For Me
swyx's site RSS Feed — Thu, 21 Mar 2019 02:00:00 GMT
My thoughts to self while weighing adding netlify functions:create to Netlify Dev. Originally a gist


Placeholder Horse Races
infrequent oscillations — Wed, 20 Mar 2019 08:15:09 GMT
More fake horse races as filler content.


Tips for your first tech conference
CRAIG KERSTIENS — Mon, 18 Mar 2019 22:55:56 GMT
I make it to a lot of conferences these days. I often see colleagues, former colleagues, and friends at these conferences. Sometimes it is friends I haven’t seen in a few years, sometimes I just saw the same person in a different country the week before. Conferences now are much easier for me, in fact it is a bit hard to recall what the experience was like when I first started attending, but I’m at least going to try to give some input so others can have a smoother first experience.


Most people know no one


There is a very strong chance you’re attending your first conference by yourself or just with a single colleague you know. Being suddenly surrounded by thousands of people you don’t know can be intimidating. The reality though is most folks there are just like you and know very few other people. Most people don’t go to multiple conferences in a year, they go to 1. There are often a lot of opportunities to get to know folks. Some of the best times are during breaks between the talk track, and whenever there is food/beverages. If there is breakfast served at the conference as much as you’d like to sleep in wake up and go have breakfast and chat with someone. At lunch if there is a half full table, go ask if a seat is taken and join the table vs. sitting at a completely open one. During the breaks don’t rush to get to the talk 10 minutes before it begins, stay and mingle with folks.
If you’re really feeling shy wonder the booth floor. Those folks are paid to be there to talk to you. Let them tell you about their product or service, it’ll at least be a start to some flowing conversation.


Speakers can be your friends too


If you’re new often the speakers can seem extra intimidating to get to know. Don’t let that be the case. If one is talking about a topic you’re excited about hang around after the talk and discuss it with them. Speakers aren’t necessarily any more technically advanced than you are they just happened to get a talk accepted.


Study the agenda


The default is to look at the list of talks being presented and pick out which ones you want to attend. But pay closer attention at so many conferences there are a bunch of other great activities to participate in. PyCon is a model example of this. In addition to all the talks there is:

PyLadies Auction - Which is a great night of dinner, drinks, and an auction of fun Python focused or other items which benefits PyLadies.
[5k run] - You wake up early in the morning and run (it should be noted that 5:30 am does not sound fun to me, but to each his own)
Evening dinners - If you’re worried about having dinner plans then this is a great option at a fun venue already coordinated.

If you’re not at something as structured as PyCon then there are a few places you can find what is happening. These days conferences often have a slack channel. Join it, find out when people get into town. Find out if people like to hike or visit breweries, join in with others. There is also twitter. Twitter things don’t get planned far in advance, but often you can find last minute folks that are coordinating things.


Stay at the conference hotel


If there is an official conference hotel or hotels, stay there. This opinion may be a little more controversial, but stay with me. First you’ll find folks hanging around the hotel bar after or just sitting around mingling. It’s a just one more way to not feel isolated. Second, it often helps the conference. Conferences can take on big risks with minimum rooms filled at a hotel. They do this to get good rates, but it also presents risk. Staying at the hotel both keeps you around others that are there for the conference, but also helps the conference itself.


You don’t have to attend all the talks


Talks are super helpful. You can learn a lot in a compressed period of time and then often get where to go next to learn further. Great ones teach you something but are also entertaining presentations as well. But, you don’t have to attend all the talks. Many conferences record the talks and make them available later. Personally I make a note of ones I wanted to see, as well as ones that others said were especially great. Then I go to youtube and watch them at 1.5 or 2x speed to get through more of them in less time.


Conclusion


Going to a conference may absolutely feel intimidating if it is your first. There will be a little of that first day of school jitters. What do you wear, who will you talk to, what if no one sits with you at lunch. Just know that for most people there it really is like the first day of school for them too. Even though it may be intimidating hopefully the above helps you jump in with a little less apprehension.


Regarding My Blogs
Ben Overmyer's Site — Mon, 18 Mar 2019 02:00:00 GMT
I've been frustrated lately with the writing side of Hugo. It's a great tool for publishing and content transformation. It plays well with CI/CD tools. My static site host, Netlify, makes hosting sites built with Hugo a breeze. However, when it comes to being able to write whenever the mood strikes, Hugo is horrible. Yes, technically, I can just use GitHub's web IDE to write a new post. I don't have to use the git+vim workflow that I often do when writing Markdown. But it feels very… clunky. Inserting images is a slow and manual process. Layout requires messing with the site theme and its CSS, making it unwieldy for individual posts. Hence, all my posts on this blog look the same.
I decided to return to the GUI blog engine world. With that decision behind me, I started looking at the options. Having been out of that particular game for awhile, I hoped that some new tool had come along that I could check out. In fact, there was - a blog engine called Write Freely. So it entered the running alongside the usual suspects (Blogger, Ghost, and WordPress). Blogger hasn't changed. It has some new themes since I last looked at it seriously ten years ago, but the interface has not aged well. Ghost charges a ridiculous amount of money for a hosted blog, and the self-hosted version still hasn't acquired some of the quality-of-life features that WordPress has.
Write Freely is interesting, especially since it's in the federation space, but it's not yet mature enough for my tastes. This, despite the fact that I contributed code to it. So that leaves WordPress. And rather than deal with hosting it myself, I decided to pay for hosting on wordpress.com. It's not ideal (the Personal version doesn't allow installing plugins), but it works well enough and gives me the GUI features I missed. At time of this writing, I'm still trying to figure out how to migrate all of my Hugo posts to WordPress. No tool exists for that purpose. I'll probably have to write something myself. Update = I did end up writing a tool to handle this. I'll write a blog post about it later.


How to remove all bars in fullscreen mode in Firefox on MacOS/OSX  ("kiosk mode")
Subclassed — Thu, 14 Mar 2019 14:32:28 GMT
Procedure
If you want to remove all toolbars from Firefox in fullscreen mode on Mac, this is the procedure:

Type about:config: in the location bar to open advanced settings. (If it is the first time you do it, you might have to click "confirm" or something like that in the scary screen that comes up first.)
search for full-screen-api.allow-trusted-requests-only and double-click the line that is found, so the value changes to false
Bring up the Bookmark Toolbar if you don't have it (menu View -> Toolbars -> Bookmark Toolbar)
Right-click on empty space in the Bookmark Toolbar, and select New Bookmark...
In the Name field, type something like "Enter Fullscreen" (it doesn't really matter, just don't leave it blank)
In the Location field, enter the following line:
javascript:document.getElementsByTagName('html')[0].mozRequestFullScreen();void(0)

Click Add to save the bookmark.


Now, when you click on that bookmark, Firefox will go "real fullscreen", hiding all toolbars. To get them back, you will have to press ESC.

Now, there is a bit of a snag. That preference in about:config: is set to true by default for a reason: it is a bit unsafe to allow any website to use fullscreen-enabling javascript. For this reason I would suggest to only do this for limited periods, then switch it back once you're done.

Long-winded explanation

FF has a preference, browser.fullscreen.autohide, that in theory should govern the behaviour of toolbars when going fullscreen. It is set to true by default, meaning toolbars should disappear. However, for some reason this property has (almost) always failed to work on MacOS/OSX builds of Firefox.
There used to be a few extensions that corrected that behaviour, but they've all gone away when Firefox switched to WebExtension APIs, which basically sandboxed extensions more rigidly and blocked them from changing the browser interface as deeply as they could before. This was the price for the dramatic performance improvement seen in Firefox 57+.
This workaround hence relies on the new Fullscreen WebAPI, which is supposed to be a new, standard way to enable fullscreen via Javascript on any browser (well, any browser that implements it - just a few at the moment). However, because of security concerns, this API is locked down by default; so, in order to use it, we first have to relax checks as described above. Once that is done, we can launch a bit of JS from a bookmark that triggers "real" fullscreen.
Credit to this AskDifferent post where I first found the trick.


My Personal Complaints about Programming in Go
Ben E. C. Boyter — Thu, 14 Mar 2019 02:00:00 GMT
Go as a language is fairly decent. However because questions about why I have issues with tends to come up often enough on the company slack programming channel (see what I did there?) I figured I would write them down and put it here so I can point people at a link when they ask what my complaints are.

For the record I have been using Go heavily for the last year or so, writing command line applications, scc, lc and API’s. These include large scale API’s for clients to a syntax highlighter that will be used in https://searchcode.com/ sometime soon.


Give me back my monolith
CRAIG KERSTIENS — Wed, 13 Mar 2019 22:55:56 GMT
It feels like we’re starting to pass the peak of the hype cycle of microservices. It’s no longer multiple times a week we now see a blog post of “How I migrated my monolith to 150 services”. Now I often hear a bit more of the counter: “I don’t hate my monolith, I just care that things stay performant”. We’ve actually seen some migrations from micro-services back to a monolith. When you go from one large application to multiple smaller services there are a number of new things you have to tackle, here is a rundown of all the things that were simple that you now get to re-visit:


Setup went from intro chem to quantum mechanics


Setting up a basic database and my application with a background process was a pretty defined process. I’d have the readme on Github, and often in an hour or maybe a few I’d be up and running when I started on a new project. Onboarding a new engineering, at least for an initial environment would be done in the first day. As we ventured into micro-services onboarding time skyrocketed. Yes, we have docker and orchestration such as K8s these days to help, but the time from start to up and running a K8s cluster just to onboard a new engineer is orders of magnitude larger than we saw a few years ago. For many junior engineers this is a burden that really is unnecessary complexity.


So long for understanding our systems


Lets stay on the junior engineer perspective for just a moment. Back when we had monolithic apps if you had an error you had a clear stacktrace to see where it originated from and could jump right in and debug. Now we have a service that talks to another service, that queues something on a message bus, that another service processes, and then we have an error. We have to piece together all of these pieces to eventually learn that service a was on version 11 and service q was expecting vesion 12 already. This in contrast to my standard consolidated log, and lets not forget my interactive terminal/debugger for when I wanted to go step by step through the process. Debugging and understanding is now inherintly more complicated.


If we can’t debug them, maybe we can test them


Continuous integration and continuous development is now starting to become common place. Most new apps I see now days automatically build and run their tests with a new PR and require tests to pass and review before check-in. These are great processes to have in place and have been a big shift for a lot of companies. But now to really test my service I have to bring up a complete working version of my application. Remember back to onboarding that new engineer with their 150 service K8s cluster? Well now we get to teach our CI system how to bring up all those systems to actually test that things are working. That is probably a bit too much effort so we’re just going to test each piece in isolation, I’m sure our specs were good enough that APIs are clean and service failure is isolated and won’t impact others.


All the trade-offs are for a good reason. Right?


There are a lot of reasons to migrate to micro-services. I’ve heard cases for more agility, for scaling your teams, for performance, to give you a more resilient service. The reality we’ve invested decades into development practices and tooling around monoliths that are still maturing. In my day to day I work with a lot of folks from all different stacks. Usually we’re talking about scaling because they’re running into limits of a single node Postgres database. Most of our conversation focuses on scaling the database.
But in all the conversations I’m fascinated to learn about their architecture. Where are they in their journey to micro-services. It has been an interesting trend to see more and more reactions “We’re happy with our monolithic app.” The road to micro-services may work fine for lots, and the benefits may outweigh the bumpy road to get there, but personally give me my monolithic app and a beach somewhere and I’ll be happy.


Why I love building developer products
CRAIG KERSTIENS — Tue, 12 Mar 2019 22:55:56 GMT
For much of my career I’ve been focused on building out developer or data focused products with the customer in some form or fashion being a developer on the other end. I fully realize now that I’m destined to spend the rest of my career in that space, either that or trying my hand at wine making. There are a few things that I personally find rewarding about the space that I’ve shared with a number of people individually lately and thought I would share more broadly.
First, software really is eating the world. Often developers and entrepeurs ask about what they could build that would be a good business. The reality is about anything that has not been modernized to as a service and improved with software could be. We have far less developers in the world than we need to execute on all the ways we could improve products and life. To me what is interesting is the last part of that last sentence. It is not that the market for developers is huge, which I do believe it is. It more that when we automate with systems we can get amazing economies of scale. I know folks that reminisce and talk about how we’re more stressed being always connected and such and that in the old days people got out and worked the fields and enjoyed the sun. They also absolutely physically exhausted their bodies in the process. The ability to make life better at scale is an interesting one and often done through systems we develop.
The second reason is the challenge and the reward. Developers are notoriously tough critics. If it feels/smells like marketing then they have an allergic reaction, and that is probably because much of it is done poorly. If they experience really positive/great marketing they latch on more than the average person. Most developers are by nature a bit skeptical… for some reason this resonates with me. But, I’ve found they are the largest/biggest supporters once they’re excited about something. Personally I’d rather folks more critical and then have a few diehard fans than a whole bunch of people I can try to some site with ads and convert them, but then never actually connect with my product/brand.
The third and final reason is, I can understand it. Having worked at small startups, side projects, and at large enterprise I can understand the challenges as each different level. Building a product that I can directly relate to and connect with is some how easier to stay motivated. I’m sure instagram has a lot of interesting technical challenges, or that scaling an ad network is extremely challenging. But building a product that I could see myself directly benefiting from makes everyone’s input on the team valuable and key. This changes the entire dynamic from product team to engineering to marketing.
Now days I dont get much chance to ship code to production, usually that is what I threaten when things are running behind, but at heart I’m still very much an engineer. Building products that make developers lives better and more productive may not change the world, but hopefully it better enables other developers to.


Hacker News Embeddings with PyTorch
Bytepawn - Marton Trencseni — Tue, 12 Mar 2019 01:00:00 GMT
A PyTorch model is trained on public Hacker News data, embedding posts and comments into a high-dimensional vector space, using the mean squared error (MSE) of dot products as the loss function. The resulting model is reasonably good at finding similar posts and recommending posts for users.




Getting Closure on React Hooks
swyx's site RSS Feed — Mon, 11 Mar 2019 02:00:00 GMT
Learn React Hooks by building a clone of React - in 30 minutes!


The Statistical Impossibility of Dropped Points in the Mind of a Football Manager
infrequent oscillations — Sun, 10 Mar 2019 17:52:32 GMT
Who avoided dropping *any* points after 30 games?


Writing a lisp compiler from scratch in JavaScript: 3. LLVM
Notes on software development — Sun, 10 Mar 2019 02:00:00 GMT

  Previously in compiler basics:
  <! forgive me, for I have sinned >
  

  1. lisp to assembly
  

  2. user-defined functions and variables
  

  

  Next in compiler basics:
  

  4. LLVM conditionals and compiling fibonacci
  

  5. LLVM system calls
  

  6. an x86 upgrade
In this post we'll extend the
compiler to emit LLVM
IR as an option instead of x86
assembly.
All source code is available on Github.
LLVM IR is a portable, human-readable, typed, assembly-like syntax
that LLVM can apply optimizations
on before generating assembly for the target architecture. Many
language implementors choose to compile to LLVM IR specifically to
avoid needing to implement sophisticated optimizations.
But the biggest reason I'm adding an LLVM backend is so that I can
punt on implementing register
allocation. This
is the technique that allows you to generically use as many registers
as possible before storing local variables on the stack. While
register allocation algorithms are not that difficult, I got
bored/lazy trying to implement this for ulisp. And LLVM IR provides
"infinite" locals that get mapped as needed to registers and the stack
-- implementing register allocation.
LLVM IR basics
In LLVM IR, all local variables must be prefixed
with %. All global variables (including function names)
must be prefixed with @. LLVM IR must be in
single-static
assignment
(SSA) form, which means that no variable is assigned
twice. Additionally, literals cannot be assigned to variables
directly. So we'll work around that by adding 0 to the
literal. Furthermore, we'll take advantage of
the add, sub, and mul
operations built into LLVM IR.
; x = 4
%x = add i32 4, 0

The type that the operation is operating on must be specified after
the operation name. In this case we are specifying
that add is operating on and returning 32-bit integers.
While this might seem very inefficient, we'll see in the end that
LLVM easily optimizes this away.
Function definition
Functions are defined at the top-level and are much simpler than x86
assembly since the details of calling conventions are handled by LLVM.
; (def plus (a b) (+ a b))
define i32 @plus (i32 a, i32 b) {
  %res = add i32 a, b
  ret i32 %res
}

In ulisp, all functions will return a result (and the only supported
type for now are 32-bit integers). So we annotate the definition with
this return type (i32 in define
i32). Finally, we return inside the function with
the ret instruction that must also specify the type
(again i32).
Generating LLVM IR
We are going to generate LLVM IR as text. But any large project will
benefit from generating LLVM IR via
API.
Supporting multiple backends
The goal is to be able to switch at compile-time between generating
x86 assembly or generating LLVM IR. So we'll need to reorganize ulisp
a little bit.
We'll edit src/ulisp.js to accept a second argument to
specify the backend (and from now on we'll default to LLVM).
const cp = require('child_process');
const fs = require('fs');

const { parse } = require('./parser');
const backends = require('./backend');

function main(args) {
  const input = fs.readFileSync(args[2]).toString();

  let backend;
  switch (args[3]) {
    case 'llvm':
    case undefined:
      backend = backends.llvm;
      break;
    case 'x86':
      backend = backends.x86;
      break;
    default:
      console.log('Unsupported backend ' + args[3]);
  }

  const [ast] = parse(input);
  const program = backend.compile(ast);

  try {
    fs.mkdirSync('build');
  } catch (e) {}
  backend.build('build', program);
}

main(process.argv);

The LLVM backend
We'll add src/backend/llvm.js and expose
compile and build functions.
compile(ast)
This will work the same as it did for the x86 backend, creating a new
Compiler helper object, creating a scope manager (which
we'll get into in more detail shortly), and generating code from the
AST wrapped in a begin.
module.exports.compile = function(ast) {
  const c = new Compiler();
  const scope = new Scope();
  c.compileBegin(ast, scope.symbol(), scope);
  return c.getOutput();
};

build(buildDir, output)
The job of build will be to clean up the build directory,
write any output as needed to the directory, and compile the written
output. Since we're dealing with LLVM IR, we first call
llc on the IR file to
get an assembly file. Then we can call gcc on the
assembly to get a binary output.
const cp = require('child_process');
const fs = require('fs');

...

module.exports.build = function(buildDir, program) {
  const prog = 'prog';
  fs.writeFileSync(buildDir + `/${prog}.ll`, program);
  cp.execSync(`llc -o ${buildDir}/${prog}.s ${buildDir}/${prog}.ll`);
  cp.execSync(`gcc -o ${buildDir}/${prog} ${buildDir}/${prog}.s`);
};

Taking advantage of locals
Before we get too far into the specifics of LLVM IR code generation,
let's build out the infrastructure to take advantage of "infinite"
locals. In particular, we want a local-manager (Scope)
with four functions:

register(local: name): for tracking user variables and mapping to safe names
symbol(): for tracking internal temporary variables
get(local: name): for returning the safe name of a user variable
copy(): for duplicating the local-tracker when we enter a new scope

It is important to track and map user variables into safe names so we
don't accidentally conflict between variable names used by the user
and names used by the compiler itself.
register(local)
When we register, we'll want to replace any unsafe characters that
Lisp allows but LLVM likely won't. For now, we'll just replace any
dashes in the name (since dashes are fine in variables in Lisp) with
underscores. Then we'll add a number to the end of the local name
until we have a safe name that doesn't exist already. Finally we
return that safe name after storing the mapping.
class Scope {
  constructor() {
    this.locals = {};
  }

  register(local) {
    let copy = local.replace('-', '_');
    let n = 1;
    while (this.locals[copy]) {
      copy = local + n++;
    }

    this.locals[local] = copy;
    return copy;
  }
}

symbol()
This is a simple function that will return one new unused safe name
that we can store things in.
class Scope {
  ...

  symbol() {
    const nth = Object.keys(this.locals).length + 1;
    return this.register('sym' + nth);
  }

  ...
}

We start off by making up a name based on the prefix sym
and a suffix of the current key length and pass that into the
register function to make sure we get a safe name.
get(local)
This function is a very simple lookup to return the safe name for a
user variable. It is up to the caller of this function to handle if
the user variable does not exist in scope (and perhaps throw a
compiler error back to the programmer).
class Scope {
  ...

  get(local) {
    return this.locals[local];
  }

  ...
}

copy()
Finally, we want to expose a copy function so we can duplicate the
local storage before entering a new scope. (A variable inside a
function should not exist in scope outside the function.)
class Scope {
  ...

  copy() {
    const c = new Scope();
    c.locals = { ...this.locals };
    return c;
  }

  ...
}

Back to codegen!
As mentioned in module.exports.compile, we're going to
use a Compiler that exposes a number of compiler helpers:

emit(depth, code): an internal helper for outputting indented lines of code
compileBegin(ast, destination, scope): compiles a begin block
compileExpression(ast, destination, scope): compiles variable references, literals, and passes on function calls
compileCall(functionName, ast, destination, scope): compiles a function call
compileDefine([functionName, parameters, ...body], destination, scope): compiles a function definition
compileOp(op): helper function for generating code for primitive operations like add
getOutput(): returns the code generated by the compiler

emit(depth, code)
Like we had in the x86 backend, this will indent the code two spaces
depth times and write it to the buffer we track generated
code.
class Compiler {
  constructor() {
    this.outBuffer = [];
  }

  emit(depth, code) {
    const indent = new Array(depth + 1).join('  ');
    this.outBuffer.push(indent + code);
  }
}

compileBegin(ast, destination, scope)
Our first compiler function actually does no code generation
itself. We'll call compileExpression on each item within
the begin block. And we'll pass the destination to the
last expression in the list so that the value of a begin block is set
to the value of its last expression. All other expressions will
receive a temporary variable to store results.
class Compiler {
  ...

  compileBegin(body, destination, scope) {
    body.forEach((expression, i) =>
      this.compileExpression(
        expression,
        i === body.length - 1 ? destination : scope.symbol(),
        scope,
      ),
    );
  }

  ...
}

Example:
(begin 1 2) ; returns 2

compileExpression(ast, destination, scope)
This is the most generic compile function. If the ast is a list
(representing a function call), it will pass compilation off to
compileCall. Otherwise the only non-function call parts
of the language are variable references and numeric literals.
class Compiler {
  ...

  compileExpression(exp, destination, scope) {
    // Is a nested function call, compile it
    if (Array.isArray(exp)) {
      this.compileCall(exp[0], exp.slice(1), destination, scope);
      return;
    }

    // If numeric literal, store to destination register by adding 0.
    if (Number.isInteger(exp)) {
      this.emit(1, `%${destination} = add i32 ${exp}, 0`);
      return;
    }

    // If is local, store to destination register similarly.
    const res = scope.get(exp);
    if (res) {
      this.emit(1, `%${destination} = add i32 %${res}, 0`);
    } else {
      throw new Error(
        'Attempt to reference undefined variable or unsupported literal: ' +
          exp,
      );
  }

  ...
}

Example:
1
...
a
...
(+ 1 a)

compileCall(functionName, arguments, destination, scope)
Most function calls will automatically compile arguments before
calling the function. However, certain control-flow primitives don't
do this (e.g. def, if, etc.). Macros in Lisp
allow you to add new control-flow primitives (even if you don't use it
to modify control-flow). But we will ignore user-defined primitives
for now.
We'll keep a list of control-flow primitives and pass off compilation
to them if the function name matches a primitive. Otherwise, we'll
look up the function name in scope (to find its safe name), compile
the arguments, and call the function with the results of the
arguments.
class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
    };
  }

  ...

  compileCall(fun, args, destination, scope) {
    if (this.primitiveFunctions[fun]) {
      this.primitiveFunctions[fun](args, destination, scope);
      return;
    }

    const validFunction = scope.get(fun);
    if (validFunction) {
      const safeArgs = args
        .map((a) => {
          const res = scope.symbol();
          this.compileExpression(a, res, scope);
          return 'i32 %' + res;
        })
        .join(', ');
      this.emit(1, `%${destination} = call i32 @${validFunction}(${safeArgs})`);
    } else {
      throw new Error('Attempt to call undefined function: ' + fun);
    }
  }

  ...
}

Yay LLVM for simplifying calls!
Example:
(foo 1)
...
(+ 1 2)

compileDefine([functionName, parameters, ...body], destination, scope)
This is the last undefined compile function we've used. The call
signature may look funny but we write less code if we keep the
primitive signatures the same. In any case, JavaScript's destructuring
makes it pretty enough.
Aside from code generation, we also need to add the function itself to
scope so we can look it up later in use. Additionally we need to
create a copy of the current scope for the body of the function. And
we'll add the parameter names themselves to the child scope.
class Compiler {
  ...

  compileDefine([name, params, ...body], destination, scope) {
    // Add this function to outer scope
    const safeName = scope.register(name);

    // Copy outer scope so parameter mappings aren't exposed in outer scope.
    const childScope = scope.copy();

    const safeParams = params.map((param) =>
      // Store parameter mapped to associated local
      childScope.register(param),
    );

    this.emit(
      0,
      `define i32 @${safeName}(${safeParams
        .map((p) => `i32 %${p}`)
        .join(', ')}) {`,
    );

    // Pass childScope in for reference when body is compiled.
    const ret = childScope.symbol();
    this.compileExpression(body[0], ret, childScope);

    this.emit(1, `ret i32 %${ret}`);
    this.emit(0, '}\n');
  }

  ...
}

Example:
(def plus (a b) (+ a b))

compileOp(op)
The last function mentioned above will help us expose some useful
primitives.  This function will take a string builtin operation and
return a function that can be used to generate code when the operation
is called.
class Compiler {
  ...

  compileOp(op) {
    return ([a, b], destination, scope) => {
      const arg1 = scope.symbol();
      const arg2 = scope.symbol();
      this.compileExpression(a, arg1, scope);
      this.compileExpression(b, arg2, scope);
      this.emit(1, `%${destination} = ${op} i32 %${arg1}, %${arg2}`);
    };
  }

  ...
}

This allows us to add some builtin ops as primitives (even though they
aren't control-flow modifying).
class Compiler {
  constructor() {
    this.outBuffer = [];
    this.primitiveFunctions = {
      def: this.compileDefine.bind(this),
      begin: this.compileBegin.bind(this),
      '+': this.compileOp('add'),
      '-': this.compileOp('sub'),
      '*': this.compileOp('mul'),
    };
  }

  ...
}

Example:
(+ 1 2)

Hello world!
Putting it all together, we'll compile this Lisp program:
(def plus-two (a b)
     (+ a (+ b 2)))

(def main ()
     (plus-two 3 (plus-two 1 1)))

To get 9.
$ node src/ulisp.js tests/function_definition.lisp
$ ./build/prog
$ echo $?
9

Generated code
The generated LLVM can be found in ./build/prog.ll:
define i32 @plus_two(i32 %a, i32 %b) {
  %sym7 = add i32 %a, 0
  %sym9 = add i32 %b, 0
  %sym10 = add i32 2, 0
  %sym8 = add i32 %sym9, %sym10
  %sym6 = add i32 %sym7, %sym8
  ret i32 %sym6
}

define i32 @main() {
  %sym6 = add i32 3, 0
  %sym8 = add i32 1, 0
  %sym9 = add i32 1, 0
  %sym7 = call i32 @plus_two(i32 %sym8, i32 %sym9)
  %sym5 = call i32 @plus_two(i32 %sym6, i32 %sym7)
  ret i32 %sym5
}

You can see all these unnecessary add, ... 0
instructions. But let's look at the x86 assembly that LLVM generates
in build/prog.s:
...

_plus_two:                              ## @plus_two
        .cfi_startproc
## %bb.0:
                                        ## kill: def $esi killed $esi def $rsi
                                        ## kill: def $edi killed $edi def $rdi
        leal    2(%rdi,%rsi), %eax
        retq
        .cfi_endproc
                                        ## -- End function
...

And we see that LLVM easily optimized the inefficiencies away. :)
Next up

Compiling conditionals
Tail call optimization
Lists and dynamic memory
Strings?
Foreign function calls?
Self-hosting?

Adding an LLVM backend to ulisp (small Lisp compiler in JavaScript) https://t.co/VIddKW1r3N
— Phil Eaton (@phil_eaton) March 10, 2019
 


Trot Trot Machine Trotting
infrequent oscillations — Thu, 07 Mar 2019 13:08:25 GMT
In which we enter the CHALLENGE STAKES CUP PACE TROT


Getting back into MMORPGs
Ben Overmyer's Site — Wed, 06 Mar 2019 02:00:00 GMT
As a result of my one-year boycott of Activision-Blizzard, I found myself without any of my usual games to play. I decided to take the opportunity to return to a genre I haven't played much of in the past few years - MMORPGs.
Aside from a month where I played World of Warcraft following the new expansion, and about three weeks playing Final Fantasy XIV after that, I haven't played MMOs much in the last three years. With Pantheon: Rise of the Fallen still a long way off, Crowfall still in alpha, and Camelot Unchained in... whatever state it's in, I decided to have a look at games I've played in the past for something to occupy my attention.
I settled on four options - Elder Scrolls Online, Black Desert Online, Project Gorgon, and Guild Wars 2. ArenaNet's layoffs killed my enthusiasm for Guild Wars 2, so that dropped off the list. Black Desert Online had some strange bugs with keyboard controls (like erasing all my key binds) that left a sour taste in my mouth. Some friends of mine decided to try playing ESO together, so I decided to join them. But, I also have renewed interest in Project Gorgon because of its influx of Pantheon fans, so I've decided to alternate between those two.
I never got very far in either game. In Elder Scrolls Online, my highest character was level eight. My character in Project Gorgon still hasn't left the starter town. Of the two, ESO seems to be holding my attention better. The story is pulling me along. That got me to thinking... what exactly do I want out of an MMORPG?
These days twitch shooters usually are my go-to. But with most people obsessed with battle arena games (which I hate) and Overwatch and Destiny 2 in my boycott, there isn't much to play that interests me. So I've decided to take a different angle. What if I actually relax with video games, instead of get a drip-feed of adrenaline? So that's what I'm trying to do with ESO and Project Gorgon: relax. Sit back and enjoy a slower pace.
So far it seems to be working.
I may write in the future about my adventures as a wood elf nightblade in Elder Scrolls Online. I'm taking to the character's playstyle pretty well. For now, I'll finish with this thought: when's the last time you relaxed with a video game?


How can I help? East coast vs. West coast mentalities
CRAIG KERSTIENS — Sun, 03 Mar 2019 22:55:56 GMT
Often times when I’m traveling on the east coast, whether it is NYC area or back home down south I try to spend some time to catch up with various people. In catching up we’ll spend some time talking about what we’re both up to, thoughts on tech or in general, and at the end I typically ask “Is there anything in particular I can help with?” More often than not the answer to this question isn’t super substantial, which is fine. But what is surprising is the stark contrast on reactions to this question and how it differs from west coast vs. east coast.
On the east coast when I ask “How can I help?” I get a look of:

What’s the catch?
Okay, what is it you want me to do for you?
There is no way you actually intend to help…

Meanwhile on the west coast the mentality is generally quite different. I often have people showing their own willingness to help. It may be someone you know personally that is offering help, but have also seen where someone gets asked for input or help, refers someone else and now you have two people that have known each other for under 5 minutes and one willing to dispense helpful advice or assistance.
The how I can help mentality of the west coast seems to follow that offering assistance is not a zero sum game, and by extending some effort now it hopefully comes back around when they need it themselves.


rxe: literate and composable regular expressions
Bytepawn - Marton Trencseni — Sat, 02 Mar 2019 01:00:00 GMT
rxe is a thin wrapper around Python's re module. The various rxe functions are wrappers around corresponding re patterns. For example, rxe.digit().one_or_more('a').whitespace() corresponds to \da+\s. Because rxe uses parentheses but wants to avoid unnamed groups, the internal (equivalent) representation is actually \d(?:a)+\s. This pattern can always be retrieved with get_pattern().




Mouldeth Thy Lord
infrequent oscillations — Fri, 01 Mar 2019 23:18:27 GMT
Moulding a tribute to Our Lord Helix for Helix Day


AOT-compilation of Javascript with V8
Notes on software development — Tue, 26 Feb 2019 02:00:00 GMT
tldr; I'm working on a AOT-compiled Javascript implementation called
jsc.
Many dynamically typed programming languages have implementations that
compile to native binaries:

Python: Cython
Common Lisp: SBCL
Scheme: Chicken Scheme

The benefits of compiling dynamically typed languages are similar to
those of compiling statically typed languages:

Simplified deployment via a single binary
Simplified foreign-function interfaces
e.g. Embedded C/C++ strings


Predictable performance compared to JIT compiling interpreters
Performance gains compared to non-JIT compiling interpreters

I (re)discovered a common technique for compiling dynamic languages
while developing BSDScheme,
an interpreter and compiler for Scheme. In this technique, you use
core parts of the runtime code as a library that is imported and
referenced by compiled code.
You save time building object-memory representations, memory
management, operations, interacting with existing libraries, etc. when
an interpreter already exists. The runtime as a library (plus existing
parser frontends) allows you to focus solely on code generation of
control flow.
The first pass
I wrote the initial version of jsc
in Rust using Dave Herman's
esprit parser (supports a subset
of ES6 that includes all of ES5).
The interesting parts of the runtime are taken care of by V8, e.g.:

V8::String - a Javascript string object
V8::String::NewFromUtf8(isolate, "hello world!") - C++ string to Javascript string object


V8::Number - a Javascript number object
V8::Number::New(isolate, 10) - C++ double to Javascript number object


Heap allocations
Calling convention

And so on.
An example
This first version of jsc could take the following Javascript:
function fib(n, a, b) {
    if (n == 0) {
        return a;
    }

    if (n == 1) {
        return b;
    }

    return fib(n - 1, b, a + b);
}

function main() {
  console.log(fib(50, 0, 1));
}

And produce the following C++:
#include <string>
#include <iostream>

#include <node.h>

using v8::Array;
using v8::Boolean;
using v8::Context;
using v8::Exception;
using v8::Function;
using v8::FunctionTemplate;
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Null;
using v8::Number;
using v8::Object;
using v8::String;
using v8::False;
using v8::True;
using v8::Value;

void fib_0(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
  Local<Value> n_1 = args[0];
  Local<Value> a_2 = args[1];
  Local<Value> b_3 = args[2];
tail_recurse_4:

  Local<Context> ctx_5 = isolate->GetCurrentContext();
  Local<Object> global_6 = ctx_5->Global();
  Local<Function> Boolean_7 = Local<Function>::Cast(global_6->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_8(n_1);
  std::string string_tmp_9(*utf8value_tmp_8);
  String::Utf8Value utf8value_tmp_10(Number::New(isolate, 0));
  std::string string_tmp_11(*utf8value_tmp_10);
  Local<Value> argv_12[] = { (n_1->IsBoolean() || Number::New(isolate, 0)->IsBoolean()) ? Boolean::New(isolate, n_1->ToBoolean(isolate)->Value() == Number::New(isolate, 0)->ToBoolean(isolate)->Value()) : ((n_1->IsNumber() || Number::New(isolate, 0)->IsNumber()) ? Boolean::New(isolate, n_1->ToNumber(isolate)->Value() == Number::New(isolate, 0)->ToNumber(isolate)->Value()) : ((n_1->IsString() || Number::New(isolate, 0)->IsString()) ? Boolean::New(isolate, string_tmp_9 == string_tmp_11) : (False(isolate)))) };
  Local<Value> result_13 = Boolean_7->Call(Null(isolate), 1, argv_12);
  if (result_13->ToBoolean()->Value()) {
    // return a;
    args.GetReturnValue().Set(a_2);
    return;


  }

  Local<Context> ctx_14 = isolate->GetCurrentContext();
  Local<Object> global_15 = ctx_14->Global();
  Local<Function> Boolean_16 = Local<Function>::Cast(global_15->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_17(n_1);
  std::string string_tmp_18(*utf8value_tmp_17);
  String::Utf8Value utf8value_tmp_19(Number::New(isolate, 1));
  std::string string_tmp_20(*utf8value_tmp_19);
  Local<Value> argv_21[] = { (n_1->IsBoolean() || Number::New(isolate, 1)->IsBoolean()) ? Boolean::New(isolate, n_1->ToBoolean(isolate)->Value() == Number::New(isolate, 1)->ToBoolean(isolate)->Value()) : ((n_1->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? Boolean::New(isolate, n_1->ToNumber(isolate)->Value() == Number::New(isolate, 1)->ToNumber(isolate)->Value()) : ((n_1->IsString() || Number::New(isolate, 1)->IsString()) ? Boolean::New(isolate, string_tmp_18 == string_tmp_20) : (False(isolate)))) };
  Local<Value> result_22 = Boolean_16->Call(Null(isolate), 1, argv_21);
  if (result_22->ToBoolean()->Value()) {
    // return b;
    args.GetReturnValue().Set(b_3);
    return;


  }

  // return fib(n - 1, b, a + b);
  Local<Value> arg_23 = (n_1->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? (Number::New(isolate, n_1->ToNumber(isolate)->Value() - Number::New(isolate, 1)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<Value> arg_24 = b_3;
  Local<Value> arg_25 = (a_2->IsString() || b_3->IsString()) ? Local<Value>::Cast(String::Concat(a_2->ToString(), b_3->ToString())) : Local<Value>::Cast((a_2->IsNumber() || b_3->IsNumber()) ? (Number::New(isolate, a_2->ToNumber(isolate)->Value() + b_3->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate)));
  Local<FunctionTemplate> ftpl_27 = FunctionTemplate::New(isolate, fib_0);
  Local<Function> fn_26 = ftpl_27->GetFunction();
  fn_26->SetName(String::NewFromUtf8(isolate, "fib_0"));
  n_1 = arg_23;
  a_2 = arg_24;
  b_3 = arg_25;
  goto tail_recurse_4;

}

void jsc_main(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
tail_recurse_5:

  // console.log(fib(50, 0, 1))
  Local<Value> dot_parent_7 = isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console"));
  Local<String> property_8 = String::NewFromUtf8(isolate, "log");
  while (dot_parent_7->IsObject() && !dot_parent_7.As<Object>()->HasOwnProperty(isolate->GetCurrentContext(), property_8).ToChecked()) {
    dot_parent_7 = dot_parent_7.As<Object>()->GetPrototype();
  }
  Local<Value> dot_result_6 = dot_parent_7.As<Object>()->Get(isolate->GetCurrentContext(), property_8).ToLocalChecked();
  Local<Value> arg_9 = Number::New(isolate, 50);
  Local<Value> arg_10 = Number::New(isolate, 0);
  Local<Value> arg_11 = Number::New(isolate, 1);
  Local<FunctionTemplate> ftpl_13 = FunctionTemplate::New(isolate, fib_0);
  Local<Function> fn_12 = ftpl_13->GetFunction();
  fn_12->SetName(String::NewFromUtf8(isolate, "fib_0"));
  Local<Value> argv_14[] = { arg_9, arg_10, arg_11 };
  Local<Value> result_15 = fn_12->Call(Null(isolate), 3, argv_14);
  Local<Value> arg_16 = result_15;
  Local<Function> fn_17 = Local<Function>::Cast(dot_result_6);
  Local<Value> argv_18[] = { arg_16 };
  Local<Value> result_19 = fn_17->Call(dot_parent_7, 1, argv_18);
  result_19;

}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

This output gets compiled (by jsc) as a Node
addon using
node-gyp.
The compiled addon is loaded by a single-line Javascript file generated by jsc:
$ rm -rf build
$ jsc fib.js
$ cat build/fib.js
require("build/Release/fib.node").jsc_main()
$ node build/fib.js
12586269025

Analysis
The code was a mess of bad formatting, unnecessary locals, inefficient
basic operations (e.g. huge, often unnecessary Boolean conversions),
and so on. The unnecessary locals was partially a by-product of
single-pass code generation. And the unnecessary conversions was
partly due to ignoring types (even types of literals that you don't
need Typescript/Flow to provide).
After I got this proof-of-concept working for basic examples, I wanted
to rewrite it around destination-driven code
generation,
a technique by Kent Dybvig used in V8's baseline compiler. And after a
few weeks not getting far in a refactor in Rust, I rewrote the
compiler in Typescript.
The second pass
Written in Typescript and using the Typescript compiler
API,
this second iteration was built to do destination-driven code
generation and leaf type propagation. Destination-driven code
generation allows a single-pass code generator to reduce redundant
reassignments. And leaf type propagation allows simple, obvious
optimizations such as just calling V8::Boolean::IsTrue()
on a statically-known boolean rather than calling
V8::Value::Equals().
Example
Given the same fibonacci Javascript program from before, this
iteration produces the following C++:
#include "lib.cc"

void tco_fib(const FunctionCallbackInfo<Value>& _args) {
  Isolate* isolate = _args.GetIsolate();
  std::vector<Local<Value>> args(_args.Length());;
  for (int i = 0; i < _args.Length(); i++) args[i] = _args[i];
tail_recurse_0:

  Local<Number> sym_rhs_4 = Number::New(isolate, 0);
  Local<Boolean> sym_anon_2 = args[0]->StrictEquals(sym_rhs_4) ? True(isolate) : False(isolate);
  if (sym_anon_2->IsTrue()) {
    _args.GetReturnValue().Set(args[1]);
    return;
  }

  Local<Number> sym_rhs_11 = Number::New(isolate, 1);
  Local<Boolean> sym_anon_9 = args[0]->StrictEquals(sym_rhs_11) ? True(isolate) : False(isolate);
  if (sym_anon_9->IsTrue()) {
    _args.GetReturnValue().Set(args[2]);
    return;
  }

  Local<Number> sym_rhs_19 = Number::New(isolate, 1);
  Local<Value> sym_arg_17 = genericMinus(isolate, args[0], sym_rhs_19);
  Local<Value> sym_arg_21 = genericPlus(isolate, args[1], args[2]);
  args[0] = sym_arg_17;
  args[1] = args[2];
  args[2] = sym_arg_21;
  goto tail_recurse_0;

  return;
}

void jsc_main(const FunctionCallbackInfo<Value>& _args) {
  Isolate* isolate = _args.GetIsolate();
  std::vector<Local<Value>> args(_args.Length());;
  for (int i = 0; i < _args.Length(); i++) args[i] = _args[i];
tail_recurse_1:

  Local<Number> sym_arg_29 = Number::New(isolate, 100);
  Local<Number> sym_arg_30 = Number::New(isolate, 0);
  Local<Number> sym_arg_31 = Number::New(isolate, 1);
  Local<Value> sym_args_32[] = { sym_arg_29, sym_arg_30, sym_arg_31 };
  Local<Function> sym_fn_33 = FunctionTemplate::New(isolate, tco_fib)->GetFunction();
  sym_fn_33->SetName(String::NewFromUtf8(isolate, "tco_fib"));
  Local<Value> sym_arg_28 = sym_fn_33->Call(sym_fn_33, 3, sym_args_32);

  Local<Value> sym_args_34[] = { sym_arg_28 };
  Local<Value> sym_parent_37 = isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console"));
  Local<Value> sym_anon_36 = sym_parent_37.As<Object>()->Get(String::NewFromUtf8(isolate, "log"));
  Local<Function> sym_fn_35 = Local<Function>::Cast(sym_anon_36);
  Local<Value> sym_anon_27 = sym_fn_35->Call(sym_fn_35, 1, sym_args_34);

  return;
}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

Analysis
Common code (genericPlus, genericMinus) and
all imports have been pulled into lib.cc for clarity. And
the entire result is run through
clang-format if it is
present on the system.
The benefit of leaf type propagation can be seen everywhere a local is
declared that is not Local and specifically in if
tests on statically known booleans:
...
Local<Boolean> sym_anon_2 = args[0]->StrictEquals(sym_rhs_4) ? True(isolate) : False(isolate);
  if (sym_anon_2->IsTrue()) {
...

It's obvious to a human that there is another optimization you could
do here by not wrapping this check in a V8::Boolean at
all. The only types tracked in destinations are V8 types, not yet C++
types. But not needing to passing this through a bool
toBoolean(Value v) wrapper is still an improvement.
In general, unboxing has not really been explore. But the ultimate
goal is to use Typescript types to produce function- or block-level
unboxed versions -- perhaps using a toggle in code to specify safety à
la Common Lisp.
Next steps
I broke tests and regressed on syntax support in the Typescript port,
so that's the first step. The second step is enough syntax to support
more interesting benchmarks than the fibonacci example (which has
comparative performance to Node.js/V8 but isn't saying much).
After that:

Unboxed expressions
Unboxed blocks
Foreign-function interface
Self-hosting
Node-API compatible runtime without Node

Companion blog post to my talk on an AOT-compiled Javascript implementation built on Typescript https://t.co/0aHVJ9UzYh
— Phil Eaton (@phil_eaton) February 26, 2019
 


Learn Enough tmux to be Productive in 5 Minutes
Ben E. C. Boyter — Sun, 24 Feb 2019 02:00:00 GMT
tmux the terminal multiplexer is one of those brilliant tools I use all day. I am not a tmux expert, nor do I have any need to be. What follows is the bare minimum you need to know to feel productive in tmux, written for some work colleagues.
The following assumes you have not configured or tweaked tmux in any way opting for the default settings.
From the command line to start or connect to tmux


Dealing with dead disks in a btrfs RAID1 array
Nemo's Home — Sun, 24 Feb 2019 02:00:00 GMT
tl;dr: Check your disk usage v/s RAID capacity to ensure that you can remove a disk before trying. If you can connect a new disk without removing the old one, run a btrfs replace - it is much faster.



My homeserver setup has a 4 disk setup:


  128GB Samsung EVO 850 SSD as the primary disk (root volume)
  A 3 Disk btrfs RAID1 Array that I use for almost everything else.


The 3 disks were:


  A WD-3.5inch-3TB that I shelled from a WD-MyBook. This was the oldest disk in the array
  2xSeagate 2.5-inch-3TB external disks that I shelled from Seagate Expansion disks.


The WD disk had been giving rising errors recently, and I was noticing hangs on the system as well:


  My Steam saves would take time, and hang the game.
  Kodi would ocassionaly hang just switching between screens as it would load images from disk.
  gitea, which writes a lot to disk would get similar issues.


I asked a question on r/archlinux and confirmed that it indeed a dead disk.

Ordered a new Seagate Barracuda 3TB the next day, but my peculiar setup caused me a lot of pain before I could remove the dead disk. The primary issue was with the limited number of SATA connectors I had (just 4). The original setup had /dev/sdb,/dev/sdc,/dev/sdd as the three RAID disks with /dev/sdb being the dying WD.

This is what all I tried:


  Removing /dev/sdb and adding a new disk the array (/dev/sde). Unfortunately, to add a disk to the array, you have to mount it first, and the setup just refused to mount in degraded mode. (It didn’t give a visibly error, so I didn’t know why)
  I tried to keep the old disk attached over USB on a friend’s suggestion, but that didn’t work either. This was likely a cable issue, and I didn’t investigate this further.
  Booting with the original three disks but replacing the dying disk with the new one post boot. Didn’t work as I kept getting read/write errors to sdb even after it was disconnected.


In short:


  the system refused to mount the raid array with a missing disk (and I didn’t want to risk a boot with the array unavailable)
  I couldn’t do a live replace because I had a limited number of SATA connectors.


What worked:

Running a btrfs device delete and leting it run overnight. It gave an error after quite a long time that finally helped me figure out the problem:

btrfs device delete /dev/sdb1 /mnt/xwing
ERROR: error removing device '/dev/sdb1': No space left on device

btrfs fi df /mnt/xwing
Data, RAID1: total=2.98TiB, used=2.98TiB
System, RAID1: total=32.00MiB, used=544.00KiB
Metadata, RAID1: total=5.49GiB, used=4.81GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


The RAID array was 2.7TBx3 disks and I was storing roughly 2.98TB of data. To switch to a RAID1 setup with just 2 disks, I needed to delete some data. I ended up clearing out a few steam games (bye bye Witcher 3) and ran another btrfs device delete to resolve the issue.

If you are faced with a situation where you have to remove a device, but can’t do a live replace, here’s what you need:


  Check that your disk removal does not impact any data storage. Your n-1 disk array should have enough capacity to store everything.
  Run a btrfs device delete
  Reboot
  Re-attach new disk, and then run a btrfs device add


As a retro, I posted a summary with the issues I faced on the btrfs mailing list



If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:


  Part 1, Hardware
  Part 2, Terraform/Docker
  Part 3, Learnings
  Part 4, Migrating from Google (and more)
  Part 5, Home Server Networking
  Part 6, btrfs RAID device replacement


If you have any comments, reach out to me


Unsupervised Learning: Information Theory Recap
swyx's site RSS Feed — Sat, 23 Feb 2019 15:45:41 GMT
A small detour to catch up on the basics of Information Theory we'll need


Unsupervised Learning: Randomized Optimization
swyx's site RSS Feed — Sat, 23 Feb 2019 14:32:43 GMT
Hill Climbing, Simulated Annealing, Genetic Algorithms, oh my!


Unsupervised Learning: Randomized Optimization
swyx's site RSS Feed — Sat, 23 Feb 2019 14:32:43 GMT
Hill Climbing, Simulated Annealing, Genetic Algorithms, oh my!


Supervised Learning: Bayesian Inference
swyx's site RSS Feed — Sat, 23 Feb 2019 11:02:44 GMT
Or, the unreasonable effectiveness of dumb rules


Post Bootcamp Mindmap
swyx's site RSS Feed — Thu, 21 Feb 2019 02:00:00 GMT
What are all the things you can do to improve as a developer after graduating from college or bootcamp?


Fixing the Exim 'certificate name mismatch' error
infrequent oscillations — Wed, 20 Feb 2019 15:23:50 GMT
Spoiler: it was manualroute to bare IPs


Goal Differences
infrequent oscillations — Tue, 19 Feb 2019 11:22:37 GMT
In which we are disappointed by goal differences.


Supervised Learning: Bayesian Learning
swyx's site RSS Feed — Sat, 16 Feb 2019 04:37:05 GMT
Lets update our priors! Oh wait we have none.


Supervised Learning: VC Dimensions
swyx's site RSS Feed — Fri, 15 Feb 2019 10:14:24 GMT
It's not venture capital, that's for sure


Supervised Learning: Computational Learning Theory
swyx's site RSS Feed — Fri, 15 Feb 2019 09:03:05 GMT
What's the big O of machine learning? Lets put some formal theory around HOW we learn!


SQL: One of the most valuable skills
CRAIG KERSTIENS — Tue, 12 Feb 2019 22:55:56 GMT
I’ve learned a lot of skills over the course of my career, but no technical skill more useful than SQL. SQL stands out to me as the most valuable skill for a few reasons:

It is valuable across different roles and disciplines
Learning it once doesn’t really require re-learning
You seem like a superhero. You seem extra powerful when you know it because of the amount of people that aren’t fluent

Let me drill into each of these a bit further.


SQL a tool you can use everywhere


Regardless of what role you are in SQL will find a way to make your life easier. Today as a product manager it’s key for me to look at data, analyze how effective we’re being on the product front, and shape the product roadmap. If we just shipped a new feature, the data on whether someone has viewed that feature is likely somewhere sitting in a relational database. If I’m working on tracking key business metrics such as month over month growth, that is likely somewhere sitting in a relational database. At the other end of almost anything we do there is likely a system of record that speaks SQL. Knowing how to access it most natively saves me a significant amount of effort without having to go ask someone else the numbers.
But even before becoming a product manager I would use SQL to inform me about what was happening within systems. As an engineer it could often allow me to pull information I wanted faster than if I were to script it in say Ruby or Python. When things got slow in my webapp having an understanding of the SQL that was executed and ways to optimize it was indespensible. Yes, this was going a little beyond just a basic understanding of SQL… but adding an index to a query instead of rolling my own homegrown caching well that was well worth the extra time learning.


SQL is permanent


I recall roughly 20 years ago creating my first webpage. It was magical, and then I introduced some Javascript to make it even more impressive prompting users to click Yes/No or give me some input. Then about 10 years later jQuery came along and while it was a little more verbose at times and something new to learn it made things prettier overall so I committed to re-learning the jQuery approach to JS. Then it just picked up pace with Angular -> React/Ember, and now I have an entire pipeline to introduce basic Javascript into my website and the reality is I’m still trying to accomplish the same thing I was 20 years ago by having someone click Yes/No.
SQL in contrast doesn’t really change. Caveat: It has changed–there is modern sql, but I’d still argue less dramatically than other language landscapes. Yes we get a new standard every few years and occasionally something new comes along like support for window functions or CTEs, but the basics of SQL are pretty permanent. Learning SQL once will allow you to re-use it heavily across your career span without having to re-learn. Don’t get me wrong I love learning new things, but I’d rather learn something truly new than just yet another way to accomplish the same task.


SQL: Seem better than you are


SQL is an underlearned skill, the majority of application developers just skip over it. Because so few actually know SQL well you can seem more elite than you actually are. In past companies with hundreds of engineers I’d get a question several times a week from junior to principal engineers of: “hey can you help me figure out how to write a query for this?” Because you’re skilled at something so few others are you can help them out which always makes life a little easier when you have a question for them.
So if you’re not already proficient what are you waiting for, do you want to seem like a SQL badass yet?
If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.


More Music
infrequent oscillations — Tue, 12 Feb 2019 20:00:47 GMT
Some more music and a lunatic from Bolton


PyTorch Basics: Solving the Ax=b matrix equation with gradient descent
Bytepawn - Marton Trencseni — Fri, 08 Feb 2019 01:00:00 GMT
I will show how to solve the standard A x = b matrix equation with PyTorch. This is a good toy problem to show some guts of the framework without involving neural networks.




Setting up a new personal blog
Ben Overmyer's Site — Wed, 06 Feb 2019 02:00:00 GMT
Having a blog built using a static site generator was nice from a maintenance perspective, but from a writing perspective, it's a bit of an inconvenience.
So now I've created a new blog on this address using WordPress. I already use WordPress for the Silver Gryphon Games website and I maintain the Geek Partnership Society website (which also uses WordPress), so I might as well add a third one to the mix.
I will probably migrate all of my existing blog posts to this site. First though I'll need to write a utility to migrate from Hugo to WordPress. That kind of tool doesn't exist yet.


Goodbye XenSever - Hello XCP-ng
smcleod.net — Tue, 05 Feb 2019 15:00:00 GMT
In 2018 I set out to replace our XenSever 7.2 based virtualisation after Citrix essentially screwed over free / open source users.
This project was to directly replace XenServer 7.2 with something supported and manageable for our traditional virtualisation needs.

High Level Selection Considerations
I evaluated a number of options, with the primary candidates below.
Key criteria (at a high level) I was evaluating:

Ease of moving from our existing XenServer 7.2 based hypervisor clusters.
Security (architecture, hardening, monitoring, logging).
Cost (both licensing if any and self-support / management costs).
VM Performance (Storage IOPs and throughput, Network latency and throughput, Processing latency, steal from over-provisioned workloads).
Management UI/UX and performance (for BAU activities).
Management / Cluster SPOFs, fail-over and redundancy.
Installation and upgrade process.
Update and security patching frequency.
Networking design and complexity.
Community (size, engagement, acceptance of suggestions / MRs).
Reliable live VM migrations.
Ease of management for a small team (Part of a low TCO).
Risk of vendor and technology lock-in.
Risk of survival (will it still be well maintained over the next 1-3 years).
Stability and reliability above practically all else.

XenServer 7.6 (w/ Paid License)
For

Potentially priority support from Citrix for issues.
Easy upgrade from 7.2.
Fresh installs and upgrades are simple, painless and easy to pxeboot, licensing can be a pain after install however.

Against

Slow moving development.
Incredibly expensive, it would cost us something like $140,800 AUD per year (shelf price) for our 32 hosts (each with 2 sockets).
Features heavily restricted by licensing model.
Mostly older-fashioned enterprise users.
No web management interface (although you can use Xen Orchestra from the folks behind XCP-ng).
Diminishing community since XenServer licensing changes in 7.3.
Poor storage performance compared to KVM based solutions.
Many outdated packages, while kind of based on CentOS 7, there are a great deal of packages from older releases or completely custom rebuilt.
Uses old technologies like EXT3 and doesn’t support SSD/Flash TRIM/DISCARD functions.
SELinux not enforcing or supported.
OpenvSwitch still uses a lot of Dom0 CPU (same as XenServer).
Dom0 often ends up limiting VM storage operations (tapdisk maxing out Dom0 CPU).
Applying updates can be painful or at least slow, requiring binary ISO files to be downloaded from Citrix.
Uses Jira for bug tracking (I just can’t stand the thing, it’s painfully laggy, give my GitLab or Github over it any day).

XCP-ng
XCP-ng is a relatively recent fork from XenServer after it was open sourced, tracking upstream but clearly prioritising modernisation and community.


SQLite Parameter Queries
infrequent oscillations — Mon, 04 Feb 2019 21:15:00 GMT
Since sqlite doesn’t support stored procedures, it’s tricky to parameterise a query...


Supervised Learning: Support Vector Machines
swyx's site RSS Feed — Fri, 01 Feb 2019 08:03:32 GMT
Transforming dimensions, nearest neighbors, and boosting, all in service of "drawing the best line".


More Football Facts And Figures
infrequent oscillations — Wed, 30 Jan 2019 22:53:52 GMT
In which we amuse ourselves with football facts.


Using mkcert to Set Up Local SSL on macOS's Built in Apache
Donat Studios — Wed, 30 Jan 2019 02:16:47 GMT
A little under a year ago I wrote a walkthrough: Set up a Self Signed Certificate on macOS's Built in Apache.
Recently a new tool showed up called mkcert has come and made life way easier.
It's a wonderful tool that instead of building Self-Signed Certificates, sets your local machine up with a certificate signing authority and creates valid certificates. 
You can read more about how it works at the link above. Needless to say, the process is much less painful than creating self signed certificates.
Installing mkcert
The easiest way to install mkcert is via Homebrew.
Beyond mkcert we should also install nss which helps mkcert set up Firefox to respect its certificates.
$ brew install mkcert nss
Once that's complete, we need mkcert to set itself up as a signing authority on our local machine.
$ mkcert -install
Creating the Certificates
Once we have mkcert set up, the next step is to generate the certificates for the intended local domain(s). 
You may include multiple domains or wildcard subdomains. You can find more information about mkcert's capabilities here.
$ mkdir /tmp/crt && cd /tmp/crt
$ mkcert local.example.com
You should get an output something like as follows
Using the local CA at "/Users/user/Library/Application Support/mkcert" ✨

Created a new certificate valid for the following names ????
 - "local.example.com"

The certificate is at "./local.example.com.pem" and the key at "./local.example.com-key.pem" ✅
Now that we have our certificate, we can move it into the apache /etc directory:
$ sudo mkdir /etc/apache2/ssl
$ sudo mv *.pem /etc/apache2/ssl
Apache Configuration
Next, all we need to do is configure Apache. 
Follow the Apache Configuration section's instructions, stopping and returning here before the "Chrome / Safari Configuration" section on my previous post.
When you reach the portion about configuring your VirtualHosts, the only difference is the filenames will look more akin to this pattern: 
  SSLEngine on
  SSLCertificateFile "/private/etc/apache2/ssl/local.example.com.pem"
  SSLCertificateKeyFile "/private/etc/apache2/ssl/local.example.com-key.pem"
Unlike the prior walkthrough which restarted Apache with a reboot, we instead now only need to restart Apache.
$ sudo apachectl restart
And we're done! Unlike setting up a self signed certificate, we do not need to add anything to our keychain or browsers. We are ready to go, your certificate should work as expected.


Iron Arachne Updates for January
Ben Overmyer's Site — Wed, 30 Jan 2019 02:00:00 GMT
It's been a dry spell for Iron Arachne for the past couple months. The holidays and work on Silver Gryphon Games meant that I had no time for coding outside of work.
Coming up in February I have a two-week trip to Thailand. I won't be coding during that, either!
Yet, I have been spooling up some projects for Iron Arachne. Here are some of the things I'll be doing this half of the year:

Fixing a few bugs with the heraldry generator
Writing a language description generator
Expanding the Æther setting generator (under the Silver Gryphon Games banner)

I plan on doing work to make publishing and discovering these generators easier. That means releasing several of them as Docker images on Docker Hub, among other things.
The generator community is a talented group. More links to authors' tools are going to appear on the Iron Arachne website.
I'm thinking about what kinds of generators I should be writing for Iron Arachne. Many of the ideas I've been running with so far have been things that've done many times before by other people. This includes name generators, character generators, and so forth. I'd like to cover new territory now that I have more experience with writing these tools.


Supervised Learning: Ensemble Learning and AdaBoost
swyx's site RSS Feed — Sun, 27 Jan 2019 23:35:27 GMT
Better together - how bootstrapping samples of data can work better than the entire dataset, and how to boost it even further, and faster


Linear Typewriters, Part 1
infrequent oscillations — Sun, 27 Jan 2019 11:24:08 GMT
Adventures in optimising a linear typewriter, part 1


Supervised Learning: Instance-based Learning and K-Nearest Neighbors
swyx's site RSS Feed — Sun, 27 Jan 2019 04:38:52 GMT
Regression isn't the only way. What if we were far, far... lazier about it?


Books I abandoned
Krishna's blog — Sun, 27 Jan 2019 02:00:00 GMT
Discover the books this reader abandoned and why, from a dictator's story to a Nobel Prize winner's account of Soviet concentration camps.


Automating a Call Center with Machine Learning
Bytepawn - Marton Trencseni — Sun, 27 Jan 2019 01:00:00 GMT
Over a period of 6 months, we rolled out a Machine Learning model to predict a customer’s delivery (latitude, longitude). During the recent holiday peak, this ML model handled most of Fetchr’s order scheduling.




Supervised Learning: Neural Networks
swyx's site RSS Feed — Tue, 22 Jan 2019 08:03:54 GMT
That one time we tried to emulate our brains with computer chips


Supervised Learning: Regression
swyx's site RSS Feed — Tue, 22 Jan 2019 06:37:24 GMT
Drawing lines among dots and more!


Supervised Learning: Classification Learning & Decision Trees
swyx's site RSS Feed — Tue, 22 Jan 2019 05:51:50 GMT
The simplest form of Classification algorithm


Machine Learning: An Overview
swyx's site RSS Feed — Tue, 22 Jan 2019 02:38:21 GMT
The three major branches of ML


My Spin on the Unix Philosophy
Ben Overmyer's Site — Tue, 22 Jan 2019 02:00:00 GMT
The Unix philosophy guides the development of both Unix and its derivatives. It's a good philosophical core to judge new projects by. I'm increasingly using the Unix philosophy in my own works. This is particularly true of new versions of Iron Arachne command line tools. There are multiple versions of the Unix philosophy bouncing around on the web. Some are longer or more stringent than others. What follows is my own interpretation of it.
Rule One: Tight Focus
Each program should do one thing well. Corollary: Each program should do only one thing.
Rule Two: Predictable Flexibility
Expect each program's output to be the input for some unknown other program. Design for flexibility. Corollary: Also design first for command-line interfaces, and avoid interactive interfaces. Assume machines will use your program more often than humans.
Rule Three: Rapid Iteration
Release early and often. Corollary: It's not only OK but desirable to have minimal change between iterations.
Rule Four: Automated Development
Write tools to assist development instead of seeking out human help. Corollary: It's fine to ask for help when you've tried the above to no success.


Use Go to Restore Glacier to S3
Ben E. C. Boyter — Tue, 22 Jan 2019 02:00:00 GMT
I had a great deal of difficulty searching for an answer to this I have included it here to hopefully save someone else the effort.
How to make a S3 request to retrieve items that have transitioned into AWS Glacier from an S3 bucket using the Go AWS SDK API.
package main

import (
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/awserr"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
		Region: aws.String("ap-southeast-2"),
	})

	svc := s3.New(sess)
	input := &s3.RestoreObjectInput{
		Bucket: aws.String("BucketName"),
		Key:    aws.String("FileKey"),
		RestoreRequest: &s3.RestoreRequest{
			Days: aws.Int64(1),
			GlacierJobParameters: &s3.GlacierJobParameters{
				Tier: aws.String("Expedited"), // Valid values: Expedited | Standard | Bulk
			},
		},
	}

	result, err := svc.RestoreObject(input)
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case s3.ErrCodeObjectAlreadyInActiveTierError:
				fmt.Println(s3.ErrCodeObjectAlreadyInActiveTierError, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}
		return
	}

	fmt.Println(result)
}


Windows
Notes on software development — Sun, 20 Jan 2019 02:00:00 GMT
It has been six years since I last used Windows for any remotely
serious software development. I've used Ubuntu, Arch, or FreeBSD
since. But eventually I spent so much time working around common
workplace tasks that I decided to put Windows 10 Pro on my work
laptop.
Windows Subsystem for Linux
Introduced in 2016, this technology allows Windows to run unmodified
Linux binaries. The core feat being syscall
translation.
It works nearly flawlessly. This means I can do all my Go, Node,
PostgreSQL development on Windows without a virtual machine using
bash, tmux, git, emacs, etc.
I've seen a few minor exceptions over the course of regular software
development in WSL:

ss/netstat does not work
vagrant does not work

More generally, Linux programs are heavily file-oriented. And Windows
I/O is not designed well for
that.
In the worst cases (installing/adding Node packages) it can take
minutes to do operations that would take Linux seconds.
Vagrant
Vagrant-Windows interoperability is abysmal.
As noted above, you cannot manage Hyper-V from vagrant within WSL. So
you're stuck using Powershell. Even then, managing synced files from
vagrant is a nightmare. The default sync method requires you to sign
in using your Windows Live username and password on every
reboot. But Node package installation attempts some file operations
that are not supported over the default synced, network filesystem.
When I switched to rsync vagrant wouldn't reliable sync when the
virtual machine went down and came back up.
After hours of trying to get some files synced with vagrant I gave up.
Hyper-V
Hyper-V's GUI is much more complex/feature-complete than VirtualBox.
It even provides a Ubuntu-quick-install that I used to jump right in.
I don't recommend using this though because it gives you no option but
an 11GB hard disk. I didn't realize this until I went through an hour
or two of post-install customization only to run out of space. Too
lazy to boot into a live CD to grow the root filesystem I reinstalled
with a more suitable 64GB drive and went through the hour-long
post-install customization process again.
Networking in Hyper-V is more complex/feature-complete than VirtualBox
as well. To access a Hyper-V machine you must create a new virtual
network interface manually and associate it. Static IP address appear
to be controlled at the host networking level (e.g. Control Panel)
instead of within the Hyper-V interface. This highlights how these
virtual interfaces are first-class, but overcomplicates the process of
getting started.
Ultimately I gave up on a static IP address and decided to reboot less
frequently.
Performance-wise Hyper-V machines are exactly as expected: excellent.
Misc
Docker support on Windows needs work. It took me a while to understand
how Docker interacts with the WSL filesystem and what I needed to do
to allow Docker to mount. The complexity is similar on macOS when you
want to mount privileged directories like /var, but the experience is
worse on Windows.
Apparently Windows does have tiling window managers, but I have not
tried one out yet.
Powershell, a language with real types, is pretty compelling. But I
have not spent enough time with it to be efficient. And since WSL is
mostly good enough I don't really plan to.
Windows doesn't allow you to delete any files that are "in use". This
is kinda cool except for that the errors you get when trying to delete
files that are in use are useless. They are even more useless when you
get the plain "could not delete directory" when you try to delete a
directory with some file inside it that is in use. I had to start
deleting files within by hand until I found the one I realized was in
use.
Conclusion
If you have never run Linux or FreeBSD, don't use this post as an
excuse not to. You should run Linux or FreeBSD for the experience. But
if you've reached diminishing returns in your Linux/FreeBSD use,
Windows as a development environment has come a long way. It may be
the best platform available for software development, the profession.
Some notes on my experience having replaced Arch Linux with Windows on my work laptop https://t.co/8asxZmspwR
— Phil Eaton (@phil_eaton) January 20, 2019
 


Writing a lisp compiler from scratch in JavaScript: 2. user-defined functions and variables
Notes on software development — Sun, 20 Jan 2019 02:00:00 GMT

  Previously in compiler basics:
  <! forgive me, for I have sinned >
  

  1. lisp to assembly
  

  

  Next in compiler basics:
  

  3. LLVM
  

  4. LLVM conditionals and compiling fibonacci
  

  5. LLVM system calls
  

  6. an x86 upgrade
In this post we'll extend the compiler to support defining functions
and variables. Additionally, we'll require the program's entrypoint to
be within a main function.
The resulting code can be found
here.
Function definition
The simplest function definition we need to support is for our main
function. This will look like this:
$ cat basic.lisp
(def main ()
     (+ 1 2))

Where compiling and running it should produce a return code of 3:
$ node ulisp.js basic.lisp
$ ./build/a.out
$ echo $?
3

Parsing function definitions
The entire language is defined in S-expressions and we already parse
S-expressions.
$ node
> const { parse } = require('./parser');
> JSON.stringify(parse('(def main () (+ 1 2))'));
'[[["def","main",[],["+",1,2]]],""]'

So we're done!
Code generation
There are two tricky parts to code generation once function
definitions are introduced:

Functions definitions are not expressions (in assembly)
Function calling conventions for the callee
Variable scope

Function definitions
A function definition looks like a function call. So we'll need to
keep a list of "primitive" functions that handle what looks like
function calls differently.
function compile_define() {
  // TODO
}

const primitive_functions = {
  def: compile_define,
};

Then in our compile_call function we need to see if the function
being "called" is in this list. If so, we allow the associated
callback to handle compilation.
function compile_call(fun, args, destination) {
  if (primitive_functions[fun]) {
    primitive_functions[fun](args, destination);
    return;
  }

  // Save param registers
  args.map((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile registers and store as params
  args.map((arg, i) => compile_expression(arg, PARAM_REGISTERS[i], scope));

  emit(1, `CALL ${BUILTIN_FUNCTIONS[fun] || scope[fun]}`);

  // Restore param registers
  args.map((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

Now we can begin thinking about compile_define. It takes args
which will be a list of three elements containing the function's:

name
parameters
and body


  It does not use destination because we're treating function
  definitions as statements for now and not as expressions. If we were
  treating it as an expression, we might store the address of the
  function in the destination register.

  We keep destination around to keep the primitive function signatures
  consistent.
Based on how we called functions before and how we defined the
hard-coded add function, we know what a function definition in
assembly generally looks like. And we know the arguments to the
function when called will be in RDI, RSI, and RDX.
function compile_define([name, parameters, body]) {
  // Function name becomes a label we can CALL
  emit(0, `${name}:`);

  // Something to do with RDI, RSI, RDX and the parameters variable?

  // We renamed compile_argument to compile_expression to be more general
  compile_expression(body[0], 'RAX');

  // Maybe some cleanup to do with RDI, RSI, RDX?

  emit(1, 'RET\n');
}

Not a bad first sketch. But how do we match up RDI, RSI, RDX and
the user-defined parameters variable names? For example in the
following:
(def plus-two (a)
     (+ a 2))

It's clear to us that a must match up to RDI. In order to do this
we need to track all variables in a scope dictionary mapping the
variable name to the register where it's stored.
Additionally, keeping track of scope can help us fail quickly in the
following scenario:
(def plus-two (a)
     (+ b 2))

The variable b is used but never defined. It has not been added to
the scope dictionary. So our compiler can fail quickly saying there is
an undefined variable being referenced.
Taking this a step further, what if we want to catch the following
too:
(def plus-two (a)
     (plus a 2))

We're trying to call plus but it has not been defined. We should be
able to fail quickly here too. But that means we're need to track the
scope of function names in addition to variables. We'll choose to
track function names and variable names in the same scope dictionary.

  This is the distinction between a lisp-1 and a lisp-2. We are a
  lisp-1 like Scheme because we have a single scope. Common Lisp is a
  lisp-2 because it stores function name scope separately from
  variable name scope.
Implementing scope
We need to revise every compile function to accept a scope dictionary
(specifically: compile, compile_expression, compile_call, and
compile_define). If a variable is referenced, we need to look up
it's location in the scope dictionary. If a variable is defined
(e.g. a function name or a function parameter) we need to add a
mapping to the scope dictionary.
Modifying compile_expression is easiest:
function compile_expression(arg, destination, scope) {
  // Is a nested function call, compile it
  if (Array.isArray(arg)) {
    compile_call(arg[0], arg.slice(1), destination, scope);
    return;
  }

  if (scope[arg] || Number.isInteger(arg)) {
    emit(1, `MOV ${destination}, ${scope[arg] || arg}`);
  } else {
    throw new Error('Attempt to reference undefined variable or unsupported literal: ' + arg);
  }
}

Next we modify compile_call:
function compile_call(fun, args, destination, scope) {
  if (primitive_functions[fun]) {
    primitive_functions[fun](args, destination, scope);
    return;
  }

  // Save param registers
  args.map((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile registers and store as params
  args.map((arg, i) => compile_expression(arg, PARAM_REGISTERS[i], scope));

  const validFunction = BUILTIN_FUNCTIONS[fun] || scope[fun];
  if (validFunction) {
    emit(1, `CALL ${validFunction}`);
  } else {
    throw new Error('Attempt to call undefined function: ' + fun);
  }

  // Restore param registers
  args.map((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

And then compile_define where we modify scope for the first time:
function compile_define([name, params, ...body], destination, scope) {
  // Add this function to outer scope
  scope[name] = name.replace('-', '_');

  // Copy outer scope so parameter mappings aren't exposed in outer scope.
  const childScope = { ...scope };

  emit(0, `${scope[name]}:`);

  params.forEach(function (param, i) {
    const register = PARAM_REGISTERS[i];
    // Store parameter mapped to associated register
    childScope[param] = register;
  });

  // Pass childScope in for reference when body is compiled.
  compile_expression(body[0], 'RAX', childScope);

  emit(1, 'RET\n');
}

And finally we need to modify the entrypoint compile:
module.exports.compile = function (ast) {
  emit_prefix();
  // Pass in new, empty scope mapping
  compile_call(ast[0], ast.slice(1), 'RAX', {});
  emit_postfix();
}

And scope-wise we're pretty good!
Function calling convention: callee
We currently have a problem that we're using parameters registers to
store local variables that messes up with how we are storing
parameters for function calls within the function itself.
Ideally we could store function local variables (including the
parameters when we get them) separately from how we store function
call parameters within the function.
Thankfully according to the calling convention we've followed, we're
given a set of registers that are callee-preserved. Of them we'll use
RBX, RBP, and R12 in that order. This allows us to mess with so
long as we store them and restore them within the function.
Applying the same storing/restoring strategy to local variables as we
did for parameters, we get:
const LOCAL_REGISTERS = [
  'RBX',
  'RBP',
  'R12',
];

function compile_define([name, params, ...body], destination, scope) {
  // Add this function to outer scope
  scope[name] = name.replace('-', '_');

  // Copy outer scope so parameter mappings aren't exposed in outer scope.
  const childScope = { ...scope };

  emit(0, `${scope[name]}:`);

  params.forEach(function (param, i) {
    const register = PARAM_REGISTERS[i];
    const local = LOCAL_REGISTERS[i];
    emit(1, `PUSH ${local}`);
    emit(1, `MOV ${local}, ${register}`);
    // Store parameter mapped to associated local
    childScope[param] = local;
  });

  // Pass childScope in for reference when body is compiled.
  compile_expression(body[0], 'RAX', childScope);

  params.forEach(function (param, i) {
    // Backwards first
    const local = LOCAL_REGISTERS[params.length - i - 1];
    emit(1, `POP ${local}`);
  });

  emit(1, 'RET\n');
}

And we're set.
Cleanup
We've still got a few messes going on:

emit_prefix wraps out entire body in _main, we're requiring our own main now
emitting to stdout instead of to a file
multiple function definitions is treated as nonsense

Starting first, we rewrite emit_prefix and emit_postfix so that
our _main just calls main.
function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  emit(0, 'plus:');
  emit(1, 'ADD RDI, RSI');
  emit(1, 'MOV RAX, RDI');
  emit(1, 'RET\n');
}

function emit_postfix() {
  emit(0, '_main:');
  emit(1, 'CALL main');
  emit(1, 'MOV RDI, RAX'); // Set exit arg
  emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`);
  emit(1, 'SYSCALL');
}

Next to deal with writing to a file instead of stdout, we need our
emit function to write to a buffer. We'll let ulisp.js write that
buffer to a file. Because we're incredibly lazy, we'll do this all
globally.
let OUT = '';

function emit(depth, args) {
  const indent = new Array(depth + 1).join('  ');
  OUT += `${indent}${args}\n`;
}

...

module.exports.compile = function (ast) {
  OUT = '';

  emit_prefix();
  compile_call(ast[0], ast.slice(1), 'RAX', {});
  emit_postfix();

  return OUT;
}

And modify ulisp.js:
const cp = require('child_process');
const fs = require('fs');

const { parse } = require('./parser');
const { compile } = require('./compiler');

function main (args) {
  const input = fs.readFileSync(args[2]).toString();
  const [ast] = parse(input);
  const program = compile(ast);

  try {
    fs.mkdirSync('build');
  } catch (e) {}
  fs.writeFileSync('build/prog.s', program);
  cp.execSync('gcc -mstackrealign -masm=intel -o build/a.out build/prog.s');
}

main(process.argv);

And we're finally ready to run a simple program.
A program!
$ cat test.lisp
(def main () (+ 1 2))
$ node ulisp.js test.lisp
$ ./build/a.out
$ echo $?
3

Hurray! Now let's try defining and calling a second function
to validate parameter behavior.
$ cat test.lisp
(def plus-two (a)
     (+ a 2))

(def main ()
     (plus-two 3))
$ node ulisp.js test.lisp
$ ./build/a.out
./compiler.js:106
    throw new Error('Attempt to call undefined function: ' + fun);
    ^

Error: Attempt to call undefined function: p2

...

We start getting some really weird errors. And the reason is because
our compiler doesn't know how to deal with sibling S-expressions.
So we'll introduce a new primitive function called begin that calls
all it's sibling functions and returns the value of the last
call. Then we'll wrap the program in an implicit begin so we don't
need to.
function compile_begin(body, destination, scope) {
  body.forEach((expression) => compile_expression(expression, 'RAX', scope));
  if (destination && destination !== 'RAX') {
    emit(1, `MOV ${destination}, RAX`);
  }
}

const primitive_functions = {
  def: compile_define,
  begin: compile_begin,
};

...

module.exports.compile = function (ast) {
  OUT = '';

  emit_prefix();
  compile_call('begin', ast, 'RAX', {});
  emit_postfix();

  return OUT;
}

And we try our test program again. :)
$ cat test.lisp
(def plus-two (a)
     (+ a 2))

(def main ()
     (plus-two 3))
$ node ulisp.js test.lisp
$ ./build/a.out
$ echo $?
5

And that's all there is to it! Stay tuned for the next post on
conditionals and tail-call optimization.
Part two on compiler basics using JavaScript: user-defined functions and variables https://t.co/XOam67HO8h
— Phil Eaton (@phil_eaton) January 20, 2019
 


STAR Apps - A New Generation of Front-End Tooling for Development Workflows
swyx's site RSS Feed — Fri, 18 Jan 2019 02:00:00 GMT
Published on CSS Tricks as STAR Apps: A New Generation of Front-End Tooling for Development Workflows


Cool Facts about Cuttlefish
swyx's site RSS Feed — Tue, 15 Jan 2019 02:00:00 GMT
This was just a short rant I prepared for Netlify's Allhands where we were asked to give a short lightning talk on any topic we wished. I picked cuttles. I actually saw one in real life diving in Fiji once!


A technologist's prayer book
jodavaho.io — Mon, 14 Jan 2019 02:00:00 GMT
I recently found a list of old “prayers”, probably inspired by the “litanies”
that are everpresent in the Warhammer 40k universe. I’ll keep a growing list
here. These are free for your use, under the creative commons license.
Chapter one

In the beginning of any focused work, and to set intention for your building.


We dedicate these hours to the advancement of understanding. We thank
humanity for this opportunity. May our children find use of our work.


On the complexity of life and the search for perfection


The universe is clockwork driven by entropy. We thank the omni-algorithms for imparting upon us turing completeness such that we emulate free will. May our meta-algorithms drive us on to perfection.


Blessed be the complexity of life’s outcomes. May we honor the meta-algorithms of others as they honor the meta-algorithms in us.


On the use of life


May we have breadth over years, and depth across days.

License
CC BY-SA 4.0


Sloc Cloc and Code a Performance Update
Ben E. C. Boyter — Wed, 09 Jan 2019 02:00:00 GMT
Update 2019-03-13
This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16
Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28
Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19
Sloc Cloc and Code a Performance Update 2019-01-09
Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab

I thought I had finished with my code counter Sloc Cloc and Code (AKA scc) https://github.com/boyter/scc/ for a while. The what I had hoped to be final blog post about it https://boyter.org/posts/sloc-cloc-code-performance/ however did mention that the building of the language features in it were a cause of slowdown,


Setting up this blog
Krishna's blog — Wed, 09 Jan 2019 02:00:00 GMT
This article provides a step-by-step guide on setting up a static blog using Hugo, Git, Nginx, and Let's Encrypt on a Linux server.


5 things that happened in Data Science in 2018
Bytepawn - Marton Trencseni — Wed, 09 Jan 2019 01:00:00 GMT
2018 was a hot year for Data Science and AI. Here we picked out 5 highlights, which in our opinion shaped the field in the past year.




Trimming the fat from a Golang binary
Ben E. C. Boyter — Sun, 06 Jan 2019 02:00:00 GMT
I have recently started work again on searchcode server with the continued plan to upgrade searchcode to use the same code-base.
One big issue with this which I recently tweeted about was that Java has a decent well documented fast library for just about everything, with the exception of a good syntax highlighter like Pygments http://pygments.org/ or Chroma https://github.com/alecthomas/chroma
It is however possible to use Pygments in Java using Jython http://pygments.org/docs/java/ which was very appealing to me as Pygments is the current highlighter used by searchcode.com


Books I read in 2018
Krishna's blog — Sun, 06 Jan 2019 02:00:00 GMT
A quick round-up of some of the best books read in 2018, ranging from fantasy to spy novels to maths and politics.


Skin in the Game
Krishna's blog — Sun, 06 Jan 2019 02:00:00 GMT
A critical review of the book Skin in the Game by Nassim Nicholas Taleb.


What I Learned And Did In 2018
infrequent oscillations — Sat, 05 Jan 2019 15:13:35 GMT
A brief list of things I learnt or created in 2018. If I missed something, please tell me! Thanks!


Building c-base @ 35C3 with Flowhub
Henri Bergius — Sat, 05 Jan 2019 02:00:00 GMT
The 35th Chaos Communication Congress is now over, and it is time to write about how we built the software side of the c-base assembly there.

c-base at 35C3

The Chaos Communication Congress is a major fixture of the European security and free software scene, with thousands of attendees. As always, the “mother of all hackerspaces” had a big presence there, with a custom booth that we spend nearly two weeks constructing.



This year’s theme was “Refreshing Memories”, and accordingly we brought various elements of the history of the c-base space station to the event. On hardware side we had things like a scale model the c-base antenna, as well as vintage arcade machines and various artifacts from over the years.

With software, we utilized the existing IoT infrastructure at c-base to control lights, sound, and drive videos and other information to a set of information displays. All of course powered by Flowhub.

This was a quite full-stack development effort, involving microcontroller firmware programming, server-side NoFlo and MsgFlo development, and front-end infoscreen web design. We also did quite a bit of devopsing with Travis CI, Docker, and docker-compose.

Local MsgFlo setup

The first step in bringing c-base’s IoT setup was to prepare a “portable” version of the environment. An MQTT broker, MsgFlo, some components, and a graph with any on-premise c-base hardware or service dependencies removed. As this was for a CCC event, we decided to call it c3-flo (in comparison to the c-flo that we run at c-base).

We already have a quite nice setup where our various systems get built and tested on Travis, and uploaded to Docker Hub’s cbase namespace. Some repositories weren’t yet integrated, and so the first step was to Dockerize them.

To make the local setup simple to manage, we decided to go with a single docker-compose environment that would start all systems needed. This would be easy to run on any x86 machine, and provide us with a quite comprehensive set of features from the IoT parts to NASA’s Open MCT dashboard.

Of course we kept adding to the system throughout 35C3, but in the end the graph looked like the following:



WiFi setup

To make our setup more portable, we decided to bring a local instance of the “c-base botnet” WiFi used to Congress. This way all of our IoT devices could work at 35C3 with the exact same firmware and networking setup as they do at c-base.

Normally Congress doesn’t recommend running your own access point. But if needed, there are guidelines available on how to do it properly if needed. As it happens, out of this year’s 558 unofficial access points, the c-base one was the only one conforming to the guidelines (commentary around the 25 minute mark).



Info displays

Like any station, c-base has a set of info screens showing various announcements, timelines, and statistics. These are built with Raspberry Pi 3s running Chrome in Kiosk Mode, with a single-page webapp that connects to our MsgFlo infrastructure over WebSockets with msgflo-browser.

Each screen has a customized rotation of different pages to show, and we can send URLs to announce events like members arriving to c-base or a space launch livestream via MQTT.



For 35C3 we built a new set of pages tailed for the Congress experience:


  Tweaked version of the normal c-base events view showing current and upcoming talks
  Video player to rotate various videos from the history of c-base
  Photo slideshow with a nice set of pictures from c-base
  Countdown screen for some event (c-base crash, teardown of the assembly at the end of Congress)


Crashing c-base

Highlight of the whole assembly was a re-enactment of the c-base crash from billions of years ago. Triggered by a dropped bottle of space soda, this was an experience incorporating video, lights, and audio that we ran several times every day of the conference.



The c-base crash animation was managed by a NoFlo graph integrated to the our MsgFlo setup with the standard noflo-runtime-msgflo tool. With this we could trigger the “crash” with a MQTT message (sent by a physical button), and run a timed sequence of actions on lights, a sound system, and our info screens.



Timeline manager

There were some new components that we had to build for this purpose. The most important was a Timeline component that was upstreamed as part of the noflo-tween animation library.



With this you can define a multi-tracked timeline as JSON or YAML, with actions triggered on each track on their appropriate second. With MsgFlo this meant we could send timed commands to different devices and create a coordinated experience.

For example, our animation started by showing a short video on all info screens. When the bottle fell in the video, we triggered the appropriate soundtrack, and switched the lights through various animation modes. After the video ended, we switched to a “countdown to crash” screen, and turned all lights to a red alert mode.

After the crash happened, everything went dark for a few seconds, before the c-base assembly was returned into its normal state.

Controlling McLighting

All LED strips we used at 35C3 were run using the McLighting firmware. By default it allows switching between different light modes with a simple WebSocket API.

For our requirements, we wanted the capability to send new commands to the lights with minimal latency, and to be able to restore the lights to whatever mode they had before the crash started in the end.



The component is available in noflo-mclighting. The only thing you need is running the NoFlo graph in the same network as the LED strips, and to send the WebSocket addresses of your LED strips to the component. After that you can control them with normal NoFlo packets.

Finally

The whole setup took a couple of days to get right, especially regarding timings and tweaking the light modes. But, it was great! You can see a video of it below:



And if you’re interested in experimenting this stuff, check out the “portable c-base IoT setup” at https://github.com/c-base/c3-flo.


Vignettes Redux Redux
infrequent oscillations — Fri, 04 Jan 2019 13:40:43 GMT
In which we reduce films down to their essence.


Football Statistics
infrequent oscillations — Thu, 03 Jan 2019 06:31:40 GMT
In which David Wagner is simulataneously right and wrong.


Annual Planning
Beetle Space — Tue, 01 Jan 2019 10:00:00 GMT
New Year’s Resolutions. It’s that time of year again. Below I’ve
accumulated some resources that give interesting tips on how to think
about them, along with my own thoughts.
In this series, I will use the words goals and resolutions interchangeably.

Links
First, a dump of the …